Experiential: Open Source Model Gateway with Smart Routing

Experiential is a new open source model gateway from experientiallabs that provides a unified OpenAI-compatible API for hosted, BYOK (bring-your-own-key), and local models. The Rust-native gateway adds under 1ms for BYOK requests and under 2ms when Experiential supplies the provider key, and comes with every major inference provider and 1000+ models refreshed daily via a Codex agent that opens a PR.

Key Features

• One control plane for closed, open-source, local, and custom models — mix and match as needed.
• No markup: Unlike other routers that charge 10% token fees, Experiential takes zero markup on their hosted gateway, and the source is fully open for self-hosting.
• Opt-in traffic-based optimization: It uses standardized OpenTelemetry traces to build a router that picks the optimal model per request based on cost/quality.
• Text world models simulate rollouts for various models, an LLM judge scores them, and a nearest-neighbor classifier on prompt embeddings decides which model to call.
• Cache optimization and new model suggestions come out of the same simulation pipeline.
• Budget controls per user, agent, and use case.

Getting Started

Install and run the local gateway:

pip install experiential
exp

On first run, the setup wizard connects providers and assigns a one-time key. Then you can call any model through the gateway:

export EXP_GATEWAY_KEY=...
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer $EXP_GATEWAY_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"opus-5","messages":[{"role":"user","content":"Help me"}]}'

Optimizing from Your Traffic

To build a custom router from your existing OTel traces:

exp build support-agent

This command walks you through providers, models, budget, and asks for your trace file. You can also fine-tune an open source model from collected traces using exp optimize model support-agent. If you just want to try it, fetch the public terminal-tasks dataset:

curl -L -o traces.otel.jsonl https://huggingface.co/datasets/experiential-labs/wmo-terminal-tasks-traces/resolve/540883e451dc13d34fb50fdd36b143cb0f1fb0db/traces.otel.jsonl

The project is under active development (450 commits, 655 stars) and includes an AGENTS.md for repo conventions. Telemetry is anonymous and can be disabled via exp config telemetry disable.

Experiential is positioned as a direct alternative to proprietary gateways like OpenRouter, with the promise that simple routing doesn’t justify a 10% token markup. It’s an option worth evaluating if you’re paying per-token fees on a commercial router or juggling multiple provider SDKs.