Freeway: self-hosted OpenAI-compatible gateway for free-tier providers
One endpoint in front of every free LLM tier, with automatic failover and no provider key ever leaving the box. Providers are data, not code: adding one is a JSON file plus an env var, never a TypeScript change. Ships 12 providers pre-written; `verify-provider --apply` derives the live model list from your own key. - core: registry, bucketed quota meter (rolling + UTC-calendar windows), key-pool store, 6-step model router with diagnostic rejection reasons - gateway: node:http, failover that commits to a stream only after 200 headers, SSE passthrough with usage sniffing, virtual keys with admin/inference roles - context engine: sqlite sessions and a 5-rung refit ladder, so a 128k thread keeps working when it fails over to an 8k model - cache: exact (sha256, safe-by-default) + opt-in semantic (flat Float32Array) - guard: inbound limits, quota forecasting, limit calibration from upstream headers, key health, secret/PII/injection scanning - dashboard: one self-contained HTML file, live SSE request log, token usage - sdk: zero-dep typed client, drop-in for the OpenAI SDK Zero runtime dependencies. 326 tests, 27-check smoke suite against a scripted mock upstream so routing is testable without spending quota. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Aryan SinghandClaude Opus 5 committed
commit
cb985dcf9c
86 files changed
+16350
No files matched your search
@@ -0,0 +1,273 @@
|
||||
# Adding a provider
|
||||
|
||||
**You do not edit TypeScript to add a provider.** You write one JSON file and set
|
||||
one environment variable. If you find yourself opening a `.ts` file, something
|
||||
has gone wrong — see [When you think you need code](#when-you-think-you-need-code).
|
||||
|
||||
This document is written to be usable by a person or by an AI agent working
|
||||
unattended, including for a provider nobody here has ever heard of.
|
||||
|
||||
---
|
||||
|
||||
## The 60-second version
|
||||
|
||||
```bash
|
||||
node scripts/new-provider.mjs acme --url https://api.acme.ai/v1 --env ACME_API_KEY
|
||||
export ACME_API_KEY=sk-...
|
||||
node scripts/verify-provider.mjs acme
|
||||
```
|
||||
|
||||
The verifier tells you what is wrong, what models it found, and what the real
|
||||
rate limits are. Paste its suggestion into the file, run it again until it says
|
||||
`PASS`, and the provider is live.
|
||||
|
||||
---
|
||||
|
||||
## The loop
|
||||
|
||||
```
|
||||
new-provider.mjs ──▶ providers/acme.json ──▶ verify-provider.mjs
|
||||
▲ │
|
||||
└────── paste suggestion ◀────┘
|
||||
```
|
||||
|
||||
`verify-provider.mjs` is the feedback signal. It checks five things in order and
|
||||
stops at the first that fails, so you always get one actionable problem rather
|
||||
than a wall of noise:
|
||||
|
||||
| Step | Checks | Typical failure |
|
||||
|---|---|---|
|
||||
| 1. file | parses, matches the schema | `limits.rpm: expected a positive number or null` |
|
||||
| 2. environment | `${ENV}` resolves, a key exists | `no API key found (set one of: ACME_API_KEY)` |
|
||||
| 3. live probe | auth accepted, `/models` readable | `auth invalid — GET /models → 401` |
|
||||
| 4. limits | headers vs declared limits | `rpm: file says 60, provider reports 30` |
|
||||
| 5. suggestion | prints paste-ready JSON | — |
|
||||
|
||||
Flags: `--validate` also sends a one-token request per model (spends real quota,
|
||||
tells you which models the key can genuinely reach). `--json` for machine output
|
||||
— use this if you are an agent.
|
||||
|
||||
---
|
||||
|
||||
## The fields that actually matter
|
||||
|
||||
Everything else has a working default. Full contract:
|
||||
[`providers/_schema.json`](../providers/_schema.json).
|
||||
|
||||
### `baseUrl` — with `${ENV_VAR}` interpolation
|
||||
|
||||
```jsonc
|
||||
"baseUrl": "https://api.groq.com/openai/v1"
|
||||
|
||||
// Some providers put an account id in the path. Interpolation is why that
|
||||
// stays data instead of becoming a special case in code:
|
||||
"baseUrl": "https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/ai/v1"
|
||||
```
|
||||
|
||||
A missing variable is not a crash. The provider renders on the dashboard as a
|
||||
card reading `missing environment variable: CLOUDFLARE_ACCOUNT_ID`.
|
||||
|
||||
### `auth` — four shapes cover everything so far
|
||||
|
||||
```jsonc
|
||||
{ "type": "bearer", "envKeys": ["ACME_API_KEY"] } // Authorization: Bearer <key>
|
||||
{ "type": "header", "header": "x-api-key", "envKeys": ["ACME_KEY"] } // a named header
|
||||
{ "type": "query", "query": "key", "envKeys": ["ACME_KEY"] } // ?key=<key>
|
||||
{ "type": "none", "envKeys": [] } // local Ollama, no credential
|
||||
{ "type": "bearer", "prefix": "", "envKeys": ["ACME_KEY"] } // bare token, no "Bearer "
|
||||
```
|
||||
|
||||
**Key pools.** One variable may hold several comma-separated keys:
|
||||
|
||||
```bash
|
||||
ACME_API_KEY=sk-aaa,sk-bbb,sk-ccc
|
||||
```
|
||||
|
||||
Quota is tracked **per key**. A 429 on one benches only that key and the request
|
||||
retries on the next; a 5xx benches the whole provider because every key would
|
||||
fail identically. Commas are the grammar precisely so this works from a plain
|
||||
`docker run -e` with no extra syntax.
|
||||
|
||||
### `limits` — null is a real answer
|
||||
|
||||
```jsonc
|
||||
"limits": { "rps": 1, "tpm": 500000, "tpmo": 1000000000 }
|
||||
```
|
||||
|
||||
| Key | Window |
|
||||
|---|---|
|
||||
| `rps` `rpm` `tpm` | rolling |
|
||||
| `rpd` `tpd` `creditsPerDay` | **calendar day, resets 00:00 UTC** |
|
||||
| `tpmo` | **calendar month** |
|
||||
|
||||
> **Do not guess a number.** `null` means "unknown or unlimited" and is always
|
||||
> the right answer when you have not verified it. The guard layer overwrites
|
||||
> nulls with values read from the provider's own rate-limit response headers, so
|
||||
> an honest null becomes accurate on its own. A wrong number does not — it
|
||||
> silently misroutes, and nothing ever corrects it.
|
||||
|
||||
Set `"limitsSource": "unverified"` when you are unsure. The dashboard badges it
|
||||
and everything still works. Mistral's file does exactly this, because Mistral
|
||||
stopped publishing free-tier figures.
|
||||
|
||||
### `models` — a few verified beats a long guess
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"id": "acme-turbo", // exactly as the provider spells it; slashes are fine
|
||||
"alias": ["fast", "cheap"], // several providers sharing an alias is what enables failover
|
||||
"context": 128000, // null if unpublished
|
||||
"caps": ["chat", "tools", "json"],
|
||||
"priority": 10, // lower is tried first
|
||||
"credits": { "perMTokIn": 2457, "perMTokOut": 18252 } // only for credit-metered providers
|
||||
}
|
||||
```
|
||||
|
||||
`caps` drives routing and is not decorative:
|
||||
|
||||
| Cap | Effect |
|
||||
|---|---|
|
||||
| `chat` | eligible for `/v1/chat/completions` |
|
||||
| `embed` | eligible for `/v1/embeddings` — and **never** for chat |
|
||||
| `tools` | required when the request body has `tools` |
|
||||
| `json` | required when `response_format.type` is `json_object` |
|
||||
| `vision` | required when a message contains an image part |
|
||||
| `reasoning` `code` `long-context` | informational, filterable in the dashboard |
|
||||
|
||||
Leave `models: []` and let discovery populate it. That is better than a stale
|
||||
list — see `together.json` and `github-models.json`.
|
||||
|
||||
### `modelsPath` — for non-OpenAI response shapes
|
||||
|
||||
```jsonc
|
||||
"modelsEndpoint": "/models",
|
||||
"modelsPath": "data[].id" // {"data":[{"id":"..."}]} ← OpenAI standard
|
||||
"modelsPath": "[].id" // [{"id":"..."}] ← GitHub Models
|
||||
"modelsPath": "models[].name" // {"models":[{"name":...}]}
|
||||
"modelsPath": "result[].name" // {"result":[{"name":...}]}
|
||||
```
|
||||
|
||||
Set `"modelsEndpoint": null` when a provider has no catalog endpoint —
|
||||
Cloudflare's OpenAI-compatible surface exposes `/chat/completions` and
|
||||
`/embeddings` but not `/models`.
|
||||
|
||||
### `dropParams` — params a provider rejects
|
||||
|
||||
```jsonc
|
||||
"dropParams": ["frequency_penalty", "presence_penalty"]
|
||||
```
|
||||
|
||||
Find them empirically: send a request, read the 400, add the offending field.
|
||||
|
||||
---
|
||||
|
||||
## Worked example: a provider that does not exist
|
||||
|
||||
Say **Acme AI** offers a free tier. OpenAI-compatible, key in `x-api-key`,
|
||||
account id in the URL, and a `/models` endpoint returning `{"result":[...]}`.
|
||||
|
||||
**1. Scaffold.**
|
||||
|
||||
```bash
|
||||
node scripts/new-provider.mjs acme \
|
||||
--url 'https://api.acme.ai/v1/org/${ACME_ORG}' \
|
||||
--auth header --header x-api-key --env ACME_API_KEY
|
||||
```
|
||||
|
||||
**2. Fix the response shape** in `providers/acme.json`:
|
||||
|
||||
```jsonc
|
||||
"modelsPath": "result[].name"
|
||||
```
|
||||
|
||||
**3. Verify.**
|
||||
|
||||
```bash
|
||||
export ACME_API_KEY=sk-live-xxxx
|
||||
export ACME_ORG=org_123
|
||||
node scripts/verify-provider.mjs acme
|
||||
```
|
||||
|
||||
```
|
||||
1. file
|
||||
✓ providers/acme.json parses
|
||||
2. environment
|
||||
✓ baseUrl resolves to https://api.acme.ai/v1/org/org_123
|
||||
✓ 1 key in the pool: sk-l…xxxx (ACME_API_KEY)
|
||||
3. live probe
|
||||
✓ auth accepted (142ms)
|
||||
✓ discovery found 3 models via /models → result[].name
|
||||
4. limits
|
||||
✓ observed from response headers: rpm=60, tpm=40000
|
||||
+ rpm: file says null, provider reports 60 — consider recording it
|
||||
5. suggested edits
|
||||
{ "models": [...], "limits": {...}, "limitsSource": "observed" }
|
||||
|
||||
PASS — 0 problems, 0 warnings
|
||||
```
|
||||
|
||||
**4. Paste the suggestion in**, add aliases and capabilities the verifier could
|
||||
not know, and re-run with `--validate` to confirm each model really answers.
|
||||
|
||||
**5. Use it.** No restart needed:
|
||||
|
||||
```bash
|
||||
curl -X POST localhost:8787/api/reload
|
||||
curl localhost:8787/v1/chat/completions -H 'content-type: application/json' \
|
||||
-d '{"model":"acme/acme-turbo","messages":[{"role":"user","content":"hi"}]}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common failures
|
||||
|
||||
| Symptom | Cause |
|
||||
|---|---|
|
||||
| Provider missing from the dashboard entirely | filename starts with `_` (reserved), or the JSON failed to parse — check the red banner, it names the file and the syntax error |
|
||||
| `unknown field — did you mean "baseUrl"?` | a typo like `baseURL`. Warnings are surfaced, never silently ignored |
|
||||
| Card shows `no API key found` | env var unset, or set in a shell the gateway did not inherit |
|
||||
| Card shows `missing environment variable: X` | a `${X}` in `baseUrl`/`headers` did not resolve. Pasting a key will not fix this |
|
||||
| `modelsPath matched nothing` | wrong response shape — `curl` the endpoint and read it |
|
||||
| Model never gets routed to | missing a required `cap`, or `context` too small for `minContext`. The 429 body's `blocked[]` says which |
|
||||
| Everything 429s immediately | a limit is set too low. Set it to `null` and let calibration find the truth |
|
||||
|
||||
Every rejection reason appears in the `blocked` array of a 429 response and in
|
||||
the dashboard. You should never have to guess why a provider was skipped.
|
||||
|
||||
---
|
||||
|
||||
## When you think you need code
|
||||
|
||||
Nearly always, you do not:
|
||||
|
||||
| "I need code because…" | Actually |
|
||||
|---|---|
|
||||
| the URL contains an account id | `${ENV_VAR}` in `baseUrl` |
|
||||
| auth is a custom header | `auth.type: "header"` |
|
||||
| the token has no `Bearer ` prefix | `auth.prefix: ""` |
|
||||
| `/models` is shaped differently | `modelsPath` |
|
||||
| there is no `/models` | `modelsEndpoint: null` |
|
||||
| it rejects a parameter | `dropParams` |
|
||||
| it needs an extra header | `headers: {}` (supports `${ENV}` too) |
|
||||
| it bills in credits, not tokens | `limits.creditsPerDay` + per-model `credits` |
|
||||
| I have several keys | comma-separate them in one env var |
|
||||
|
||||
A genuinely new **wire format** — not OpenAI-compatible at all — is the one case
|
||||
that needs a new `adapter`. That field exists and is validated, but only
|
||||
`"openai"` is implemented. Adding one is a real code change; everything above is not.
|
||||
|
||||
---
|
||||
|
||||
## Repo conventions that will bite you
|
||||
|
||||
- **No TypeScript parameter properties.** `constructor(private readonly x: T)`
|
||||
throws `ERR_UNSUPPORTED_TYPESCRIPT_SYNTAX` under Node's strip-only type
|
||||
stripping. Declare fields explicitly and assign in the constructor.
|
||||
- **Zero runtime dependencies.** Dev deps are `typescript` and `@types/node`.
|
||||
Stop and ask before adding anything.
|
||||
- `noUncheckedIndexedAccess` is on: indexing returns `T | undefined`. Use guards
|
||||
or `?? fallback`, never `!`.
|
||||
- Never log a key. Everything goes through `mask()` in
|
||||
[`packages/core/src/util.ts`](../packages/core/src/util.ts).
|
||||
- Tests are `node:test`. Point them at `scripts/mock-upstream.js`, never at a
|
||||
real provider — that is what it is for.
|
||||
Reference in new issue
Block a user