Freeway: self-hosted OpenAI-compatible gateway for free-tier providers
One endpoint in front of every free LLM tier, with automatic failover and no provider key ever leaving the box. Providers are data, not code: adding one is a JSON file plus an env var, never a TypeScript change. Ships 12 providers pre-written; `verify-provider --apply` derives the live model list from your own key. - core: registry, bucketed quota meter (rolling + UTC-calendar windows), key-pool store, 6-step model router with diagnostic rejection reasons - gateway: node:http, failover that commits to a stream only after 200 headers, SSE passthrough with usage sniffing, virtual keys with admin/inference roles - context engine: sqlite sessions and a 5-rung refit ladder, so a 128k thread keeps working when it fails over to an 8k model - cache: exact (sha256, safe-by-default) + opt-in semantic (flat Float32Array) - guard: inbound limits, quota forecasting, limit calibration from upstream headers, key health, secret/PII/injection scanning - dashboard: one self-contained HTML file, live SSE request log, token usage - sdk: zero-dep typed client, drop-in for the OpenAI SDK Zero runtime dependencies. 326 tests, 27-check smoke suite against a scripted mock upstream so routing is testable without spending quota. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
commit
cb985dcf9c
86 files changed
+16350
No files matched your search
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"strategy": "priority",
|
||||
"maxAttempts": 3,
|
||||
"cooldownSeconds": 60,
|
||||
"timeoutMs": 60000,
|
||||
"streamTimeoutMs": 300000,
|
||||
"host": "127.0.0.1",
|
||||
"port": 8787,
|
||||
|
||||
"keys": [],
|
||||
"adminKey": null,
|
||||
|
||||
"//aliases": [
|
||||
"Empty on purpose. Aliases live in the provider files, next to the models",
|
||||
"they name — so a model id and its alias can never drift apart. A config",
|
||||
"alias here OVERRIDES those and pins one exact model, which is what you want",
|
||||
"only when you deliberately never want failover for that name.",
|
||||
"Example: \"fast\": \"mistral/mistral-small-latest\""
|
||||
],
|
||||
"aliases": {},
|
||||
|
||||
"guard": {
|
||||
"perKey": { "rpm": null, "tpm": null, "rpd": null, "concurrency": null },
|
||||
"perIp": { "rpm": null, "concurrency": null },
|
||||
"maxBodyBytes": 8388608,
|
||||
"maxMessages": 500,
|
||||
"scan": { "mode": "flag", "secrets": true, "pii": true, "injection": true },
|
||||
"calibrate": true,
|
||||
"keyHealthIntervalMs": 21600000
|
||||
},
|
||||
|
||||
"context": {
|
||||
"enabled": true,
|
||||
"maxTurns": 12,
|
||||
"reserveOutputTokens": 1024,
|
||||
"compactModel": "fast",
|
||||
"autoCompactAt": 0.8
|
||||
},
|
||||
|
||||
"cache": {
|
||||
"mode": "safe",
|
||||
"ttlMs": 86400000,
|
||||
"maxEntries": 5000,
|
||||
"semantic": {
|
||||
"enabled": false,
|
||||
"threshold": 0.92,
|
||||
"model": "embed",
|
||||
"maxEntries": 5000
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user