optaris-core

English · 简体中文

A pure-forwarding LLM API routing engine, shipped as an embeddable Go library — no main, no server binary. You mount its http.Handlers on your own mux, wrap them with your own auth middleware, and it routes each request to the best upstream, with failover, session affinity, and full lifecycle observability.

Module: github.com/getoptaris/optaris-core   ·   Go 1.26

What it is

optaris-core is the data plane of an LLM gateway, and nothing more. It receives requests in four inbound API formats (OpenAI Chat, OpenAI Responses, Claude, Gemini), picks an upstream channel, forwards the request byte-for-byte, retries and fails over on error, and surfaces every stage of the request lifecycle back to you.

It deliberately does not handle storage, auth, tenancy, or billing — those are yours. Three disciplines define the boundary:

• Zero storage — the engine only holds an in-memory config snapshot, loaded via New / LoadConfig and read back via Snapshot. Persistence (sqlite / redis / file / none) is entirely your job. The engine never touches disk on the request path.
• Zero auth / zero tenancy — the engine knows nothing about API keys or "users". Which group a request routes to is injected by you, per-request, via a RouteContext. Who is consuming lives entirely on your side, correlated back through an optional RequestID.
• Pure observability — every request-lifecycle stage surfaces synchronously through OnEvent (including optional raw capture). The engine itself persists nothing.

Features

• Four inbound formats, one pipeline — OpenAI Chat / OpenAI Responses / Claude / Gemini, each an http.Handler, all sharing one forwarding core.
• Weighted routing — a 4-metric, ratio-normalized score in (0,1]: price (static) + success rate + first-token latency + output speed (dynamic, measured per (channel, model)). Ties broken by ε-greedy random pick.
• Failover — per-attempt retry on the same upstream, then cooldown-and-switch to the next candidate, with an overall time budget.
• The commit line — for streaming, failures before the first upstream event are invisible to the client and freely retried; after commit, the 200 is locked in and a mid-stream failure is written as a format-correct in-stream error.
• Session affinity — optionally stick a session to one upstream to maximize prompt-cache hits (a cost optimization), with automatic degradation when there's no session.
• Hot config reload — LoadConfig swaps the whole snapshot atomically; new requests use it instantly, in-flight requests keep their old snapshot.
• Lifecycle events — synchronous OnEvent callbacks at every stage, plus exactly-once Completed per request (the natural "one request, one summary" hook), with optional full-chain diagnostic capture.

Installation

go get github.com/getoptaris/optaris-core

Requires Go 1.26+.

Quick start

You bring the mux and the auth middleware; the engine brings the handlers. The one wiring contract: your middleware must inject a RouteContext into the request context via WithRoute.

package main

import (
	"net/http"

	optaris "github.com/getoptaris/optaris-core"
	"github.com/getoptaris/optaris-core/model"
	"github.com/getoptaris/optaris-core/settings"
)

func main() {
	cfg := optaris.Config{
		Channels: []model.Channel{{
			ID:      "ch_openai",              // you generate the id (ch_ prefix)
			Name:    "OpenAI",                 // display-only
			BaseURL: "https://api.openai.com", // upstream base (before /v1, /v1beta)
			APIKey:  "sk-upstream-secret",     // upstream credential, swapped in on forward
			Models:  []string{"gpt-4o"},       // models this channel serves (exact match)
			Enabled: true,
			// AllowedClients: []string{"claude_code"}, // optional: restrict this channel to specific client types (empty = all)
		}},
		Groups: []model.Group{{
			ID:         "grp_default",
			Name:       "default",
			ChannelIDs: []string{"ch_openai"},
		}},
		Settings: settings.Default(),
	}

	eng := optaris.New(cfg)

	// Subscribe to the lifecycle. Callbacks run synchronously in the request
	// goroutine — keep them non-blocking (enqueue and return).
	eng.OnEvent(func(ev optaris.Event) {
		if ev.Phase == optaris.PhaseCompleted {
			// Persist billing / logs on your side, correlated by ev.ReqID.
			// ev.Usage (non-nil on success) has normalized token counts.
		}
	})

	mux := http.NewServeMux()
	mux.Handle("POST /v1/chat/completions", auth(eng.OpenAIChatHandler()))
	mux.Handle("POST /v1/responses", auth(eng.OpenAIResponsesHandler()))
	mux.Handle("POST /v1/messages", auth(eng.ClaudeHandler()))
	mux.Handle("POST /v1beta/models/{modelAndMethod}", auth(eng.GeminiHandler()))

	_ = http.ListenAndServe(":8080", mux)
}

// auth is YOUR middleware. Authenticate the caller against your own store,
// decide which group the request routes to, then inject the RouteContext the
// engine reads. Missing route context or an empty GroupID is a fail-loud wiring
// error, not a client error.
func auth(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		// ... verify the caller's API key, look up quota, etc. ...
		rc := optaris.RouteContext{
			GroupID:   "grp_default", // whichever group this caller maps to
			RequestID: "",            // optional; engine generates one if empty
			SessionID: "",            // optional; engine extracts one per format if empty
		}
		r = r.WithContext(optaris.WithRoute(r.Context(), rc))
		next.ServeHTTP(w, r)
	})
}

Core concepts

Config: groups, channels, settings

Config is the engine's entire world — the full snapshot it holds in memory.

• model.Channel — one upstream. Carries its own BaseURL + APIKey (the credential swapped in on forward), the list of Models it serves (exact string match), a PriceWeight, an Enabled flag, and an optional AllowedClients allowlist (empty = every client; non-empty = only the listed client types may use this channel — see How routing works). IDs are consumer-generated (ch_ prefix); the engine never mints IDs.
• model.Group — a set of channels (ChannelIDs). A request routes to exactly one group; candidates are that group's channels that serve the requested model.
• settings.Settings — global tunables: routing weights, ε, stats window, retry N + client rules, cooldown, body-size caps, the five timeouts, session-affinity switches, and capture switches. Client rules classify a request's User-Agent into a named client type, which drives both the per-request retry N and each channel's AllowedClients filter. Each name in a channel's AllowedClients must exactly match a rule's Name; an unmatched name (a typo, or client rules that predate the Name field) makes that channel unreachable for every client. settings.Default() gives you a sane baseline. Its JSON tags (snake_case) are the external config contract, so you can serialize it straight into your own store.

Config is read-only to the engine. You own persistence: read config from your store, New or LoadConfig it in; when it changes, save it, then load the new Config. Snapshot() reads the current config back out.

RouteContext: per-request injection

This is the seam that replaces a traditional gateway's token mechanism. In your middleware — after auth, DB lookup, quota calculation — you build a RouteContext and inject it with WithRoute. The engine's handler reads it with RouteFrom.

Field	Required	Meaning
GroupID	✅	Which group to route to. Empty / nonexistent → fail-loud, in-format error.
RequestID	—	Your own correlation id, echoed verbatim as Event.ReqID. Engine generates one if empty.
SessionID	—	Session-affinity key. Engine extracts one per format if empty (see Hooks).

Handlers & formats

Four data-plane entry points, one http.Handler each, all sharing the same internal serve pipeline:

Format	Handler	Suggested mount path
openai_chat	OpenAIChatHandler()	POST /v1/chat/completions
openai_responses	OpenAIResponsesHandler()	POST /v1/responses
claude	ClaudeHandler()	POST /v1/messages
gemini	GeminiHandler()	POST /v1beta/models/{modelAndMethod}

Format drives protocol adaptation only — auth-header position, Gemini query params, response validation, usage extraction. It is not a routing input: routing selects channels purely by model. A format/upstream mismatch surfaces as a forward failure that triggers failover, not a routing filter.

Lifecycle events

OnEvent(fn func(Event)) registers a subscriber, invoked synchronously in the request goroutine (the engine recovers panics; keep callbacks non-blocking). A typical sequence for one request:

Received → (AttemptStart → [FirstToken] → [Committed] → AttemptEnd → [Failover])* → Completed

Completed fires exactly once for every request — including pre-flight rejections — after the response is fully written. If you only care about "one request, one summary" (billing, logging), subscribe to Completed alone; the other phases are for real-time tracking. Event carries request-level fields (ReqID, GroupID, Model, Stream), attempt-level fields (ChannelID, Attempt), the result (Outcome, HTTPStatus, Usage), and optional diagnostic Capture.

How routing works

For each request, selectNext runs a pure decision chain (all of internal/router is unit-testable pure logic):

1. Candidates — channels in the group that serve the requested model (exact match), are enabled, permit the request's client type (a channel's AllowedClients, classified from the User-Agent via client rules; empty = all clients), and aren't cooled/failed.
2. Session affinity short-circuit — if affinity is on and the session is still bound to a candidate, reuse it directly (skip scoring). Sticking a session to one upstream maximizes prompt-cache hits.
3. Score — a weighted score in (0,1] from four ratio-normalized metrics: price (static, lower better) + success rate + first-token latency + output speed (dynamic, per (channel, model)). Non-streaming degrades to price + success. Cold-start metrics score as the optimum and don't skew the normalization.
4. Select — take the max; anything within ScoreEpsilon of it is a tie, broken by an injected random pick (ε-greedy).

Cooldown is per (channel, model), not per channel. selectNext has a recovery path: if excluding cooled+failed leaves no candidates but excluding only failed does, it clears those cooldowns and retries — a channel is never permanently lost to cooldown alone.

Failover & the commit line

The heart of the engine (serve.go): select an upstream → up to N+1 attempts on it → all failed → cool down that (channel, model) and switch upstream → aggregate 502 when exhausted or the overall budget expires. N is resolved per-request from the User-Agent via client rules (e.g. Claude Code / Codex default to N=0).

For streaming, every attempt is split at the commit line — the moment the upstream's start event arrives:

• Before commit — bytes are buffered, not forwarded. Failures here are invisible to the client: retry or switch upstream freely. Guarded by the T1 timeout.
• At commit — write the 200 header, flush the buffer, lock in this upstream, fire Committed.
• After commit — no more retries. A mid-stream failure is written as a format-specific in-stream error on the already-committed 200, and the stream terminates (outcome failed, but HTTP status stays 200). Guarded by the T2 idle timeout plus the MaxStreamDuration hard cap.

Non-streaming never commits mid-way. A committed-but-failed streaming attempt is a terminal state — it breaks out of the retry loop.

Timeouts

Five distinct timeouts, all in settings.Settings (T3 rides the injectable clock; the proxy-level stream/response timeouts use real wall-clock time):

Setting	Scope
T1FirstEventTimeout	Streaming, pre-commit: waiting for the start event.
T2IdleTimeout	Streaming, post-commit: max gap between upstream bytes. Any upstream byte (including Claude pings) resets it — a pure socket-liveness detector, so a long-thinking model that only emits pings stays alive.
MaxStreamDuration	Streaming, post-commit: a hard wall-clock cap on the whole committed stream, armed at commit and never reset. Unlike T3 it does interrupt an in-flight committed attempt (writes an in-stream error on the 200). 0 disables it.
T3FailoverTimeout	Overall failover budget. Soft — checked only at upstream-switch boundaries, never interrupts an in-flight attempt.
NonStreamTimeout	Waiting for a full non-streaming response body.

Options

New(cfg, ...Option) accepts a few optional seams, all with built-in defaults:

• WithClock(now func() time.Time) — inject the clock (defaults to time.Now). Mainly for tests driving T1/T2/T3, cooldowns, and the stats window with a fake clock.
• WithHooks(Hooks) — override optional behavior. Currently Hooks.ExtractSession lets you override per-format session-id extraction (default: OpenAI reads a session_id/session-id header, Claude reads metadata.user_id from the body, Gemini has none). Called only when RouteContext.SessionID is empty.
• WithAffinityStore(affinity.Store) — inject a custom session-affinity ledger (defaults to in-process memory). Reserved for multi-instance sharing (Redis etc.) without touching routing logic.

Observability & capture

Beyond the phase events, the engine can hand you the full-chain raw content of a request via the Completed event's Capture field: the user→gateway request, each gateway→upstream attempt and its response, and enough flags to reconstruct what was returned to the user. Auth headers are redacted; bodies are plaintext. Capture is off by default (CaptureEnabled), scoped by CaptureMode (failed_only / all), and the engine imposes no retention itself — keeping it is your subscriber's call.

Package layout

The root package optaris is the public API; everything under internal/* is implementation and cannot be imported by consumers.

• optaris (root) — Engine, Config, RouteContext / WithRoute / RouteFrom, Event / Phase / CaptureData, Hooks, the Options. serve.go is the orchestrator.
• model — zero-dependency domain kernel: Channel, Group, Format.
• settings — global settings + Default()s; JSON tags are the external config contract.
• usage — normalizes the four upstreams' divergent token-usage shapes into Normalized.
• internal/* — the snapshot cache, forwarding proxy, SSE framing, sliding-window stats + cooldown, routing decision, error envelopes, affinity ledger, id generation.

Development

No Makefile, no linter config, no CI — just the raw go toolchain:

go build ./...                              # build everything
go test ./...                               # run all tests
go test -race ./...                         # race detector (concurrency-heavy code)
go test ./internal/router/                  # a single package
go test ./internal/router/ -run TestScore   # a single test (regex match)
go vet ./...                                # vet
gofmt -l .                                  # list unformatted files (must be empty)

The black-box harness (harness_test.go, package optaris_test) drives the engine exactly as a real consumer would — New + the four handlers + a middleware that injects RouteContext + a configurable mock upstream — verifying both behavior and that the public API is sufficient for a real embedder.