Model Providers

Eparch routes each model to a provider through a registry, so you can use any mix of self-hosted and hosted models at once. Almost every option is OpenAI-compatible — self-hosted vLLM / Ollama / TGI / LM Studio, DigitalOcean serverless inference, OpenAI / Together / Groq / Azure OpenAI — plus native Anthropic.

Two ways to configure

Zero-config (single upstream)

For one endpoint, just set env vars — Eparch synthesizes a one-provider registry:

LLM_GATEWAY_URL=http://vllm:8000/v1/chat/completions   # any OpenAI-compatible endpoint
LLM_EMBEDDING_URL=http://vllm:8000/v1/embeddings
LLM_API_KEY=                                           # optional bearer key for the upstream

Unknown model names pass through to that upstream, so LLM_DEFAULT_MODEL and any model your server hosts just work.

Multiple providers (`models.yaml`)

Point LLM_CONFIG at a config file to route different models to different providers. Copy deploy/models.yaml.example:

default_chat: llama3.3
providers:
  digitalocean:  { type: openai, base_url: https://inference.do-ai.run/v1, auth: bearer, api_key_env: DO_INFERENCE_KEY }
  local-vllm:    { type: openai, base_url: http://vllm:8000/v1 }         # auth: none (trusted network)
  anthropic:     { type: anthropic, base_url: https://api.anthropic.com, auth: x-api-key, api_key_env: ANTHROPIC_API_KEY }
models:
  - { alias: llama3.3,      provider: digitalocean, model: llama3.3-70b-instruct }
  - { alias: claude-sonnet, provider: anthropic,    model: claude-sonnet-4-5 }
embeddings:
  - { alias: embed, provider: ollama, model: nomic-embed-text, dim: 768 }

LLM_CONFIG=/etc/eparch/models.yaml   # mount the file into the container at this path

Fields: type is openai (any OpenAI-compatible endpoint) or anthropic. auth is none / bearer / x-api-key. api_key_env reads the secret from an environment variable (keep secrets out of the file).

DigitalOcean

DigitalOcean’s serverless inference is OpenAI-compatible, so it’s a type: openai provider with a bearer model-access key:

providers:
  digitalocean: { type: openai, base_url: https://inference.do-ai.run/v1, auth: bearer, api_key_env: DO_INFERENCE_KEY }
models:
  - { alias: llama3.3, provider: digitalocean, model: llama3.3-70b-instruct }

Set DO_INFERENCE_KEY (create a model-access key in the DO console). Use the exact model name DO lists. Running your own model on a DO GPU Droplet is the same recipe with base_url pointing at your Droplet.

Self-hosted (air-gap)

providers:
  vllm:   { type: openai, base_url: http://vllm:8000/v1 }
  ollama: { type: openai, base_url: http://ollama:11434/v1 }

vLLM: python -m vllm.entrypoints.openai.api_server --model /models/llama-3.3-70b
Ollama: ollama serve with pre-pulled models

No API key needed on a trusted network (auth: none, the default for openai).

Embeddings

Each embedding model declares its dim (vector dimension). RAG collections bind to that dimension, so an embedding model and its collections stay consistent. For the quickstart mock, the dimension defaults to RAG_EMBEDDING_DIM (3072).