Provider Setup

Configure LLM providers for use with InferXgate.

This guide covers setting up each LLM provider with InferXgate.

Anthropic

  1. Get an API key from console.anthropic.com
  2. Add to environment:
ANTHROPIC_API_KEY=sk-ant-api03-...
  1. Test:
curl -X POST http://localhost:3000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "claude-3-haiku-20240307", "messages": [{"role": "user", "content": "Hi"}]}'

OpenAI

  1. Get an API key from platform.openai.com
  2. Add to environment:
OPENAI_API_KEY=sk-...
OPENAI_ORG_ID=org-...  # Optional
  1. Test:
curl -X POST http://localhost:3000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Hi"}]}'

Google Gemini

  1. Get an API key from makersuite.google.com
  2. Add to environment:
GEMINI_API_KEY=...
  1. Test:
curl -X POST http://localhost:3000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gemini-1.5-flash", "messages": [{"role": "user", "content": "Hi"}]}'

Azure OpenAI

  1. Create a resource in Azure Portal
  2. Deploy models to your resource
  3. Add to environment:
AZURE_OPENAI_API_KEY=...
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com
AZURE_OPENAI_API_VERSION=2024-02-01
AZURE_OPENAI_DEPLOYMENTS=gpt-4:my-gpt4-deployment
  1. Test with azure- prefix:
curl -X POST http://localhost:3000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "azure-gpt-4", "messages": [{"role": "user", "content": "Hi"}]}'

Verify All Providers

Check which providers are configured:

curl http://localhost:3000/health
{
  "providers": {
    "anthropic": "healthy",
    "openai": "healthy",
    "gemini": "healthy",
    "azure": "healthy"
  }
}

Multiple Keys Per Provider

For load balancing:

ANTHROPIC_API_KEYS=key1,key2,key3
OPENAI_API_KEYS=key1,key2