Configure AI Gateway ​

Set up AI Gateway to connect your LLM providers, create routes that determine which model serves each request, and issue access keys that control who can use them.

AI Gateway configuration builds on three objects. Create them in the following order:

1. Provider: A connection to an LLM service, such as Anthropic or Azure OpenAI, along with the credentials AI Gateway uses to authenticate.
2. Route: The target that serves a request. A route resolves to a provider and model, either directly or through rules.
3. Access key: The credential an application presents to AI Gateway. Each access key grants access to one or more routes.

Policies are optional. Create a policy when you need to cap the request rate or token usage of the routes and access keys assigned to it, or restrict access by IP address.

Add a provider ​

A provider stores the credentials AI Gateway uses to authenticate with an LLM service. Providers are stored in a project, and you can reuse the same provider connection in genies.

AI Gateway supports the following LLM providers: Anthropic, OpenAI Compatible, Azure OpenAI, and AWS Bedrock. The fields you configure depend on the provider you select.

Complete the following steps to add a provider:

The provider displays in the Providers list with a Connected status and the date it was added.

Edit a provider ​

Complete the following steps to edit a provider:

Delete a provider ​

Complete the following steps to delete a provider:

Create a route ​

A route determines which provider and model serves a request. Applications reference a route by its route ID in prompt calls.

Complete the following steps to create a route. The fields depend on the route type you select:

Route details ​

Click a route in the Routes list to open its details page. The details panel displays the following:

• Type: The route type, either Direct, Rule-based, or Intelligent.
• Route ID: The identifier to use in your LLM prompt calls.
• Location: The project or folder that stores the route.
• Description: An optional description for the route. Click Edit to add or change it.
• Created: The date and time the route was created.

Click the edit icon next to the route name to rename the route.

Edit a route ​

Editing a route opens the route builder, where you configure the route on a canvas instead of in a form.

Complete the following steps to edit a route:

Refer to Route builder to add rules to a rule-based or intelligent route.

Delete a route ​

Complete the following steps to delete a route:

Create an access key ​

An access key is the credential an application presents to AI Gateway. Each key grants access to one or more routes, which determine the provider and model the key can use.

Complete the following steps to create an access key:

{
  "model": "<route-id>"
}

The access key displays in the Keys list with its routes, policy, creation date, and a masked key value. You can filter the list by route and by policy.

Edit an access key ​

Complete the following steps to edit an access key:

Refresh an access key ​

Refresh an access key to rotate its value. Refreshing generates a new token and revokes access for all current users of the key.

Complete the following steps to refresh an access key:

Delete an access key ​

Complete the following steps to delete an access key:

Test a route ​

Send a request to AI Gateway's endpoint to confirm a route works.

Authenticate with an access key and pass the route ID as the model value:

curl <ai-gateway-url> \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <access-key>" \
  -d '{
    "model": "<route-id>",
    "messages": [
      {
        "role": "user",
        "content": "Hello"
      }
    ]
  }'

Replace <ai-gateway-url> with the URL shown in the Access key generated dialog when you created the access key.

AI Gateway resolves the request to the route's provider and model, applies any assigned policy, and returns an OpenAI-compatible chat completion response, regardless of which provider served the request.

Create a policy ​

A policy defines the rate limits, token limits, and IP access restrictions that apply to the routes and access keys assigned to it.

Complete the following steps to create a policy:

Validation messages ​

AI Gateway returns the following validation messages: