Rate limits

Avey enforces two kinds of limit: a count of requests, and a count of tokens per minute.

Request limits

Each row is its own budget. Model calls do not use up file uploads. The budgets belong to your account, not to a key: every key you create draws on the same ones.

Each budget refills continuously. You can send a full burst from idle. After that, requests are admitted at the sustained rate. A rejected request costs you nothing.

If you need more, write to support@avey.ai and describe your workload.

Response headers

X-RateLimit-Limit and X-RateLimit-Remaining appear on responses. Retry-After appears only on a 429.

Token limits

Each model has a per-minute limit on input tokens and a separate one on output tokens. The values depend on the environment and can change. A rejected request returns the current values in its headers.

A request is admitted while both token budgets are above zero. Before the model runs, the request reserves its estimated input and its output ceiling. The ceiling is max_tokens times n, or the model's maximum output of 16,384 tokens when max_tokens is not set. When the request finishes, the reservation is replaced with the actual usage.

So an oversized max_tokens is not billed, but it holds headroom in your token limits while the request runs. Set max_tokens to what you need.

A request over a token limit returns 429 with "code": "rate_limit_exceeded". The message has this form:

rate limit reached for <model>: <n> input and <n> output tokens per minute; retry in <n> s

A request-count 429 has no body.

A 429 that is not a rate limit

When your account has no credit left, or reaches a spending cap, requests return 429 with "type": "insufficient_quota". Retrying does not help. The code field names the reason, and there is no Retry-After. Add credit on the API platform, or contact support.

{
  "error": {
    "message": "API access disabled",
    "type": "insufficient_quota",
    "param": null,
    "code": "insufficient_credit"
  }
}

Handling a 429

Wait Retry-After seconds, then retry. Retrying sooner gets the same 429.

This loop retries a rate-limit 429 up to five times and gives up at once on insufficient_quota. It sets max_retries=0 so the SDK does not retry on its own, and adds jitter so workers rejected together do not retry together.

import os
import random
import time

from openai import OpenAI, RateLimitError

client = OpenAI(
    base_url="https://api.avey.ai/llm",
    api_key=os.environ["AVEY_API_KEY"],
    max_retries=0,
)


def create_with_retry(prompt: str, attempts: int = 5):
    for attempt in range(attempts):
        try:
            return client.chat.completions.create(
                model="olive",
                messages=[{"role": "user", "content": prompt}],
            )
        except RateLimitError as err:
            # Out of credit: retrying cannot succeed.
            if err.type == "insufficient_quota" or attempt == attempts - 1:
                raise
            retry_after = int(err.response.headers.get("retry-after", 5))
            time.sleep(retry_after + random.uniform(0, 1))

Do not retry on a fixed short interval. Do not run parallel workers that each retry on their own, because they compete for the same limits.