Batch API

The OpenAI-compatible Batch API sends asynchronous groups of inference requests to Inference.net, with nearly unlimited rate limits and fast completion times. It's built for processing a large number of jobs that don't need an immediate response.

#When to use it

Some workloads need synchronous requests, but many don't need an immediate response, or rate limits keep you from running a large number of queries quickly. Batch jobs help with use cases like:

1. Extracting structured data from a large number of documents.
2. Generating synthetic data for training.
3. Translating a large number of documents into other languages.
4. Summarizing a large number of customer interactions.

The Batch API has endpoints to upload a batch of requests, start a batch job, check its status, and retrieve the collected results when the batch is complete.

Compared to the standard endpoints, the Batch API has:

1. Higher rate limits: substantially more headroom than the synchronous APIs.
2. Fast completion times: each batch completes within 24 hours (and often much more quickly).
3. Lower prices: batch requests are billed at least 25% below standard rates.

#Get started

You need an Inference.net account and API key. See the quickstart to create an account and a key.

Install the OpenAI SDK for your language. To connect to the Batch API with the OpenAI SDK, set the base URL to https://batch.inference.net/v1. These examples read the API key from the INFERENCE_API_KEY environment variable.

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://batch.inference.net/v1",
  apiKey: process.env.INFERENCE_API_KEY,
});

#Run a batch job

#Prepare your batch file

Prepare a .jsonl file where each line is a separate JSON object that represents one request. Each JSON object must be on a single line and cannot contain line breaks. Each object must include these fields:

• custom_id: a unique identifier for the request. Use it to match the request to its result after completion. It must be unique for each request in the file.
• method: the HTTP method for the request. Only POST is supported.
• url: the URL to send the request to. Only /v1/chat/completions and /v1/completions are supported.
• body: the request body. The parameters in each line's body are the same as the parameters for the endpoint in url. See the quickstart for an example request.

An input file with 2 requests to /v1/chat/completions:

{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "kimi-k3-fast", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the capital of France?"}], "max_tokens": 1000}}
{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "kimi-k3-fast", "messages": [{"role": "system", "content": "You are an unhelpful assistant."}, {"role": "user", "content": "What is the capital of Belgium?"}], "max_tokens": 1000}}

An input file using /v1/completions:

{"custom_id": "request-1", "method": "POST", "url": "/v1/completions", "body": {"model": "kimi-k3-fast", "prompt": "What is the capital of France?", "max_tokens": 1000}}
{"custom_id": "request-2", "method": "POST", "url": "/v1/completions", "body": {"model": "kimi-k3-fast", "prompt": "What is the capital of Belgium?", "max_tokens": 1000}}

#Upload the input file

Upload your input file before you create a batch.

import fs from "fs";

const batchInputFile = await client.files.create({
  file: fs.createReadStream("batchinput.jsonl"),
  purpose: "batch",
});

console.log(batchInputFile);

The response looks similar to this:

{
  "id": "file-abc123",
  "object": "file",
  "bytes": 1024,
  "created_at": 1714508499,
  "filename": "batch.jsonl",
  "purpose": "batch",
  "status": "processed"
}

#Create the batch

Use the ID of the uploaded file to create a batch. These examples use the file ID file-abc123.

The completion window accepts any value ending in h, m, s, or ms (for example 24h), bounded between 24 hours and 7 days; 24h is the default.

To associate custom metadata with the batch, pass the optional metadata parameter. Inference.net does not use it to complete requests, but includes it when you retrieve the batch status.

import type { BatchCreateParams } from "openai/resources/batches";

const batch = await client.batches.create({
  input_file_id: batchInputFile.id,
  endpoint: "/v1/chat/completions",
  completion_window: "24h",
  metadata: {
    description: "nightly eval job",
  },
  // Optional. Must be HTTPS.
  webhook_url: "https://example.com/my_webhook",
} as BatchCreateParams);

console.log(batch);

The request returns a batch object with metadata about your batch:

{
  "id": "batch_abc123",
  "object": "batch",
  "endpoint": "/v1/chat/completions",
  "errors": null,
  "input_file_id": "file-abc123",
  "completion_window": "24h",
  "status": "validating",
  "output_file_id": null,
  "error_file_id": null,
  "created_at": 1714508499,
  "in_progress_at": 1714508500,
  "expires_at": 1714536634,
  "completed_at": null,
  "failed_at": null,
  "expired_at": null,
  "request_counts": {
    "total": 2,
    "completed": 0,
    "failed": 0
  },
  "metadata": {
    "description": "nightly eval job"
  }
}

If no metadata is provided when the batch is created, the metadata field is null.

#Get notified with a webhook URL

Set webhook_url to receive a notification when the batch reaches a final state: completed, failed, expired, or cancelled. The webhook_url must be an HTTPS URL on a public host that can receive POST requests.

The webhook is a POST with Content-Type: application/json. Your endpoint has 10 seconds to respond. Any 2xx response counts as delivered. Other responses and timeouts are retried, for up to 3 attempts.

Your webhook receives a POST with a JSON body like this:

{
  "batch_id": "batch_abc123",
  "status": "completed",
  "metadata": {
    "description": "nightly eval job"
  }
}

Batches take a webhook_url directly. /v1/slow and Group requests use the webhook_id of a webhook registered in the dashboard instead; see Webhooks.

#Check the status of a batch

Retrieve a batch at any time to get its current batch object. Use the batch ID Inference.net assigned to it (here batch_abc123).

const retrievedBatch = await client.batches.retrieve(batch.id);
console.log(retrievedBatch);

A batch can have any of these statuses:

A batch moves to cancelling when you cancel it, and to cancelled when it closes. See Cancel a batch.

#Retrieve the results

You receive an email notification when the batch is complete.

Once the batch is complete, download the output from the Files API using the output_file_id field of the batch object. Retrieve the error file (all failed requests) the same way with the error_file_id field.

const fileResponse = await client.files.content(batch.output_file_id);
const fileContents = await fileResponse.text();

console.log(fileContents);

The output .jsonl file has one response line for every successful request line in the input file. Failed requests have their error information written to the error file found through the batch's error_file_id.

Don't rely on order to process results. Use the custom_id field, present on each line of the output file, to map requests in your input to results in your output.

{"id": "batch_req_123", "custom_id": "request-2", "response": {"status_code": 200, "request_id": "req_123", "body": {"id": "chatcmpl-123", "object": "chat.completion", "created": 1711652795, "model": "kimi-k3-fast", "choices": [{"index": 0, "message": {"role": "assistant", "content": "Hello."}, "logprobs": null, "finish_reason": "stop"}], "usage": {"prompt_tokens": 22, "completion_tokens": 2, "total_tokens": 24}, "system_fingerprint": "fp_123"}}, "error": null}
{"id": "batch_req_456", "custom_id": "request-1", "response": {"status_code": 200, "request_id": "req_789", "body": {"id": "chatcmpl-abc", "object": "chat.completion", "created": 1711652789, "model": "kimi-k3-fast", "choices": [{"index": 0, "message": {"role": "assistant", "content": "Hello! How can I assist you today?"}, "logprobs": null, "finish_reason": "stop"}], "usage": {"prompt_tokens": 20, "completion_tokens": 9, "total_tokens": 29}, "system_fingerprint": "fp_3ba"}}, "error": null}

#List batches

List all your batches, newest first. Use limit (1–100, default 20) and after to paginate: pass the last_id of one page as after to get the next.

const list = await client.batches.list({
  limit: 10,
  after: batch.id,
});

for await (const b of list) {
  console.log(b);
}

#Manage files

List, inspect, and delete your batch input and output files with the standard OpenAI Files endpoints. Files are listed newest first; purpose is batch for inputs and batch_output for result and error files. A file can't be deleted while it is the input of a batch that hasn't finished.

curl 'https://batch.inference.net/v1/files?limit=20' -H "Authorization: Bearer $INFERENCE_API_KEY"
curl https://batch.inference.net/v1/files/file_abc123 -H "Authorization: Bearer $INFERENCE_API_KEY"
curl -X DELETE https://batch.inference.net/v1/files/file_abc123 -H "Authorization: Bearer $INFERENCE_API_KEY"

#Cancel a batch

Cancel an in-progress batch with POST /v1/batches/{batch_id}/cancel:

const cancelled = await client.batches.cancel("batch_abc123");

What happens:

1. The batch moves to cancelling at once. The response is the batch object.
2. No more requests start. Requests that are already running finish.
3. Within about 10 minutes the batch moves to cancelled, and your webhook_url receives "status": "cancelled".
4. output_file_id holds the requests that finished. error_file_id holds failed requests and the requests that did not finish, with error code batch_cancelled.

Cancelling a batch that is already in a final state returns it unchanged.

#Batch expiration

A batch that does not finish within its completion window closes as expired:

1. Requests that have not started are not run, and you are not charged for them.
2. output_file_id holds the requests that finished. error_file_id holds failed requests and the requests that did not finish, with error code batch_expired.
3. Your webhook_url receives "status": "expired".

Requests that were never dispatched before the batch closed do not appear in either file.

#Rate limits

Batch API rate limits are separate from per-model rate limits. A batch input file may include up to 1,000,000 requests (one JSONL line each); see Rate limits for request limits. If you need higher limits, contact support@inference.net.

#Model availability

Batch processing works with every model that accepts asynchronous requests; a model that is only available for synchronous requests returns a 400. See the model catalog for the full list of models.