OpenAI

Braintrust integrates with OpenAI so you can call GPT models from the Braintrust playground, API, and SDKs. Braintrust also traces OpenAI SDK calls from your application, including streaming, structured outputs, and function calling.

Add OpenAI as an AI provider

To use OpenAI models in the Braintrust playground, API, and gateway, connect OpenAI as a provider in your organization or project AI providers.

Go to Settings > AI providers.
Click Organization provider or Project provider, depending on whether you want the provider to be available across every project in the organization or just the current project.
Under Model providers, click OpenAI.
Choose your authentication method:
- API key: Visit OpenAI’s API platform, create a new API key, and paste it into the Secret field.
  API keys are stored as one-way cryptographic hashes, never in plaintext.
- Workload identity federation: Exchange a Braintrust-signed OIDC token for an OpenAI access token, instead of storing a long-lived OpenAI API key in Braintrust.
  Workload identity federation is available only for organization-level providers on Braintrust-hosted organizations with the Braintrust gateway enabled. Project-level providers and self-hosted deployments must use API key authentication.
If you chose Workload identity federation, use the setup values shown in Braintrust to configure OpenAI:
1. Create a workload identity provider in OpenAI. Enter a descriptive Name, use the OIDC issuer URL and Audience shown in Braintrust, and leave uploaded JWKS and attribute transformations disabled.
2. From the workload identity provider details page, create a mapping. Use sub as the Key and the subject pattern shown in Braintrust as the Value. Add a mapping attribute for each additional claim shown in Braintrust. Choose the OpenAI Project, Service account, and Permissions Braintrust should use.
3. Paste the OpenAI IDs back into Braintrust:
  - Identity provider ID: The workload identity provider ID configured for Braintrust.
  - Service account ID: The OpenAI service account ID Braintrust should use.
  - Subject suffix: A stable suffix for this OpenAI connection. It must match the final part of the subject pattern used in OpenAI.
For general OpenAI concepts and dashboard details, see OpenAI’s workload identity federation docs.
Click Save.

For the GPT-5 family, whether the temperature parameter is configurable depends on the reasoning effort. GPT-5.1 and later accept temperature only when reasoning effort is set to none. At any higher reasoning effort, and for older GPT-5 models (gpt-5, gpt-5-mini, gpt-5-nano) and GPT-5 Pro, temperature isn’t configurable and is disabled in the Braintrust UI.

Tracing

Braintrust traces OpenAI calls automatically with the braintrust/hook.mjs import hook, or manually with wrapOpenAI. Either path produces the same spans. Auto-instrumentation is the recommended path for most users.

Using the OpenAI Agents SDK? See the OpenAI Agents SDK framework docs.

Setup

Install the Braintrust SDK alongside the OpenAI SDK, then configure your API keys.

Install packages

pnpm add braintrust openai

npm install braintrust openai

Get an OpenAI API key

Visit OpenAI’s API platform and create a new API key, then add it as a Braintrust AI provider.

Set environment variables

.env

OPENAI_API_KEY=<your-openai-api-key>
BRAINTRUST_API_KEY=<your-braintrust-api-key>

# For organizations on the EU data plane, use https://api-eu.braintrust.dev
# For self-hosted deployments, use your data plane URL
# BRAINTRUST_API_URL=<your-braintrust-api-url>

Auto-instrumentation

To trace OpenAI calls without modifying your application code, initialize Braintrust, create a normal OpenAI client, then run your app with Braintrust’s import hook to patch the OpenAI SDK at startup.

Initialize Braintrust and call OpenAI

import OpenAI from "openai";
import { initLogger } from "braintrust";

initLogger({
  projectName: "My Project", // Replace with your project name
  apiKey: process.env.BRAINTRUST_API_KEY,
});

const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

const result = await client.chat.completions.create({
  model: "gpt-5-mini",
  messages: [{ role: "user", content: "What is machine learning?" }],
});

Run with the import hook

node --import braintrust/hook.mjs trace-openai-auto.js

The auto-instrumentation example uses plain JavaScript so node --import can run the file directly. The Braintrust APIs work the same in TypeScript projects — compile your TypeScript to JavaScript, then run the compiled file with the import hook.

If you’re using a bundler, see Trace LLM calls for plugin and loader setup.

Manual instrumentation

To trace OpenAI calls manually, wrap your client with wrapOpenAI yourself. Once wrapped, every chat.completions.create call (including streaming) emits a span.

import OpenAI from "openai";

// Initialize the Braintrust logger
const logger = initLogger({
  projectName: "My Project", // Your project name
  apiKey: process.env.BRAINTRUST_API_KEY,
});

// Wrap the OpenAI client with wrapOpenAI
const client = wrapOpenAI(
  new OpenAI({
    apiKey: process.env.OPENAI_API_KEY,
  }),
);

// All API calls are automatically logged
const result = await client.chat.completions.create({
  model: "gpt-5-mini",
  messages: [
    { role: "system", content: "You are a helpful assistant." },
    { role: "user", content: "What is machine learning?" },
  ],
});

For more control over tracing, learn how to customize traces.

Streaming

wrapOpenAI can automatically log metrics like prompt_tokens, completion_tokens, and tokens for streaming LLM calls if the LLM API returns them. Set include_usage to true in the stream_options parameter to receive these metrics from OpenAI.

model: "gpt-5-mini",
  messages: [{ role: "user", content: "Count to 10" }],
  stream: true,
  stream_options: {
    include_usage: true, // Required for token metrics
  },
});

for await (const chunk of result) {
  process.stdout.write(chunk.choices[0]?.delta?.content || "");
}

Structured outputs

OpenAI’s structured outputs are supported with the wrapper functions.

import { z } from "zod";

// Define a Zod schema for the response
const ResponseSchema = z.object({
  name: z.string(),
  age: z.number(),
});

const completion = await client.beta.chat.completions.parse({
  model: "gpt-5-mini",
  messages: [
    { role: "system", content: "Extract the person's name and age." },
    { role: "user", content: "My name is John and I'm 30 years old." },
  ],
  response_format: {
    type: "json_schema",
    json_schema: {
      name: "person",
      // The Zod schema for the response
      schema: ResponseSchema,
    },
  },
});

Function calling and tools

Braintrust supports OpenAI function calling for building AI agents with tools.

const tools = [
  {
    type: "function" as const,
    function: {
      name: "get_weather",
      description: "Get current weather for a location",
      parameters: {
        type: "object",
        properties: {
          location: { type: "string" },
        },
        required: ["location"],
      },
    },
  },
];

const response = await client.chat.completions.create({
  model: "gpt-5-mini",
  messages: [{ role: "user", content: "What's the weather in San Francisco?" }],
  tools,
});

To trace multimodal content, attachments, errors, and masking sensitive data, see the customize traces guide.

Gateway

To call OpenAI through the Braintrust gateway, point your client at the gateway base URL and use your Braintrust API key for authentication. Use any supported provider’s SDK to call OpenAI models.

const client = new OpenAI({
  baseURL: "https://gateway.braintrust.dev/v1",

  apiKey: process.env.BRAINTRUST_API_KEY,
});

const response = await client.chat.completions.create({
  model: "gpt-5-mini",
  messages: [{ role: "user", content: "What is a proxy?" }],
  seed: 1, // A seed activates the proxy's cache
});

The gateway also supports the OpenAI-compatible /embeddings endpoint for generating embeddings. See Generate embeddings for an example.

What Braintrust traces

Braintrust instruments the OpenAI client and emits an LLM span per call. Chat and responses spans capture the request messages or input, the response output, and the request parameters as metadata. Embeddings and moderation spans capture their inputs and results.Spans

Span	Coverage
`Chat Completion`	`chat.completions.create`, `.parse`, and `.stream` (including streaming)
`Embedding`	`embeddings.create` (output records the embedding length, not the raw vector)
`Moderation`	`moderations.create`
`openai.responses.create`	`responses.create` and the `responses.stream` helper (including streaming)
`openai.responses.parse`	`responses.parse` (structured outputs)
`openai.responses.compact`	`responses.compact` (OpenAI SDK v6.10.0 or later)

Metrics

Metric	Description
`prompt_tokens`	Input tokens
`completion_tokens`	Output tokens
`tokens`	Total tokens
`prompt_cached_tokens`	Tokens read from the prompt cache
`completion_reasoning_tokens`	Reasoning tokens
`time_to_first_token`	First-token latency (chat and responses spans)
`cached`	Whether the response was served from the Braintrust cache

Tracing resources

Evals

Evaluations help you distill the non-deterministic outputs of OpenAI models into an effective feedback loop that enables you to ship more reliable, higher quality products. Braintrust Eval is a simple function composed of a dataset of user inputs, a task, and a set of scorers. To learn more about evaluations, see the Experiments guide.

Basic eval setup

Evaluate the outputs of OpenAI models with Braintrust.

import { Eval } from "braintrust";
import { OpenAI } from "openai";

const client = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

Eval("OpenAI Evaluation", {
  // An array of user inputs and expected outputs
  data: () => [
    { input: "What is 2+2?", expected: "4" },
    { input: "What is the capital of France?", expected: "Paris" },
  ],
  task: async (input) => {
    // Your OpenAI LLM call
    const response = await client.chat.completions.create({
      model: "gpt-5-mini",
      messages: [{ role: "user", content: input }],
    });
    return response.choices[0].message.content;
  },
  scores: [
    {
      name: "accuracy",
      // A simple scorer that returns 1 if the output matches the expected output, 0 otherwise
      scorer: (args) => (args.output === args.expected ? 1 : 0),
    },
  ],
});

Learn more about eval data and scorers.

Use OpenAI as an LLM judge

You can use OpenAI models to score the outputs of other AI systems. This example uses the LLMClassifierFromSpec scorer to score the relevance of the outputs of an AI system.Install the autoevals package to use the LLMClassifierFromSpec scorer.

pnpm add autoevals

npm install autoevals

Create a scorer that uses the LLMClassifierFromSpec scorer to score the relevance of the outputs of an AI system. You can then include relevanceScorer as a scorer in your Eval function (see above).

import { LLMClassifierFromSpec } from "autoevals";

const relevanceScorer = LLMClassifierFromSpec("Relevance", {
  choice_scores: { Relevant: 1, Irrelevant: 0 },
  model: "gpt-5-mini",
  use_cot: true,
});

AI providers

SDKs

Developer tools

Add OpenAI as an AI provider

Tracing

Setup

Auto-instrumentation

Manual instrumentation

Streaming

Structured outputs

Function calling and tools

Gateway

What Braintrust traces

Tracing resources

Evals

Basic eval setup

Use OpenAI as an LLM judge

Cookbooks

​Add OpenAI as an AI provider

​Tracing

​Setup

​Auto-instrumentation

​Manual instrumentation

​Streaming

​Structured outputs

​Function calling and tools

​Gateway

​What Braintrust traces

​Tracing resources

​Evals

​Basic eval setup

​Use OpenAI as an LLM judge

​Cookbooks

Add OpenAI as an AI provider

Tracing

Setup

Auto-instrumentation

Manual instrumentation

Streaming

Structured outputs

Function calling and tools

Gateway

What Braintrust traces

Tracing resources

Evals

Basic eval setup

Use OpenAI as an LLM judge

Cookbooks