Testing

Agent tests that hit a real model are slow, flaky, and expensive, so they end up not being written. just-another-sdk/testing ships the helpers to avoid that — the same ones the SDK's own offline test suite uses, which is why they run in about 300ms with no key and no network.

# nothing to install — it is part of the package

Scripting a model

mockProvider takes a script of turns. A turn with toolCalls makes the agent execute tools; a turn with only text ends the run.

import { describe, expect, it } from 'vitest'
import { Agent } from 'just-another-sdk'
import { mockProvider } from 'just-another-sdk/testing'

it('answers using the weather tool', async () => {
  const model = mockProvider([
    { toolCalls: [{ toolName: 'get_weather', input: { city: 'Paris' } }] },
    { text: 'It is 18°C and clear in Paris.' },
  ])

  const agent = new Agent({ name: 'test', model, tools: [getWeather] })
  const result = await agent.run('Weather in Paris?')

  expect(result.text).toBe('It is 18°C and clear in Paris.')
  expect(result.stopReason).toBe('finish')
  expect(result.turns).toBe(2)
})

Asserting on what was sent

Every request is recorded, so you can test the prompt as well as the answer:

expect(model.calls).toHaveLength(2)
expect(model.calls[0].system).toBe('Be concise.')
expect(model.calls[0].tools?.map((tool) => tool.name)).toEqual(['get_weather'])

// The second call must carry the tool-call turn and its result
const second = model.calls[1].messages
expect(second.at(-2)?.role).toBe('assistant')
expect(second.at(-1)?.role).toBe('tool')

Prop

Type

Streaming

The mock implements stream() as well as generate(), and both derive their response from the same place — so "the deltas concatenate to the final text" is guaranteed by construction rather than something your test has to assume:

const model = mockProvider([{ text: 'Hello there.' }])
const stream = new Agent({ name: 'test', model }).stream('hi')

let streamed = ''
for await (const chunk of stream.textStream()) streamed += chunk

expect(streamed).toBe((await stream).text)

To test the fallback for providers that do not stream, omit the method entirely:

const model = mockProvider([{ text: 'ok' }], { supportsStreaming: false })
// agent.stream() still works; `model.streamCallCount` stays 0.

Tool calls stream as two fragments each, so argument reassembly is genuinely exercised. streamCallCount tells you which path a run took.

Dynamic responses

Pass a function to react to the incoming request:

const model = mockProvider((request, index) => {
  const lastMessage = request.messages.at(-1)
  if (index === 0)
    return { toolCalls: [{ toolName: 'search', input: { q: 'x' } }] }
  return { text: `I saw ${request.messages.length} messages.` }
})

Catching extra model calls

By default the last turn repeats when the script runs out, which keeps a loop test from failing for the wrong reason. Use 'throw' to assert an exact call count:

const model = mockProvider([{ text: 'done' }], { onExhausted: 'throw' })
// A second model call now fails the test loudly.

Asserting on events

collectEvents records the run so you can test what the agent did, not just what it returned. An ordering assertion is the cheapest way to catch a regression in the loop:

import { collectEvents, mockProvider } from 'just-another-sdk/testing'

const collected = collectEvents()

await new Agent({ name: 'test', model, tools: [getWeather] }).run('go', {
  onEvent: collected.listener,
})

expect(collected.types()).toEqual([
  'run.start',
  'model.request',
  'model.response',
  'tool.start',
  'tool.end',
  'model.request',
  'model.response',
  'run.finish',
])

expect(collected.first('tool.end')?.isError).toBe(false)
expect(collected.last('run.finish')?.stopReason).toBe('finish')
expect(collected.count('model.request')).toBe(2)

Testing the paths that matter

The interesting tests are the failure ones, and they are all offline.

it('recovers when a tool throws', async () => {
  const broken = tool({
    name: 'broken',
    description: 'Always fails.',
    execute: () => {
      throw new Error('upstream down')
    },
  })

  const model = mockProvider([
    { toolCalls: [{ toolName: 'broken' }] },
    { text: 'Sorry, that service is unavailable.' },
  ])

  const result = await new Agent({ name: 'test', model, tools: [broken] }).run(
    'go',
  )

  // The run completed rather than throwing
  expect(result.stopReason).toBe('finish')
  expect(result.steps[0].toolResults[0].isError).toBe(true)
})

it('stops at maxTurns instead of looping forever', async () => {
  // A model that always calls a tool — the pathological case
  const model = mockProvider([
    { toolCalls: [{ toolName: 'get_weather', input: { city: 'X' } }] },
  ])

  const result = await new Agent({
    name: 'test',
    model,
    tools: [getWeather],
    maxTurns: 3,
  }).run('go')

  expect(result.stopReason).toBe('max_turns')
  expect(model.calls).toHaveLength(3)
})

it('rejects invalid tool input without calling the handler', async () => {
  const execute = vi.fn()
  const strict = tool({
    name: 'strict',
    description: 'x',
    inputSchema: z.object({ city: z.string() }),
    execute,
  })

  const model = mockProvider([
    { toolCalls: [{ toolName: 'strict', input: { city: 123 } }] },
    { text: 'recovered' },
  ])

  await new Agent({ name: 'test', model, tools: [strict] }).run('go')

  expect(execute).not.toHaveBeenCalled()
})

it('surfaces a provider failure as a typed error', async () => {
  const model = mockProvider([{ error: new Error('boom') }])
  const agent = new Agent({ name: 'test', model })

  await expect(agent.run('go')).rejects.toMatchObject({
    code: 'provider_error',
  })
})

Structured output, including the repair

A malformed-then-valid script is the repair path — no extra mock capability needed:

it('repairs an answer that failed the schema', async () => {
  const model = mockProvider(
    [
      { text: 'Sure! ```json\n{"severity":"high","summary":"x"}\n```' },
      { text: '{"severity":4,"summary":"x"}' },
    ],
    { onExhausted: 'throw' }, // fails loudly if a third call happens
  )

  const result = await new Agent({
    name: 'test',
    model,
    outputSchema: z.object({ severity: z.number(), summary: z.string() }),
  }).run('go')

  expect(result.output.severity).toBe(4)
  expect(result.steps.map((s) => s.kind)).toEqual(['turn', 'repair'])
  expect(result.turns).toBe(1) // a repair is not a turn
})

Drop maxOutputRetries: 0 in to assert the failure instead — it rejects with an InvalidOutputError carrying issues and rawText.

Guardrails and approval

The two assertions worth writing: a rejected input costs nothing, and a blocked tool call leaves the run recoverable.

it('rejects before spending a token', async () => {
  const model = mockProvider([{ text: 'never reached' }])
  const agent = new Agent({
    name: 'test',
    model,
    inputGuardrails: [{ name: 'too-long', check: () => ({ reject: 'Nope.' }) }],
  })

  await expect(agent.run('hi')).rejects.toThrow(GuardrailError)
  expect(model.calls).toHaveLength(0) // the receipt
})

it('suspends and resumes', async () => {
  const error = (await agent
    .run('refund 500')
    .catch((e) => e)) as ApprovalRequiredError

  const final = await agent.resumeApproval(error.suspension, [
    {
      toolCallId: error.suspension.pending.calls[0].toolCallId,
      approved: true,
    },
  ])

  expect(final.steps.map((s) => s.kind)).toEqual(['resume', 'turn'])
})

Testing a provider

Providers accept an injected fetch, so you can exercise the real translation and error-mapping code with no network:

const fetchMock: typeof fetch = (url, init) =>
  Promise.resolve(
    new Response(
      JSON.stringify({
        choices: [{ finish_reason: 'stop', message: { content: 'hi' } }],
        usage: { prompt_tokens: 1, completion_tokens: 1, total_tokens: 2 },
      }),
      { status: 200, headers: { 'content-type': 'application/json' } },
    ),
  )

const model = openrouter('anthropic/claude-opus-5', {
  apiKey: 'sk-test',
  fetch: fetchMock,
})

Use it to pin error mapping too:

const rateLimited: typeof fetch = () =>
  Promise.resolve(
    new Response('{}', { status: 429, headers: { 'retry-after': '30' } }),
  )

await expect(agent.run('go')).rejects.toMatchObject({
  code: 'rate_limit_error',
  retryable: true,
  retryAfterMs: 30_000,
})

Should you ever test against a real model?

Yes — but keep it in a separate suite, gated on a key being present, so CI stays fast and deterministic:

// The whole suite disappears when there is no key.
describe.skipIf(!process.env.OPENROUTER_API_KEY)('live', () => {
  it('round-trips a real tool call', async () => {
    const agent = new Agent({
      name: 'live',
      model: openrouter('openai/gpt-4o-mini'), // cheapest tool-capable model
      tools: [getWeather],
      maxOutputTokens: 128,
    })

    const result = await agent.run('What is the weather in Paris?')
    expect(result.stopReason).toBe('finish')
    expect(result.usage.inputTokens).toBeGreaterThan(0)
  }, 60_000) // real calls need a real timeout
})

This is worth the effort. A mock provider proves the loop is correct; only a real model proves the wire format is — request translation, tool-call round-tripping, and provider error mapping. This SDK's own live suite caught a bug the entire offline suite missed: a run-level timeout was being reported as a network error, because real fetch rejects with the abort reason object while the mock rejected with a generic AbortError.

The SDK's own version of this is packages/core/test/live.test.ts — 12 tests covering parallel tool calls, failure recovery, forced tool choice, maxTurns, authentication failure, cancellation, and multi-turn context. Total cost per run is a fraction of a cent.

Events & tracing

The event stream you are asserting on.

Examples

Runnable projects in the repository.