Feedback signals

Feedback signals capture what users think of a specific AI response (thumbs up/down, free-text comments, numeric scores, or corrected output) and link the signal back to the exact span of that response. This lets you analyze response quality across users, chats, cohorts, and prompt variants.

Linking feedback to an AI response

The key challenge: feedback happens in the browser (user clicks thumbs up) but the span ID that identifies the AI response is created on the server. You need to bridge the two.

Step 1 — Capture the span ID (server)

After a wrapped AI call, getSpanId() returns the Otis span ID for that response:

import { getSpanId } from "@runotis/sdk";

const result = await generateText({
  model: anthropic("claude-sonnet-4-6"),
  prompt: "...",
});
const spanId = getSpanId(result);

Pass spanId to the client alongside the response. How depends on your framework:

• Vercel AI SDK useChat: include it as a message annotation or data stream annotation
• REST API: include it in the JSON response body
• Server-rendered: inject it as a data-span-id attribute on the response element

Step 2 — Send the signal (browser or server)

When the user clicks a feedback button, call sendFeedbackSignal with the span ID:

import { sendFeedbackSignal } from "@runotis/sdk";

sendFeedbackSignal(spanId, "thumbs_up");

This works from both server and browser. On Next.js, sendFeedbackSignal is also exposed from the useOtis() hook so client components can send feedback directly.

If span ID transport is impractical

If bridging the span ID to the client is too complex for your current architecture, you can call sendFeedbackSignal server-side in the same request where the AI call happens — for example, in an onFinish callback or a follow-up API endpoint that receives the feedback and still has the span ID in scope.

Variants

// Thumbs down with a comment
sendFeedbackSignal("span-abc123", "thumbs_down", {
  comment: "Answer was off-topic",
});

// Numeric scores (0–1 range)
sendFeedbackSignal("span-abc123", "rating", {
  scores: { relevance: 0.3, accuracy: 0.8 },
  comment: "Mostly accurate but missed the point",
  userId: "user-123",
});

// User correction — capture the edit the user actually wanted
sendFeedbackSignal("span-abc123", "user_edit", {
  type: "edit",
  expected: "The corrected response text that the user actually wants.",
  comment: "Reworded to be clearer",
  userId: "user-123",
});

Built-in signal names

thumbs_up, thumbs_down, confused, helpful, hallucination_suspected, user_edit. Custom names are fine too; they're freeform.

Signal types

Categorize the signal beyond its name. Stored as signalType on the span:

Fields

The comment and expected fields are automatically scanned for PII before they reach analytics storage. See Privacy.