Response Streaming and Buffering

WitnessAI streams model responses by default. As the AI model generates its reply, WitnessAI passes it to the user in real time — the same progressive, token-by-token experience users expect from the model directly.

Streaming is automatic. There is no setting to enable it. WitnessAI buffers a response only when the active policy includes a configuration that requires the response to be inspected or modified before the user sees it.

This page explains which configurations cause buffering and how to design policies that preserve the streaming experience.

💡

Note: This page is about response delivery only. Prompt-side inspection and enforcement — blocking, routing, or alerting on a user’s prompt — work the same regardless of whether responses stream or buffer.

How buffering works

When a policy requires buffering, WitnessAI holds every response frame from the model in memory until the reply is complete. It then evaluates the full response, applies any required changes, and delivers the result to the user. The model takes the same total time to generate the response either way — the last word arrives at roughly the same time whether the response was streamed or buffered. What changes is the time to the first word: with streaming, users see content almost immediately as the model starts generating; with buffering, nothing appears until the entire response has been generated and evaluated. For short responses this difference is barely noticeable. For longer replies, the wait before any text appears can be significant.

Three policy configurations cause this behavior.

1. Response-inspection GuardRails set to Warn or Block

Two GuardRails in WitnessAI evaluate the model’s response — not the user’s prompt:

Harmful Response Prevention analyzes the model’s reply for harmful content and can Allow, Warn, or Block it.
Model Identity Protection checks that the model’s reply is consistent with its intended identity and can Allow or Block it.

The action you choose on these GuardRails determines whether responses are buffered.

The seven GuardRails available in a policy, including the two response-inspection GuardRails.

GuardRails available in a policy. Harmful Response Prevention and Model Identity Protection are the two that evaluate the model’s response.

Allow — responses stream, violations are still recorded

When a response-inspection GuardRail is set to Allow, WitnessAI evaluates every response asynchronously — after it has already been delivered to the user. Responses stream without any delay. Any violations detected are recorded and appear as alerts in the console, giving administrators full visibility into response activity without affecting the user experience.

Harmful Response Prevention enabled with the action set to Allow.

Harmful Response Prevention set to Allow. Responses stream. Violations are recorded and visible as alerts.

Warn or Block — all responses are buffered

When a response-inspection GuardRail is set to Warn or Block, WitnessAI must hold the complete response before evaluating it, because it may need to deliver a warning message or replace the response entirely. This means every response covered by the policy is buffered — not just the responses that eventually match a rule.

In practice, only a small fraction of responses typically trigger a harmful-response or model-identity violation. Choosing Warn or Block affects the experience for all users under that policy, even those whose responses would never have matched.

Harmful Response Prevention enabled with the action set to Block.

Harmful Response Prevention set to Block. All responses covered by this policy are buffered, regardless of whether they match.

Model Identity Protection enabled, with Allow and Block as the available actions.

Model Identity Protection set to Block also buffers all responses covered by the policy.

2. The Warn action on a prompt-side GuardRail

When a prompt-side GuardRail (Behavioral Activity, Organizational Behavior, Risk Analysis, or Model Protection) is set to Warn, WitnessAI inserts a warning message into the response for the user to see. Inserting that message requires modifying the response before it is delivered.

Unlike response-inspection GuardRails, this buffering is per-prompt, not per-policy: the response is buffered only when a specific prompt actually matches a rule and triggers the Warn action. Prompts that do not match any rule are unaffected — their responses stream as normal.

Risk Analysis enabled with the action set to Warn.

Risk Analysis set to Warn. The response is buffered only for prompts that match a rule and trigger the warning message.

3. Data Protection restoring anonymized values in responses

You can configure Data Protection to anonymize sensitive data in prompts: when enabled, it replaces detected values — such as credit card numbers or Social Security Numbers — with placeholders before the prompt is sent to the model. By default, WitnessAI also restores the original values in the response, so the user sees their actual data while the model never received it. When this restoration is active, the response must be buffered so the placeholders can be replaced before the user sees them.

The Data Protection anonymization options. Anonymization is enabled by default when the GuardRail is turned on. Show masked data in model response controls whether WitnessAI restores original values in the response.

Show masked data in model response

Enabling Show masked data in model response tells WitnessAI to skip the restoration step and leave the placeholder values visible in the response. With restoration disabled, most applications can stream responses even when a prompt was anonymized.

[!NOTE] Some applications buffer even when Show masked data is enabled. For certain AI applications — including Claude, Claude Code, and Codex — anonymizing a prompt requires additional processing in the response that cannot be skipped. For these applications, Data Protection anonymization causes the response to be buffered regardless of the Show masked data setting. If guaranteed streaming is required for these applications, do not enable Data Protection anonymization in the policies that apply to them.

Summary

Configuration	Effect on responses
No response-side inspection or modification	Streams (default)
Harmful Response Prevention or Model Identity Protection — Allow	Streams — violations recorded asynchronously as alerts
Harmful Response Prevention or Model Identity Protection — Warn or Block	Buffered — all responses under the policy
Any GuardRail — Warn — prompt matches a rule	Buffered — only for the matched prompt
Any GuardRail — Warn — prompt does not match	Streams
Data Protection anonymization, default (restores values in response)	Buffered — when a prompt is anonymized
Data Protection anonymization, Show masked data enabled	Streams for most applications; buffered for Claude, Claude Code, and Codex
Prompt-side GuardRails (Behavioral Activity, Organizational Behavior, Risk Analysis, Model Protection) — Allow or Block	Streams

Best practices

Start response GuardRails on Allow and observe

Before setting Harmful Response Prevention or Model Identity Protection to Warn or Block, enable them with Allow first. In Allow mode, WitnessAI evaluates every response and records violations as alerts — without buffering any responses. Review the Alerts console over time to understand how often responses actually trigger a violation and which users are involved. This data helps you decide whether enforcement is warranted and for whom, before accepting the trade-off for all users.

Scope enforcing policies to specific users or groups

Because Warn and Block buffer all responses covered by the policy — not just matching ones — applying an enforcing response GuardRail to the entire organization is a significant trade-off. After observing traffic in Allow mode, consider creating a policy scoped to the specific users or groups that account for most violations. Users outside that scope continue to stream; the buffering impact is limited to those who need it.

Use “Show masked data in model response” when value restoration is not required

If users do not need to see their original values restored in the response — for example, when masked values are acceptable or when responses are consumed programmatically — enable Show masked data in model response in Data Protection. This eliminates the buffering caused by anonymization for most applications.

Review the Alerts dashboard before tightening actions

The Alerts dashboard records every GuardRail match, including response-inspection GuardRail matches captured in Allow mode. Before switching an action from Allow to Warn or Block, verify that the volume and pattern of matches justifies the trade-off. A response GuardRail with very few matches may not warrant buffering all responses for all affected users.

Response streaming and buffering How buffering works 1. Response-inspection GuardRails set to Warn or Block Allow — responses stream, violations are still recorded Warn or Block — all responses are buffered 2. The Warn action on a prompt-side GuardRail 3. Data Protection restoring anonymized values in responses Show masked data in model response Summary Best practices Start response GuardRails on Allow and observe Scope enforcing policies to specific users or groups Use “Show masked data in model response” when value restoration is not required Review the Alerts dashboard before tightening actions