Ai Keeper/Guides/Lab and evaluation
← All workflows
Collection 10 · Test before trust

Lab and evaluation

Send raw requests, exercise stateful Responses, compare models, benchmark performance, keep regression checks in QA Lab, transcribe audio, and inspect Harness evidence. Browser and Canvas evaluation workflows open from Chat.

10 workflowsBeginner-first

Find one workflow

Filter this collection by name, outcome, provider, engine, channel, mode, or command.

10 shown
Workflow 01

Send a raw request in Playground

Test an endpoint and payload without writing a separate script.

Intermediate5–10 minResult · Request and response are inspectable✓ Source-audited workflow
Current Lab Playground with request method, endpoint, headers, JSON body, and response panel.
Start with a harmless non-streaming request and inspect the response before testing stateful or tool payloads.
Before you start
  • A Ready route.
  • Know which capability you are testing.

  1. Open Lab > Playground and choose the target instance/model and endpoint type.

  2. Start with the generated example payload. Change only the prompt and one parameter.

  3. Send and inspect status, headers, body, timing, and streaming events if enabled.

  4. For embeddings or reranking, choose the matching specialist instance and verify vector/ranking output rather than chat text.

  5. Copy a working request only after it succeeds here.

You are done when
The response has the expected status and schema for the selected endpoint, and Activity > Requests shows the same call.
If something looks wrong
400 responseRead the returned validation message and compare payload shape to the generated example.
Testing raw instance URLUse the proxy target when you want to validate real app routing and transforms.
Workflow 02

Test the stateful Responses API

Exercise response state, compacted history, and WebSocket transport against a local runtime.

Advanced10–20 minResult · A multi-turn response continues by state ID✓ Source-audited workflow
Lab navigation showing Responses beside Playground, Compare, Benchmark, QA Lab, Transcription, and Harness.
Use Responses for stateful calls and Playground for a raw one-request baseline.
Before you start
  • A Ready compatible route.
  • Basic chat completion already works.

  1. Open Lab > Responses and select the model.

  2. Send a first request and record the response/state identifier shown by the UI.

  3. Send a follow-up linked to the previous response instead of resending the whole history.

  4. Test compacted-history payload handling with the built-in example, then switch transport to WebSocket when available.

  5. Inspect events and ensure the response remains tied to the same process generation.

You are done when
The follow-up remembers the first turn, compacted input is accepted, and WebSocket events complete without crossing to a stale process.
If something looks wrong
Instance restarted mid-sessionStart a new stateful response; generation-pinned state should not be silently reused across processes.
Workflow 03

Compare two models side by side

Run one prompt with matched settings and judge quality, format, and behavior fairly.

Beginner5–20 minResult · A documented model choice✓ Source-audited workflow
Current Lab hub; Compare sits alongside Playground in the tab bar.
Seven Server-mode Lab tools share this hub: Playground, Responses, Compare, Benchmark, QA Lab, Transcription, and Harness.
Before you start
  • Two Ready chat models/routes.
  • One representative prompt and success criteria.

  1. Open Lab > Compare and choose model A and model B.

  2. Use the same system context, prompt, output limit, temperature/profile, and tool availability for both.

  3. Run the prompt and wait for both sides to finish.

  4. Score correctness, instruction following, useful detail, format, latency, and any tool behavior. Do not choose only by writing style.

  5. Repeat with at least one different representative prompt before changing a production route.

You are done when
You can explain the choice using observed criteria and more than one prompt, not a single subjective impression.
If something looks wrong
One model had different tools/contextThe comparison is invalid. Match inputs and policy before interpreting output quality.
Workflow 04

Benchmark TTFT, throughput, and latency

Measure repeatable performance after warm-up and watch memory pressure while the run executes.

Intermediate10–20 minResult · A repeatable performance baseline✓ Source-audited workflow
Current Lab hub tab bar including Benchmark.
Benchmark is where TTFT and throughput are measured; do not infer speed from chat feel.
Before you start
  • A stable Ready instance.
  • No unrelated heavy workloads if you want comparable numbers.

  1. Open Lab > Benchmark, choose the instance, prompt size, output size, and repeat count.

  2. Run one warm-up that you do not treat as the final number.

  3. Run the repeat set while watching Monitor for memory/CPU pressure.

  4. Record time to first token, tokens per second, and end-to-end latency together.

  5. After a tuning change, rerun the exact same benchmark and environment.

You are done when
Repeated warm runs are in a reasonably consistent range and comparisons use the same prompt/output settings.
If something looks wrong
Cold and warm results mixedLabel them separately; model load and first compilation/cache work distort the first run.
Only tokens/sec improvedCheck TTFT, memory, and output quality too.
Workflow 05

Create and inspect a managed browser session

Open a page, review the visible state, perform one action, and audit what the model did.

Intermediate5–15 minResult · A browser action is visible and logged✓ Source-audited workflow
Chat Browser session tool showing a loaded isolated research board, connected state, profile, URL bar, and New Tab.
The local demonstration page proves navigation completed while keeping the session disposable and free of personal browsing data.
Before you start
  • A web task that is allowed by the site and your policy.
  • No need to expose credentials to the model unless explicitly approved.

  1. Open Chat and choose Browser from the session tools; create or select a browser profile and session.

  2. Navigate to a harmless page and inspect the preview/DOM state before acting.

  3. Run one simple action such as following a link or filling a non-sensitive test field.

  4. Review the LLM action log and screenshot/preview to confirm what changed.

  5. Stop or reset the session when finished.

You are done when
The visible page reflects the intended single action and the action log attributes it correctly.
If something looks wrong
Model wants to submit/purchase/sendRequire explicit confirmation and inspect the final target and payload.
Page changed unexpectedlyStop and capture the current state before retrying; do not stack blind clicks.
Workflow 06

Create and snapshot a Canvas artifact

Render live HTML, charts, or interactive output, inspect source, and preserve a known-good snapshot.

Intermediate10–30 minResult · A live artifact and snapshot exist✓ Source-audited workflow
Chat Canvas session tool showing a generated release-readiness dashboard with summary cards and a request-volume chart.
Use Source or Snapshot after a visual artifact renders correctly; Reset is available when you want to start over.
Before you start
  • HTML/chart/interactive source from Chat, an agent, or your own edit.

  1. Open Chat and choose Canvas from the session tools; create or load the artifact source.

  2. Render/preview it and interact with the important controls.

  3. Open the source view and correct any visible error rather than repeatedly asking the model to regenerate everything.

  4. Create a snapshot when the artifact is working. Make one change, then use history/reset to prove recovery.

  5. Export or hand the artifact to the next workflow only after the snapshot test passes.

You are done when
The artifact renders, key interactions work, source is inspectable, and the known-good snapshot can be restored.
If something looks wrong
Arbitrary script from an untrusted sourceReview code and use an isolated test artifact. A live Canvas is executable content, not a static screenshot.
Workflow 07

Create a QA regression check

Save a focused probe so a model, prompt, route, or workflow change can be tested again later.

Advanced15–30 minResult · A test suite detects a deliberate failure✓ Source-audited workflow
Lab navigation showing QA Lab as the quality-check destination.
Establish a passing Playground request first, then preserve it as a repeatable QA check.
Before you start
  • A behavior with a clear expected result.
  • A stable test input that contains no sensitive production data.

  1. Open Lab > QA Lab and create a suite named for the behavior, not the implementation.

  2. Add one small probe with input, target, expected condition, and useful failure message.

  3. Run it against the known-good setup and save the result.

  4. Temporarily change the test expectation or target so the probe fails, confirming it is capable of detecting a problem.

  5. Restore the correct setup and rerun before relying on the suite.

You are done when
The probe passes in the intended setup, fails in the controlled bad setup, and passes again after restoration.
If something looks wrong
Test only checks HTTP 200Assert the behavior or content you care about; a successful status can still contain the wrong answer or missing tool call.
Workflow 08

Transcribe, review, and promote local audio

Import, record, or capture audio; review timestamped segments; and move durable material into Knowledge.

Intermediate5–30 minResult · A reviewed transcript and optional Knowledge document✓ TranscriptionWorkspaceView import/record/system-audio, waveform, chapters, exports, and Knowledge promotion verified
Lab Transcription showing import and local capture entry points for timestamped audio work.
Use only the capture permission you need, stop cleanly, then review speakers and segments before promotion to Knowledge.
Before you start
  • The local transcription model selected in Settings.
  • Microphone or Screen Recording permission only for the capture method that needs it.

  1. Open Lab > Transcription. Choose Import Audio for a file, Record for the microphone, or system-audio capture when a meeting/app stream is the source.

  2. Stop the recording cleanly and wait for local transcription. The workspace keeps the copied audio, model identifier, waveform, timestamped segments, and automatically grouped chapters.

  3. Play from a segment and correct text or speaker labels when the local acoustic clustering guessed incorrectly.

  4. Generate a summary or action items only after the transcript is readable, then export as text, Markdown, JSON, SRT, or WebVTT.

  5. Choose Add to Knowledge for material that future chats should retrieve; leave one-off recordings in Transcription.

You are done when
The audio plays, timestamps seek correctly, edits persist after reopening the document, and an added transcript becomes searchable in Knowledge.
If something looks wrong
Recording is silentGrant the relevant microphone or Screen Recording permission and capture a short test before a long session.
Speaker labels are wrongThey are lightweight local acoustic clusters, not identity recognition; edit them manually.
Summary misses a decisionCorrect the transcript first, then regenerate the derived summary or action items.
Workflow 09

Inspect Harness experiments, traces, and context budgets

Use one evidence surface to connect a regression case to prompt composition, timing spans, tool contracts, and runtime environment state.

Advanced10–25 minResult · A failure has attributable evidence✓ HarnessLabView Experiments/Context/Causal Traces/Tool Contracts/Environments verified
Lab Harness showing health, experiments, causal traces, context budgets, tools, environments, rules, and scratch workspaces.
Start from the experiment and follow its evidence chain before changing a learned rule or tool contract.
Before you start
  • At least one Chat or agent request; richer pages appear after experiments and tools have run.

  1. Open Lab > Harness and start at Overview to see regression cases, trace spans, prompt manifests, tool issues, environments, and pending rules.

  2. Use Experiments to compare run counts and recent keep rate for a named case rather than relying on a single anecdotal result.

  3. Open Context and inspect the token share of system, memory, skills, tools, files, and other prompt components. Estimated counts are identified separately from tokenizer counts.

  4. Open Causal Traces to follow the complete span chain and duration/error count; export completed traces to the configured OTel destination only when sharing is intended.

  5. Use Tool Contracts to review lint issues and call reliability, then compare the active environment fingerprint before blaming a model or prompt.

You are done when
You can name the failing case, exact context composition, error span, tool-contract status, and environment fingerprint from recorded evidence.
If something looks wrong
A page is emptyGenerate the corresponding evidence first: a request for context/traces, tool synchronization for contracts, or a validated build for environments.
Trace export shares too muchReview the configured diagnostics destination and retention policy before exporting.
Workflow 10

Review learned rules and use a disposable scratch workspace

Keep feedback-derived behavior provisional and prototype risky work outside the real project until explicitly saved.

Advanced10–30 minResult · A time-bounded rule or isolated prototype is reviewed✓ HarnessLabView Rule Drafts and Scratch confirmation flows verified
Lab Harness showing learned rules and scratch workspaces alongside causal evidence.
Promote only reviewed, repeatable findings; scratch work remains disposable.
Before you start
  • An open Chat session for scratch creation.
  • Evidence attached to any proposed learned rule.

  1. Open Lab > Harness > Rule Drafts and read the draft body, evidence count, and security-sensitive warning before deciding.

  2. Approve a useful rule for 90 days, or reject it. Do not turn one user correction into permanent unreviewed system context.

  3. Open Scratch, give the prototype a clear label, and create the app-managed disposable workspace from the active chat session.

  4. Inspect its isolated path and preview. Choose Save As… only for a result worth keeping; otherwise choose Discard and confirm the exact workspace.

  5. Return to Overview and confirm no security-sensitive rule is pending unintentionally and no abandoned scratch remains ready.

You are done when
The rule has an explicit reviewed state and expiry, while the scratch prototype is either saved to a chosen path or removed from the disposable list.
If something looks wrong
Create Scratch is disabledOpen a Chat session first; scratch ownership is session-scoped.
Rule has weak evidenceReject it and gather more observations instead of approving a brittle global behavior.
Discard could remove workSave a deliberate copy first; the confirmation removes only the app-managed scratch, not a previously saved copy.

Keep going

← Images, video, voice, screen, and cameraPrevious collectionRemote access and distributed use →Next collection