Fin's performance figures are averages computed by its vendor, on content that is not yours. The only benchmark worth a decision is the one you run on your questions, with your sources. Fin ships the tools for it, and most teams use them once, on launch day.
Three testing tools, three uses
Fin distinguishes Previews, Batch tests and Simulations. They do not replace each other.
| Preview | Batch test | Simulations | |
|---|---|---|---|
| Principle | You type, Fin answers live | Up to 50 questions per group, which you rate | An AI plays the customer and judges the outcome |
| Scope | Every setting | All Fin customers | Procedures only |
| Verdict | None, you judge | Manual: Good, Acceptable, Poor | Automated: Pass or Fail |
| Calls your live APIs | Yes | No, sample data | No |
| Rerunnable | No | Yes, with CSV export | Yes, with "Run all" |
| Cost | No limits | Free | Included, 250 to 12,500 runs per month |
For an answer-quality benchmark, the tool is the Batch test: it is the only one that covers a broad question set, keeps a record and reruns identically. Simulations validate the logic of a Procedure, not the quality of an answer drawn from your content.
Building the 50-question set
The Batch test offers four ways to build a group: generate from recent conversations (30 to 90 days), generate by topic, upload a CSV, or type them in. Automatic generation is tempting, but it mirrors your volume, and therefore your easy questions. A useful test bench is composed by hand, in four families.
| Family | Share | What it tests |
|---|---|---|
| Frequent questions | 20 | Baseline coverage, generated from your conversations |
| Trap questions | 10 | Topics with no answer in your sources: Fin must abstain or escalate, not invent |
| Context-dependent questions | 10 | The same question for two different plans or countries |
| Sensitive questions | 10 | Refunds, security, data, cancellation |
Trap questions teach the most. An "if you don't know, escalate" guidance protects nothing, as we explain in our Fin configuration guide. The only way to know what Fin does when facing a gap is to ask it the question.
Running the test under the right conditions
By default, the Batch test runs as a generic preview user, with access to all content and no targeting. That is the most favorable case, and the least realistic. The test settings let you choose:
- A real user or lead, to see how Fin tailors its answer to their attributes (language, country, plan).
- An audience, to check targeted content and guidance. In this mode automations do not run.
- A brand, if your workspace handles several.
Run the ten context-dependent questions once per segment. If Fin serves the same answer to a free customer and an enterprise customer, your targeting is not working, and no dashboard would have shown you.
Rate, then trace the cause
Each answer is rated Good, Acceptable or Poor. For a Poor answer, Fin asks you to pick a cause. This is the most useful part of the tool, because each cause points to a different setting.
| Cause offered by Fin | Indicator | Fix |
|---|---|---|
| Didn't use the correct content | Off-topic source, old policy | Fix or disable the source, pin the right article |
| Didn't clarify the question | Multi-intent or vague request | "Context and clarification" guidance |
| Used the content incorrectly | Steps out of order, wrong plan's rule | Restructure the article, target by audience |
| Tone wasn't right | Too cheerful on a sensitive topic | Tone in Basics, style guidance |
| Answer length too long or short | Wall of text for a quick FAQ | Length in Basics, per-channel guidance |
| Wrong language | English answer to a French speaker | Supported languages, real-time translation |
Make it a benchmark, not a launch test
A test becomes a benchmark the day you rerun it. The protocol we recommend comes down to four rules.
- Freeze the question set. Same 50 questions, same segments, every run. You only change the set by adding to it.
- Rerun after every change. New content, new guidance, a length change: one run before, one run after.
- Keep the CSV export. It holds your ratings and notes. That is your history, outside the platform.
- Track a single number. The share of Poor answers per question family. Poor answers on trap questions are your hallucinations.
The evaluation guide published by Fin gives orders of magnitude: 50 to 100 known-answer queries for factual correctness, and an audit of at least 200 responses for hallucination rate. One group of 50 questions is therefore a floor. Nothing stops you from creating several.
What the test bench will never measure
- Real conversations. A test question is clean and isolated. A real customer writes in three messages, backtracks, gets annoyed. The Batch test does not test the conversation, it tests one answer.
- Real data. Connectors use sample payloads there. Only Preview calls your APIs.
- Drift. Your product changes every week, your 50 questions do not. An answer that is right in October can be wrong in December without a single test turning red.
The test bench tells you whether the configuration holds before going live. You then need a check on production: that is the purpose of our weekly audit ritual, and it is what ResponZ quality scoring automates: the same rubric, applied to every real Fin answer, continuously.