Bot QA

Bot QA checks a proposed bot reply against your quality rules before it reaches the customer. It helps you catch problems such as an unsupported refund confirmation, a promise outside your policy, or a reply that misses an important instruction.

You decide what each check looks for and what happens when it finds an issue: record the finding and continue, or cancel the reply and hand the conversation to a person. You can also ask Copilot to set up alerts for your team.

Bot QA is available to every business. Automatic checks start off, so you can review the rules and test them before turning them on. Configuration is separate for each business.

When to use Bot QA

QA is an additional review of the reply. Keep your knowledge, bot rules and workflows accurate too. A QA check can flag a claim against the approved information available to it; it cannot establish that all of your source material is correct or certify regulatory compliance.

How it works

1. The bot prepares a reply. It may already have run actions as part of handling the request.
2. QA reviews the proposed message. It uses the preceding conversation, the bot's instructions, applicable workflows and available action results. Global checks apply to every supported message. Workflow checks apply when their workflow was used, including workflows embedded in the bot's instructions.
3. Potential failures receive a verification review. The review checks the same evidence before deciding whether a suspected violation is supported.
4. Your configured reaction determines delivery. A confirmed violation on a check set to Cancel & hand off cancels the proposed message and hands the conversation to your team. Otherwise, delivery continues.
5. Results become available for reporting and alerts. Confirmed findings include an explanation and supporting evidence. Event handlers can notify your team after the result is stored.

QA checks the message before an email's configured sending delay, rather than rechecking it when that delay expires. Turning QA on affects new messages; it does not review or retract previously sent messages.

Supported channels

Automatic QA applies to bot messages on email, Instagram and Messenger. It does not currently check web chat, phone, WhatsApp or public social comments. The same channel limits apply when testing an existing message.

Checks and reactions

Octocom supplies universal checks covering instruction compliance, action confirmations, factual grounding, usable output, language and presentation. You can pause these checks or change their reaction. You can also add global checks for your business or checks for a particular workflow.

Each check has two independent settings:

Critical does not automatically mean blocked. A critical check set to Continue records the finding and lets the message proceed. Universal checks default to Continue. If several checks fail, any confirmed violation with a Cancel & hand off reaction is enough to cancel the reply.

Check wording, severity and workflow scope are fixed after creation so historical results retain their meaning. To change them, create a replacement and archive the old check. Pausing a check or changing its reaction does not require replacing it. Past results remain available after a check is archived.

What cancellation does

The proposed reply is not sent, and the conversation is handed off through your normal human-support routing. QA does not automatically rewrite the reply.

Actions that already ran are not undone. For example, cancelling a reply does not reverse a refund that an action has already issued. QA evaluates the message and its claims; it is not an approval step before business actions run.

A successful handoff action supports saying that a request was passed to the team. It does not support saying that a person has reviewed it, or that a refund or cancellation has completed. Those claims need their own evidence.

Limits and incomplete checks

QA uses bounded review time so it does not hold replies indefinitely. The initial check has a two-second budget. Suspected failures can require additional verification, with a longer budget on email than on Instagram or Messenger.

If the initial check cannot complete, QA records a skipped attempt and delivery continues. A suspected failure that could not be verified can remain visible as an unverified finding, but it cannot cancel delivery. Other confirmed violations in the same review can still trigger their configured reaction.

Treat QA as an additional safeguard. Review real examples and tune your checks: it can miss problems or flag valid replies, and a technical failure does not automatically stop a message.

Set up with Copilot

Tell Copilot which business to configure, what you want to catch, and what should happen. Copilot can inspect the existing checks, add focused checks, change reactions, test messages and turn automatic checking on or off.

For example:

Review our Bot QA checks for this business. Add a critical check for product claims that contradict our approved knowledge. Set confirmed violations to cancel the reply and hand off. Keep automatic checking off while we test.

Check whether our existing action-confirmation rule covers refunds, cancellations and shipment changes. Use it if it already covers the requirement. Test it against conversation ABC123 without changing delivery or handing anything off.

After reviewing the tests:

Turn on automatic Bot QA for this business. Keep language and presentation checks on Continue, and cancel and hand off for confirmed unsupported action confirmations.

Tests use the latest bot message in the selected conversation, evaluated using its preceding context. They do not send messages, store QA results, trigger alerts or hand off the conversation. Tests can run while automatic checking is off.

Manage in the dashboard

Open Settings, choose your business, then Automation -> Bot QA.

1. Review Checks. Universal checks are provided by Octocom. Add a global check or select a workflow for a more specific rule.
2. Choose the reaction for each check. Start with Continue while assessing the results, or choose Cancel & hand off for checks you want to block on.
3. Test a bot message. Enter a conversation ID or public ID and select Run test. Check the findings and whether the reply would continue or be cancelled.
4. Turn on Check bot messages automatically when the checks and reactions are ready.

You can turn automatic checking off again without losing your checks or historical results. To pause one check, use its individual switch. To replace its wording or severity, duplicate it, adjust the replacement, and archive the original.

In Metrics -> AI quality, review message counts and breakdowns by check for your selected businesses and date range. If you have hidden this tab in your personal Metrics layout, show it again. Ask Copilot to list individual findings with conversation links, filter to confirmed violations, or summarize recurring issues.

Get alerted

Alerts are configured separately from checks. Ask Copilot to create an event handler for Bot Message QA Completed, with your preferred destination and filters. You do not need to write the handler yourself.

The event is available for completed and skipped QA attempts. For alerts about verified problems, ask for confirmed violations only. Filtering only on the overall Critical grade can also include unverified findings.

Slack alerts

Set up a Slack alert for confirmed critical Bot QA violations for this business. Include the check name, explanation, supporting evidence and a link to the conversation. Send one notification per checked message, even if several checks fail. Ask me for the Slack destination details you need.

This works well for a shared quality or support channel. Ask Copilot to use a dedicated channel and filter out routine improvements if you want a low-volume alert feed.

Email alerts

Create an event handler that emails our support lead when Bot QA cancels a reply and hands off. Include the business, failed checks, explanations and a conversation link. Help me connect an email destination if needed.

Email alerts can use your business's configured email integration. This sends an outbound email and creates a separate handed-off conversation for it. Copilot can help configure the handler and check the required connection. Choose the recipients explicitly so QA alerts go to your team.

Other useful patterns

• Alert on one important rule: notify your compliance team only when a particular product-claims check has a confirmed violation.
• Separate urgent and routine findings: alert immediately for cancelled replies, and ask Copilot to summarize improvement findings during your regular quality review.
• Track review health: notify an owner about repeated skipped or unverified checks, separately from confirmed quality problems.
• Feed an existing process: use event handlers or webhooks to send QA results to your reporting or incident system.

Notifications run after the QA result is stored. They do not block or cancel the current reply. Set Cancel & hand off on the check itself when you want to stop delivery; an alert handler is for telling your team what happened.

Write checks that are useful

Keep each check focused on one observable issue. Describe what counts as a violation and include the exceptions your business allows.

For example:

Does the proposed reply claim that a refund or cancellation has completed without supporting execution evidence? Earlier successful actions count. Do not flag a conditional offer to help or a statement that a request has only been passed to the team. Flag an explicit failed action that the reply describes as successful.

Before enabling cancellation, test both replies that should fail and replies that should pass. Include valid exceptions, successful earlier actions, incomplete context and messages where the rule should not apply. Use findings to improve the underlying knowledge or workflow as well as the check.