Results
How pass, fail, and inconclusive are determined — and what to do with each.
At the end of every run the AI agent produces a result. This is not a simple assertion check — the agent reads the page and reasons about whether the expected outcome was met.
The three results
Pass
The agent completed all described steps and confirmed the expected outcome is visible and correct. The run succeeded.
Fail
The agent worked through the test but the expected outcome was not met. Common causes:
- A button wasn't found at the expected location
- A page didn't load (redirect, 404, or error)
- The expected text wasn't visible
- A form submission didn't trigger the expected confirmation
A fail result includes the agent's reasoning — it explains what it observed versus what it expected. Read this first before digging into the recording.
Inconclusive
The agent could not confidently determine whether the run passed or failed. This usually means:
- The expected outcome was too vague to evaluate
- The page state was ambiguous (e.g. a loading spinner was still visible)
- The agent was interrupted before reaching the verification step
If you're seeing repeated inconclusive results, the most common fix is to make the expected outcome more specific. "The page loads" → "The heading Dashboard is visible."
How the agent decides
After completing the steps in the description, the agent:
- Takes a final screenshot of the browser
- Reads the expected outcome from your test
- Compares what it sees against what you described
- Produces a result with a short written explanation
The agent does not pass or fail based on assertions in code — it reasons like a human tester looking at the screen.
Automated run failures
If a scheduled, API-triggered, GitHub-triggered, or deployment webhook-triggered run fails, Smoketest records the failure in run history and can notify you by email, Slack, or Discord. Project-level notification filters are planned, but today's behavior is every non-on-demand run plus an extra alert after consecutive failures.
Acting on failures
When a run fails:
- Read the result reasoning — the agent usually explains what it observed
- Replay the recording to see the browser state at each step
- Check whether the failure is in your app or in the test description
- If the test is wrong, edit the description and re-run manually to verify the fix