Run regression suites in Chromium, Firefox, and WebKit on every push and PR. Combine results into PASS, CONDITIONAL, or FAIL. Unstable checks are quarantined, and failures include screenshots and replay videos.
Run #48 · Pre-releaseTrigger: PR · Approved by QA lead
48 / 48
Cases
Chromium
Firefox
WebKit
API
Session expiry after login
Payment amount validation
Block access to another branch's orders
Order list pagination Quarantined · 25%
24-hour order cancellation rule
Immediate points refund
1 failed · Minor · 3 evidence types
3Browser engine matrix
3levelsRelease decision
20runsRun history for reliability tracking
80%Requirements coverage target
Case maintenance
Regression suites keep up with code changes.
PR changes suggest additions, updates, and removals. Approved cases join the suite. Repository, branch, and changed-path rules automate runs on pushes and PRs.
PR analysis → Case suggestions → Approve or reject
Run types: smoke · regression · new feature · pre-release
Await approval → Approve execution
Deduplication and Korean, English, and Japanese translation
PR #142Add a payment methodSuggested: 2 new · 1 update · 1 removalApproveReject
Scenario
Priority
Automation
Source
Language
Session expiry after login
High
Automated
Document
KO EN JA
Payment amount validation — zero, negative, limit
Critical
Automated
Document
KO EN JA
Block access to another branch's orders
High
Automated
Code
KO EN JA
Order list pagination
Medium
Automated
Document
KO EN JA
New payment method · Card expiry validation Awaiting approval
High
Automation candidate
Code
KO
Shared checklist: 13 / 17
Test reliability
Quarantine inconsistent checks.
Track result changes on the same commit over the last 20 runs. A flip rate of at least 20% triggers quarantine. Checks still run and report but cannot cause FAIL.
A different result on the same commit counts as a flip
Quarantine appears as a reason for CONDITIONAL
Per-engine quarantine: Firefox-only flips quarantine Firefox only
Quarantined checks continue running and appear in reports
Test reliabilityLast 20 runs · Per engine
Order list paginationFirefox · 25% flip rate
Quarantined
Payment amount validationChromium · 0% flip rate
Session expiry handlingWebKit · 5% flip rate
Quarantined checks continue running and reporting, but cannot cause a FAIL verdict.
Release decision
One verdict, with its reasons.
Critical or major failures produce FAIL. Minor/trivial failures, quarantine, or inconclusive checks produce CONDITIONAL. PASS requires every check to pass on its first attempt. Coverage below 80% produces CONDITIONAL even when all checks pass.
CONDITIONAL means human review is needed, not a soft pass
Requirements, checklist, and screen coverage alongside verdicts
Browser notifications for FAIL, quarantine, and coverage gaps
Revalidation of fixed commits connects to earlier verdicts
All executed tests passed first try · Requirements coverage 83% (target 80%)
48 cases47 passed · 0 failed · 1 skipped
Quarantined tests1 · Passed this run · No effect on decision
Security & supply chain0 critical · 0 major
Screens never opened2 screens
Decision · Ready to releaseQA lead sign-off
Run #48CONDITIONAL
1 minor failure · 1 quarantined · Requirements coverage 76% → Human review needed
Frequently asked questions
Before starting regression validation
We have existing test cases. Must we recreate them?
Case generation supports documents, code, and people. Cases derived from repository code and manually written cases join the same regression suite and verdict.
When do checks run?
Repository, branch, and changed-path rules trigger runs on pushes or PRs. With approval enabled, runs wait for authorization. Use this alongside your GitHub Actions pipeline.
Can a FAIL verdict block a PR automatically?
FAIL appears immediately in browser notifications and verdict cards, and failures can become Jira issues. PR checks and deployment-blocking release gates are in preview.
Who determines whether a failure is a real defect?
Each failure includes screenshots, replay video, and logs for your team to review. With Managed QA, experts triage results, isolate reproducible defects, and revalidate fixes.
How is requirements coverage calculated?
Extract testable statements from requirements documents and calculate the share backed by actual execution. This differs from code coverage; unexecuted cases do not count.
What if only one browser fails?
Results and quarantine are recorded separately for Chromium, Firefox, and WebKit. Engine-specific failures retain the engine name and affect the verdict according to severity.
Get a verdict for your next release.
A staging URL and one repository usually get you to your first run in five minutes. See the verdict card in a demo.