A green end-to-end suite, a nightly run that hadn’t failed in weeks, and a checkout flow that shipped visibly broken anyway. The tests weren’t wrong. They were testing the wrong thing, and they’d been testing the wrong thing so reliably that everyone trusted them.
TL;DR: Tests pinned to the implementation (CSS selectors,
nth-childpaths, DOM structure) pass as long as the markup is unchanged, even when the behavior is broken, and fail the instant someone refactors the markup even when the behavior is fine. Both are false signals. Derive your tests from the intent (the spec, the acceptance criteria, the user-visible contract) and assert on roles, labels, and outcomes. The same shift is what turns a flaky test from a mystery into a named race condition.
Green, and wrong
The failure was embarrassing in its simplicity. A company name field that used to be required had been made optional, but a downstream handler still assumed it was present, so checkout threw for anyone who left it blank. Users could no longer check out if they omitted an optional field. Real users hit it within the hour.
The end-to-end suite had not said a word. I opened the test that “covered” checkout expecting a gap. There was no gap; there was a test, and it was green. It filled in every field (including the now-optional one, because the test was written back when it was required), clicked a button located at .checkout-form > div:nth-child(3) > button.primary, and asserted that an element with class .confirmation appeared.
Every one of those assertions passed. None of them was about checkout working. They were about the page being built the way it was built the day the test was written. The test and the code had drifted apart on the one axis that mattered (behavior) while staying perfectly aligned on the one that didn’t: structure.
Two failure modes of the same mistake
Pinning a test to the implementation breaks in both directions, and both directions lie to you:
False green: broken behavior, passing test. The markup is unchanged, so the selectors still resolve and the assertions still pass, even though the feature is broken. This is the checkout case. It’s the dangerous one, because it actively launders broken code as tested.
False red: working behavior, failing test. Someone refactors the markup (wraps a div, renames a class, reorders the layout) and every nth-child and .class selector shatters. Nothing about the user experience changed. You spend an afternoon “fixing tests” that were never testing anything real. Do this enough times and the team learns the deepest anti-lesson in testing: when the suite goes red, update the selectors until it’s green. Which is how you get back to the false green.
Both come from the same root: the test encodes how the page is currently built instead of what the page is supposed to do.
Assert on the contract a user relies on
The fix is to write assertions against the things that are supposed to be stable (because they’re the actual contract with the user) and to stop touching the things that are supposed to change, because they’re just implementation.
What’s stable is the accessible contract: this thing is a button named “Place order”; this is a textbox labeled “Email”; after I place the order, I see text that says my order is confirmed. Those hold across a markup refactor, because they’re what the feature means, not how it’s assembled. A user doesn’t find the button at nth-child(3); they find the thing that says “Place order.” Test the way the user perceives the page, and your test survives everything except the behavior actually breaking, which is the one time you want it to fail.
The same checkout step, pinned to structure vs. pinned to intent:
Implementation-coupled Intent-coupled
────────────────────── ──────────────
.checkout-form > div:nth-child(3) getByRole('button', { name: 'Place order' })
> button.primary getByLabel('Email')
assert .confirmation exists assert text 'Order confirmed' is visible
breaks when: markup is refactored breaks when: checkout actually breaks
stays green when: checkout breaks stays green when: markup is refactored // Reads like the acceptance criterion it came from.
await page.getByLabel('Email').fill('a@b.com')
await page.getByRole('button', { name: 'Place order' }).click()
await expect(page.getByText('Order confirmed')).toBeVisible()Those locators aren’t just more stable, they exercise the accessibility surface your users and their assistive technologies actually rely on. A test that can’t find the button by its accessible name is quietly telling you a screen reader can’t either. Correctness and accessibility end up asserted by the same line.
That test would have caught the checkout bug (with the field left blank, “Order confirmed” never appears) and it would have survived the layout refactor that the old test would have died on.
Derive the test from the spec, not the screen
If the assertions should come from intent, so should the test itself. The richest source of intent isn’t the finished page: the page is already just one implementation of the specification. It’s the specification you want: the design, the acceptance criteria, the PRD, the user story. Whether it’s a Figma flow or a line of Gherkin, the spec describes behavior, not markup, which is exactly what you want a test to encode. That’s where “what checkout is supposed to do” is written down before anyone chose a class name.
This is why generating end-to-end tests from a design plus its acceptance criteria (rather than by recording clicks on the built page) produces such different tests. A recorder can only observe what happened once; it can’t infer what should happen always. It captures the DOM as it happens to exist and bakes the implementation straight into the assertions, you’ve automated the false-green machine. Deriving from the spec captures the intended flow and the user-visible checkpoints (“the user provides an email, places the order, and sees a confirmation”) and only then binds those to roles and labels. The test encodes the contract, because it was generated from the document that defines the contract.
It’s the same principle underneath every good test: derive from the authoritative source (what it should do), don’t accumulate incidental detail from a convenient one (how it’s currently built). It’s the same derive-don’t-accumulate discipline I keep coming back to. The spec is authoritative. The current DOM is a snapshot that’s already going stale.
The flaky-test corollary
There’s a second dividend, and it’s the one that quietly saves the most time. When a test asserts on outcomes (“the confirmation is visible”) instead of timing-coupled structure, flakiness stops being a ghost.
A flaky test is almost never random. It’s a race condition you haven’t named yet: the assertion runs before the state it’s checking has settled. Implementation-coupled tests hide this behind arbitrary sleep(500) calls and brittle selectors, so the race is invisible and you “fix” it by bumping the sleep. Intent-coupled tests, asserting on a user-visible outcome with a proper wait (“wait until the text Order confirmed is visible”), make the race explicit: the test waits for the actual condition, so either the condition happens and the test passes deterministically, or it never happens and the test fails for a real reason. The flake has a name now, and a named race is a fixable one.
When you hand flaky-test triage to an agent, this is exactly what makes it tractable: it can look at an outcome-based assertion and reason about which state wasn’t awaited, where a sleep-and-nth-child test gives it nothing but noise to guess at.
The one line to remember
The nightly suite was green for weeks while the thing it claimed to test was broken, because it had been quietly measuring the wrong quantity (the shape of the markup) with total reliability. A test that stays green while the feature is broken isn’t giving false confidence. It is the false confidence.
Assert on what the user is promised, not on how the page is built. Derive the test from the spec, bind it to roles and outcomes, and your green means working, your red means broken, and your flake has a name.