Building an E2E Explorer That Doesn't Know Your Codebase
This tool exists because of two problems, not one, and they weren't sequential. Travelstart runs something close to 800 country and market specific marketing sites, WordPress hosted, each with its own flight search widget that hands off to a separate booking engine entirely outside WordPress the moment someone actually searches. None of those sites had any automated coverage, and real failing conditions had already turned up on them by hand, well before anyone sat down to automate anything. At that scale there was never a realistic way to keep re-checking all of them manually. At Kiyoh, the shape of the problem was almost the opposite: a real QA process already existed, it just lived in a spreadsheet and in whoever remembered a given edge case, rather than in anything that ran itself.
Kiyoh's version of the problem had a specific shape worth spelling out too, because it's why the backlog mattered as much as it did. The QA process that already existed wasn't ad hoc, it was 336 test cases, each one a row in an Excel sheet carrying its own history of whether it had passed or failed the last time somebody actually ran it by hand. Three hundred and change is small enough to feel manageable and large enough that nobody was rerunning all of them before every release. The backlog wasn't a lack of process. It was a process that had outgrown the format it was trapped in.
Two different companies, two different flavors of the same underlying gap: neither one had a safety net that ran itself. Building one engine that could be pointed at either, rather than two separate throwaway scripts, was the only version of the answer that didn't mean solving the same problem twice. The first real constraint I wrote down for that engine, which I ended up calling E2E Explorer, had nothing to do with WordPress or admin panels specifically. It was about what the tool would and wouldn't be allowed to look at while doing its job.
Treating every site like a stranger's website
The README for the tool states the constraint plainly: it's an external, black box E2E reliability explorer built on Playwright, not connected to the codebase of any site it tests. Every target is treated as an outside URL, the same way a real visitor would experience it. That line reads like a limitation, and in a narrow sense it is one. The tool can't introspect component state, can't read a backend's route table, can't assert against an internal data model. All it has is what loads in a browser: the DOM, the network requests a real page fires, the console, the rendered result of clicking something.
I made that trade because the failure mode I'd already lived through was worse than the one I was accepting. A test that reads the source can fail because the source changed shape, even when the actual customer experience didn't change at all. A test that only ever looks at rendered pages fails when something a real visitor would notice actually broke. That's a stricter bar to pass, but it means a green run means something. It also means the same engine can point at a site it has never seen configuration for beyond a single typed file, because it isn't relying on anything specific to that codebase in the first place.
Core is an engine, sites are just data
That black box rule shaped the whole layout of the project. core/ is a shared engine, reused by every site the tool is pointed at, and it contains no logic specific to any one site. Inside it: core/engine/flow-context.ts, which runs checkpointed test steps, core/checks/, one file per generic check that applies to any site, core/auth/login.ts, a login flow driven by data, core/selfheal/selector-fallback.ts, a fallback chain of locator strategies for when a primary selector stops matching, plus a logger, an errors module, shared types, and a stub for an AI maintainer I haven't finished.
Everything that's actually about a particular company lives under sites/<company>/ instead: a typed site.config.ts that is data, not logic, along with raw recordings, promoted scenarios, change requests, and proposed diffs for that one site. The dividing line is strict. If I ever find myself writing an if branch inside core/ that only applies to one company, that's a sign the thing I'm building belongs in that company's config file, not in the engine.
The piece that connects the two is deliberately tiny. generic-checks/ wires the generic checks in core/checks/ into a runnable Playwright spec per site, and a real file in there is three lines long: import a site's config, call registerGenericChecks(config). That's the entire contract a new site has to satisfy to get the full set of generic checks running against it. Onboarding a new company means writing a config file, not writing new test infrastructure.
I didn't just assume that split would hold up. When Travelstart became the second real deployment of the engine, the first thing I did wasn't write any code, it was diff core/ between the two copies. I expected to spend the afternoon reading through everything that needed to change before a single test could run. Instead the diff came back almost entirely empty, a handful of files different out of close to thirty: one company's WordPress specific checks that don't apply to the other, one company's extra credentials handling, nothing that touched the shared engine's actual behavior. That's the moment I stopped thinking of the two as one tool and a rebuild of it, and started thinking of them as exactly what they actually were: the same engine, running twice, in two places at once, a design proven general enough to reuse rather than one I just hoped was.
Steps that fail one at a time
The part of the architecture I'm proudest of is how a flow actually runs. Early on I wrote flows as one long Playwright test: load the page, click through a dozen things, assert at the end. When step three of that chain failed, the report told me the test failed. It didn't tell me steps one and two had gone fine, or that steps four through twelve never even got a chance to run. I'd have to open the trace and reconstruct that myself every time, which is exactly the kind of manual work a reliability tool is supposed to remove.
So flows run as a sequence of named, checkpointed steps through a runFlowSteps() helper instead of one flat pass or fail. A real, illustrative scenario spec looks like this:
const steps = await runFlowSteps(page, [ { name: "1. Load travel-deals page", run: async (p) => { /* ... */ } }, { name: "2. Open flight search widget", run: async (p) => { /* ... */ } }, { name: "3. Fill search form", run: async (p) => { /* ... */ } }, { name: "4. Submit search", run: async (p) => { /* ... */ } }, { name: "5. Wait for results to render", run: async (p) => { /* ... */ } }, { name: "6. Click first deal, confirm handoff URL is allowed", run: async (p) => { await p.getByTestId("deal-card").first().click(); await p.waitForURL(/.+/); assertAllowedRedirect(p.url(), config.allowedRedirectHosts); } }, ]);
When step three fails now, the report shows steps one and two as passed, step three as failed, and steps four through six as skipped rather than attempted. That sounds like a small change in how results are displayed, but it changed how I read reports. I stopped opening a trace file every time something broke, because the step list itself usually tells me where the flow actually diverged from what a customer would experience, and how far the customer would have gotten before hitting the same wall.
Because core/checks/ has no logic specific to any one site, every generic check in there, broken links, console errors, exposed internal endpoints, form sanity, load time, basic accessibility, runs against every site the tool is configured for automatically, with no extra setup beyond that three line wiring file. None of these are exotic. They're the kind of thing a careful person clicking around a site by hand would eventually notice, which is the point: the tool is standing in for that person, not for a static analyzer that understands the codebase.
Recording a flow without hand writing one
None of that architecture matters if getting a flow into the system still means an engineer sitting down to write a Playwright spec by hand for every one of those 336 rows. The recorder exists so it doesn't have to.
Playwright ships its own recorder as a CLI command, playwright codegen. Point it at a URL, it opens a browser, and every click and fill and navigation gets turned into generated code in a side panel as you go. The problem is that codegen doesn't expose a way to launch maximized, and a fixed viewport size isn't the same thing as a real maximized OS window. Without it, people recording flows kept clicking on things positioned differently than they expected, because the responsive layout they were recording against wasn't the size their real users would see.
So I stopped going through the CLI. Playwright's codegen is really just a thin wrapper around a recorder that lives inside the library itself, and that recorder is reachable directly if you're willing to call it yourself:
const context = await browser.newContext({ viewport: null, // let the real window size drive layout, not a fixed viewport }); await context._enableRecorder({ mode: "recording" });
Launch the browser maximized, skip the CLI, call _enableRecorder on the context yourself. The underscore in the name is not decoration. This is an internal API, not part of Playwright's public surface, not covered by their semver guarantees, and not something their docs mention at all. I found it by reading Playwright's own source rather than any documentation. It works today, against the versions I've tested it with. It could stop working the next time Playwright refactors their recorder internals, with no deprecation notice warning me first. That's a real cost, and I'd rather be upfront about it than pretend calling a private API is free just because it happens to work.
Whatever gets generated during a session lands untouched in a folder for that site, as plain Playwright code, using whatever selector Playwright's own recorder decided to generate at that moment. That matters because a raw recording is not a test. It's a replay. It clicks the things you clicked and fills the things you filled, in order, and it will happily "pass" against a page that's completely broken, as long as every element it's told to click still exists somewhere on the page. Nothing in a raw recording checks that the right thing actually happened.
Closing that gap is what I call "promote," and it's deliberately not automatic. A pass, done by a human or assisted by AI, swaps the generated CSS selectors for locators built around role and label that survive a class name changing far better than a brittle CSS path does, adds real assertions where the recording only assumed a click worked, parameterizes whatever got hardcoded during the recording so the scenario isn't permanently tied to the exact values on screen that day, and wraps the whole thing in the checkpointed step runner. None of that is exotic. It's the same cleanup pass any team does when it inherits a spec someone wrote quickly and needs to actually rely on it, done here as a named, repeatable stage instead of something ad hoc.
Even a carefully promoted locator can stop resolving, a page's markup shifts, an attribute gets renamed, a component gets swapped for one that renders similar but not identical output. Rather than fail the instant that happens, a function works through a fallback chain of locator strategies before giving up, trying the next strategy down the chain and only failing the step once everything's been tried. It's not magic and doesn't fix every kind of drift, but it buys tolerance for the specific, common case where a page changed a little without changing what a user would actually recognize as the same element.
Two bugs from actually running these recordings stuck with me more than the bigger architectural decisions did, because they were so specific. The first was a date picker rendering two month grids side by side, current and next, where a naive selector meant to find "the 15th" matched a hidden filler cell in the inactive month instead of the real, visible one. Both cells technically matched the selector. Only one was actually on screen. The fix was a :not(.hidden) condition so the selector only matched cells inside the visible grid. The second was a confirmation dialog that appeared while still animating in, failing Playwright's "stable" actionability check because its position was still changing frame to frame. The fix was either { force: true } to skip that check, or an explicit wait for the animation to finish where I wanted the check to stay meaningful. Neither fix is clever. Both came from watching a recording fail in Watch mode, the same real Chromium running visibly and slowed down, and actually looking at what the browser was doing at the moment it failed, rather than guessing from a stack trace alone.
Since a flow name ends up driving a subprocess spawn, flow names get validated against path traversal and shell injection before that happens, and the subprocess is always spawned with an argument array, never a shell string that could let a crafted name get reinterpreted as something else. Redirect targets, like the handoff URL in the step six example earlier, get checked against an explicit allowlist of hosts before the tool trusts them, because a black box tool that follows a page wherever it redirects is one bad redirect away from testing something it was never pointed at. And every captured log goes through PII redaction before it's stored, since logs from a real user flow can easily contain something a user typed that shouldn't sit in a recording on disk. None of these came from a specific incident. They came from treating a recorded flow, and any AI assisted edit to one, as something that shouldn't be able to quietly redirect execution or leak something it captured.
The part I'm still honest about not having solved: letting someone non technical fix a single broken step of an existing recording, one bad click or one stale field, without recording the whole flow again from the start. Digging into Playwright's own source, its Recorder class only ever captures actions when the browser was actually launched through the real codegen CLI code path, and I couldn't find a way to flip that internal flag once a context had been launched programmatically, which is exactly what the maximize trick depends on. The same undocumented call that solved one problem quietly closed the door on this one. Streaming the page over CDP's screencast protocol has no authentication built in and real, noticeable latency. Diffing a scrape against a previous one looked promising until I tried it, it's genuinely ambiguous whether a detected difference is a structural change worth fixing or content that legitimately changes between visits. So that feature doesn't exist. Fixing one step today means editing the generated code directly or recording the flow again.
Not everything belongs in an automated run
Some flows are unsafe to run automatically, a real payment capture, for instance, and some are simply impractical to automate reliably. For those, the tool supports manual test steps written in plain text, added once, explicitly excluded from every automated run, and surfaced separately in a review UI so a human can walk through them deliberately. I'd rather have an honest manual step documented and skipped on purpose than a flaky automated version of it that people learn to ignore.
All of this is driven from a CLI, but there's also a local recorder and player, a plain Express server with around thirty REST endpoints, paired with a plain HTML and vanilla JS frontend, no framework, served locally with automatic reloading during development. That's the side that doesn't require writing code: starting and stopping a recording, replaying it, watching a flow run, promoting a raw recording into an official scenario, entering a manual test, reviewing findings. I kept it deliberately simple rather than reaching for a frontend framework, because the recorder's job is to stay out of the way of the engine underneath it. I built that side specifically so it wasn't only me who could add coverage.
Turning an Excel backlog into an actual baseline
Before any of this existed, Kiyoh already had a manual QA process, and that process already had a spreadsheet, an actual named file, v2026.06.1 - new ui_framework.xlsx, with a Baseline sheet: 331 rows originally, each one a test case someone had walked through by hand at some point, each carrying its own pass or fail history. Five more rows got added later covering an area with zero prior coverage, bringing the real total to 336. That spreadsheet was the actual seed of the automated suite, the accumulated knowledge of what the QA process already checked, sitting in a format nobody but a person could run.
The honest current state, rather than a tidy finished number: 329 of those 336 rows have been recorded, driven live through the real login and click through flow and written as a Playwright spec per row, checked against that row's own manual QA history so a case that used to fail intermittently under manual testing didn't just quietly disappear into "converted, therefore fine." None of those 329 have been promoted into the trusted scenario folder yet. That's not a demo milestone dressed up as more than it is, it's the actual size of the backlog and the actual place the conversion currently sits, recorded but not yet the thing a broken run would fail loudly against in CI.
What it actually costs to automate 300 rows by hand, and what didn't
Converting one row by hand is easy to estimate. Converting hundreds of them is a different problem, and my first instinct, a pipeline with three separate roles where one agent explored the flow, a second wrote the spec, and a third verified it, turned out to be worse than it sounded on paper once I actually measured it instead of assuming a more elaborate pipeline would obviously do better.
So I ran an actual comparison, separately from the real Kiyoh rows, against a public Playwright practice site so I could iterate on the comparison itself without burning real conversion time. Fifteen different architectures for driving an AI agent through Playwright MCP to author specs, measured for accuracy, cost, and wall clock time on the same test cases. My first pass at the cost numbers was wrong in a specific, embarrassing way, I'd priced cached tokens at the same rate as fresh ones and only counted the most expensive turn instead of summing the whole conversation, which made every variant look more expensive than it actually was and made the comparison meaningless until I fixed it.
Once the accounting was corrected, two variants stood out ahead of the other thirteen, both hitting full accuracy on the test set. A single agent handling everything sequentially, no subagents, no parallelism, was the cheapest option overall. A batched version, three rows per call, the explorer and writer roles merged with the writer self verifying instead of a separate verifier, five parallel sessions running at once, cost a bit more per batch but finished roughly twice as fast. Neither was strictly better, they traded cost for time depending on whether wall clock speed mattered for that particular run, which is why the rule I wrote down afterward isn't "always use the cheap one," it's to actually choose on purpose every time rather than defaulting to whichever one got used last. The single agent method is the one actually driving the real Kiyoh conversion work today, the 329 rows recorded so far went through it directly, not through the batched variant, since correctness on real production credentials mattered more there than shaving the wall clock time down.
Porting the same engine to a second company
Once the diff between the two core/ copies came back nearly empty, most of the local recorder app came across intact too, the Express server and its frontend, with only the smallest adjustments needed to point it at a different site's config. What was actually specific to each company turned out to be a short list: the config file, obviously, since that's where a company's identity lives inside the tool; each company's own recordings and scenarios, since a recorded flow through an admin login has no equivalent on a travel booking site; the WordPress specific checks, relevant only to Travelstart's marketing sites, since Kiyoh's admin tool doesn't run WordPress anywhere in its stack; and each company's own credentials and environment file, never shared, for reasons that have nothing to do with the tool and everything to do with not mixing production secrets between two unrelated organizations.
Everything on that list is data or configuration, not logic, and confirming that rather than just assuming it going in was the most relieving part of the whole port. A typed site config is the entire adapter: auth holds login selectors plus credentials keyed by environment variable names, requiredCookies exists because one company's app needs a routing cookie set first or an edge router sends the request to the wrong internal frontend entirely, and environments lets the same recorded flow run against a different hostname of what is, underneath, the identical application. The recording doesn't know or care which hostname it's pointed at. The config is the only thing that has to know.
325 out of 329
The best story from the entire port isn't really about the port at all, it's about something Kiyoh's own config was already quietly protecting against, well before anyone went looking at the worker count.
Kiyoh's Playwright config deliberately restricts test execution to a single worker, workers: 1, and for a long time I treated that as an inherited setting rather than something worth questioning. Then, while looking into whether that restriction still made sense, someone ran the full baseline suite at 9 parallel workers instead. 325 of 329 tests failed, on a suite that passed cleanly at one worker.
The failure signature was almost suspiciously consistent: a getByRole('heading', { name: 'Tenant dashboard' }) timeout, the same early page load assertion, over and over, in tests that had nothing else in common. My first instinct was a Playwright bug, or a suite that had been flaky all along and the single worker setup had been quietly hiding it. Neither was true. The DEV backend was getting saturated under that many concurrent real browser sessions at once. Nine real Chromium instances logging in and loading dashboards simultaneously was more concurrent load than that backend had ever been asked to handle, and it buckled in a way that looked, from the test runner's side, exactly like a broken test framework.
The lesson I wrote down afterward was blunt on purpose: before raising a worker count on any project, measure it first, and treat a sudden, nearly total failure rate concentrated on the exact same early assertion as the signature of backend saturation, not of the framework being broken. A framework bug doesn't usually pick one specific line of one specific assertion and fail it across 325 unrelated tests. A backend falling over under load does exactly that, because that one early assertion is the first thing every single test touches.
When I set up Travelstart's config, I did not copy that same worker restriction over, and I want to be specific about why, because it would have been the easier and more cautious looking thing to do. Travelstart's own backend was confirmed to handle that level of concurrency fine. Copying workers: 1 anyway, just because it came from the same tool and the same author, would have meant treating a fix for one company's infrastructure limit as if it were a universal property of the tool itself. It isn't. The tool doesn't care how many workers it runs with. The backend behind it does, and that's a fact about each company's own infrastructure, not about E2E Explorer.
What's real and what's still an idea
The Travelstart side is honest scaffolding right now: the engine is wired up and pointed at the first of those marketing sites, but the scenarios sitting in the repo are illustrative placeholders, written ahead of a real Codegen recording rather than promoted from one. The Kiyoh side is further along but not finished either, 329 of 336 rows recorded, none yet promoted into the trusted scenario folder that CI actually runs. An AI maintainer that would look at a broken flow and propose a fix automatically exists only as a stub, not wired into anything. I'd rather say plainly where each piece actually stands than let a folder name or a site config imply more than the code does.
There's also one open question I don't have a settled answer to. The two companies live in two separate repos today, with new engine improvements written into one first and manually carried over to the other. That works, but every sync is a small chance for the two to drift apart again the way they didn't during the initial port. The obvious fix is a shared package, an npm workspace or a private package both repos pull from instead of copying between. I haven't done that, because it's only worth the migration cost if a third company becomes likely, and building real versioning and release discipline for two consumers when a third might never show up would be solving a problem I don't actually have yet.
the actual lesson
None of this started as an infrastructure project for its own sake. It started because two different companies had the same shaped hole at the same time, one with nothing at all across 800 sites, one with a QA process too big for the people available to rerun it by hand, and neither problem waited politely for the other to get solved first. The black box discipline kept the tool honest about what a real visitor would actually experience, at either company. Measuring the cost of AI authoring instead of guessing at it kept the Kiyoh conversion affordable enough to actually make progress on. Proving the port with a diff instead of an assumption is what let me trust the engine was general rather than accidentally coupled to whichever company I'd built it against first. None of those were the interesting part on their own. Together they're the reason one engine made sense instead of building the same tool twice.