Hardening a Payment Checkout: Trust Boundaries, Races, and a Gateway That Lied
I was running through the checkout flow myself, paying with a real test card end to end instead of mocking any of it, when I hit a result that made no sense on its face. The payment gateway had confirmed the charge, our internal payments microservice was reporting the transaction as SUCCESSFUL, and the booking behind it was still sitting in Pending. Money had moved. The trip hadn't been booked. I opened the logs for the endpoint that was supposed to close that gap and found a plain HTTP 200 staring back at me, looking entirely pleased with itself. That response was lying, and figuring out exactly how it was lying became the most useful debugging session I'd had in months.
What the client gets to touch
The payment flow in the app I'd rebuilt from scratch, a Nuxt rewrite serving multiple tenants that I've written about elsewhere, sits behind a payments module on the server that does a specific, bounded set of things. It creates an embedded checkout session with our payment provider, Peach. It handles redirect callbacks coming back from several external gateways: Ozow, Mobicred, Paystack, Payflex. It runs a card BIN lookup so the frontend can show a discount banner specific to the bank. It polls payment status from our internal payments microservice. It proxies capture, refund, and void actions. And it has a finalize endpoint, the one that had just lied to me, which transitions a booking from Pending to Paid by calling our legacy booking API once the payment provider reports success.
None of that is unusual for a checkout flow. What mattered more, once I sat down to actually harden it, was drawing a hard line around what the client was allowed to assert and what the server had to verify for itself.
The amount, the currency, and the payment reference for a given flow always come out of a Redis store on the server, never from anything the client sends. A request can ask the server to create a checkout session, but it can't say how much that session is for. The HMAC secret used to sign requests to the payment provider never leaves the server either, so nothing running in a browser can forge a request that looks authentically signed. Before the endpoint takes any action on a payment, capture, refund, void, even a plain status check, it checks that payment's ID against an ID stored in Redis and scoped to the flow first, so a request tied to one user's booking flow can't reach into a payment that belongs to somebody else's. Checkout sessions get cached in Redis too, so a retried request returns the same session instead of quietly minting a second one against the same booking. And when the payment provider hands back an SDK URL for the client to load a script from, that URL gets checked against a strict allowlist regex matching only real subdomains of the provider's own domain, before the client is ever told to trust it.
Written out like that, it sounds like a tidy list of rules I sat down and designed in one sitting. It wasn't. Most of it came from staring at two very different failures and asking, each time, what exactly the server had just believed that it shouldn't have.
A race the UI alone couldn't close
The first failure was quieter than the one that opened this piece, and quieter in a way that made it easy to miss until someone actually looked for it. The checkout endpoint's normal guard against a repeat payment lived entirely in the frontend: once a payment succeeded, the app redirected the user away from the checkout page, and that redirect was supposed to be the thing preventing a second charge. Most of the time it worked exactly like that.
A redirect on the client is a suggestion, not a lock. A customer with a second tab open on the same booking could hit checkout again after the first tab had already succeeded, before that tab's redirect had a chance to fire. Someone calling the API directly, skipping the UI entirely, wouldn't be affected by a redirect at all, because there was no redirect to bypass in the first place. Either path led to the same place: a request for a new checkout session against a booking that was already paid.
The fix was to stop treating the redirect as the whole guard and add a check the server enforced regardless of what the client did or didn't do. Before creating a new session, the checkout endpoint now looks up whether the flow's booking has already been finalized as paid, and if it has, it refuses outright with a 409 rather than starting a second session. The redirect is still there and still does its job most of the time, but it's no longer the only thing standing between a customer and an accidental double charge.
We locked that behavior into a test with a name that says exactly what it checks: "throws 409 and never calls payments service when the flow already finalized." A neighboring test, "returns cached session without calling payments service when cache hit," covers the related idempotency guarantee, that retrying a checkout request in the middle of the flow returns the same session instead of minting a new one. Neither of those is a subtle assertion. That was the point. I wanted the next person reading the test file, including future me, to see immediately what behavior we were protecting and why.
The 500 that was almost the interesting bug
The second failure is the one I actually opened this piece with, and it took two passes to fully understand.
The first pass looked simple enough. The finalize endpoint calls our legacy booking API, WAPI, to transition the booking from Pending to Paid, and every so often that call came back as a flat, opaque 500 with no useful detail in the body. Tracing it back to WAPI's own source showed the actual cause: a request missing certain fields, in this case payment data that our endpoint hadn't reliably persisted and forwarded, would hit an unguarded null dereference deep inside WAPI's booking logic. WAPI didn't validate that the data was present before using it. It just used it, and when the data wasn't there, the whole request blew up into a generic server error that told us nothing about which field was missing.
That's a familiar shape of bug. The fix was familiar too: check, on our side, before ever making the call, that every field WAPI's booking logic depends on is actually present, passengers, contact details, product data, the price snapshot, the payment method snapshot, the selected payment method itself. If any of it is missing, the endpoint now fails fast with a 422 that names the field, instead of forwarding an incomplete request and letting WAPI turn that gap into an unreadable crash. We wrote a test for exactly that boundary too: "throws 422 and does not call finalizeWapiBooking when selectedPaymentMethod was never persisted."
I want to be honest that this first bug, on its own, wasn't the interesting one. Missing required fields causing a downstream 500 is a bug I'd fixed a dozen times before in other systems. What made it worth writing about is what fixing it uncovered underneath.
The 200 that lied
Once the 500s caused by missing fields stopped, a quieter and much harder problem surfaced. WAPI's schema for the field that names which payment method to book against technically allows a whole GROUP of payment methods, not just one. Our endpoint had been persisting that entire group from an earlier step in the flow and forwarding it unchanged, rather than narrowing it down to the single method the customer had actually selected and paid with.
WAPI's validation didn't reject that. It accepted the group, deserialized it with every field coming back null because a group doesn't map cleanly onto the shape WAPI's booking logic expects for a single method, and then returned a plain HTTP 200 reporting a status of SUCCESSFUL. Every signal our endpoint was checking at the time said the call had worked. The status code said success. The reported status field said SUCCESSFUL. Underneath all of that, WAPI had never actually booked anything, because its own validation at the business level had rejected the payment method as invalid, just not in a way that surfaced as an error anywhere we were looking.
That's the run that opened this piece. I paid, the gateway confirmed it, our payments service confirmed it, and the booking sat there unbooked, because the one system we were treating as the final word on success had answered a question we hadn't actually asked. We'd asked "did this request complete," and WAPI answered that honestly. We needed the answer to "did this booking succeed," and WAPI had that answer sitting in a field called validationResults.isValid that we had never once looked at.
The real fix had two parts. First, stop sending WAPI the whole group and resolve the single entry the customer actually used, matched against the payment brand our own payments service reports for that specific payment, never against anything the client claims. Second, and this is the part I keep coming back to, stop treating any HTTP 200 from WAPI as proof of success at all. The finalize call now explicitly checks validationResults.isValid inside WAPI's response body before treating the booking as genuinely finalized, and logs an entry at error level whenever the resolved payment method doesn't match what the payments service reported, so a mismatch shows up in our own logs immediately instead of surfacing later as a booking stuck on Pending after a payment that had actually gone through.
That second part is the one worth sitting with. A 200 tells you a server accepted your request and produced a response. It does not tell you the response means what you hoped it would mean, especially against a legacy system whose real validation logic lives several layers below the HTTP status it eventually returns. If a response can lie to you, even unintentionally, even because a legacy schema was more permissive than it should have been, you have to check what it actually claims, not just whether it answered at all.
Writing the lesson into tests instead of memory
The part of this I'm proudest of isn't the fix itself. It's that none of these lessons live only in a code comment or in my own memory of that debugging session. They live in a test suite that would fail loudly if anyone, including me, ever reintroduced the same class of mistake.
A few of those tests, beyond the ones I already mentioned, capture exactly the boundaries this whole module depends on. One asserts that the finalize flow "does not use gateway callback params to determine success, always calls WAPI," which closes off a shortcut where a redirect query parameter might get trusted instead of verifying status again against the source of truth. Another checks that a postMessage back to the parent window "posts with an explicit targetOrigin, not a wildcard," a small detail that matters enormously the moment you're passing anything related to a payment across a window boundary. Another confirms the endpoint "clamps the persisted TTL to whatever the main flow has left, not a fresh FLOW_TTL," so a payment record can't accidentally outlive the booking flow it belongs to. And one covers the exact path for a brand mismatch from the WAPI incident directly: "logs the brand mismatch at error level and 422s without calling finalizeWapiBooking when no method matches the reported brand."
None of those read like abstract security theater. Each one is a direct, named answer to a specific way this system had already gone wrong once.
The lesson
If I had to compress everything this module taught me into one rule, it's this: a downstream system telling you it succeeded is a claim, not a fact, and the only claims worth trusting are the ones you've checked against a field the system actually uses to mean success, not against the signal at the transport level that it merely responded. WAPI's 200 meant "I processed your request." It never meant "I did what you asked." Every trust boundary in this module, what the client can send, what a gateway callback can claim, what an HTTP status code implies, exists because at some point one of those things told us something that sounded true and turned out to be false. The fix was never to trust that category of signal a little more carefully next time. It was to stop trusting it at all, and check the actual claim underneath it instead.