Observability and testing in production

What this is not

"Testing in production" does not mean skipping the testing before it. Every argument in the previous eleven lessons still holds.

It means accepting something staging cannot fix. Staging has a fraction of the data, none of the traffic, mocked third parties, a different network, and none of your users. Some defects only exist where those things are real — and they are found in production either way. The only question is whether you find them before your users do, or read about them in a support ticket.

The previous lesson was about getting confidence without a full environment before deploy. This one is about the half that comes after.

Two prerequisites, and neither is optional

You are allowed to test in production when you can do both of these:

See what happens. Logs, metrics and traces, reachable by you, not just by the platform team. Without them you are not testing, you are poking.

Limit the damage. A feature flag, a canary, and a rollback that takes minutes. Without those, every production test is a bet on being right.

A team with neither should treat "we test in production" as a description of what is happening to them, not a strategy.

Observability, in tester's terms

	Answers	You use it to
Metrics	Is something wrong?	Notice a change: error rate, latency, throughput
Traces	Where is it wrong?	Follow one request across services
Logs	What happened?	Read the detail of the specific failure

The practical version for a tester: when you reproduce something in production, capture the trace id. A bug report carrying a trace id skips the entire "can you give us a timestamp and we'll go looking" round trip, and it is the single highest-value habit in this lesson.

The four golden signals are what to watch when you do not know what to watch: latency, traffic, errors, saturation. Most production surprises show up in one of those four before anyone files anything.

And the numeric version of T2's rule — a number without its conditions is an opinion — is the SLO: "99.5% of checkout requests under 800ms over 30 days". That sentence has a target, a scope and a window, so it can be met or missed rather than argued about. The gap between it and 100% is the error budget, and a spent budget is a legitimate reason to stop shipping features.

Deploy and release are different events

Separating them is the biggest testability win available in production, and it is what makes everything below safe.

Feature flags ship the code dark and turn it on for whoever you choose — your own account first, then internal users, then a percentage. Testing with real data, real integrations and real traffic, with an off switch that does not need a deploy.

Three things flags demand in return:

• Test both states. The off path is the rollback path. A flag whose off branch was never exercised is a rollback that fails at the worst moment.
• They multiply state space. Ten independent flags are 1024 combinations. Nobody tests 1024 combinations, so keep the number of simultaneously live flags small and know which ones interact.
• They are debt. A flag that has been on for everyone for six months is dead configuration and untested branches. Removing it is a task, and it should be on the board.

Canary releases send a small share of real traffic to the new version and compare its error rate and latency against the old one. This is automatable and usually under-automated: the comparison is the test, and the rollback should be its assertion failing.

Blue-green keeps two full environments and switches traffic. Faster rollback, more infrastructure, and — the part people forget — your database migration has to work for both versions at once, which is a testing problem before it is an operations one.

Synthetic monitoring: where your E2E suite goes to live

Take the five or six tests that cover your critical paths — login, search, checkout — and run them against production on a schedule from a couple of regions. They answer a question no dashboard does: does the thing work right now, whether or not anyone has tried it yet.

Rules that keep it from becoming a liability:

• A dedicated, identifiable test account, never a real customer's.
• Read-only where possible, and where not, clean up after yourself.
• Tag the traffic so it is excluded from analytics, conversion metrics and revenue reporting. Synthetic checkouts in the sales figures is an easy mistake to make once.
• Nothing destructive, no third-party systems (a synthetic payment is a real payment), and no test data left where support will find it and open a ticket.

Synthetic and real-user monitoring answer different questions and you want both: RUM tells you what your users actually experienced on their real devices, synthetics tell you whether a specific journey is working at 4am on a quiet Sunday.

Production is also your best source of test ideas

This is the part testers under-use. Before designing the next test round, go and read:

• The most-used flows. Effort should follow usage, and the ranking is almost never what the team assumes.
• The real browser and device mix. T2's compatibility lesson said to build the matrix from your own analytics rather than from a market-share chart — this is that data.
• The endpoints carrying most of the traffic, which is where a performance regression hurts most.
• The errors already happening. Most applications log failures nobody has triaged. Reading a week of them is often the highest-yield hour in a sprint.
• The searches returning nothing, the forms abandoned at one particular step, the retries.

And after every incident: an incident that does not produce a test is an incident you have agreed to have again. Write the regression check while the post-mortem is still open, not from the ticket three weeks later.

Where TestForge fits

Point the scheduled synthetic run at a project of its own and upload each result through /api/v1/junit the way the T3 capstone did. The value is not the individual run, it is the record: a suite named after each critical journey, one result per interval, so "was checkout working last Tuesday at 03:00" becomes a query rather than a memory.

The honest limitation: this is a test-management system, not an alerting platform. It will hold the history and show you the pattern; it will not page anyone at 3am. Wire the alert to your monitoring stack and keep TestForge for the record that survives the incident.

Next: AI in QA — where it genuinely helps, and where a plausible-looking test is worse than no test at all.