Your SRE agent should be working before the pager wakes you up

The alert fires. Someone acknowledges it, opens a laptop, finds the incident channel, and asks, "What changed?" Someone else starts looking for the dashboard. The engineer who knows the service is still joining.

Meanwhile, customers are still getting errors.

I think every company running customer-facing software should have an agent watching for customer impact and starting an investigation as soon as the alert fires. When the incident commander and engineers arrive, the first useful data should already be there. Making a person assemble that starting point every time is a poor use of the most expensive minutes in an outage.

That's why I expect observability to become conversational. I want to join an investigation that's already underway and ask, "Is this limited to the new release?" The agent should check, bring back the evidence, and keep working as we decide what to do.

The first five minutes should produce something useful

Imagine checkout starts failing at 2:03 a.m. A monitor catches the increase in failed requests and pages the on-call engineer. That same event starts the agent. The page goes out immediately, without waiting for the investigation.

The agent compares checkout failures with the previous healthy period. It checks which regions and versions are affected, looks for recent deployments and configuration changes, and pulls a few failed request traces. It also finds the service owner and the checkout runbook.

Here's the kind of brief I'd want waiting in the incident channel. This is an illustrative scenario, with invented numbers:

Since 2:03 a.m., 14% of checkout requests in the US region have failed, compared with 0.2% in the preceding 30 minutes. The number of distinct customers affected is still unknown.
Release 482 began rolling out at 2:01 a.m. Failures are concentrated on that version. Instances still running 481 remain near their earlier error rate.
Sampled failed traces show timeouts while acquiring a database connection. Database CPU has stayed near its earlier level. Lock contention has not been checked yet.
The release includes a connection-pool configuration change. That makes it a lead worth checking, but the investigation has not established the cause.
The checkout team owns the service. The next check is to compare pool settings and connection counts across the two versions, then review rollback safety with the service owner.

In production, each finding needs a link to its graph, saved query, trace, or deployment record, with the time window attached. A paragraph that says "probably the database" gives the team another opinion to argue about. This brief gives them checks they can verify.

The incident commander can start assessing impact and getting the right people involved. Engineers can investigate the version difference without spending their first ten minutes finding it.

Datadog Bits AI already does part of this

Datadog's Bits AI SRE, now called Bits Investigation, is a concrete example of this direction. It can gather context from the monitor message, consult linked runbooks and past investigations, and let responders ask follow-up questions about its findings.

Datadog documents automatic investigations for supported monitors. Enable Auto-Investigate, and a transition into the alert state starts the work. Bits queries telemetry to test explanations and can report an inconclusive result when the evidence isn't enough.

For a team already using Datadog, I'd start with a checkout error monitor or a synthetic test that exercises a customer journey. Add the service and environment context, attach the runbook, and review what the agent finds during an incident exercise.

Then try the questions people actually ask on a bridge. "Did failures begin before or after the deploy?" "Are customers on the old version affected?" "What have you checked that makes the payment provider less likely?" Those answers are useful only if the team can inspect the evidence behind them.

What I'd borrow from Pstack

Pstack is a collection of engineering skills that give an agent instructions for doing particular kinds of work. The closest match for diagnosing a live problem is the runtime forensics playbook in /poteto-mode. It tells the agent to capture a runtime signal, narrow it to the responsible behavior, test the explanation, and locate the relevant code.

Take a service that's using CPU while traffic is flat. The useful result is a profile showing where the CPU time went, followed by a check of why that code keeps running. For an existing profile or heap snapshot, Pstack's trace forensics playbook covers analyzing the capture and comparing it with a healthy one when available.

I'd also borrow /why for the history around a problem. It searches available sources such as PRs, tickets, docs, team chat, and observability tools in parallel, then distinguishes evidence from inference. In the checkout example, that method could uncover the earlier incident that led someone to choose the old connection-pool limit.

These are pieces of an investigation method. To use them for continuous SRE work, a team still needs to connect alert events to an agent runner, give it access to the relevant systems, and deliver its findings to responders. Installing a skill alone doesn't set that up.

I would adapt the runtime playbook before putting it near production. It includes techniques such as injecting instrumentation and hotfixing live code to test an explanation. My first version would use existing telemetry and approved captures with read-only access. Any experiment that changes the running service would need a separate decision.

Start with the thing customers can't do

Pick one journey. Customers can't sign in. Checkout fails. Orders stop reaching fulfillment. Write down how you detect that failure and the questions your on-call engineer asks first. Give the agent tools to answer those questions.

A busy CPU might explain a problem, but the investigation should establish whether customers are affected. A successful HTTP response can still contain an empty search result or a failed business operation. Include failed synthetic journeys, application errors, and business signals where they help show what customers are experiencing.

Keep one investigation for related alerts. If checkout produces twenty notifications, the team needs one evolving brief. Have the agent publish partial findings as they arrive, mark what it couldn't check, and update the brief when someone provides new context.

Give it specific instructions from your runbook. "Compare the current release with the previous release." "Check whether the backlog is growing or draining." "Find the last time this error appeared and link the incident." Those are tasks an engineer can review and improve after each outage.

Also put a time and query budget on the investigation. An agent that searches every log in the company while checkout is down can create more work for the same systems you're trying to recover.

Make the MTTR claim earn its place

I expect this to cut recovery time substantially where teams spend a large part of an incident gathering context. By MTTR here, I mean mean time to restore service. The number that matters is when customers can use the product again.

Consider an illustrative 40-minute outage. The team spends 20 minutes finding evidence and another 20 choosing a mitigation, applying it, and confirming recovery. If an agent reduces evidence gathering to five minutes, that outage takes 25 minutes, assuming the rest stays the same. Fifteen fewer minutes of customer impact is a large improvement. It is also an example of the opportunity, not a measured result from Bits AI or Pstack.

Test it against your own incidents. During a replay, give the agent only the information available at that point in the outage. Keep the eventual postmortem out of reach. Measure how long it takes to produce the first finding an engineer can verify, and record the wrong leads that would have wasted time.

In live use, track time to mitigation and confirmed customer recovery too. A fast investigation report is helpful only if it helps the team restore service sooner.