ilert seamlessly connects with your tools using our pre-built integrations or via email. ilert integrates with monitoring, ticketing, chat, and collaboration tools.
See how industry leaders achieve 99.9% uptime with ilert
Organizations worldwide trust ilert to streamline incident management, enhance reliability, and minimize downtime. Read what our customers have to say about their experience with our platform.
When you get paged at 3am, it takes about 30 seconds for the notification to reach you and maybe two minutes until you're in front of a laptop, awake enough to read. What you see then is usually a raw alert. A metric name, a threshold, a link to a dashboard. Then the ritual starts: open the dashboard, check what deployed in the last few hours, grep the logs for the first error, ask in Slack whether anyone touched the database.
Most of that time is search. Once you've found the needle in the haystack, the fix is usually fast, and most of the time it's a rollback. It's the search that takes 20, 45, sometimes 60 minutes of your night.
The goal for ilert AI SRE was simple to state: by the time you open the laptop, the search should be done. We opened the closed beta about a year ago, back then under the name ilert Responder, with a few teams. Since then we've rebuilt the agent three times, and each rebuild taught us something the previous version didn't know: where an agent actually helps, what a good harness looks like, and what "production-ready" means when the agent is on call next to you. Today it's generally available on every paid plan and during trials.
What it does
Give AI SRE an alert from any of our 150+ integrations and it investigates the way you would, just faster. It looks at logs, metrics and traces. It looks at what changed, because change is the number one cause of incidents: deployments, config changes, merged pull requests, and, if you've connected your repository, the actual diff. If 50 alerts fire at once, it first triages them into clusters and investigates each cluster on its own.
A few minutes later, you're not looking at a blank alert. You're looking at a root cause analysis.
Every investigation has the same shape. A hypothesis for the root cause, with a confidence label: high, medium or low. Five or six key findings, and every finding links to its evidence: the deployment it's referring to, the log line that produced the out-of-memory error, the metric that broke first. A short list of things it ruled out. And, where it makes sense, a proposed action: roll back to the last healthy version, double the memory on that pod, restart the service.
Then it waits for you. You make the call. Nothing changes the state of your system without an engineer clicking approve.
At GA, you start the investigation. There are three ways in: from an alert, from an incident, or by describing what you're seeing in the agent's chat. Plain language is enough; "customers are reporting that metrics on their status pages are stale" is a complete brief, and it's how the incident in the next section was handed to the agent. Investigations that start on their own the moment an alert fires, so the analysis is already waiting when you pick up the phone, are the next step, not this one.
A real one
A while ago we had an external penetration test. It found a blind SSRF in the metrics feature of our status pages: customers can point a status page at their Datadog or Prometheus to display API response times, and an attacker could try to make that fetcher call internal URLs instead. We fixed it the same week with a Kubernetes network policy. The network policy was too broad. The metrics service could no longer talk to its own database, and metrics on customer status pages went stale.
I like this incident as a test case for two reasons. First, no runbook on earth covers "after a pentest fix, status page metrics go stale." Novel incidents don't have runbooks. Second, the symptom is ambiguous: tell an agent "customers say metrics stopped working" and there are a dozen internal things called metrics it could chase, including our entire Prometheus setup. The agent had to find candidate pods, find the logs showing a service that couldn't reach its database, then walk the recent changes until it landed on the network policy. That's the kind of search that eats an hour of a human's night, and it's exactly the kind of search the agent is good at.
Why there is no autonomous mode at GA
We think about autonomy in levels. Observe only: the agent has read-only access and produces an analysis. Propose: the agent suggests an action and a human approves it. Pre-approved actions: a class of low-risk actions the agent may run on its own when its confidence is high. Fully autonomous: you only get paged when the agent is stuck.
GA ships the first two. I want to be direct about why, because the usefulness of an agent grows with its autonomy. A self-driving car that needs you watching the road is helpful; one that doesn't is a different product. I believe we'll get there, and I think production agents will reach the maturity coding agents reached over the last year. But I don't have a way to do full autonomy safely today, so I wouldn't run it today, and I don't think you should either. We run demos where the agent does everything end to end, from investigation to status page update to fix, and we always add: don't do this at home.
Observe-and-propose is not a consolation prize. Read-only access means that even if the agent goes badly wrong, the blast radius is an incorrect document. And most of the time you spend on an incident is the root cause analysis. Cutting that from 45 minutes to a few is the bulk of the value, before the agent ever touches anything.
What it reads, and what it deliberately doesn't
The best documentation of how a system actually works is the code and the live telemetry. Everything else goes stale. So AI SRE doesn't start from your Confluence, your Notion or your runbooks. Pointing an agent at a wiki with hundreds of runbooks, the last of which was updated three years ago, causes more damage than it produces results.
What it does read is everything you already send to ilert, plus what you connect: logs from Elastic or CloudWatch, dashboards and metrics from Grafana, Prometheus or InfluxDB, your Kubernetes cluster, your GitHub repositories and CI/CD pipeline. That breadth is the point. Every observability vendor is building an AI SRE right now, and each one sees its own slice. We sit on top of all of them, we stay vendor-neutral, and if your observability tool produces its own RCA, we take it as one input among many. Nobody wants another siloed tool that only does RCA for one data source.
Beyond what you connect, the agent runs a discovery phase when you set it up, and again whenever it's been idle for a while. It builds a service topology from live tracing data, so it understands your dependencies without anyone maintaining a service catalog by hand. That topology is also what allows our agent to do cost-effective alert triage before running an investigation with powerful (but more expensive) reasoning models. If you don't have tracing instrumented, and most companies don't, you can drop an eBPF collector into your cluster and get most of the way there without touching code.
Over time the agent keeps a lightweight long-term memory of the tribal knowledge it picks up, the "this service always hits its limits at midnight because of the batch jobs" kind of thing. It's plain text. There's no vector database to keep in sync.
How we test it
There is no recipe for root cause analysis, which makes it hard to test. We do three things.
First, a public benchmark. OpenRCA, from Microsoft Research, is the hardest public test of root cause analysis I know of: 335 real failure cases from three production systems, 68 GB of telemetry, scored all-or-nothing per task. What counts as a full point depends on the task: some require the right component, reason, and time; others are scored on a smaller subset, sometimes just the component. At paper release, the best published baseline, an agent on Claude 3.5 Sonnet, solved about 11% of the cases. By the time we ran our own benchmark, later leaderboard results had moved on, Opus 4.6 sat at roughly 36%, so treat that 11% as the paper's number, not today's frontier.
On a 50-task sample across all three systems, our first prototype solved 26% under that strict rule, and matched the ground truth in 30% of cases by the broader measure, at roughly nine minutes per investigation. I'd treat the number as a floor, and the sample size as the caveat it is, but two things we learned matter more than the number. Handing the agent a plain shell on the raw telemetry nearly doubled the strict pass rate compared to handing it our eight production-shaped Grafana and Prometheus tools, 26% against 14%, with fewer tool calls. Tool design is not a detail. And OpenRCA is telemetry only: no deployment history, no config changes, no alerts. In production, changes are the agent's strongest signal, so this benchmark measures it with one hand tied behind its back. In the same run, Claude Code carrying our investigator prompt scored 38%, ahead of our own harness. We publish that because it's the test we hold ourselves to internally: if a general coding agent with your prompt beats your harness, your harness has work to do.
Second, chaos. We inject real faults into a real environment and hand the agent only what an on-call engineer would see: the symptoms. It's never told what we broke. Across a dozen scenarios covering bad deploys, config regressions, resource exhaustion, dependency failures:
Correct root cause identified in all runs
Median time from alert to first finding: 194 seconds
Proposed remediation matched what our engineers would have done in 90% of runs
Wrong hypothesis presented with high confidence: Zero. This is the number we watch most closely.
Third, replay. Every production investigation is recorded: every tool call, every response, the final RCA. When we change a model or a prompt, we replay those recordings and compare the outcome, with an LLM as judge and with embedding similarity. This catches regressions when a new model comes out. It doesn't catch everything: a recording only covers the tools the original run happened to use.
The agent is sometimes wrong. Not hallucinating-wrong; in a highly contextualized environment we see very little of that. Plain wrong: a confident line from the symptoms to the wrong cause. The confidence label and the evidence links exist so you can see that in under a minute, and either redirect it with a follow-up question or do the analysis yourself.
What it doesn't do
It won't act on its own. Every action requires approval.
It only sees what you connect. Most of the value of an agent is in the context it gets. If it can't see that something deployed, it can't blame the deploy.
It won't fix not having observability. Some people hope the agent lets them skip that step. It doesn't, at least not yet.
It's not great at backing out of a wrong path on its own. When the search space is large and the starting point is vague, it can get stuck. That's when you ask for a follow-up or take over.
What this changes for the team
The conversations we're having with engineering leaders about this are different from the ones we used to have about MTTR. A pattern I keep seeing: a small, strong SRE team runs first-line on-call across many product teams, with the agent doing the triage and the first investigation. Fair rotations normally force every service team to staff around five people just to cover on-call. Centralizing first-line on an AI-augmented team removes that constraint per team. Engineers spend more of their time shipping and less of it in 25-person incident bridges. Same headcount, more services, lower cost per incident.
That's the version of "AI in operations" I find worth building: not fewer engineers, but engineers who were hired to build things getting to build things. Nobody was hired to be on call full-time.
Your data
All AI workloads run on dedicated infrastructure in our EU regions, Frankfurt and Stockholm, and models are called through regional endpoints. We don't train on your data and have opted out of training with every model provider we use. Read-only API keys are all the agent needs to investigate. Personal and user-level data isn't shared with external models. If your security team wants to bring your own model API keys behind your own guardrails, we support that. Details: https://docs.ilert.com/trust-center
Availability
AI SRE is on for every paid plan and for trials starting today, running on the AI credits included in your plan: 250 per month on Pro, 1,000 on Scale, 5,000 on Enterprise. You can see your baseline and decide what happens when you run out.
Existing customers: nothing to migrate. Enable it under Account settings → AI features, create an agent and connect it with your tools, and launch your first investigation from the next incident.
Where this goes
The end state we're building toward is that you're not paged at 3am at all. You wake up to a report: the agent stopped the bleeding, verified for an hour that the symptoms were gone, and there's a pull request for the real fix. That's not GA. GA is the search taking a few minutes instead of an hour, started with one click, with the fix one click behind it. The trust for the rest gets earned one investigation at a time, and this is where it starts.
If you have a scenario you think it will get wrong, I want to hear about it.
For years, ilert has given our customers extensive analytics across their alerts, notifications, and on-call activity, a comprehensive overview of how their teams and services respond to incidents. These capabilities were backed by a separate analytical database running on Google BigQuery. It held the numbers behind every reporting dashboard in ilert, and for a long stretch it was perfectly fine.
Then three problems grew too big to ignore:
It was the slowest experience on our platform. Analytics dashboards aren’t on ilert’s hot path, but they were noticeably sluggish, several seconds on average, and up to twenty seconds for our largest customers. Not the experience we wanted to offer.
We paid per query. BigQuery bills by bytes scanned, and our workload is the same handful of dashboards run over and over, so we kept paying, again and again, to answer questions we’d already asked.
Updating a row was difficult. Some of our analytics aren’t append-only, for example, an alert’s state, TTA (time-to-accept) and TTR (time-to-resolve) all change as the alert is acknowledged and resolved, and we need the stored row to follow. BigQuery is built around appends and full rewrites; row-level updates there are slow and awkward.
There was a bonus reason too: BigQuery happened to be the last workload keeping us on Google Cloud, so retiring it let us drop a whole provider, and a subprocessor from our tech stack.
So we moved our analytics to ClickHouse, running on our own AWS infrastructure. Queries that used to take around ten seconds now come back in under two. Here’s how we got there.
Our workloads
Our analytical workload consists mainly of:
Events: the raw alerting events ilert ingests and handles. A high-volume, append-only stream, the source data everything else is built from.
Statistics: the aggregates behind our core capabilities: alerts, notifications, and on-call reports.
Entity state transitions: the history of how our core entities move through their lifecycle: call flows, event flows, incidents. We log what happened over time, with each transition as its own distinct row, one per entity per change.
Those three workloads shaped both the problems above and the design below. What we needed was a fast, columnar database, tuned for tenant-scoped, time-range reads, with a table engine that could also handle mutations. The shortlist also included Apache Druid, Apache Pinot, StarRocks, and TimescaleDB. But ClickHouse matched our access pattern most closely, and just as important, it wasn’t a risky bet. We had been following the project for years and watching it mature. In that time it has become a well-established choice for large enterprises like Uber, Cloudflare, and Cisco.
Why not just tune BigQuery?
It’s the fair question, and we asked it first. BigQuery has real answers for slow interactive dashboards, BI Engine, reservations, materialized views, and any of them would have helped with the speed, and probably the cost too.
But every one of them keeps us on Google Cloud. By this point BigQuery was our only remaining GCP workload; everything else already ran on our own AWS infrastructure. Tuning BigQuery would have fixed two of our three problems while permanently cementing a second cloud provider, and a second subprocessor, into the stack. Moving the workload instead let us fix the speed and the cost and collapse two providers into one. That is what decided it: migrate, rather than optimize in place.
Getting data in, Kafka in the middle
As is common in the industry, our operational and analytical data stores are decoupled and only eventually consistent, with a buffer in between. We were already using Kafka for that buffer, so it was the natural choice for feeding data into ClickHouse too.
The realtime flow:
Source services own their operational data as they always have.
A dedicated CDC (change data capture) service reads change events and exports them onto Kafka topics, one logical stream per workload.
Consumer workers read from Kafka, batch rows client-side, and insert into ClickHouse.
The backfill flow reused the same path, with a different source:
An export job reads the historical BigQuery tables, one by one, and streams the rows into a Kafka topic.
A backfill consumer batches those rows and inserts them into ClickHouse, the same way the realtime consumers do.
Those consumer workers needed to do one thing well: insert rows into ClickHouse efficiently. ClickHouse is fastest with large, infrequent inserts and slow with many small ones, so each consumer batches rows client-side, flushing on whichever comes first, a size threshold or a short timer, and commits its Kafka offset only after the batch lands. Offset and insert move together; otherwise a crash could acknowledge rows that never reached the table.
Why route through Kafka instead of dual-writing?
Decoupling. The source services don’t know or care that ClickHouse exists. If the analytical store is down for maintenance, events queue in Kafka; nothing upstream blocks.
Replayability. Getting a schema or a transform wrong is normal during a migration. With the stream retained in Kafka, we can rewind offsets and re-ingest into a corrected table instead of asking every source service to send its history again.
Backfill and cutover in parallel. We backfilled history into ClickHouse while the live stream kept flowing, then flipped reads over once the two agreed.
Natural batching boundary. ClickHouse wants large, infrequent inserts, not a trickle of single rows. A Kafka consumer is the ideal place to accumulate a batch and flush it.
Designing the ClickHouse schema
One of the main decisions each ClickHouse adopter has to make is the choice of table engine, which comes down to one question: can the data be mutated, and in what way.
Beyond the engine, our tables share a few conventions. Each one is sharded across the cluster and replicated for durability, partitioned by month, and sorted by tenant and then time, so a tenant’s date-range query touches the smallest slice of data possible. Columns are compressed with type-appropriate codecs, and every table carries a TTL that ages data out on its own retention schedule instead of letting it grow forever.
Analyzing our use cases, there is one that clearly requires mutability:
Alert analytics: as an alert moves from triggered to acknowledged to resolved, we simply insert it again with its new state; ClickHouse keeps the newest version and drops the older ones as it merges. Reads ask for the current state with FINAL, which removes those old versions at query time. This is ReplicatedReplacingMergeTree.
But keeping in mind the need to backfill, we decided to use the “replacing” engine for all other migrated tables as well.
Why this shape makes fixing migration issues safe
Migrating data is messy. You almost always get halfway through, notice a transform was subtly wrong, and have to run the whole backfill again. On a plain append-only table, re-running means every row you already inserted lands a second time, so re-ingesting the backfill turns into a truncate-and-start-over task, and it is risky every time.
The “replacing” engine changes this. Because it deduplicates by the full ORDER BY tuple and keeps the row with the highest version column, re-ingesting the same logical rows from the stream simply replaces them, while any rows missing the first time just get filled in. Re-ingesting the backfill stops being cleanup and becomes something you can safely repeat: rewind the consumer, let it run, and the table ends up in the correct state. Nothing duplicates, nothing needs truncating, and FINAL gives you the clean view even before background merges catch up.
The FINAL isn’t free, but it is acceptable for us because none of the APIs backed by these tables are on a hot path of the platform, so in practice the overhead is negligible.
The tricky part: porting the queries
Setting up the tables was the easy half. Porting the analytical queries from BigQuery’s SQL dialect to ClickHouse’s took far more attention, and the dangerous bugs weren’t the ones that raised an error. They were the ones that returned a perfectly plausible number that happened to be wrong. Even with the help of AI assistants, it took several iterations to map each one properly, leaning on our test data to make the new queries functionally identical to the old ones.
A few of the traps that bit us:
%M is the month, not the minute. In ClickHouse’s formatDateTime, %M prints the full month name and %i is the minute. We ended up using the provided shortcuts: %F = %Y-%m-%d, %T = %H:%i:%S
Non-aggregated columns. BigQuery lets you SELECT a column that isn’t in the GROUP BY, as long as it is functionally determined by it. ClickHouse refuses. Just adding the column to the GROUP BY silently splits rows and double-counts if that column ever varies within the group. The correct fix is to wrap it with aggregates like any() or min().
Indexing starts at 1. ClickHouse arrays and tuples are 1-based. BigQuery’s [OFFSET(0)] for the first element becomes [1], and tuple fields are t.1, t.2.
Averaging over “skipped” rows. In our data a TTA or TTR of 0 means the alert was never acknowledged or resolved, not that it took zero seconds. A plain AVG would count those zeros and drag the number down, so we had to use the following expression: COALESCE(ROUND(AVG(CASE WHEN x != 0 THEN x END)), 0).
The trade-offs we accepted
Self-hosting ClickHouse brings complexities of its own, and we took them on consciously:
We operate it now. Replication, ZooKeeper, server upgrades, backups, capacity planning, the things BigQuery did invisibly are our team’s job today. We took that on deliberately; it’s the cost of controlling performance and consolidating onto one provider.
We size for peak. There’s no serverless elasticity. The cluster is provisioned for our busy periods and sits under-used the rest of the time. For a workload as predictable as ours, that still comes out cheaper than paying per query.
The results
The number we cared about most improved the way we hoped: p95 dashboard query latency dropped from around ten seconds to under two, roughly a 5× speedup.
Everything else followed from the design:
Cost: per-query scan billing is gone, replaced by the fixed, predictable cost of a cluster running on hardware we already operate.
A leaner schema: being able to update the alert-state field through ReplacingMergeTree let us drop to a more compact table layout, instead of the append-and-rewrite workarounds BigQuery required.
Operational resilience: with Kafka buffering ingestion, a ClickHouse maintenance window is a queue that drains later, not lost data, and a bad transform is a replay, not an incident.
One less dependency: BigQuery was our last workload on GCP. Retiring it dropped a whole cloud provider from our tech stack.
All in all, we’re happy with the migration. The wins above were the goal, but the bigger payoff is what it opens up: with fast, affordable, real-time analytics as our new baseline, richer reporting and deeper insights are suddenly cheap to build. Expect a lot more analytical capabilities landing in the ilert platform soon.
Not all alerts are created equal. Some are resolved quickly by the on-call engineer. Others signal something serious enough to affect your business and require your whole team to coordinate. That is why we redesigned incidents as a dedicated coordination workspace for the alerts that have the most business impact.
Until now, incidents in ilert were used to communicate status updates to customers and stakeholders. Creating one meant publishing to your status page. We have separated the two. Incidents are now internal. Status updates remain the way you communicate externally, and they go out only when you choose to post one.
Inside an incident, your team can create an incident channel, run parallel escalations, track the timeline, manage status, and create post-mortems. Nothing reaches your customers until a status update is published.
Here is what changed and what is new.
Incidents are now your internal coordination workspace
Previously, creating an incident in ilert meant publishing an update to your status page directly. Incident coordination lived inside alerts, and information went public the moment an incident was created.
That's no longer the case, incidents are internal; nothing is public until you post a status update. Creating an incident opens an internal workspace for your response team, and nothing reaches your status page until you choose to publish a status update. The two are fully decoupled.
Alerts, incidents, and status updates
With this change, ilert now has three clearly separated objects, each with a different purpose:
Alerts have not changed. They signal issues and page your team.
Incidents are now a dedicated internal workspace for coordinating your response. They are no longer tied to your status page.
Status updates are now separate from incidents. They are how you communicate with customers and stakeholders when you are ready.
Your existing incidents will be migrated automatically to status updates. Nothing has been deleted.
Declare incidents with severity and a dedicated channel
Two new fields appear when you declare an incident: severity and incident channel.
Severity
Severity lets you classify the business impact from the moment of declaration, which is useful for routing, reporting, and setting expectations with your team. When declaring an incident, severity is set to SEV3 by default and can be changed depending on your context.
Incident channel
A dedicated channel can be created when you declare an incident. Previously this was done from alerts. It now lives in the incident, where all the context for that response is kept in one place.
A full response workspace: parallel escalation, incident commander, status, timeline, and post mortems
Once declared, the incident becomes the single place your team coordinates the response.
Parallel escalation
Page multiple teams and individuals at the same time when you declare an incident. You do not have to wait for one escalation chain to step through before paging the next.
You can trigger all of these simultaneously:
An escalation policy
An on-call schedule
A full team, which pages every member at once
A specific user, directly
When paging a team directly, the first member to join the incident stops the page for others, who receive a "page resolved" notification.
Incident commander
Assign a named person to own the response, giving the team a single point of accountability from declaration through to resolution.
Incident status
Status can be tracked and updated as the team works on the situation, keeping everyone internally informed on where the response stands.
Incident timeline
The incident timeline records every action, page, and change automatically in real time. Useful while the incident is running, and essential when writing the post mortem.
Post mortems
Post mortemshave moved from status updates to incidents, where the full record of what happened actually lives. When you resolve an incident, a post mortem can be created and populated from the timeline, incident channel messages, and linked alerts.
Status updates: communicating with stakeholders
What was previously called an incident on your status page is now called a status update. The functionality is unchanged. Status updates now have their own dedicated page in ilert.
There are two ways to create a status update.
Post status update from status updates page
The status updates page gives you a dedicated view of all your published updates. Filter by service, status page, and creation date, and post new updates directly from there.
Post status update from incident (recommended)
Work the incident internally, then publish a status update when you are ready to communicate externally. The status update is linked to the incident, keeping your internal coordination and external communication connected.
In short, incidents now serve as an internal coordination layer with a dedicated set of tools to support your team during response. Status updates remain unchanged and are the communication layer used to inform your customers and stakeholders.
If you have any questions or need support navigating the new setup, reach out to your Customer Success Manager or contact our support team.
New to ilert? Try it for free and see how your team can coordinate faster when it matters most.
