Why Observability Must Become Proactive on Kubernetes
Observability is evolving faster than the organizations that practice it. As systems spread across Kubernetes, serverless, and managed services, telemetry volumes explode, yet context becomes ever harder to piece together. Static thresholds, siloed tools, and per-metric alerts produce noise, slow down root cause analysis, and burn out operations teams.
In 2026, the challenge is no longer collecting more signals: OpenTelemetry has established itself as the instrumentation standard, and the central question has become correlation, connecting a technical anomaly to real business impact, and doing so before the customer notices. This is precisely the territory of AIOps: anomaly detection learned from data, impact-based prioritization, and, increasingly, investigation assisted by AI agents.
This article details how we used Datadog on Amazon EKS to move a global manufacturing client from permanent firefighting to proactive, AI-driven monitoring: monitors as code, unified tagging, chaos-engineering validation, and alert designs tied to business impact, delivering 80% less alert noise and a mean time to restore (MTTR) cut in half.
Alert Fatigue: Diagnosing a Monitoring Platform Running on Empty
Our client, a global manufacturing leader, was operating an observability platform that had become noisy, fragmented, and purely reactive. Instead of producing actionable insights, it poured out a continuous stream of irrelevant, redundant, or misleading alerts.
Five symptoms of monitoring that has lost the teams' trust
The diagnosis revealed five recurring patterns: false positives, where benign events like a brief CPU spike were flagged as critical; non-actionable alerts on predictable conditions such as scheduled maintenance; duplicate alerts, with multiple notifications pointing to the same root cause; flapping alerts oscillating between OK and ALERT; and static thresholds triggered by harmless fluctuations, ignoring traffic seasonality and business context. Perfectly normal temporary spikes thus triggered false urgency without any real degradation of the user experience.
The result was classic alert fatigue: high volume with no prioritization, overwhelming the operations teams. This constant noise slowed down root cause analysis, multiplied manual troubleshooting, and obscured the genuinely critical issues. The downstream impact was severe, order-processing delays, repeated interventions during peak hours, and a growing operational load that ultimately degraded both customer experience and business performance.
Why Datadog to Unify Metrics, Logs, and Traces
The limitations of the existing platform made the need obvious: a unified, intelligent observability solution able to cut noise, add context, and provide end-to-end visibility across a modern distributed environment. Datadog was selected for its ability to address these gaps directly.
Four criteria weighed on the decision: a unified platform offering a single interface for metrics, logs, and traces, eliminating tool-switching and speeding up investigation; AI-driven monitoring, with built-in anomaly detection and dynamic thresholds; full visibility through native integrations with EKS, AWS services, and in-house applications, including OpenTelemetry telemetry; and business-technical correlation, meaning the ability to ingest custom business metrics and tie them to technical signals.
Correlating technical signals with business impact
That last point was decisive. An alert announcing "p99 latency rising" is only valuable if you know which customer orders it puts at risk. The combination of Datadog's functional breadth, integration depth, and AI capabilities made it more than a replacement for the legacy platform: the foundation of a proactive, business-context-aware observability model, aligned with the AIOps practices the client wanted to establish.
Phase 1: Industrialized Observability Foundations on EKS
We approached the transformation in two deliberate phases. The first aimed to build consistent, scalable foundations, around five capabilities that would underpin all subsequent observability and AIOps work.
Unified telemetry and RED dashboards
We deployed the Datadog agents and admission controller across a shared EKS cluster, so that every service was automatically instrumented for metrics, logs, and traces. Unified service tagging (env, service, version) was applied from the start: every signal, whether from infrastructure, an application, or a custom metric, can be filtered and correlated consistently across environments. We then adopted the RED framework (rate, errors, duration) as the application-health baseline: every service gets a standard dashboard template, complemented by a cluster-wide infrastructure view, a shared performance language across development, operations, and the business.
Monitors as code and GitOps pipelines
Alert definitions were expressed as Kubernetes custom resources managed by the Datadog Operator, allowing monitors to be versioned, reviewed, and managed like any other code. Environment-specific metadata, thresholds, service identifiers, is stored as JSON on GitHub, and GitHub Actions pipelines deploy and update the monitors automatically. No more configuration drift: a monitor change becomes as fast and reliable as an application deployment.
Validating reliability through chaos engineering
Before going to production, we ran targeted chaos-engineering experiments: deliberately injecting faults into microservices to verify that dashboards lit up in the right places and alerts fired at the right time. This step gave the team confidence that the monitoring stack was trustworthy before depending on it in production.
Phase 2: AIOps, Business Metrics, and Natural-Language Access via MCP
The second phase extended the platform with deeper integrations and advanced capabilities. With the foundations, telemetry, tagging, monitors as code, in place, we could connect more systems, enrich the data model with business context, and shift from reactive monitoring to proactive incident prevention.
AWS integrations: RDS, SQS, and Lambda under watch
We enabled Datadog's native integrations with the AWS services critical to order processing: Amazon RDS to surface slow queries, connection limits, and bottlenecks; Amazon SQS to monitor queue depth, processing rates, and error handling; AWS Lambda to track cold starts, execution timeouts, and business error codes. Each integration comes with a dedicated dashboard: engineers troubleshoot EKS containerized workloads and managed AWS services in a single motion.
From static thresholds to anomaly detection and business metrics
We first captured business metrics directly from traces and logs, order-creation volumes, error-code occurrences, key events of operational success. Alerts now fire when successful orders drop significantly, not merely when a technical threshold is crossed. Distributed tracing then accelerated root cause analysis: following an order through every microservice pinpoints exactly where it slows down or fails, a slow SQL query, a failing service, or a degraded third-party payment gateway. Finally, Datadog's anomaly-detection models, including Watchdog, replaced static thresholds: they learn normal patterns, seasonality included, and flag deviations before customers notice them.
Querying observability in natural language with Datadog's MCP server
Finally, we made the platform accessible to AI agents by connecting Datadog's official MCP server to the team's assistants, Claude Code and GitHub Copilot in agent mode. An on-call engineer asks an operational question in natural language,"which services had error spikes in the last hour?", and gets an actionable answer without writing a complex query.
Measured Results: Alert Noise Down 80%, MTTR Cut in Half
The transformation produced measurable improvements. Alerts were reclassified into P1 to P4 tiers based on business impact: critical issues get immediate attention, low priorities are handled without disrupting key workflows. This reclassification, combined with context-aware monitors, cut alert noise by 80%; every alert now carries the context needed to start the investigation without wasted time.
On the operations side, MTTR dropped by 50% thanks to trace-driven analysis and enriched alert context. Lessons from past incidents were converted into predictive monitors, rate-limit and quota alerts at the API gateway level, to prevent recurring issues before they reach customers. Unified dashboards let engineers isolate a problem, assess its business impact in real time, and resolve it without switching tools.
On the business side, the gains translated into a 15% drop in support tickets related to order processing, a 10% increase in completed orders during demand peaks thanks to proactive bottleneck resolution, and roughly 20% fewer engineering hours spent each week on manual troubleshooting, time reinvested in higher-value work.
Toward Agentic Observability: The Roadmap Beyond AIOps
Moving from a noisy, reactive stack to a unified, AI-assisted platform is only a beginning. The next steps extend the logic: broadening anomaly detection to more critical workflows, refining composite alerts for complex cross-service patterns, and automating remediation for recurring incidents. Early trials of autonomous investigation agents, such as Bits AI SRE, which pre-analyzes alerts and proposes verified root-cause hypotheses, show that first-level triage can largely be delegated to the machine, under human supervision.
Natural-language access also opens the platform beyond engineering: support and product teams can query service health directly without depending on a Datadog expert. In the longer run, the goal is to connect observability to codebases, customer sentiment, change management, and deployment pipelines, closing the detection-resolution-prevention loop. In this model, operations stop being a reactive cost center and become a genuine intelligence layer guiding decisions across the entire business.
STAY POSTED
Get our next analyses and field notes straight to your inbox.
