AI Agent Monitoring on Datadog: What It Does Well and Where the Gaps Are
By Brad Bell, Principal Observability Lead
Let me start where most posts like this refuse to: Datadog’s AI observability is very good. It is the deepest product in the category, it has shipped at a pace nobody else matches, and the AI announcements at this year’s DASH (an autonomous tier of agents, an agent console with spend attribution, behavioral security for AI workloads) are real engineering, not slideware. If you came here for a takedown, you will be disappointed, and you should be suspicious of anyone offering one.
What this post is actually about is structure. Every observability platform has a shape: a region where it is excellent and a boundary past which it cannot see. Datadog’s shape is unusually easy to describe, and understanding it is the difference between adopting the product with clear eyes and discovering its edges during an incident. This is written for the team that already runs Datadog or is about to, and wants to know exactly what they are getting and what they still need to cover themselves.
Give the product its full due
Inside a Datadog-instrumented environment, the AI observability story is genuinely strong, and the strength compounds with how much of your stack already lives there. Conversation traces join your existing APM and infrastructure telemetry, so an agent’s behavior sits in the same view as the services it calls. The quality-review workflow, pairing automated trace routing with structured human annotation queues, is the best of its kind currently available; no competitor has matched that particular piece. Agent monitoring now spans detection through remediation, the console attributes spend across coding agents, and security signals for AI workloads correlate into the same pane. For an organization that has committed to Datadog as its platform of record, this is a coherent, powerful, fast-to-value capability, and we will tell any client exactly that.
None of what follows contradicts a word of that. It qualifies it, the way understanding any tool’s working envelope qualifies the tool.
Gap one: the value is bounded by the instrumentation
Datadog’s AI observability delivers its full value when your workloads are instrumented with Datadog’s tracer and your telemetry flows through Datadog’s ingest. That is not a criticism; it is how the product is architected, and the tight integration is precisely what makes the in-footprint experience so good. But it has a consequence worth stating plainly: the quality of your AI observability is bounded by the completeness of your Datadog instrumentation.
Enterprises are heterogeneous by nature. Acquisitions bring stacks that were never on Datadog. Legacy systems predate the standardization. Third-party SaaS that your agents call is outside your instrumentation entirely. An edge deployment runs where the agent does not reach. For each of those, Datadog’s AI observability has nothing to say, and here is the subtle part: the product does not warn you about its own blind spots. Its confidence does not drop at the boundary of what it can see. Its coverage does. A dashboard that looks complete can be complete only for the half of your AI estate that happens to live inside the circle.
Gap two: retrieval is not in the box
The turnkey AI observability story across the whole industry, Datadog included, centers on the model interaction: the prompt, the response, the tokens, the latency. That is the visible, easy-to-instrument part. It is also not where most RAG systems fail.
Retrieval (embedding the query, searching the vector store, reranking, assembling context) is where the majority of production AI quality incidents actually originate, and it is the part that demands custom instrumentation work no platform fully automates. You can absolutely instrument retrieval and send those signals to Datadog; the point is that it is engineering you do, not a checkbox the product fills. Teams that adopt Datadog’s AI observability expecting end-to-end coverage and then debug a wave of wrong answers often discover the retrieval pipeline was dark the whole time, because nothing in the default story drew their attention to it. This gap is not unique to Datadog. It is just easy to miss inside a product that does so much else for you automatically.
Gap three: the instrumentation is not yours to take
This is the structural one, and it is the most important for any decision with a multi-year horizon. When you instrument your AI workloads with Datadog’s tracer, that instrumentation is Datadog-shaped. It does its job well inside the platform. It does not travel. If your platform strategy changes in two years (a renewal that prices differently than you hoped, a consolidation decision, a regulatory requirement Datadog cannot meet), the instrumentation work does not come with you. You re-instrument, which means the switching cost is not just the new tool; it is rebuilding the telemetry layer you already paid to build once.
Compare that to instrumenting on the OpenTelemetry gen_ai conventions, which Datadog consumes natively through OTLP. You give up nothing in the Datadog experience today, and you keep the telemetry portable tomorrow. This is the single highest-leverage decision in the whole adoption, and it is invisible until the day it matters, which is exactly why it is worth making deliberately on the way in rather than regretting on the way out.
How to adopt Datadog AI observability well
The conclusion is not to avoid the product. For committed Datadog shops it is often the right production platform, and the in-footprint experience is genuinely excellent. The conclusion is to adopt it with three deliberate moves that cost you nothing and protect you from all three gaps.
- Instrument on the open standard, not the proprietary tracer. Datadog ingests OpenTelemetry gen_ai telemetry natively. Use that path and your instrumentation investment survives any future decision. This is the move that turns a one-way door into a reversible one.
- Map your boundary before you trust the dashboard. Inventory which AI workloads sit inside the Datadog-instrumented footprint and which do not, so you know what the product is covering and what you still own. The dashboard will not tell you this; you have to draw it.
- Instrument retrieval explicitly. Do not assume the turnkey story covers your RAG pipeline. Build retrieval telemetry as deliberate engineering and route it into the platform, so your most failure-prone component is not your darkest one.
The honest bottom line
Datadog AI Observability is a strong product with a clearly shaped working envelope. Adopt it knowing where the edges are, instrument on open standards so the edges are not permanent, and cover the retrieval and out-of-footprint gaps deliberately rather than discovering them in a postmortem. Do that, and you get the genuine strength of the product without the lock-in and blind spots that catch teams who adopted it assuming it did everything.
If you are evaluating Datadog’s AI observability now (and if you run Datadog, that conversation is probably already on your calendar), the most useful thing you can bring to the demo is an honest map of your own estate and the two questions that flow from it: what share of my AI workloads does this actually see, and what does my retrieval pipeline look like in it today. Our three-minute self-assessment helps you answer the first before you walk into the room; it scores your coverage across the five signals regardless of which platform you choose. Going in with the map is how you buy a powerful tool on your terms instead of its.
About the Author
Brad Bell is the Principal Observability Lead at TekStream Solutions, a Digital Resilience partner that helps organizations operate, recover, and adapt with confidence by modernizing, securing, and optimizing their digital environments. His 29 years of experience spans large-scale telecommunications infrastructure, enterprise cloud strategy, and operational transformation across industries ranging from financial services to quick-service restaurants — work that included scaling monitoring platforms to 12,000+ nodes, architecting next-generation networks, and turning reactive ops teams into reliability engineering organizations. A Grafana Certified Solutions Architect and PreSales Solution Architect, he teaches SRE immersion courses at major financial institutions and consults on observability strategy at the enterprise level, helping clients navigate the convergence of cost pressure, tool sprawl, and AI-driven operations that is reshaping how enterprises think about reliability. His perspective isn’t theoretical — it comes from having been the person on the other end of a 2 AM page, and from watching organizations make expensive, avoidable mistakes with their observability investments. At TekStream, his focus is helping enterprises build platforms that solve today’s problems and hold up against where the industry is headed.