Expedia Deploys Deterministic AI for Observability and Incident Response

Expedia Group has introduced an internal AI-assisted observability platform, the Service Telemetry Analyzer, into its production environment to streamline incident investigation and reduce recovery times. As reported in the company’s engineering blog, the system utilizes large language models within a strictly defined, deterministic framework to provide engineers with structured root cause assessments based on standardized infrastructure telemetry.

The platform operates as a FastAPI application that bridges service metrics from Datadog with an internal generative AI gateway. This architecture ensures that all diagnostic processes remain within the company’s security perimeter while managing the authentication and access requirements for LLM providers. The system avoids the unpredictability of autonomous agents by relying on predefined diagnostic workflows that execute specific, repeatable analytical tasks.

Technical implementation relies on an asynchronous model, utilizing Redis as both a message broker and result backend to handle the high volume of I/O-bound operations. This design choice allows the platform to manage concurrent analysis requests while respecting the rate limits imposed by external monitoring tools and internal AI gateways. The system architecture supports prompt chaining, which enables the platform to perform specialized, sequential analyses before consolidating findings into a final report for human review.

Prompt chaining logic within the analyzer allows the system to decompose complex incidents into smaller, manageable diagnostic segments. By executing specialized prompts for different telemetry subsets, the platform avoids the performance bottlenecks associated with single-pass analysis. This modular approach ensures that the LLM receives context-specific data, which significantly improves the accuracy of the generated root cause assessments compared to generalized queries.

Integration with Datadog requires the platform to handle diverse data streams, including request throughput, latency, and error rates across HTTP, gRPC, and GraphQL protocols. The system must normalize these disparate metrics into a coherent format that the generative models can interpret effectively. This normalization layer is critical for maintaining consistency across services developed in diverse programming languages and frameworks, ensuring that the AI provides reliable insights regardless of the underlying technology stack.

Infrastructure metrics form the core data set for the analyzer, covering critical system health indicators including CPU utilization, memory pressure, container restart events, and Kubernetes readiness probes. By focusing on these standardized telemetry points, the tool maintains consistency across services developed in diverse programming languages and frameworks. This standardization is essential for the platform to function as a reliable diagnostic aid in a complex, distributed microservices environment.

The engineering team at Expedia designed the platform to function as an assistive tool rather than an autonomous decision-maker. Engineers retain full responsibility for reviewing the generated findings and validating the recommended next steps before taking any corrective action. This human-in-the-loop approach mitigates the risks associated with automated remediation in complex, distributed environments.

Performance tracking and prompt evaluation are managed through Langfuse, providing the necessary visibility into how the system interprets telemetry data. The platform has already seen deployment across production incident investigations, post-incident analysis, and specific troubleshooting scenarios like JVM memory diagnostics. This practical application demonstrates the utility of applying LLMs to structured observability data within a controlled, enterprise-grade environment.

The shift toward deterministic AI workflows reflects a broader trend in enterprise architecture where reliability and predictability are prioritized over the potential flexibility of autonomous systems. By constraining the AI to specific diagnostic paths, Expedia limits the surface area for hallucinations or incorrect operational decisions. This approach allows the organization to leverage the analytical speed of generative models while maintaining the rigorous governance required for high-availability infrastructure.

Future development phases for the platform include the integration of service dependency mapping and additional operational metadata to improve diagnostic depth. The team is also exploring the use of the Model Context Protocol to facilitate broader tool integrations and conversational interfaces for engineers. Evaluating the platform within the context of chaos engineering experiments remains a key priority for the next phase of the project.

Leaders should monitor how the integration of AI-assisted observability impacts the long-term maintenance burden of internal diagnostic tools. The reliance on prompt chaining and predefined workflows suggests that the effectiveness of such systems is tied directly to the quality of the underlying domain-specific prompts. Organizations considering similar implementations must weigh the benefits of reduced time-to-recover against the ongoing effort required to maintain and evaluate these diagnostic models.