Ray Summit 2024

In-Person Agenda

Back to Session CatalogBack to Session Catalog
Beyond Black Boxes: Evaluating AI Agents
This talk addresses key challenges in evaluating LLM-powered AI agents: behavioral instability, comprehensive testing, and cascading failures. We'll explore advanced techniques including multi-dimensional metrics and automated scenario generation. Gain insights gleaned from hundreds of AI engineering teams in production into implementing agent-specific observability systems and designing robust evaluation pipelines that can handle the complexities of modern AI agents, from detecting subtle regressions to quantifying performance across diverse, dynamically generated test cases.

Speakers

  • Tuesday, Oct 11:00 PM - 1:30 PM PDT
    Yerba Buena Salon 14

READY TO REGISTER?

Come connect with the global community of thinkers and disruptors who are building and deploying the next generation of AI and ML applications.

Join the Conversation

Hashtag it 

#RaySummit

Don't wait for the conference to get the convo going. Join the Ray community now – ask a question in the forums, open a pull request or simply share why you’re excited. Create some buzz!