As LLMs become more common, ensuring they're reliable, secure, and compliant is more important than ever. With many guardrail solutions popping up, how do AI engineers figure out which solutions work best for them?

In this talk, Shreya will cover the first-ever AI Guardrails Index — a thorough evaluation of top guardrail solutions. Based on systematic assessment of 6 key AI risk categories, this benchmark helps AI developers, platform teams, and decision-makers better understand the landscape of LLM safety.

Key takeaways:

• The Current State of AI Guardrails and why they matter for responsible AI use
• See how different guardrail solutions stack up in terms of precision, recall, accuracy, and speed.
• Find out which guardrails work best for specific needs, from finance to healthcare to content filtering

Whenever you read about taking Retrieval Augmented Generation beyond simple vector search on embeddings, graphs are almost sure to come up. But what graphs? Old-school knowledge graphs, with entities and their relationships, or document-centric graphs, with text snippets as nodes? And how do you use them to improve your retrieval? Nearest neighborhood? PageRank? Something else?

I will provide an overview of what's happening in that space, including what I'm doing, and give you a tour of the different options, with their pros and cons.

Large Language Models (LLMs) are transforming industries, but their success hinges not just on cutting-edge models, but on the ability to efficiently train, deploy, and manage them at scale. Training state-of-the-art models like GPT/LLAMA requires thousands of GPUs running for weeks, posing significant challenges for MLOps teams.

This session explores how LLMs are built, the scale of training required, and the growing infrastructure demands placed on MLOps teams. It will cover the key operational challenges of AI workloads, including distributed training, cost optimization, and multi-tenancy. Finally, the talk will highlight how Rafay’s GPU PaaS enables seamless AI infrastructure management, helping organizations scale AI efficiently without bottlenecks.

Attendees will gain a deeper understanding of LLM training, GPU workload management, and practical strategies to optimize AI infrastructure.

Learn how the best companies use evals to guide AI development and build a virtuous cycle of product improvement!

We’ll cover the fundamentals of how to use evaluation driven development to build reliable applications with Large Language Models (LLMs).

Building an AI application, RAG system, or agent involves many design choices. You have to choose between models, design prompts, expose tools to the model and build knowledge bases for RAG.

If you don’t have a good evaluation in place then it’s likely you’ll waste a lot of time making changes but not actually improving performance. Post-deployment, evaluation is essential to ensure that changes don’t introduce regressions to your product.

Using real world case-studies from AI teams at companies like Gusto, Filevine and Fundrise, who are building production-grade Agents and LLM applications, we’ll cover how to design your evaluators and use them as part of an iterative development process.

At the end of the session you should understand the pitfalls and best practices of how to construct evaluators in practice, as well as the process of evaluation driven AI development.

Building a GenAI chatbot for millions of users? This session reveals the secret sauce: best practices in LLM orchestration, agentic workflows, and grounded responses, all while prioritizing brand safety. Learn key architectural decisions for balancing latency and quality, and discover strategies for scaling to production.

As enterprises scale AI adoption, the infrastructure demands of GenAI workloads are pushing the limits of traditional ML platforms. This panel will explore the key differences in architecture, resource management, and operational challenges when building platforms for GenAI versus traditional ML. How do data, compute, and model lifecycle management change? What trade-offs exist between flexibility and control?

This talk is about making AI agents truly useful by fixing how we handle authorization. Right now, we depend on API keys and static tokens stored in environments variables that tie actions to single users, which isn't flexible or secure for bigger operations. I'll cover why this holds us back from letting AI do important tasks, like sending emails or managing sensitive data, autonomously. We'll explore simple ways to update these systems, so AI can work for us without constant human intervention. This is all about moving beyond flashy demos to real-world, impactful AI applications.

A well-crafted prompt is essential for obtaining accurate and relevant outputs from LLMs (Large Language Models). Prompt design enables users new to machine learning to control model output with minimal overhead.

To facilitate rapid iteration and experimentation of LLMs at Uber, there was a need for centralization to seamlessly construct prompt templates, manage them, and execute them against various underlying LLMs to take advantage of LLM support tasks.

To meet these needs, we built a prompt engineering toolkit that offers standard strategies that encourage prompt engineers to develop well-crafted prompt templates.

The centralized prompt engineering toolkit enables the creation of effective prompts with system instructions, dynamic contextualization, massive batch offline generation (LLM inference), and evaluation of prompt responses. Furthermore, there’s a need for version control, collaboration, and robust safety measures (hallucination checks, standardized evaluation framework, and a safety policy) to ensure responsible AI usage.

The advent of Large Language Models (LLMs) has significantly transformed the landscape of recommendation systems, marking a shift from traditional discriminative approaches to more generative paradigms. This transition has not only enhanced the performance of recommendation systems but also introduced a new set of challenges that need to be addressed. LLMs has several practical use-cases in modern recommendation systems, including retrieval, ranking, embedding generation for users and items in diverse spaces, harmful content detection, user history representation, and interest exploration and exploitation.

However, integrating LLMs into recommendation systems is not without its hurdles. On the algorithmic front, issues such as bias, integrity, explainability, freshness, cold start, and the integration with discriminative models pose significant challenges. Additionally, there are numerous production deployment and development challenges, including training, inference, cost management, optimal resource utilization, latency, and monitoring. Beyond these, there are unforeseen issues that often remain hidden during A/B testing but become apparent once the model is deployed in a production environment. These include impact dilution, discrepancies between pre-test and backtest results, and model dependency, all of which can affect the overall effectiveness and reliability of the recommendation system. Addressing these challenges is crucial for harnessing the full potential of LLMs in recommendation systems.

The Dark Web: an estimated $3T USD flows annually through shell corps and tax havens worldwide -- serving as the perpetua mobilia for oligarchs, funding illegal weapons transfers, mercenaries, human trafficking at scale, anti-democracy campaigns, cyber attacks at global scale, even illegal fishing fleets. Tendrils of kleptocracy extend through the heart of London, reaching into many of the VC firms in Silicon Valley, and now into the White House.

The people who "catch bad guys" – investigative journalists, regulators, gov agencies – leverage AI apps to contend with the overwhelming data volumes. Few of those who do "bad guy hunting" get to speak at tech conferences. However, our team provides core technology for this work, and we can use open source, open models, and open data to illustrate. How technology gets used to stick the moves of the world's worst organized crime, how to fight against the oligarchs who use complex networks to hide their grift.

This talk explores known cases, tradecraft employed, and open data sources for fighting against kleptocracy. Moreover, we'll look at where AI and Data professionals are very much needed, where you can get involved.

Erik will talk about how he went into a deep infrastructure rabbit hole to fix the developer experience working with the cloud and GPUs. This involves building a custom file system, their own scheduler, and much more.