What is the state of the art on RAG quality evaluation? How much attention should you pay to embedding model benchmarks? How to establish and evaluate objectives for your information retrieval system before and after you launch? The journey to Quality AI starts with measurement. The second, third and 100th step are then an iteration and improvement against that measurement - what can we learn from the search & relevance industry that has been around for decades? What challenges are specific to embedding powered retrieval? And how to actually improve your vector search? Let's talk about it!

By integrating LLM-based entity extraction into unstructured data workflows, knowledge graphs can be created which illustrate the named entities and relationships in your content. Kirk will discuss how knowledge graphs can be leveraged in RAG pipelines to provide greater context for the LLM response, as well as utilizing entity extraction for better content filtering. Using GraphRAG, developers can build enriched user experiences - pulling data from a wide variety of sources, not just what is accessible with standard RAG vector search retrieval.

The rapid advancement of AI and generative AI is transforming industries and empowering new applications. However, the widespread adoption of these technologies necessitates robust protection of the sensitive data that fuels them. Retrieval Augmented Generation (RAG) is a popular technique to enhance Large Language Models LLMs by grounding their responses in factual data, thereby improving the quality, accuracy, and reliability of AI-generated outputs This lightning talk will explore practical techniques to safeguard sensitive data and ensure trustworthy AI-driven applications. We will demonstrate how to filter out sensitive or toxic responses from LLMs using Sensitive Data Protection (SDP) and NLP. We will also showcase how to leverage Google-standard authentication with Identity-Aware Proxy to control access so users can seamlessly connect to your LLM front end and Jupyter notebooks.

Open Generative AI (GenAI) models are transforming the AI landscape. But which one is right for your project? What are the quality metrics for one to evaluate his/her own trained model? For application developers and AI practitioners enhancing their applications with GenAI, it’s critical to choose and evaluate the model that meets both quality and performance requirements. This talk will examine customer scenarios and discuss the model selection process. We will explore the current landscape of open models and collection mechanisms to measure model quality. We will share insights from Google’s experience. Join us to learn about model metrics and how to measure them.

Nik joins the podcast to discuss an interesting trend: the anti-AI hype. Dive into what many companies might be missing when non-technical management rushes to roll out machine learning initiatives without bringing on the technical experts who can set them up for success. It's great to converse about bridging the gap between management and tech to make AI work effectively!

Neha Sharma, leader of the notebooks and code authoring teams at Databricks, discusses their journey with the Databricks Assistant powered by large language models (LLMs), showcasing its ability to interpret queries, generate code, and debug errors. She explains how the integration of code, user data, network signals, and the development environment enables the assistant to provide context-aware responses. Neha outlines the three evaluation methods used: helpfulness benchmarks, side-by-side evaluations, and tracking user interactions. She highlights ongoing improvements through model tuning and prompt adjustments and discusses future plans to fine-tune models with Databricks-specific knowledge for personalized user experiences.

This is my story of building KinConnect, a tool designed to help hackathon participants form effective teams using AI-driven participant profiles and matching algorithms. The project was developed during the MongoDB GenAI Hackathon using tools like Google Forms, Pipedream, FireworksAI, Modal Labs, and MongoDB Hybrid Search. Some lessons learnt are the importance of experimentation, prompt engineering, and leveraging synthetic data.

Eric Landry discusses the integration of AI in healthcare, highlighting use cases like patient engagement through chatbots and managing medical data. He addresses benchmarking and limiting hallucinations in LLMs, emphasizing privacy concerns and data localization. Landry maintains a hands-on approach to developing AI solutions and navigating the complexities of healthcare innovation. Despite necessary constraints, he underscores the potential for AI to proactively engage patients and improve health outcomes.