Being an LLM-native is becoming one of the key differentiators among companies, in vastly different verticals. Everyone wants to use LLMs, and everyone wants to be on top of the current tech but - what does it really mean to be LLM-native? LLM-native involves two ends of a spectrum. On the one hand, we have the product or service that the company offers, which surely offers many automation opportunities. LLMs can be applied strategically to scale at a lower cost and offer a better experience for users. But being LLM-native not only involves the company's customers, it also involves each stakeholder involved in the company's operations. How can employees integrate LLMs into their daily workflows? How can we as developers leverage the advancements in the field not only as builders but as adopters? We will tackle these and other key questions for anyone looking to capitalize on the LLM wave, prioritizing real results over the hype.

Bending the Rules: How to Use Information Extraction Models to Improve the Performance of Large Language Models Generative AI and Large Language Models (LLMs) are revolutionizing technology and redefining what's possible with AI. Harnessing the power of these transformative technologies requires careful curation of data to perform in both cost-effective and accurate ways. Information extraction models including linguistic rules and other traditional text analytics approaches can be used to curate data and aid in training, fine-tuning, and prompt-tuning, as well as evaluating the results generated by LLMs. By combining linguistic rule-based models with LLMs through this multi-modal approach to AI, we can help to improve the quality and accuracy of LLMs and enable them to perform better on various tasks while cutting costs. We will demonstrate this innovation with a real-world example in public comment analysis. Scaling Large Language Models in Production Open source models have made running your own LLM accessible many people. It's pretty straightforward to set up a model like Mistral, with a vector database, and build your own RAG application. But making it scale to high traffic demands is another story. LLM inference itself is slow, and GPUs are expensive, so we can't simply throw hardware at the problem. Once you add things like guardrails to your application, latencies compound. BAML: Beating OpenAI's Structured Outputs We created a new programming language that allows us to help developers using LLMs get higher quality results out of any model. For example, in many scenarios, we can match GPT-4o performance with GPT-4o-mini using BAML. We'll discuss some of the algorithms that BAML uses, how they improve the accuracy of models, and why function calling is good and bad.

Simba dives into how feature stores have evolved and how they now intersect with vector stores, especially in the world of machine learning and LLMs. He breaks down what embeddings are, how they power recommender systems, and why personalization is key to improving LLM prompts. Simba also sheds light on the difference between feature and vector stores, explaining how each plays its part in making ML workflows smoother. Plus, we get into the latest challenges and cool innovations happening in MLOps.

In our September 12th MLOps Community Reading Group session, live-streamed at the Data Engineering for AI/ML Virtual Conference, we covered the paper "Hybrid RAG: Integrating Knowledge Graphs and Vector Retrieval for Information Extraction." The panel discussed using hybrid RAGs to pull information from unstructured financial documents. The paper's approach combines knowledge graphs with vector-based retrieval for better results. We critiqued the context-mixing methods and lack of graph pruning techniques. Overall, it was a solid session with great insights on improving financial data extraction using hybrid models.

Delivery Hero innovates locally within each department to develop the most effective MLOps practices in that particular context. We also discuss our efforts to reduce redundancy and inefficiency across the company. Hear about our experiences creating multiple micro feature stores within our departments, and our goal to unify these into a Global Feature Store that is more powerful when combined.

Data can't be split into microservices! But teams should own their data! But there should be one definition for metrics! But teams can bring their own architectures! Data platform teams have a tough job: they need to find the right balance between creating reliable data services and decentralizing ownership -- and rarely do off-the-shelf architectures end up working as expected. In this talk, I'll discuss Whatnot's journey to providing a suite of data services -- including machine learning, business intelligence, and real-time analytics tools -- that power product features and business operations. Attendees will walk away with a practical framework for thinking about the maturation of these services, as well as patterns we've seen that make a big difference in increasing our adoption while reducing our maintenance load as we've grown.

Dive into what makes Retrieval-Augmented Generation (RAG) systems tick—and it all starts with the data. We’ll be talking with an expert in the field who knows exactly how to transform messy, unstructured enterprise data into high-quality fuel for RAG systems. Expect to learn the essentials of data prep, uncover the common challenges that can derail even the best-laid plans, and discover some insider tips on how to boost your RAG system’s performance. We’ll also touch on the critical aspects of data privacy and governance, ensuring your data stays secure while maximizing its utility. If you’re aiming to get the most out of your RAG systems or just curious about the behind-the-scenes work that makes them effective, this episode is packed with insights that can help you level up your game.

Utilizing LLMs in high impact scenarios (e.g., healthcare) remains difficult due to the necessity of including private/ sensitive information in prompts. In many scenarios, AI/prompt engineers might want to include few shots examples in prompts to improve LLM performance, but the relevant examples are sensitive and need to be kept private. Any leakage of PII or PHI into LLM outputs could result in compliance problems and liability. Differential Privacy (DP) can help mitigate these issues. The Machine Learning (ML) community has recognized the importance of DP in statistical inference, but its application to generative models, like LLMs, remains limited. This talk will introduce a practical pipeline for incorporating synthetic data into prompts, offering robust privacy guarantees. This approach is also computationally efficient when compared to other approaches like privacy-focused fine-tuning or end-to-end encryption. I will demonstrate the pipeline, and I will also examine the impact of differentially private prompts on the accuracy of LLM responses.