LLMs in Production Conference Part II

Large language models are fluent text generators, but they often make errors, which makes them difficult to deploy in high-stakes applications. Using them in more complicated pipelines, such as retrieval pipelines or agents, exacerbates the problem. In this talk, Matei will cover emerging techniques in the field of “LLMOps” — how to build, tune and maintain LLM-based applications with high quality. The simplest tools are ones to test and visualize LLM results, some of which are now being incorporated into MLOps frameworks like MLflow. However, there are also rich techniques emerging to “program” LLM pipelines and control LLMs’ outputs to achieve desired goals. Matei discusses Demonstrate-Search-Predict (DSP) from my group as an example programming framework that can automatically improve an LLM-based application based on feedback, and other open-source tools for controlling outputs and generating better training and evaluation data for LLMs. This talk is based on their experience deploying LLMs in many applications at Databricks, including the QA bot on our public website, internal QA bots, code assistants, and others, all of which are making their way into our MLOps products and MLflow.

What do we need to be aware of when building for production? In this talk, we explore the key challenges that arise when taking an LLM to production

In the last few years, DeepSpeed has released numerous technologies for training and inference of large models, transforming the large model training landscape from a system perspective. Technologies like ZeRO, and 3D-Parallelism have become the building blocks for training large models at scale, powering LLMs like Bloom-176B, Megatron-Turing 530B, and many others. Heterogenous memory training systems like ZeRO-Offload and ZeRO-Infinity have democratized LLMs by making them accessible with limited resources. DeepSpeed-Inference and DeepSpeed-MII have made it easy to apply powerful inference optimizations to accelerate LLMs for deployment. As a result, DeepSpeed has been integrated directly into platforms like HuggingFace, PyTorch Lightning, and Mosiac ML. Similarly, the ZeRO family of technologies and 3D-Parallelism are offered as part of PyTorch, Colossal-AI, Megatron-LM, etc. In this talk, Samyam shares the journey of DeepSpeed as they navigated through the large model training landscape and built systems to extend it beyond what was possible. Samyam shares their motivations, insights, aha moments, and stories behind the technologies that are now part of DeepSpeed and have become the fundamental building blocks for training and inferencing large language models at scale.

Evaluating LLM-based applications can feel like more of an art than a science. In this workshop, we'll give a hands-on introduction to evaluating language models. You'll come away with knowledge and tools you can use to evaluate your own applications, and answers to questions like: Where do I get evaluation data from, anyway? Is it possible to evaluate generative models in an automated way? What metrics can I use? What's the role of human evaluation?

Copilots embedded within SaaS applications have become one of the dominant ways of leveraging LLMs within products. In this lightning talk, Tristan reviews some of the dominant UI paradigms and features, general design patterns and system architectures, and top challenges and future frontiers of production copilot systems.

Generalized models solve general problems. The real value comes from training a large language model (LLM) on your own data and finetuning it to deliver on your specific ML task. Now you can build your own custom LLM, trained on your data and fine-tuned for your generative or predictive task in ten lines of code with Predibase and Ludwig, the low-code deep learning framework developed and open-sourced by Uber, now maintained as part of the Linux Foundation. Using Ludwig’s declarative approach to model customization, you can take a pre-trained large language model like LLaMA and tune it to output data specific to your organization, with outputs conforming to an exact schema. This makes building LLMs fast, easy, and economical. In this session, Travis Addair, CTO of Predibase and co-maintainer of open-source Ludwig, shares how LLMs can be tailored to solve specific tasks from classification to content generation, and how you can get started building a custom LLM in just a few lines of code.

Retrieval augmented generation with embeddings and LLMs has become an important workflow for AI applications. While embedding-based retrieval is very powerful for applications like 'chat with my documents', users and developers should be aware of key limitations, and techniques to mitigate them.

Humanloop has now seen hundreds of companies go on the journey from playground to production. In this talk, we’ll share case studies of what has and hasn’t worked. Raza shares what the common pitfalls are, emerging best practices, and suggestions for how to plan in such a quickly evolving space.

Here’s the truth: troubleshooting models based on unstructured data is notoriously difficult. The measures typically used for drift in tabular data do not extend to unstructured data. The general challenge with measuring unstructured data drift is that you need to understand the change in relationships inside the unstructured data itself. In short, you need to understand the data in a deeper way before you can understand drift and performance degradation. In this presentation, Claire Longo presents findings from research on ways to measure vector/embedding drift for image and language models. With lessons learned from testing different approaches (including Euclidean and Cosine distance) across billions of streams and use cases, she will dive into how to detect whether two unstructured language datasets are different — and, if so, how to understand that difference using techniques such as UMAP.

This workshop focuses on the crucial task of constructing and managing datasets specifically designed for reinforcement learning from human feedback (RLHF) and large language model (LLM) fine-tuning. Let's explore the utilization of Argilla, an open-source data platform that facilitates the integration of human and machine feedback. Participants will learn effective strategies for dataset construction, including techniques for data curation and annotation. The workshop aims to equip attendees with the necessary knowledge and skills to enhance the performance and adaptability of RLHF and LLM models through the use of Argilla's powerful data management capabilities.

Most of us are using LLMs and some of us are getting to the point where LLMs are going to production. Honeymoon phase is going to get over soon and practical realities like cost & maintainability are going to become mainstream. However, the cost of running LLMs is not well understood or often not put in perspective. In this talk, we will dive deep into what type of costs are involved in building LLM-based apps. How do these compare when you run RAG vs Fine-tuning, what happens when you use Open Source vs Commercial LLMs? Spoiler- If you wanted to summarize the entire Wikipedia to half its size using GPT-4 8k context window, it would cost a whopping $360K! If you used a 32k context window your cost would be $720k!

This talk is all about quantifying hallucinations -- the most critical evaluation metric for LLMs. In this talk, we'll dive deeper into what Hallucinations mean in modern-day LLM workflows, and how they affect model outcomes (and downstream consumers of LLMs). We will also discuss novel and efficient metrics and methods to eagerly detect hallucinations early on in the process, so we help disinformation and poor/biased outcomes from large language models, thereby increasing trust in your LLM systems. I'm currently working on the slides/content and should upload it in the next day or two. Let me know if you have any questions or need something else from my side!

The main problems faced by LLMs such as hallucinations, lack of domain knowledge, or outdated info are all data problems. How do we fix these data problems? Add a layer on top of the LLM with the ability to search the data we need to use.

How to use a Large Language Model (LLM) to create memes? Let's discuss the unique dataset of ImgFlip, the selection, and fine-tuning of a commercially usable LLM, and associated challenges. Of course, we’ll also demonstrate the model prototype itself. We will also discuss the challenges we anticipate facing with the productionization of an LLM that is used by millions of users.

While we've seen great progress on Open Source LLMs, we haven't seen the same level of progress on systems to serve those LLMs in production contexts. In this presentation, I work through some of the challenges of taking open-source models and serving them in production.

In the last LLM in Production event, Sam spoke on some of the ways they've seen people use a vector database for large language models. This included use cases like information/context retrieval, conversational memory for chatbots, and semantic caching. These are great and make for flashy demos, however, using this in production isn't trivial. Oftentimes, the less flashy side of these use cases can present huge challenges such as: Advice on prompts? How do I chunk up text? What if I need HIPAA compliance? On-premise? What if I change my embeddings model? What index type? How do I do A/B tests? Which cloud platform or model API should I use? Deployment strategies? How can I inject features from my feature platform? Langchain or LlamaIndex or RelevanceAI??? This talk details a distillation of a year+ worth of deploying Redis for these use cases for customers and distill it down into 20 minutes.

We are in the midst of a true technological revolution as every CEO and business is beginning to heavily invest in AI to remain competitive. Over the last year, businesses and even private individuals dreamt up seemingly limitless possibilities of how to apply LLM in their daily lives. By putting these technologies into the hands of business users, they learned how their tasks can be made easier, and their output and products better — resulting in more productivity, automation, and intelligence. This is the true promise of democratization of AI, which has been central to Salesforce’s journey. Whether it is in Sales, Service, Marketing, or Developers, Sarah shares how they have unlocked the power of LLMs for our teams, and therefore their customers, to drive the next chapter in AI — changing the way they work.

Language models are very complex thus introducing several challenges in interpretability. The large amounts of data required to train these black-box language models make it even harder to understand why a language model generates a particular output. In the past, transformer models were typically evaluated using perplexity, BLEU score, or human evaluation. However, LLMs amplify the problem even further due to their generative nature thus making them further susceptible to hallucinations and factual inaccuracies. Thus, evaluation becomes an important concern.

The journey from LLM PoCs to production deployment is fraught with unique challenges, from maintaining model reliability to effectively managing costs. In this talk, we delve deep into these complexities, outlining design patterns for successful LLM production, the role of vector databases, strategies to enhance reliability, and cost-effective methodologies.

Building a chatbot is not easier. We need: An embedding model that translates questions to a matrix. A Vector database to search. LLm to generate the answers. We can orchestrate the job using Langhcain with minimum development.

Using Wardley Maps we can understand value chains and map out the landscape. Using this to develop strategies and understand where to target our efforts.

As Foundation Models (FMs) continue to grow in size, innovations continue to push the boundaries of what these models can do on language and image tasks. This talk describes our work on applying foundation models to structured data tasks like data linkage, cleaning, and querying. We discuss the challenges and solutions that these models present for production deployment in the modern data stack.

Autonomous AI agents have gotten a lot of attention recently, but they're mostly just toys. What are the primitives that we need to build more reliable agents, and what are the main business use cases that agentic automation will enable over the next few years?

Proprietary LLMs are difficult for enterprises to adopt because of security and data privacy concerns. Open-source LLMs can circumvent many of these problems. While open LLMs are incredibly exciting, they're also a nightmare to deploy and operate in the cloud. Aqueduct enables you to run open LLMs in a few lines of vanilla Python on any cloud infrastructure that you use.

There are key areas we must be aware of when working with LLMs. High costs and low latency requirements are just the tip of the iceberg. In this panel, we hear about common pitfalls and challenges we must keep in mind when building on top of LLMs.

It’s silly to think of training and using large LANGUAGE models without any sort of input from the study of language itself. Linguistics is not the only field of knowledge that improve LLMs, as they are the intersection of several fields, however, they can help us not only improve current model performance but also clearly see where future improvements will come.

This session provides an overview of the evolving landscape of Generative AI, with a focus on the latest trends and technologies that shape this field. Designed with startups in mind, the talk offers practical insights on how to adapt and leverage these advancements to enhance their products. Attendees will acquire valuable knowledge to navigate the dynamic landscape of Generative AI, enabling them to stay up-to-date and harness untapped potential for the success of their startups.

The rapid adoption of large language models (LLMs) is transforming how businesses communicate, learn, and work, prioritizing AI safety and security. This captivating and insightful talk will delve into the challenges and risks associated with LLM adoption and unveil AIShield.GuArdIan – a game-changing technology that enables businesses to leverage ChatGPT-like AI without compromising compliance. AIShield.GuArdIan's unique approach ensures legal, policy, ethical, role-based, and usage-based compliance, allowing companies to harness the power of LLMs safely. Join us on this riveting journey as we reshape the future of AI, empowering industries to unlock the full potential of LLMs securely and responsibly. Don't miss this opportunity to be at the forefront of responsible AI usage – reserve your seat today and take the first step towards a secure AI-powered future!

Large Language Models require a new set of tools... or do they? K8s is a beast and we like it that way. How can we best leverage all the battle-hardened tech that k8s has to offer to make sure that our LLMs go brrrrrrr. Let's talk about it in this chat.

Large Language Models are an especially exciting opportunity for Operations: they excel at answering questions, completing sentences, and summarizing text while requiring ~100x less training data than the previous generation of models. In this talk, Sophie discusses lessons learned productionising Stripe’s first application of Large Language Modelling - providing answers to user questions for Stripe Support.