LLMs in Production Conference Part III

This tutorial starts by surveying the different ways we can use LLMs. Then, we will take a deeper dive into various LLM finetuning strategies, such as low-rank adaptation, and learn how we can create custom LLMs using open-source software.

The future of Generative AI will be shaped by many factors like scaling laws, the evolution of agents, multi-modality, open-source contributions, etc. However, challenges such as GPU and talent shortages and regulations could pose obstacles. Join us as we delve into the fascinating world of Generative AI and explore the key drivers that will shape its development in the next three years.

Martian is focused on building a model router to dynamically route every prompt to the best LLM for the highest performance and lowest cost. Corti, the Al Co-Pilot for health care uses Al to improve patient care, demonstrating the potential of Al in healthcare and medical decision-making. They recently raised $60M, with Prosus being one of the lead investors. Transforms is pioneering in synthetic entertainment, showing how Al can transform the way we create and consume media.

This talk dives into using Large Language Models (LLMs) for data annotation automation. We'll cover techniques and workflows to run automation with LLMs, and discuss how to interpret, validate, and use these results in subsequent machine learning pipelines, such as model finetuning. Join to understand the practical side of LLM-driven data annotation.

Explore the transformative role of AI in EdTech, discussing its potential to enhance learning experiences and personalize education. The panelists share insights on AI use cases, challenges in AI integration, and strategies for building a differentiated business model in the evolving AI landscape. The discussion looks ahead at how the latest wave of GenAI is set to shape the future of education. Join us to understand the exciting prospects and challenges of AI in EdTech.

Making LLMs reliable is hard. You can't debug or unit test them, not in the traditional sense at least. Instead, you'll need to turn to the practice of Observability, by instrumenting your feature to produce rich telemetry and analyzing behavior from that data. Observability can also act as a key source of data for evaluations.

Unlock the power of real-world LLM use cases and learn how to keep them grounded and deliver accurate results through techniques such as Retrieval Augmented Generation (RAG). Leverage Databases with vector support as a bridge between LLMs and your enterprise Gen AI apps.

So you’ve built your first LLM product. Now what? If you’re a Product person, you need to understand how people are using it, and how it's performing. That’s where product analytics come in. But it's a totally different problem to product analytics for graphical user interfaces - you need to understand mountains of text. This talk will cover the key considerations for building great end-user experiences with LLMs, from a Product Managers perspective.

Getting the right LLM inference stack means choosing the right model for your task, and running it on the right hardware, with proper inference code. This talk will go through popular inference stacks and set-ups, detailing what makes inference costly. We'll talk about the current generation of open-source models and how to make the best use of them, but we will also touch on features currently missing from the open-source serving stack as well as what the future generations of models will unlock.

Prosus AI, a top-tier applied AI centre, drives rapid experimentation and implementation of AI throughout Prosus's global portfolio, which includes over 80 technology companies with more than 800 AI experts. In this talk, we show how AI is harnessed for discovery within the Prosus network. We will share insights gained from 10,000 colleagues who utilise generative AI daily across the group, significantly enhancing the impact of their work.

AutoGPT sparked the imaginations of millions, it’s exciting because you can see what they will be able to do just by talking to them like you would with a human. It blew up because of this, not because of actual use case. First, what is an interact through natural language, performs actions in the real world. RPA. What agents could do in the future. But they suck right now. Why no actual use case yet? Reliability Memory, llm model complexity, architecture, tokens per second But ultimately we need a loss function to do tdd and improve agents. Getting thousands of prs and no way to test them. Don't know how to step if you don't know where you're going. How to get there, related to building language models Performance (benchmark) Safety (monitor) Standardization (agent protocol) Research pedigree is no longer the barrier to making an impact in the space, creativity is. And now there's a clear way to make it happen, mention and some of the work there AutoGPT.

Discover how GetYourGuide navigates the dynamic landscape of LLMs and delivers products valuable to consumers and business. Will cover decision-making process of when to opt for LLMs over supervised models, offering practical insights into implementation and how these are put into production at GetYourGuide. On top of this will go deeper into strategic LLM prioritisation, streamlining their integration into product processes, and ensuring safe deployment to consumers.

How can companies best build useful and differentiated applications on top of language models? Many of the products and companies built do this by providing the relevant context to LLMs and asking them to reason appropriately. In this talk, Harrison will discuss the different types of context you should be aware of, the different levels of cognitive architectures that are emerging, and how LangChain and LangSmith are built to help with this journey.

A video-game themed walkthrough of LLM products in today's markets.

Deploying LLMs to products is no easy feat. It's common to have dozens of model variants when trying things out. As usage scales up, cost-to-serve and latency become primary concerns. In this talk, we will dive into how Fireworks.aI GenAI Platform helps developers on the journey from early experimentation to highly loaded production deployments without breaking the bank.

Data quality is the foundation of successful Generative A, traditional ML, and data-driven initiatives. In this talk, I will share our research results on this as there is a tangible impact of poor data quality on model performance and training cost.

In this talk, Pieter covers some key things to keep in mind when building ML applications using LLMs regarding privacy.

What Effy learned from building tools for self-driving cars, and how might we apply those learnings to building LLM applications?

The rapid advancements in Natural Language Processing (NLP) have paved the way for the deployment of Large Language Models (LLMs) in real-world production systems. This talk aims to provide a succinct overview of the current state of Large Language Models (LLMs) in production, emphasizing their capabilities, deployment strategies, and the challenges encountered.

Have you ever had someone suggest "ohhh we could use LLMs for that?" And you knew the idea had some painpoints...but you couldn't put your finger on the sore point? Hopefully my little checklist can help you out. In this talk I will introduce a simple tool for assessing the value and feasibility of an LLM use case. It will make it easier to discuss these. It's not rocket science, but it does help when discussing use cases with stakeholders of different experience levels.

We will start with a brief overview of synthetic data generation. Next, we will inspect its effectiveness by referring to some of the latest works in this space. Then we will move on to see how this can be leveraged in evaluating LLM-driven applications pipelines.

Discover the highs and lows of building a Retrieval Augmented Generation (RAG) system as we walk through the crucial challenges: data quality, query engines, and contextualization. Gain key insights into the pitfalls and best practices that can help you make informed decisions in your own projects.

Join us for an introduction to NSQL, a new family of open-source foundation models with up to 7B parameters automating SQL generation tasks. We will explore the limitations of existing open and closed-source foundation models for enterprise use, including issues of customization, quality, and privacy. We will highlight how NSQL addresses these challenges with its open-source nature, specialized training for SQL tasks, and a range of model sizes to accommodate diverse hardware configurations. Included in the talk will be NSQL's data generation process and GPU training approach, underlining its advantages over other foundation models for SQL generation. We will demonstrate how the NSQL models outperform existing open source models for SQL generation and, by starting from the newest LLama2 commercially available model, we even beat closed source models.

In this talk, we will cover how to develop and deploy RAG-based LLM applications for production. We will cover how the major workloads (data loading and preprocessing, embedding, serving) can be scaled on a cluster, how different configurations can be evaluated and how the application can be deployed. We will also give an introduction to Anyscale Endpoints which offers a cost-effective solution for serving popular open-source models.

Time series—data ordered chronologically—constitutes the underlying fabric of systems, enterprises, and institutions. Its impact spans from measuring ocean tides to tracking the daily closing value of the Dow Jones. This type of data representation is indispensable in sectors such as finance, healthcare, meteorology, and social sciences. However, the current theoretical and practical understanding of time series hasn't yet achieved a consensus among practitioners that mirrors the widespread acclaim for generative models in other fundamental domains of the human condition, like language and perception. Our field is still divided and highly specialized. Efforts in forecasting science have fallen short of fulfilling the promises of genuinely universal pre-trained models. In this talk, we will introduce TimeGPT, the first pre-trained foundation model for time series forecasting that can produce accurate predictions across various domains and applications without additional training. A general pre-trained model constitutes a groundbreaking innovation that opens the path to a new paradigm for the forecasting practice that is more accessible and accurate, less time-consuming, and drastically reduces computational complexity. We will show how to use TimeGPT in a live demo.

An opinionated view of how to build production LLMs.

The rapid adoption of Large Language Models (LLMs) has swept across various industries, inspiring companies to incorporate them into their products, regardless of the industry's domain. Unlike organizations like OpenAI, Google, or Microsoft, most entities view LLMs as powerful tools rather than standalone products. In this context, it remains paramount to uphold core product principles where customer satisfaction reigns supreme. In this talk, we will see how to pick practical use cases for using LLMs, where they are not essentially user-facing chatbots but valuable tools to build new features into existing products - with examples from the cybersecurity domain. We will also see a sample product strategy for how to plan and roadmap LLM features, along with key performance metrics to gauge success.

A product-minded engineering perspective on UX/design patterns, product evaluation, and building with AI.

Conversational AI demands low latency for a seamless dialogue between humans and AI. However, engineers face the dilemma that some latency is inherently required in order to process human speech and craft a response. Some incremental wins to shave off milliseconds involve trade-offs against how the AI response could be enriched during the additional processing time. Others simply refactor out inefficiency to obtain more performant results from AI devtools. This talk presents best practices of designing streaming speech-to-text applications, as well as reasons to accept extra latency for the sake of an enhanced product experience.

AI Squared is an AI platform designed for product owners, data scientists, and enterprise leaders. We empower you to accelerate both predictive and generative AI projects, measure their benefits, and drive significant revenue growth and cost reduction. The largest gap in the LLM developer stack is creating different experiences for users to leverage LLM results. Many companies have only utilized generative AI inside of chat applications. AI Squared has developed a framework that empowers our customers to harvest context and connect to additional content as well as other AI models from across the organization while integrating these insights directly into currently existing tools and applications.