MLOps Reading Group

Retrieval-Augmented Generation and agentic AI are increasingly common in enterprise deployments, but real enterprise environments introduce challenges largely absent from academic treatments and consumer-facing APIs: multiple tenants with heterogeneous data, strict access-control requirements, regulatory compliance, and cost pressures that demand shared infrastructure. This paper identifies a fundamental problem underlying existing RAG architectures in these settings. Retrieval systems rank documents by relevance, not by authorization, so a query from one tenant can surface another tenant’s confidential data simply because it scores highest. The authors formalize this relevance-authorization gap alongside related shortcomings (tool-mediated disclosure, context accumulation across turns, client-side orchestration bypass) and introduce a layered isolation architecture combining policy-aware ingestion, retrieval-time gating, and shared inference, enforced through server-side orchestration. They validate it through an open-source implementation in OGX, a vendor-neutral OpenAI-compatible Responses API, showing empirically that ABAC gating eliminates cross-tenant leakage while introducing negligible overhead.

We present LingBot-World, an open-sourced world simulator stemming from video generation. Positioned as a top-tier world model, LingBot-World offers the following features. (1) It maintains high fidelity and robust dynamics in a broad spectrum of environments, including realism, scientific contexts, cartoon styles, and beyond. (2) It enables a minute-level horizon while preserving contextual consistency over time, which is also known as "long-term memory". (3) It supports real-time interactivity, achieving a latency of under 1 second when producing 16 frames per second. We provide public access to the code and model in an effort to narrow the divide between open-source and closed-source technologies. We believe our release will empower the community with practical applications across areas like content creation, gaming, and robot learning.

AI agents aren’t “helping” devs code anymore—they’re starting to run the workflow. This panel pokes at the uncomfortable question: are engineers still in control, or just supervising very confident machines that are slowly replacing how we think, design, and build software?

Language models are getting better at reasoning but their ability to verify their own outputs still lags behind. This paper tackles that challenge head-on by introducing Weaver, a framework that combines multiple weak verifiers into a single, stronger verifier without relying heavily on labeled data. ​Weaver uses weak supervision to estimate verifier reliability, normalize inconsistent outputs, and filter low-quality signals, resulting in a unified score that better reflects true response quality. In practice, this approach significantly boosts reasoning and math task performance rivaling models several times larger, such as achieving o3-mini-level accuracy using only Llama 3.3 70B as the generator.

As AI agents become more capable, their real-world performance increasingly depends on how well they can coordinate tools. This month's paper introduces a benchmark designed to rigorously test how AI agents handle multi-step tasks using the Model Context Protocol (MCP) — the emerging standard for tool integration. ​The authors present 101 carefully curated real-world queries, refined through iterative LLM rewriting and human review, that challenge models to coordinate multiple tools such as web search, file operations, mathematical reasoning, and data analysis.

LLM performance isn’t just about the model—it’s about the scaffolding we build around it. “Context Engineering” reframes the conversation: prompt design is the toy problem, while the real frontier is systematically engineering the information environments that shape model behavior. Surveying 1,400+ papers, this work defines the field’s taxonomy—retrieval, generation, processing, management—and shows how it powers RAG, memory, tool use, and multi-agent systems. The survey also reveals a paradox: LLMs can absorb increasingly complex contexts but remain clumsy at producing equally complex outputs. This tension signals a coming split between research obsessed with cramming more into context windows and the harder question of whether models can ever match the sophistication of what they’re given.

When Bigger Isn’t Always Better: How Context Length Can Break Your LLM ​Longer context windows are the new bragging rights in LLMs — now stretching into the millions of tokens. But can models really handle the first and the 10,000th token equally well?

This paper challenges the LLM-dominant narrative and makes the case that small language models (SLMs) are not only sufficient for many agentic AI tasks—they’re often better. ​🧠 As agentic AI systems become more common—handling repetitive, task-specific operations—giant models may be overkill. The authors argue that: ​SLMs are faster, cheaper, and easier to deploy ​Most agentic tasks don't require broad general intelligence ​SLMs can be specialized and scaled with greater control ​Heterogeneous agents (using both LLMs and SLMs) offer the best of both worlds ​They even propose an LLM-to-SLM conversion framework, paving the way for more efficient agent design.

We break down key insights from the paper, discuss what these findings mean for AI’s role in the workforce, and debate its broader implications. As always, our expert moderators guide the session, followed by an open, lively discussion where you can share your thoughts, ask questions, and challenge ideas with fellow MLOps enthusiasts.

We dive deep into this groundbreaking paper, breakdown its key insights, and discuss what makes DeepSeek-R1 so special. Our expert moderators guide the session, followed by a lively round-robin discussion where everyone shares their thoughts, asks questions, and debates the implications with fellow MLOps enthusiasts. ​This is the reading group for anyone passionate about MLOps, from seasoned practitioners to the AI-curious. We meet every month on the second Thursday, and trust us—you don’t want to miss this one.

In the December Reading Group session, we explored A Taxonomy of Agents for Enabling Observability of Foundation Model-Based Agents. Key participants discussed the challenges of building agentic AI systems, focusing on four key capabilities: perception, planning, action, and adaptation. The paper highlighted issues like lack of controllability, complex inputs/outputs, and the difficulty of monitoring AI systems. Early-stage insights drew on DevOps and MLOps practices, and the need for improved tools and evaluation strategies for agent observability. The session fostered a collaborative exchange of ideas and practical solutions.

This November Reading Group conversation covers advanced retrieval techniques, strategies like iter-drag and hyper-drag for complex queries, and the impact of larger context windows on model performance. The Reading Group also examines challenges in generalizing these methods.

October 2024 MLOps reading group session explores the role and relevance of small language models in an era dominated by large language models (LLMs). The author of a recent survey paper on small models joins to discuss motivations for using smaller models, including resource constraints, efficiency, and unique capabilities they bring to certain tasks. Key discussion points include the advantages of small models in specific contexts (e.g., edge devices and specialized tasks), their role in complementing large models, and emerging techniques for leveraging small models to enhance model efficiency and mitigate issues like out-of-vocabulary words. The group also touches on methods for compressing models and the challenges in balancing model size with generalization and task-specific performance.

In our September 12th MLOps Community Reading Group session, live-streamed at the Data Engineering for AI/ML Virtual Conference, we covered the paper "Hybrid RAG: Integrating Knowledge Graphs and Vector Retrieval for Information Extraction." The panel discussed using hybrid RAGs to pull information from unstructured financial documents. The paper's approach combines knowledge graphs with vector-based retrieval for better results. We critiqued the context-mixing methods and lack of graph pruning techniques. Overall, it was a solid session with great insights on improving financial data extraction using hybrid models.

MLOps Reading Group meeting Reading Group Session about Ben Wilson's Machine Learning Engineering in Action book. https://www.manning.com/books/machine-learning-engineering-in-action https://www.amazon.com/Machine-Learning-Engineering-Action-Wilson/dp/1617298719