Traditional NLP models are trainable, deterministic, and for some of them, explainable. When we encounter an erroneous prediction that affects downstream tasks, we can trace it back to the model, rerun the inference step, and reproduce the same result. We obtain valuable information like confidences (prediction probabilities) as a measure of the model’s ability to perform the task given the inputs (instead of silently hallucinating) and retrain it to patch its understanding of the problem space. By replacing them with large language models (LLMs), we trade the controllability of machine learning (ML) systems for their flexibility, generalizability, and ease of use.

Note: by LLMs, I am referring to managed models like OpenAI’s GPT-4. Self-hosted open-sourced LLMs (via Hugging Face or otherwise) usually allow users to set a seed for reproducibility.

While LLMs are powerful, we should be cognizant of these risks and take appropriate steps to mitigate them. Below we discuss some of these methods, but they are non-exhaustive in this quickly-evolving space.

Defensive prompting

We start with the most straightforward method to guard against hallucination and possibly malicious jailbreaking is to add a defensive component within the prompt. I’m not sure if there’s a name for this, but I’ll simply call this approach defensive prompting. The simplest variant (that you’ve probably seen before) looks like this:

Specifically for preventing jailbreaks, we can set up a prompt like the following:

For cases where we want the LLM to output different “error messages” for varying cases, we can introduce “codes” for each.

In downstream applications or code, we can check for the presence of HVD20AB or 06YVM98 and handle these cases separately.

Note: If you’re using OpenAI Chat Completion models, separate these instructions into the system and user messages as appropriate.

These are quick and easy prompt engineering tricks to nudge LLMs to be more predictable, but as a prompt-level intervention, this of course doesn’t solve the reproducibility problem. There’s no guarantee that LLMs will be fully reliable even with these additional clauses. In the next section, we look towards explicit, reproducible guardrails.

Guardrails

Guardrails are checks on top of LLM outputs to ascertain they meet predetermined criteria before being used in downstream services or exposed to the customer. If these checks fail, we can devise retry mechanisms to query the LLM again.

The simplest way is a proxy LLM approach: given the query and an LLM response, we make another query to the LLM to ask if the response is “good enough” in answering the query. For example, in a system where we use LLMs to generate email replies to sales leads, we might do the following:

With this guard, we can allow the response to be sent out if this query outputs 2, and send a separate query to the LLM to improve the reply if the response is 1. This approach is also extensible in a way such that we can cover more special cases and special instructions by appending to the TAKE NOTE section in the above prompt.

I found this method to be quite good in scoring the appropriateness of LLM responses. However, the most glaring drawback is that this introduces yet another LLM call — the very element we’re trying to build reliability for in this post. This self-check mechanism may be effective most of the time, but it is ultimately not robust and reproducible.

A promising trend in the LLM community is the emergence of declarative frameworks for LLM output verification. One open-source project is the Guardrails Python library. Essentially, this package provides wrappers around OpenAI calls to validate LLM outputs, e.g., data types, data characteristics (such as two-word strings, valid URLs), or even more sophisticated checks (e.g. similarity to document below a threshold, profanity-free outputs, relevance for question-answering, etc).

We provide a RAIL spec, an XML document (or string) comprising an output schema, and the prompt. The framework helps inject prompts instructing the model to convert XML to JSON so that the LLM’s output follows a certain JSON structure, which will be checked against using the RAIL spec.

For example, this RAIL spec (from the project docs):

will enforce the LLM output having this JSON structure:

Within the RAIL spec, we can specify quality checks, such as a certain string value to be only one of the $n$ choices. We can also set corrective actions to take, like re-asking OpenAI, filtering out certain values, etc. I recommend spending some time in the docs if you’re interested to find out more.

At the time of writing this post, there are other alternatives as well, like NVIDIA’s NeMo guardrails.

Human feedback

In my previous blog post, I discussed the value of human-in-the-loop machine learning and how human feedback (whether implicit or explicit) is crucial for monitoring ML systems in production. We can apply the same approach here, especially for LLMs that try to perform traditional ML tasks, like text classification and generation. Model performance based on human preferences is the ultimate benchmark of the utility of ML systems.

Note: This section is not about RLHF. We’re not training LLMs; as consumers from a product-building perspective, we can only tweak our systems that are built on top of these LLMs, but tweak them in a targeted way.

We can consider human verification for a random sample of LLM outputs, rating them (most commonly on a Likert scale) based on how well they answer the prompt. This allows us to collect data points (at least perform a qualitative assessment) on LLM performance: how the model performs with certain prompts characteristics, its tone, its helpfulness, or even just how good it is at answering questions over time. This is similar to monitoring the “data drift” problem in classical ML.

In retrieval-augmented LLM systems (where similar pieces of content to the query are retrieved from a vector database and injected into the prompt), this also gives a qualitative view of any gaps in knowledge, and any inadequacies in the retrieval process, so we can patch them appropriately.

The big challenges here are 1) how can we turn this human feedback into a quantitative measure (alongside qualitative inspection) so that we can analyze these results and monitor them more efficiently. and 2) maintaining a comprehensive set of guidelines so that human evaluation is fair across annotators (if there is more than one) and across time.

ML-based response evaluators

A faster and more scalable way to evaluate response is to train ML models to score these outputs. Recent dialogue response evaluation metrics include ADEM and RUBER, which go beyond word-overlap metrics like BLEU and METEOR commonly used in machine translation since they don’t correlate well with human judgments for dialogue response evaluation [1].

Automatic Dialogue Evaluation Model (ADEM) takes as inputs the dialogue context vector $c$, candidate response vector $r$, and reference response vector $r$. These vectors are embeddings from a pre-trained RNN model. ADEM computes the score with:

$\mathrm{ADEM}(c, r, \hat{r}) = ({c}^\top M{\hat{r}}+{r}^\top N{\hat{r}}-\alpha)/\beta$

where $M, N$

are learned matrices,

are scalar constants used to initialize the model’s predictions in the range

[2]. The score is a sum of a referenced metric and an unreferenced metric.

I won’t go into further details, but Referenced metric and Unreferenced metric Blended Evaluation Routine (RUBER), as its name suggests, also uses both metrics but in a different way: a combination of a similarity score between

and

, and a trained neural network predicting an “appropriateness” score between

and

. However, the main criticism for both ADEM and RUBER is that they tend to produce scores with very low variation due to the referenced metric [3].

More recently in 2020, Zhao et al devised a simple method without involving the use of the referenced metric. In this study, a pretrained RoBERTa encoder was used to obtain an embedding

given context

and candidate response

, upon which a multi-layer perceptron is trained on. Specifically, from the paper,

where RoBERTa’s parameter ϕ and the MLP’s parameter θ can both be optimized during training [4].

Despite the obvious latency and scalability benefits of automating evaluation with ML models, I have to mention that there are also several complicating points to consider. Firstly, we encounter the classic cold-start problem: we need sufficient data to train specialized evaluators, ideally, human-annotated labels to ensure data quality. Secondly, depending on how many LLM calls we invoke in the process, we might want to build different evaluators for different tasks, which can quickly become a hassle to manage. Thirdly, we will still need to monitor the performance of these models in production and retrain them when necessary. This, ultimately, is likely to involve human validation, but random sampling should suffice.

Monitoring LLMs

Like with any piece of software, it is also good practice to monitor the usage and performance of LLMs. In the previous section, we’ve seen ways in which we can derive automatic metrics for LLM evaluation; these will be very helpful for monitoring. In a chatbot use-case, for example, metrics like latency, session duration, hallucination rate (if we can detect hallucination reliably), the most commonly-raised topics, and the most accessed documents (if it is search-enabled) already give us a good sense of how the chatbot performs over time. Together with human feedback, we can derive metrics on the usefulness of the chatbot to our customers.

We want to be in a position where we can trace each step and have a clear picture of how things work. While we cannot guarantee things will go as expected especially in non-deterministic systems, it would be helpful to at least be alerted if something does go wrong so that we can take corrective action. The key would be to devise accurate metrics and alerts, specifically first minimizing false negatives (to eliminate uncaught critical errors), then minimizing false positives (so we can better trust our alerts and avoid alert fatigue). These could also serve as service-level indicators for the LLM-enabled system.

With good metrics, monitoring LLMs gives us a grasp on how reliable our system is, sheds light on any performance bottlenecks, and how we can improve the system further.

Conclusion

The Generative AI space has changed significantly in recent months, galvanized by OpenAI’s ChatGPT and its mass adoption by the world. Though many researchers have their efforts aimed at LLMs’ performance against benchmarks, there is also a distinct opportunity space where product engineers can quantify and manage the reliability and quality of LLM’s outputs while harnessing their immense generative abilities to delight customers.

Thanks to my friend Fan Pu for reviewing this post and offering helpful suggestions!

This post first appeared on my blog. Click here to read more.

References

[1] Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In EMNLP 2016, The 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122–2132

[2] Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic turing test: Learning to evaluate dialogue responses. In ACL 2017, The 55th Annual Meeting of the Association for Computational Linguistics, volume 1, pages 1116–1126.

[3] Ananya B. Sai, Mithun Das Gupta, Mitesh M. Khapra, and Mukundhan Srinivasan. 2019. Re-evaluating adem: A deeper look at scoring dialogue responses. In AAAI 2019, The 33rd AAAI Conference on Artificial Intelligence, pages 6220–6227.

[4] Tianyu Zhao, Divesh Lala, and Tatsuya Kawahara. 2020. Designing Precise and Robust Dialogue Response Evaluators. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 26–33, Online. Association for Computational Linguistics.

LinkedIn