Automated Evaluations

These notes are taken from this free course: https://learn.deeplearning.ai/courses/building-evaluating-advanced-rag/lesson/1/introduction Huge thanks to the folks at <DeepLearning.ai> Introduction To productionize a high quality RAG system you need: Advanced retrieval techniques to get highly relevant sources Automated evals to measure responses Goal of the course: Teach you how to build production ready LLM based system Course covers: Advanced RAG Sentence Window Retrieval Auto Merging Retrieval Evaluations (the RAG triad) Context Relevance Groundedness Answer Relevance Advanced RAG Pipeline Overview of simple (naive) RAG pipeline Ingestion: Take a doc -> chunk it up -> embed the chunks -> store them in an index...

Automated testing is critical to delivering high quality, stable software. Evals are the new paradigm on testing which allow us to automatically validate the functionality of LLM based applications. Why do we need a new paradigm of automated testing for LLMs? Automated tests validate that, given some setup, the same output is always achieved. This doesn’t work for LLM based applications, since give one input, you should get a different output each time....