Design Process for Retrieval Augmented Generation Systems

Generative AI (GenAI) refers to artificial intelligence systems capable of developing and creating new content such as text, images, audio, or videos based on learned patterns from existing data. Integrating GenAI deeply into more complex software systems poses challenges such as hallucination, out-dated knowledge, non-removable knowledge, and non-traceable reasoning process. Retrieval-Augmented Generation (RAG) has emerged as a promising solution for enabling GenAI systems to incorporate knowledge from external sources. However, there is a lack of systematic guidance on designing a RAG system and identifying critical decision points. This paper presents a design process for a RAG system, including a suitability analysis based on the requirements of specific scenarios and the inherent characteristics of RAG. It outlines the key steps to design a RAG system and discusses critical design decisions. Insights are provided on the impact of design alternatives on the quality attributes of GenAI systems and their tradeoffs. Finally, a feasibility study examines an LLM-based chatbot designed for Q&A in taxation scenarios.