A Retrieval-Augmented Generation Information System for the Oil and Gas Industry

Research Context: The oil and gas industry relies on extensive and complex technical documentation, making manual information retrieval slow and inefficient for engineers and technicians who require quick and reliable answers for decisions. Scientific and/or Practical Problem: Generalist Large Language Models (LLMs) often struggle with specialized domains, leading to inaccuracies, contextual errors (”hallucinations”), and an inability to handle specific technical jargon. Proposed Solution and/or Analysis: This paper presents ”RAG Petrolês”an information system assistant built using Retrieval-Augmented Generation (RAG) on ”Petrolês”, a collection of Portuguese theses and dissertations on the topic. It uses the IBM Granite model for embeddings, FAISS for similarity search, and a combination of the DeepSeek R1 and Mistral Small 3.2 LLMs to generate and refine answers. Related IS Theory: The work is grounded in the Information Processing theory where a Retrieval-Augmented Generation (RAG) system integrates external knowledge bases with LLMs to enhance accuracy, contextual relevance of their responses and reduce hallucinations. Research Method: A quantitative and qualitative evaluation was performed. The system’s performance was tested against a custom dataset of 1500 questions. The evaluation involved a manual analysis of 150 answers and a broader statistical analysis of all 1500 responses using an ”LLM-as-a-Judge”approach combined with Prediction-Powered Inference (PPI) to ensure robust results. Summary of Results: The proposed system achieved an accuracy of 88.88% in manual evaluation. The statistical analysis resulted in an accuracy range from 81.48% to 97.71% within a 95% confidence interval. Outperforming baseline models and demonstrating superior performance compared to a direct adaptation of a similar existing framework. Contributions and Impact to IS area: This research demonstrates the effectiveness of a specialized RAG system in a technical, non-English domain. It provides a viable architecture for creating highly accurate, specialized information systems assistants without resorting to retrain foundational models.