From Naive RAG to Context Engineering

The secret to building truly effective Agentic AI is to create tools that augment them with the highest quality context possible. This is what makes ‘Context Engineering’ one of the hottest topics in the AI Engineering world today.

The concept of AI ‘context’ goes beyond just a simple prompt. We need to consider all the key elements listed below, which are tailored to form the context.

Instructions / System Prompt
User Prompt
State / History (short-term Memory)
Long-Term Memory
Grounding in Data Sources
Available Tools

While all of these are important, one particular category above is significantly challenging to implement but also has the highest value in terms of the organizational information it encodes.

Grounding in Data Sources: External, up-to-date organizational knowledge, relevant information from documents, databases, or APIs to answer specific questions.

Previously, a solution that addressed the above need was called RAG (Retrieval Augmented Generation). However, Naive RAG techniques quickly ran into several well-known limitations and are no longer considered robust enough to address the complexities of how large real-world organizations store and process data. This is why the AI world has now moved beyond Naive RAG to far more advanced context-engineering tools.

AI Context Engineering Tools at Redica

At Redica, we spent countless hours focusing on advanced context engineering to ground our AI applications in our organizational data, eliminate AI hallucinations, and continuously improve relevance by producing high-quality AI responses supported by verifiable data from our data stores.

To that end, we developed the following set of context engineering architecture and components to AI-enable all our Redica data, both structured and unstructured.

We started by building robust Data Foundations for AI:

The Redica Knowledge Graph to encode deeply graphical structures and interrelationships in Regulatory entities.
An AI Query Layer for LLMs to easily consume and interpret data stored in Relational databases.
A high-quality Document Parsing and Advanced Vectorization Solution to make the total content of our entire document corpus AI-discoverable, and suitable to address a wide variety of access patterns.

Corresponding to each of the Data Foundations above, we then built an elaborate custom Agentic AI toolset for efficient retrieval of data from the above foundational constructs:

A custom Text-to-Cypher AI Tool. For efficient retrieval of insights from the Knowledge graph via natural language.
A custom Text-to-SQL AI Tool. For low-latency and accurate retrieval of data from our AI query layer via natural language.
Multimodal Hybrid Agentic RAG Tool. Enable multiple access patterns to the content of all Regulatory documents (Unstructured data) – Document Similarity, Topic Modeling, Document-to-Document Comparison, Semantic Search, Text summarization, Custom document workflows, just to name a few. This includes both Regulatory Intelligence documents and Enforcement Documents.

What makes Redica AI tools different from off-the-shelf solutions?

We have deeply researched each of these problems, only to find that no off-the-shelf solution was robust enough to handle data complexities beyond toy data models. Also, none of them were customizable enough to unlock value from diverse data asset types. Some of these challenges are discussed below.

1. Natural Language Querying (NLQ) with Knowledge Graphs & SQL databases

LLMs are excellent at working with text (unstructured data). You can hand over any text to an LLM in its context and provide processing instructions. However, you cannot just hand over a database connection to an LLM and expect it to derive insights.

As we went through our own development journey, as well as case studies across organizations, we noticed some common design pitfalls and challenges that make this a highly complex task.

In this section, we discuss some challenges in building AI accessible Knowledge Graphs and highly interpretable Relational data schemas, as well as creating AI Agents/Tools that can translate natural language to native database query languages (Cypher or SQL)

1. Scalability problems for databases of practical size. Handling structured data gets increasingly complex for larger enterprises as their production data stores have complex schemas having hundreds, if not thousands, of tables and other DB objects. Specialized components need to be developed so that Text-to-SQL or Text-to-Cypher models can handle the size and complexity of these schemas effectively.

2. Data Catalog is not rich enough. Well-curated metadata plays a key role in LLM interpretability of data stores.

3. Limitations of automated migration solutions. When building a new knowledge graph, one is tempted to translate the data structures in relational databases to graph structures using off-the-shelf automation solutions(model tables as nodes and foreign keys as relationships in a graph). This approach of directly converting from relational to graphical structures almost always fails. Building effective Knowledge Graphs requires rethinking your data model altogether, combining domain expertise with Cypher and Graph database expertise.

4. Not designing a graph schema based on access patterns. Access patterns are one of the key factors commonly ignored while designing knowledge graphs, but are critical for accurate LLM-based access and low-latency node traversal.

5. LLMs are unable to map domain-specific entities to Graph/Relational database attributes. Even highly intelligent foundation models struggle to map terms and entities often referenced in common regulatory language to attributes on the graph nodes/relationships or relational database tables. We need to build dedicated custom solutions to bridge this gap for high-accuracy LLM retrieval.

6. Handling complex analytical queries in Graphs. Graph databases are great at traversing relationships fast, but are not best suited for very complex aggregations and heavily analytical or reporting needs. Again, special dynamic routing capabilities should be built into Agentic workflows that handle such query patterns via alternate paths.

7. Handling complex data structures in Cypher. Handling complex data structures (e.g, heavily nested JSON objects) is not as easy to handle with Cypher as it is with its relational counterparts. Both data models and agentic solutions need to be designed accordingly.

8. Lack of Reference Data Understanding is a common problem in applying foundational models to company data.

9. Lack of Sample Data Understanding results in low accuracy retrieval queries (bad filtering predicates) on both text-to-SQL and text-to-Cypher solutions, and specialized components need to be built to bake in this additional understanding.

10. Schema interpretability issues. For an effective natural language query model, graph schema objects (nodes, relationships, attributes) or relational schema objects (tables, columns, keys) must be made easily interpretable by LLMs.

11. SQL dialect challenges. Different SQL databases/warehouses have slightly varying SQL dialects, including their own set of supported custom functions (There are differences between SQL dialects of Postgres, MySQL, Snowflake, BigQuery, etc). To make the best out of the native functionalities provided by your specific database/data warehouse, complex modules need to be built to generate dialect-specific SQL with high accuracy.

Redica Text-to-Cypher and Text-to-SQL tools address all the above challenges and many more, using specialized modules, carefully designed data models, and a robust Multi-agent AI system. All these components have been built with thorough HiTL reviews, evaluation through robust LLM evaluation frameworks, and extensive testing of hundreds of query patterns.

2. Advanced Document Parsing, Vectorization & Agentic RAG

Redica Platform houses a huge corpus of Regulatory Intelligence and Enforcement Documents, HiTL-processed for clean information, augmentation of data from secondary sources, and metadata-enriched with quality and regulatory labeling/tagging models built by domain experts.

Redica AI has the following tools, purpose-built to derive all content from this document corpus and related metadata and make it AI discoverable, so that customers can request insights on demand in natural language.

Smart Parsing solutions for extracting document content in various formats.
Advanced Vectorization pipelines for extracted document content.
Agentic RAG solution for generating highly contextual and relevant AI responses.

Given below are a set of real-world challenges and considerations that make this end-to-end solution highly complex.

Document Parsing Challenges

Need for state-of-the-art OCR capabilities. Regulatory documents present a plethora of complexities for parsing solutions.
- They range in size from one-pagers to 1000+ page documents.
- Many documents are scanned images without text layers on them.
- There are old, poorly scanned, handwritten documents.
- There are a variety of factors that can throw off an OCR software, including badly spaced borders, watermarks, and wrong text alignments
Multimodal Parsing needs. Regulatory documents often contain complex elements such as tables, charts, and images. Traditional OCR software simply misses capturing the information contained in many of these complex elements. Multimodal AI-based parsing needs to be augmented with traditional OCR to extract semantic information from these elements.
Layout and coordinate awareness. Many downstream document workflows often require section-specific processing via HiTL forms, text highlighting, and commenting capabilities that require the parsing process to include exact sectional layout (bounding block coordinates of sections)
Support multiple extract formats. The parsed content of a document may have different downstream applications that require different extract formats. For usage in an AI application, an LLM-preprocessed markdown (.md) could be a great extract format due to the added semantic context of markdown tags. For search applications, a simple text extract (.txt) is required to maintain the fidelity of search results.
Long-running infrastructure-intensive processes. Often requires a one-time, large-batch effort to parse a large corpus of documents. Since this is both a compute-intensive as well as a memory-intensive process, it requires heavy infrastructure provisioning, monitoring, quality-checking via HiTL, and reprocessing, often amounting to months of effort.

Redica AI uses robust tools for parsing all types of complex documents, addressing the above parsing needs and more.

Vectorization and AI Retrieval Considerations

Summarization questions versus Needle-in-a-haystack questions. The vectorization/indexing/chunking processes need to ensure that both Top-k retrieval for pointy questionsand full-text-based use cases like summarization and multiple-document comparison are supported. Agentic retrieval systems should efficiently route to the appropriate index based on the query intent.
Support Graph RAG capabilities. Extracting business/domain entities and their linkages using LLM-based extraction and storing them in a graph index, along with vectorized content, often improves performance for large, complex regulatory documents.
Document-to-Document Comparison. Vectorization/Retrieval processes should support use cases involving understanding the relationships between common themes and entities across documents, or inferring that one is relevant to the other, using Agentic workflows.
Extract accurate Semantic context out of complex document elements like Tables, Images, and Charts.
The vectorization/retrieval process should understand the document section hierarchy and, by extension, the relationships between topics and subtopics, and answer queries with relevance.
The vectorization/retrieval process should support the Generation of accurate citations pointing to relevant document sections to help the AI response.

Redica’s multi-agent architecture supports workflows and strategies that cater to the above retrieval patterns and provide building blocks to unlock capabilities for even more complex document processing needs.