As promised, I’ve prepared some notes about the security risks and protections related to Retrieval-Augmented Generation, or RAG, systems.
Remember that RAG systems combine powerful language models with external knowledge retrieval to give more accurate and relevant responses. But this introduces some security concerns.
One risk is passage injection. It is possible that someone could manipulate the data that a RAG system retrieves—which could lead to misleading or even harmful outputs. Zou et al. recently published a paper introducing their ‘PoisonedRAG’ attack framework demonstrating the consequences of malicious text injection. This could be done by getting malicious websites indexed by the retriever, or compromising enterprise databases.
In a paper by Xiang et al, you can learn about their ‘RobustRAG’ framework and the security enhancements that come from a different process of aggregating sources. In some RAG designs, sources are concatenated and passed to the LLM. In RobustRag, a ‘secure text aggregation technique’ processes sources independently and removes suspicious entries.
Data leakage is another concern. In a recent podcast, Simon Suo from LllamaIndex notes that organizational information is often segmented and that people are starting to raise concerns about information being passed to an LLM and then shared with employees that it is not intended for.
Check out the articles I’m linking in this post to learn more about the security implications of RAG and let me know your thoughts in the comments!
