The StreamRag paper from Meta proposes running a RAG system concurrently while a user is speaking to reduce latency in conversational voice AI.

The StreamRag method was shown to decrease latency by 0.5 seconds for synthetic datasets and 1.5 seconds for human-spoken datasets without a loss in accuracy.