How SURGE substantially improves knowledge relevance through targeted augmentation while retaining language fluency
Generative pre-trained models have shown impressive fluency and coherence when used for dialogue agents. However, a key limitation they suffer from is the lack of grounding in external knowledge. Left to their pre-trained parameters alone, these models often generate plausible-sounding but factually incorrect responses, also known as hallucinations.
Prior approaches to mitigate this have involved augmenting the dialogue context with entire knowledge graphs associated with entities mentioned in the chat. However, this indiscriminate conditioning on large knowledge graphs brings its own problems:
Limitations of Naive Knowledge Graph Augmentation:
- Much of the 1-hop context may be irrelevant to the dialogue, inserting unnecessary noise
- Encoding entire knowledge subgraphs strains sequence length limits
- No guarantee model will use the relevant facts for generation
- Risk of hallucination still exists despite knowledge grounding
To overcome this, Kang et al. 2023 propose the SUbgraph Retrieval-augmented GEneration (SURGE) framework, with three key innovations:
- Context-Relevant Subgraph Retriever: Retrieving the most relevant knowledge graph facts to the dialogue context using a graph neural network retriever.
- Efficient Graph Encoding: Perturbing token embeddings based on relations while encoding just subgraph entities instead of all triplets. Maintains permutation and inversion invariance.
- Graph-Text Contrastive Learning: Ensuring consistency between retrieved knowledge graph and generated response via contrastive loss.
This allows providing precisely the requisite factual context to the dialogue without dilution from irrelevant facts or model limitations. Experiments show SURGE reduces hallucination and improves grounding.