Mitigating U-Shaped Attention Recall Decay in Long Context Prompts
Structuring Information Density to Overcome Position-Dependent Attention Loss
Mathematical intuition
Transformers exhibit maximum retrieval fidelity at context boundaries (0-15% and 85-100%). Pruning redundant middle prose elevates critical sentences into higher attention regions.
- 01Information positioned in the middle 30-70% of long context prompts suffers from systematic recall degradation.
- 02Extracting central informational sentences reduces total prompt volume and moves key facts closer to instruction boundaries.
- 03Combines query relevance scoring with position preservation to maintain narrative coherence.
1. The Lost-in-the-Middle Phenomenon
Research by Liu et al. (2023) established that decoder language models exhibit a U-shaped performance curve when retrieving information from input documents:
- **Head Bias**: High recall accuracy when target information resides near the initial system directives ($0-15\%$ of context).
- **Tail Bias**: High recall accuracy when target information resides adjacent to the final prompt query ($85-100\%$ of context).
- **Middle Degradation**: Statistically significant drop in recall performance when critical facts reside in the middle $30-70\%$ region.
By identifying and removing non-essential filler sentences from document contexts, LLMSlim reduces overall sequence length, effectively moving mid-document facts closer to high-attention boundaries.
2. Query-Aware Document Compression Implementation
Using `compress_documents()` to extract relevant sentences across retrieved vector chunks:
rag_pruner.py
from llmslim import compress_documents
retrieved_chunks = [
"Doc Chunk 1: Background corporate history founded in 2012...",
"Doc Chunk 2: Q3 Financial Results: Net operating income reached $4.2M, representing a 14% YoY increase...",
"Doc Chunk 3: Additional administrative overhead details and disclaimers..."
]
query = "What was the Q3 net operating income?"
# Compress documents with query-focused sentence scoring
compressed_docs = compress_documents(retrieved_chunks, query=query, target_ratio=0.4)
for idx, doc in enumerate(compressed_docs):
print(f"--- Chunk {idx+1} ---")
print(doc.compressed_text)