Skip to main content
LLMSlim
Try Studio
All engineering notes
Context Engineering 9 min read Yashvardhan Thanvi July 15, 2026

Mitigating U-Shaped Attention Recall Decay in Long Context Prompts

Structuring Information Density to Overcome Position-Dependent Attention Loss

Mathematical intuition
Transformers exhibit maximum retrieval fidelity at context boundaries (0-15% and 85-100%). Pruning redundant middle prose elevates critical sentences into higher attention regions.
Key takeaways
  1. Information positioned in the middle 30-70% of long context prompts suffers from systematic recall degradation.
  2. Extracting central informational sentences reduces total prompt volume and moves key facts closer to instruction boundaries.
  3. Combines query relevance scoring with position preservation to maintain narrative coherence.

1. The Lost-in-the-Middle Phenomenon

Research by Liu et al. (2023) established that decoder language models exhibit a U-shaped performance curve when retrieving information from input documents:

  • Head Bias: High recall accuracy when target information resides near the initial system directives ($0-15\%$ of context).
  • Tail Bias: High recall accuracy when target information resides adjacent to the final prompt query ($85-100\%$ of context).
  • Middle Degradation: Statistically significant drop in recall performance when critical facts reside in the middle $30-70\%$ region.

By identifying and removing non-essential filler sentences from document contexts, LLMSlim reduces overall sequence length, effectively moving mid-document facts closer to high-attention boundaries.

2. Query-Aware Document Compression Implementation

Using compress_documents() to extract relevant sentences across retrieved vector chunks:

rag_pruner.pypython
from llmslim import compress_documents

retrieved_chunks = [
    "Doc Chunk 1: Background corporate history founded in 2012...",
    "Doc Chunk 2: Q3 Financial Results: Net operating income reached $4.2M, representing a 14% YoY increase...",
    "Doc Chunk 3: Additional administrative overhead details and disclaimers..."
]

query = "What was the Q3 net operating income?"

# Compress documents with query-focused sentence scoring
compressed_docs = compress_documents(retrieved_chunks, query=query, target_ratio=0.4)

for idx, doc in enumerate(compressed_docs):
    print(f"--- Chunk {idx+1} ---")
    print(doc.compressed_text)
References
  1. [Liu et al. (2023)] Lost in the Middle: How Language Models Use Long Contexts (Transactions of the Association for Computational Linguistics)