LongMemEval-S
Answer from about 50 earlier chat sessions
Within noise of the whole history at 1/62 of the cost (list prices)
76.7%
accuracy, 82.2 with whole history
- Whole history82.2
- LLMSlim + reranker76.7
- LLMSlim68.9
- Most recent sessions14.4
Results, October 2026*
Cut the context.
Keep the answers.
Seven public benchmarks, run against full context and plain truncation, plus a head-to-head with the best-known open compressors on NaturalQuestions. Only the context changed between runs.
- 62×
- cheaper long-chat memory at Haiku 5.5 list prices (14× fewer tokens), within noise of the whole history
- 3×
- fewer tokens on long documents, for the same score
- 100×
- faster per question than LongLLMLingua, and level with it on retrieval
One question, one long memory
How LLMSlim compresses a long chat history
- 01
The history
Fifty chat sessions. 114,006 tokens.
A real long-memory history. The question needs a few lines of it.
- 02
The read
Every round, scored against the question.
Lexical ranking with no model at all, or a 22M-parameter reranker for a sharper read.
- 03
The cut
Seven percent stays, in its original order.
Kept rounds go to the model whole, with their session headers. Everything else is left out.
- 04
The answer
76.7% right. The whole history scored 82.2%.
At 1/62 of the cost at Haiku 5.5 list prices, with 14× fewer tokens. On a fresh sample it scored 76.7% against 72.2% for the whole history, within noise.
- Context
- 114,006 tokens
- Per 1,000 questions
- $90.72
Seven benchmarks
Memory, retrieval, documents, and agents.
Each one public, each one scored by its own official metric. Same reader and prompt for every run.
NaturalQuestions
20 retrieved passages, the answer in one
Finds the answer wherever it sits
75%
accuracy, 86.3 with every passage
- Every passage86.3
- LLMSlim + reranker75
- LLMSlim65.7
- Passages in order52
HotpotQA
Two of ten paragraphs hold the answer
93% of the full-context F1 on 43% of the tokens
69.4
answer F1, 74.9 with all ten paragraphs
- All ten paragraphs74.9
- LLMSlim + reranker69.4
- LLMSlim67.5
- Cut to the same size49.5
LongBench v1
Seven tasks over 15.5k-token documents
The same score on a third of the tokens
54.9
macro score, 54.9 with full context
- Full context54.9
- LLMSlim54.9
- LLMSlim + reranker54.2
- Middle truncation45.2
LongBench v2
Multiple choice over 30k-token documents
72% fewer tokens, the best compressed score
55.6%
accuracy, 60 with full context
- Full context60
- LLMSlim + reranker55.6
- LLMSlim53.9
- Middle truncation48.3
SWE-bench Verified
50 real GitHub issues, graded by the official harness
No measured harm inside a coding agent
38
issues resolved, 38 with no compression, best run
- No compression, best run38
- LLMSlim 0.838
- No compression, second run36
- LLMSlim 0.7.234
InjecAgent
1,054 tool results carrying an injected instruction
Agent turns in order: zero repeated calls
0
repeated tool calls, 1 with native tool-use history
- Request after the tool results50
- Calls beside their results43
- Native tool-use history1
- Platform agent turn0
Head to head
Level with a 7B compressor. About 100× faster per question.
NaturalQuestions with 20 retrieved passages and the same token budget for every compressor. We ran LLMLingua-2 and LongLLMLingua ourselves, with their published settings.
- 1LLMSlim + reranker22M cross-encoder78.7
- 2LongLLMLinguaLLaMA-2-7B, 4-bit78.0
- 3LLMLingua-2XLM-RoBERTa-large69.0
- 4LLMSlimNo model68.7
- 5Cut to fitNo model48.3
Over all 300 cases (100 questions, each with the answer placed 1st, 10th and 20th), LLMSlim with the reranker and LongLLMLingua are level: +0.7 points, 95% interval −4.7 to +6.0. LLMSlim leads LLMLingua-2 by 9.7 (3.7 to 15.7).
LLMLingua-2 paper (arXiv 2403.12968) · LongLLMLingua paper (arXiv 2310.06839) · LLMLingua code (microsoft/LLMLingua) · Per-question results · Script to rerun the competitors
Fresh questions
It held on questions nobody had seen.
Settings were tuned on separate development questions. After the code was frozen, every suite ran once more on a fresh sample.
OriginalFresh
OriginalFresh
OriginalFresh
OriginalFresh
LLMSlim with the reranker Full context
Run it on your context.
Paste a long history, a stack of passages or a CI log, and watch what stays.
* Every context was read by Claude Haiku 5.5 with the same prompt; only the context changed. Confidence intervals are paired bootstrap 95% over the same questions. Times are to compress one question on one machine. The reranker is optional: pip install "llmslim[semantic]", then relevance="semantic".
Full context still leads on open-domain retrieval and multi-hop questions, by about 5 to 12 points. Every table and method.