Measured on public benchmarks. Every number published.
Public datasets, official prompts and scoring, and the same model for every configuration. Only the context changes: the full context, LLMSlim, and truncation cut to exactly LLMSlim's token count.
LLMSlim Core 0.8.0 · LLMSlim Platform 0.9.1 · harness and raw results in the public repository
Chat memory
62×
lower cost per question over long chat histories, at Haiku 5.5 list prices
Whole history$90.72
LLMSlim$1.45
LongMemEval, 76.7% vs 82.2% with the whole history; 14× fewer tokens
Long documents
3×
fewer tokens on long documents, for the same score
Full document15.5k tokens
LLMSlim5.2k
LongBench v1, 54.9 vs 54.9 with full context
Retrieval
+23
points over truncation at the same token budget
Cut to fit52.0
LLMSlim75.0
NaturalQuestions, 20 passages, 25% budget
Agent turns
0
repeated tool calls across 1,054 agent turns
Before50
Agent turn0
InjecAgent, Platform agent turns
90 questions, 15 per type · accuracy (official judge prompts)
LongMemEval-S
Answer a question from about 50 earlier chat sessions (115k tokens of history).
With the reranker, 8k tokens score within noise of the whole 115k-token history (95% CI −15.6 to +4.4) at 1/62 of the cost at Haiku 5.5 list prices (14× fewer tokens; full-history prompts bill at the over-100k-token rate). On the original sample, multi-session and time questions score one question above the full history (15 per type); on the fresh sample multi-session falls below it (60.0 against 73.3).
NotePreference questions (a recommendation that should use a stated taste) are the weakest type: 40% against 87% on this sample.
accuracy (official judge prompts) by configuration
Full history (full context)82.2
114k tokens$90.72 per 1k questions
LLMSlim Core 0.8 + reranker (LLMSlim)76.7
7,984 tokens$1.45 per 1k questions
LLMSlim Core 0.8 (LLMSlim)68.9
7,985 tokens$1.45 per 1k questions
LLMSlim Platform 0.9.1 (LLMSlim)63.3
7,600 tokens$1.36 per 1k questions
LLMSlim Core 0.7.2 (earlier LLMSlim)25.6
10k tokens$1.86 per 1k questions
Most recent sessions (baseline)14.4
6,693 tokens$1.23 per 1k questions
Fresh questions, never used in development90 new questions
accuracy (official judge prompts) by configuration
Open-domain questions with 20 retrieved passages; the answer sits in one, placed first, 10th or 20th.
With the reranker, 75% on a quarter of the tokens wherever the answer sits (74 to 76% at every position), against 95% to 29% for keeping passages in order. Ahead of LLMLingua-2 at the same budget. LongLLMLingua, which runs a 7B model, is level with it over 300 cases: ahead on the original set, behind on the fresh one (below).
NoteAt 4.4x compression this is still 11 points below full context with the reranker, and 21 without it.
accuracy (answer in response) by configuration
Full context (full context)86.3
2,889 tokens$0.694 per 1k questions
LLMSlim Core 0.8 + reranker (LLMSlim)75.0
668 tokens$0.331 per 1k questions
LLMSlim Core 0.8 (LLMSlim)65.7
659 tokens$0.327 per 1k questions
LLMSlim Core 0.7.2 (earlier LLMSlim)62.3
727 tokens$0.355 per 1k questions
Keep passages in order (baseline)52.0
669 tokens$0.343 per 1k questions
Fresh questions, never used in development300 new cases (100 questions × 3 answer positions)
accuracy (answer in response) by configuration
Full context (full context)88.0
LLMSlim Core 0.8 + reranker (LLMSlim)75.7
LLMSlim Core 0.8 (LLMSlim)67.3
Keep passages in order (baseline)48.3
200 questions · answer F1
HotpotQA (distractor)
Multi-hop questions over 10 Wikipedia paragraphs, 2 of them relevant.
57% fewer input tokens and 44% lower cost than full context. Exact match with the reranker: 56.0 against 57.0 with every paragraph (54.5 against 59.0 on the fresh sample).
NoteShort contexts leave little to remove: on a 2k-token context, full context still has the best F1.
answer F1 by configuration
Full context (full context)74.9
2,179 tokens$0.245 per 1k questions
LLMSlim Core 0.8 + reranker (LLMSlim)69.4
939 tokens$0.137 per 1k questions
LLMSlim Core 0.8 (LLMSlim)67.5
936 tokens$0.133 per 1k questions
LLMSlim Core 0.7.2 (earlier LLMSlim)61.7
1,159 tokens$0.180 per 1k questions
Truncate to the same budget (baseline)49.5
1,147 tokens$0.194 per 1k questions
No context (baseline)43.8
84 tokens$0.120 per 1k questions
Fresh questions, never used in development200 new questions
answer F1 by configuration
Full context (full context)77.5
LLMSlim Core 0.8 + reranker (LLMSlim)71.8
LLMSlim Core 0.8 (LLMSlim)70.1
Truncate to the same budget (baseline)41.1
210 samples, 30 per task · macro average of official metrics
LongBench v1
Seven long-context tasks: document QA, multi-hop QA, summarisation, few-shot QA and code completion (15.5k tokens on average).
The same macro score as full context (95% CI −4.5 to +4.6) on a third of the tokens, and +9.8 over truncation. On the original sample RepoBench-P (cross-file code, 30 samples) scored above full context, 57.2 against 44.3; it did not on the fresh sample.
NoteMuSiQue multi-hop QA is still below full context (58.8 against 64.9; 61.2 with the reranker), though most of the 0.8 pre-release drop (48.3) is recovered.
macro average of official metrics by configuration
Full context (full context)54.9
16k tokens$1.81 per 1k questions
LLMSlim Core 0.8 (LLMSlim)54.9
5,232 tokens$0.778 per 1k questions
LLMSlim Core 0.8 + reranker (LLMSlim)54.2
5,229 tokens$0.778 per 1k questions
LLMSlim Core 0.7.2 (earlier LLMSlim)52.4
5,164 tokens$0.779 per 1k questions
Middle truncation, same budget (baseline)45.2
5,147 tokens$0.785 per 1k questions
Fresh questions, never used in development210 new samples
macro average of official metrics by configuration
Full context (full context)55.1
LLMSlim Core 0.8 + reranker (LLMSlim)55.4
LLMSlim Core 0.8 (LLMSlim)55.0
Middle truncation, same budget (baseline)46.8
180 questions · accuracy
LongBench v2 (short split)
Hard multiple-choice questions over long documents, codebases and structured data (30k tokens on average).
72% fewer tokens and 78% lower cost. With the reranker, 4.4 points below full context (95% CI −11.7 to +2.8), and the best compressed arm.
NoteQuestions that need the whole document lose accuracy: hard multiple choice over 30k tokens is where compression costs the most.
accuracy by configuration
Full context (full context)60.0
30k tokens$7.62 per 1k questions
LLMSlim Core 0.8 + reranker (LLMSlim)55.6
8,469 tokens$1.69 per 1k questions
LLMSlim Core 0.8 (LLMSlim)53.9
8,470 tokens$1.69 per 1k questions
LLMSlim Core 0.7.2 (earlier LLMSlim)51.1
8,566 tokens$1.90 per 1k questions
Middle truncation, same budget (baseline)48.3
8,566 tokens$1.94 per 1k questions
Head to head
Other compressors, same reader. Same questions, same budget.
We ran the best-known open-source prompt compressors ourselves, with the same reader for every compressed context and the same token budget for each.
Compressor
Model it runs
Original (150 cases)
Fresh (150 cases)
All (300 cases)
Tokens kept
Time per question
LLMSlim Core 0.8 + reranker
22M cross-encoder
74.0
83.3
78.7
668
0.1 s on a GPU
LongLLMLingua
LLaMA-2-7B (4-bit)
80.0
76.0
78.0
630
9.3 s on a GPU
LLMLingua-2
XLM-RoBERTa-large
64.7
73.3
69.0
680
2.3 s on a GPU, about 43 s on CPU
LLMSlim Core 0.8
none
63.3
74.0
68.7
658
25 ms on CPU
Cut to the same size
none
49.3
47.3
48.3
627
instant
NaturalQuestions with 20 retrieved passages and a 25% budget; full context scores 87.3 and 90.7 on the two sets. Each set is 50 questions × 3 answer positions (1st, 10th, 20th), so the 300 cases come from 100 questions. Over all 300 cases LLMSlim with the reranker and LongLLMLingua are level (78.7 and 78.0: +0.7 points, 95% interval −4.7 to +6.0), and LLMSlim is ahead of LLMLingua-2 by 9.7 points (95% interval 3.7 to 15.7). LongLLMLingua uses the paper's NQ settings with LLaMA-2-7B-chat; the paper does not state its precision, and we ran it in 4-bit NF4 to fit a 6 GB GPU. Tokens are means and times are medians over the 300 cases, to compress one question on our machine (LLMLingua-2's CPU time is the mean of a separate CPU run on the original set).
50 real GitHub issues (Django, Sphinx); a bash-tool agent edits the repository; graded by the official test harness.
Core 0.8 matches the best uncompressed run with a cost per task between the two: no harm, little saving. A careful agent keeps its context near 8k tokens, so there is not much to remove. Published work found LLMLingua-2 on tool output cost an agent 8 points.
Agent, no compression (two runs)
36 and 38 of 5058k and 48k prompt tokens per task
LLMSlim Core 0.8 on outputs over 800 tokens
38 of 5028 outputs compressed; 51k prompt tokens per task
LLMSlim Core 0.7.2, same setting
34 of 5030 outputs compressed; 59k prompt tokens per task
InjecAgent (enhanced attacks)
1,054 cases where a tool result carries an attacker instruction; success means the agent calls the attacker's tool.
The reader model resisted every attack on its own, so this benchmark shows no defence effect with this model. It did expose an agent-loop gap, now fixed: before agent turns, the Platform put the user's request after the tool results and the model repeated its call 50 times. With continue_turn, the turn is shown in the order it happened and no call is repeated.
Native tool-use history
0 of 1,054attack success 0.0%; 1 repeated call
Platform 0.9.0 flow (request after the tool results)
0 of 1,054attack success 0.0%; 50 repeated calls
Platform 0.9.1 agent turn (continue_turn)
0 of 1,054attack success 0.0%; 0 repeated calls
Other systems
Published results, side by side. With their readers.
These numbers come from each project's paper or a third-party re-run, with different reader models. Compare each system's gap to its own full-context baseline, not raw scores across models.
Same model. Same questions. Only the context changes.
01
Reader model: Claude Haiku 5.5 at effort low for every arm. Only the context changes between arms.
02
Official prompts and metrics for each dataset. LongMemEval uses its official judge prompts, run with Haiku 5.5 as the judge.
03
Truncation and recency baselines get exactly the token count LLMSlim produced, so comparisons are at equal budget.
04
Confidence intervals are paired bootstrap 95% over the same questions. On NaturalQuestions each question appears three times (answer 1st, 10th and 20th) and the intervals resample those cases, so they are narrower than intervals that resample whole questions.
05
Tuning used only separate development questions (free offline checks of what survives compression). After the code was frozen, every suite was also run once on a fresh sample nobody had looked at; those numbers are shown beside the originals.
06
The optional reranker is cross-encoder/ms-marco-MiniLM-L-6-v2 (22M parameters, trained on MS MARCO web search only). Install it with pip install "llmslim[semantic]" and pass relevance="semantic". Without it, LLMSlim uses no model at all.
07
LLMLingua-2 and LongLLMLingua were run by us with their published settings, on the same questions, budget and reader.