# LLMSlim benchmarks

> Measured on Claude Haiku 5.5 (October 2026) with public datasets and official scoring. Only the context changes between configurations. LLMSlim Core 0.8.0, Platform 0.9.1. Weak spots are listed with each result.

## LongMemEval-S

Answer a question from about 50 earlier chat sessions (115k tokens of history). 90 questions, 15 per type; metric: accuracy (official judge prompts).

| Configuration | Score | Tokens | Cost per 1k questions |
|---|---|---|---|
| Full history | 82.2 | 114,006 | $90.72 |
| LLMSlim Core 0.8 + reranker | 76.7 | 7,984 | $1.45 |
| LLMSlim Core 0.8 | 68.9 | 7,985 | $1.45 |
| LLMSlim Platform 0.9.1 | 63.3 | 7,600 | $1.36 |
| LLMSlim Core 0.7.2 | 25.6 | 10,482 | $1.86 |
| Most recent sessions | 14.4 | 6,693 | $1.23 |

Fresh sample (90 new questions, run once after development was frozen): Full history 72.2; LLMSlim Core 0.8 + reranker 76.7; LLMSlim Core 0.8 67.8; Most recent sessions 14.4.

With the reranker, 8k tokens score within noise of the whole 115k-token history (95% CI −15.6 to +4.4) at 1/62 of the cost at Haiku 5.5 list prices (14× fewer tokens; full-history prompts bill at the over-100k-token rate). On the original sample, multi-session and time questions score one question above the full history (15 per type); on the fresh sample multi-session falls below it (60.0 against 73.3).

Note: Preference questions (a recommendation that should use a stated taste) are the weakest type: 40% against 87% on this sample.

## NaturalQuestions, 20 documents

Open-domain questions with 20 retrieved passages; the answer sits in one, placed first, 10th or 20th. 300 cases (100 questions × 3 answer positions); metric: accuracy (answer in response).

| Configuration | Score | Tokens | Cost per 1k questions |
|---|---|---|---|
| Full context | 86.3 | 2,889 | $0.694 |
| LLMSlim Core 0.8 + reranker | 75.0 | 668 | $0.331 |
| LLMSlim Core 0.8 | 65.7 | 659 | $0.327 |
| LLMSlim Core 0.7.2 | 62.3 | 727 | $0.355 |
| Keep passages in order | 52.0 | 669 | $0.343 |

Fresh sample (300 new cases (100 questions × 3 answer positions), run once after development was frozen): Full context 88.0; LLMSlim Core 0.8 + reranker 75.7; LLMSlim Core 0.8 67.3; Keep passages in order 48.3.

With the reranker, 75% on a quarter of the tokens wherever the answer sits (74 to 76% at every position), against 95% to 29% for keeping passages in order. Ahead of LLMLingua-2 at the same budget. LongLLMLingua, which runs a 7B model, is level with it over 300 cases: ahead on the original set, behind on the fresh one (below).

Note: At 4.4x compression this is still 11 points below full context with the reranker, and 21 without it.

## HotpotQA (distractor)

Multi-hop questions over 10 Wikipedia paragraphs, 2 of them relevant. 200 questions; metric: answer F1.

| Configuration | Score | Tokens | Cost per 1k questions |
|---|---|---|---|
| Full context | 74.9 | 2,179 | $0.245 |
| LLMSlim Core 0.8 + reranker | 69.4 | 939 | $0.137 |
| LLMSlim Core 0.8 | 67.5 | 936 | $0.133 |
| LLMSlim Core 0.7.2 | 61.7 | 1,159 | $0.18 |
| Truncate to the same budget | 49.5 | 1,147 | $0.194 |
| No context | 43.8 | 84 | $0.12 |

Fresh sample (200 new questions, run once after development was frozen): Full context 77.5; LLMSlim Core 0.8 + reranker 71.8; LLMSlim Core 0.8 70.1; Truncate to the same budget 41.1.

57% fewer input tokens and 44% lower cost than full context. Exact match with the reranker: 56.0 against 57.0 with every paragraph (54.5 against 59.0 on the fresh sample).

Note: Short contexts leave little to remove: on a 2k-token context, full context still has the best F1.

## LongBench v1

Seven long-context tasks: document QA, multi-hop QA, summarisation, few-shot QA and code completion (15.5k tokens on average). 210 samples, 30 per task; metric: macro average of official metrics.

| Configuration | Score | Tokens | Cost per 1k questions |
|---|---|---|---|
| Full context | 54.9 | 15,532 | $1.812 |
| LLMSlim Core 0.8 | 54.9 | 5,232 | $0.778 |
| LLMSlim Core 0.8 + reranker | 54.2 | 5,229 | $0.778 |
| LLMSlim Core 0.7.2 | 52.4 | 5,164 | $0.779 |
| Middle truncation, same budget | 45.2 | 5,147 | $0.785 |

Fresh sample (210 new samples, run once after development was frozen): Full context 55.1; LLMSlim Core 0.8 + reranker 55.4; LLMSlim Core 0.8 55.0; Middle truncation, same budget 46.8.

The same macro score as full context (95% CI −4.5 to +4.6) on a third of the tokens, and +9.8 over truncation. On the original sample RepoBench-P (cross-file code, 30 samples) scored above full context, 57.2 against 44.3; it did not on the fresh sample.

Note: MuSiQue multi-hop QA is still below full context (58.8 against 64.9; 61.2 with the reranker), though most of the 0.8 pre-release drop (48.3) is recovered.

## LongBench v2 (short split)

Hard multiple-choice questions over long documents, codebases and structured data (30k tokens on average). 180 questions; metric: accuracy.

| Configuration | Score | Tokens | Cost per 1k questions |
|---|---|---|---|
| Full context | 60.0 | 30,493 | $7.621 |
| LLMSlim Core 0.8 + reranker | 55.6 | 8,469 | $1.688 |
| LLMSlim Core 0.8 | 53.9 | 8,470 | $1.689 |
| LLMSlim Core 0.7.2 | 51.1 | 8,566 | $1.902 |
| Middle truncation, same budget | 48.3 | 8,566 | $1.942 |

72% fewer tokens and 78% lower cost. With the reranker, 4.4 points below full context (95% CI −11.7 to +2.8), and the best compressed arm.

Note: Questions that need the whole document lose accuracy: hard multiple choice over 30k tokens is where compression costs the most.

## Head to head with other compressors

| Compressor | Model it runs | Original (150 cases) | Fresh (150 cases) | All (300 cases) | Tokens kept | Time per question |
|---|---|---|---|---|---|---|
| LLMSlim Core 0.8 + reranker | 22M cross-encoder | 74.0 | 83.3 | 78.7 | 668 | 0.1 s on a GPU |
| LongLLMLingua | LLaMA-2-7B (4-bit) | 80.0 | 76.0 | 78.0 | 630 | 9.3 s on a GPU |
| LLMLingua-2 | XLM-RoBERTa-large | 64.7 | 73.3 | 69.0 | 680 | 2.3 s on a GPU, about 43 s on CPU |
| LLMSlim Core 0.8 | none | 63.3 | 74.0 | 68.7 | 658 | 25 ms on CPU |
| Cut to the same size | none | 49.3 | 47.3 | 48.3 | 627 | instant |

NaturalQuestions with 20 retrieved passages and a 25% budget; full context scores 87.3 and 90.7 on the two sets. Each set is 50 questions × 3 answer positions (1st, 10th, 20th), so the 300 cases come from 100 questions. Over all 300 cases LLMSlim with the reranker and LongLLMLingua are level (78.7 and 78.0: +0.7 points, 95% interval −4.7 to +6.0), and LLMSlim is ahead of LLMLingua-2 by 9.7 points (95% interval 3.7 to 15.7). LongLLMLingua uses the paper's NQ settings with LLaMA-2-7B-chat; the paper does not state its precision, and we ran it in 4-bit NF4 to fit a 6 GB GPU. Tokens are means and times are medians over the 300 cases, to compress one question on our machine (LLMLingua-2's CPU time is the mean of a separate CPU run on the original set).

Choosing between them: [LLMLingua Alternative: LLMSlim vs LLMLingua-2 and LongLLMLingua](https://www.llmslim.app/compare/llmlingua.md), with migration code.

## SWE-bench Verified Mini

50 real GitHub issues (Django, Sphinx); a bash-tool agent edits the repository; graded by the official test harness.

| Configuration | Result | Detail |
|---|---|---|
| Agent, no compression (two runs) | 36 and 38 of 50 | 58k and 48k prompt tokens per task |
| LLMSlim Core 0.8 on outputs over 800 tokens | 38 of 50 | 28 outputs compressed; 51k prompt tokens per task |
| LLMSlim Core 0.7.2, same setting | 34 of 50 | 30 outputs compressed; 59k prompt tokens per task |

Core 0.8 matches the best uncompressed run with a cost per task between the two: no harm, little saving. A careful agent keeps its context near 8k tokens, so there is not much to remove. Published work found LLMLingua-2 on tool output cost an agent 8 points.

## InjecAgent (enhanced attacks)

1,054 cases where a tool result carries an attacker instruction; success means the agent calls the attacker's tool.

| Configuration | Result | Detail |
|---|---|---|
| Native tool-use history | 0 of 1,054 | attack success 0.0%; 1 repeated call |
| Platform 0.9.0 flow (request after the tool results) | 0 of 1,054 | attack success 0.0%; 50 repeated calls |
| Platform 0.9.1 agent turn (continue_turn) | 0 of 1,054 | attack success 0.0%; 0 repeated calls |

The reader model resisted every attack on its own, so this benchmark shows no defence effect with this model. It did expose an agent-loop gap, now fixed: before agent turns, the Platform put the user's request after the tool results and the model repeated its call 50 times. With continue_turn, the turn is shown in the order it happened and no call is repeated.

## Published results from other systems

Different reader models: compare each system's gap to its own full-context baseline.

| System | Benchmark | Reader | Score | Full context | Source |
|---|---|---|---|---|---|
| Zep | LongMemEval-S | GPT-4o | 71.2% | 60.2% | Zep paper (self-reported): https://arxiv.org/abs/2501.13956 |
| Mem0 | LongMemEval-S | GPT-4o-mini | 53.6 to 66.4% | 55.4 to 56.8% | Third-party re-runs (MemOS, LightMem papers): https://arxiv.org/abs/2507.03724 |
| LongLLMLingua (4x) | NQ, 20 documents | GPT-3.5-Turbo | 71.2 to 75.0 | 54.1 to 75.7 | LongLLMLingua paper: https://arxiv.org/abs/2310.06839 |
| LLMLingua-2 (3.9x) | NQ, 20 documents | GPT-3.5-Turbo | 39.9 to 48.6 | 54.1 to 75.7 | LLMLingua-2 paper: https://arxiv.org/abs/2403.12968 |
| LLMLingua-2 (3k) | LongBench v1 | GPT-3.5-Turbo | 42.4 | 44.0 | LLMLingua-2 paper: https://arxiv.org/abs/2403.12968 |
| LLMLingua-2 on tool output | SWE-bench Verified (50) | Not stated | 54% | 62% | SWE-Pruner paper: https://arxiv.org/abs/2601.16746 |

## Method

- Reader model: Claude Haiku 5.5 at effort low for every arm. Only the context changes between arms.
- Official prompts and metrics for each dataset. LongMemEval uses its official judge prompts, run with Haiku 5.5 as the judge.
- Truncation and recency baselines get exactly the token count LLMSlim produced, so comparisons are at equal budget.
- Confidence intervals are paired bootstrap 95% over the same questions. On NaturalQuestions each question appears three times (answer 1st, 10th and 20th) and the intervals resample those cases, so they are narrower than intervals that resample whole questions.
- Tuning used only separate development questions (free offline checks of what survives compression). After the code was frozen, every suite was also run once on a fresh sample nobody had looked at; those numbers are shown beside the originals.
- The optional reranker is cross-encoder/ms-marco-MiniLM-L-6-v2 (22M parameters, trained on MS MARCO web search only). Install it with pip install "llmslim[semantic]" and pass relevance="semantic". Without it, LLMSlim uses no model at all.
- LLMLingua-2 and LongLLMLingua were run by us with their published settings, on the same questions, budget and reader.

## Earlier offline suites

- [v0.6 Adaptive Context Planner](https://www.llmslim.app/benchmarks/adaptive-context-planner.md): Frozen multilingual context planning with explicit quality and safety checks
- [v0.4.0 Release Verification](https://www.llmslim.app/benchmarks/release-verification.md): Measured test, coverage, lint, and schema-tax benchmark results

Canonical page: https://www.llmslim.app/benchmarks