# LLMSlim vs LLMLingua-2 vs LongLLMLingua: A Reproducible Prompt Compression Benchmark

> One reader, one token budget, 300 paired cases on NaturalQuestions, and every script and per-case result published

By Yashvardhan Thanvi (LLMSlim Author & Core Maintainer) · Published October 12, 2026 · Updated October 12, 2026 · Benchmarking Methodology

## Abstract

We compared LLMSlim 0.8 with LLMLingua-2 and LongLLMLingua on 20-document NaturalQuestions at a 25% token budget, with Claude Haiku 5.5 as the reader. With its optional 22M-parameter reranker, LLMSlim was level with LongLLMLingua (78.7 vs 78.0) and 9.7 points ahead of LLMLingua-2, with a median compression time per case of 0.09 s against 9.3 s and 2.3 s.

## Intuition

Every comparison is paired: both systems answer the same cases, so each case gives a score difference of −1, 0 or +1. The 95% interval resamples those differences with replacement 4,000 times, by case or by whole question, and reads the 2.5th and 97.5th percentiles of the resampled means.

## 1. Why long-context compression is hard

Retrieval pipelines hand a model more text than it needs. In the NaturalQuestions setup used here, each question comes with 20 retrieved Wikipedia passages, about 2,900 tokens, and one of them holds the answer. A compressor working to a 25% budget has to drop three quarters of that text without seeing the answer, and the passage that matters can be first, tenth or last.

Three things make this hard:

- **Relevance depends on the question.** A passage that is noise for one question answers another. A compressor that never reads the question can only keep what looks generally informative.
- **Naive cuts fail on position.** With the full context, Claude Haiku 5.5 scored 88.0, 87.0 and 84.0 with the gold passage 1st, 10th and 20th (test_a, 300 cases). Keeping passages in retrieval order until the budget ran out scored 95.0 when the answer came first, and 32.0 and 29.0 when it came 10th or 20th.
- **What survives has to stay readable.** A compressor can keep the answer string and still strip the context a reader needs to commit to it. Section 10 shows a case where this appears to have decided one of the two test splits.

There is a measurement problem as well: compressors state budgets in different units. Asked for `rate=0.25`, LLMLingua-2 kept 16% of the context as LLMSlim counts tokens (469 tokens), because its rate is measured on its own tokens. A comparison only means something when every system is held to the same token count, in the same tokenizer, with the same reader. That is the whole design of this benchmark.

## 2. What LLMSlim is, and which part was tested

LLMSlim Core is an MIT-licensed Python library on PyPI (`pip install llmslim`); these runs used Core 0.8.0. It runs inside your process and returns the text to send, and your application keeps control of the model call.

Given a query, Core 0.8 ranks the context's structural units (documents, chat rounds, paragraphs) against it, keeps the best units whole and in their original order inside the token budget, and fills what is left sentence by sentence from the next best unit. Relevance comes in two modes, and both are in the results:

- **Lexical, the default:** BM25 with light stemming and one round of pseudo-relevance feedback. No model, no download, deterministic output.
- **Reranker, optional:** `relevance="semantic"` adds `cross-encoder/ms-marco-MiniLM-L-6-v2`, a 22M-parameter cross-encoder trained on MS MARCO web-search passages only. It rescores the 80 strongest lexical candidates, so it works on top of BM25 rather than replacing it, and it is installed with the `semantic` extra.

LLMSlim Platform is a separate, private beta built on Core, for agent state, decision traces and learned context policies. It is proprietary and not on PyPI, and it played no part in the NaturalQuestions comparison. The benchmark README reports it in two places only, LongMemEval-S and InjecAgent, and section 6 quotes both.

## 3. The two baselines from Microsoft Research

LLMLingua-2 and LongLLMLingua come from the LLMLingua project at Microsoft Research, whose code is open source at [github.com/microsoft/LLMLingua](https://github.com/microsoft/LLMLingua). They are the best-known open-source prompt compressors, which is why we chose them. We ran their public code ourselves; Microsoft did not run, review or endorse anything in this article.

- **LLMLingua-2** (Pan et al., 2024, [arXiv:2403.12968](https://arxiv.org/abs/2403.12968)) treats compression as token classification. An encoder, here XLM-RoBERTa-large fine-tuned on MeetingBank transcripts with labels distilled from GPT-4, decides for each token whether to keep it. The checkpoint has about 560M parameters according to its Hugging Face metadata (the paper's text gives 355M). It is task-agnostic: it does not read the question.
- **LongLLMLingua** (Jiang et al., 2023, [arXiv:2310.06839](https://arxiv.org/abs/2310.06839)) targets long, question-specific contexts. A small causal language model scores how much each document and token matters given the question. Compression runs coarse to fine, documents are reordered by relevance, and each document gets its own compression ratio. Its NaturalQuestions experiments use LLaMA-2-7B-chat as that small model.

Both papers report results on this dataset family with GPT-3.5-Turbo as the reader. LongLLMLingua at 4x compression reported 71.2 to 75.0 against a full-context range of 54.1 to 75.7; LLMLingua-2 at 3.9x reported 39.9 to 48.6. With a different reader only each system's gap to its own full-context baseline is comparable, not the raw scores, so we reran both under one reader.

## 4. Dataset and methodology

**Data.** The 20-document NaturalQuestions files from the "lost in the middle" study (Liu et al., 2023; `nelson-liu/lost-in-the-middle`). Each question exists in three versions, with the gold passage placed 1st, 10th or 20th among the 20. In this article a **case** is one question at one gold position, so 100 questions make 300 cases.

**Splits.** Fixed in `splits.json` by `splits.py`, disjoint by question:

- **test_a:** 100 questions (300 cases), the questions reported since the first round.
- **dev:** 400 questions, for development only.
- **test_b:** 100 fresh questions (300 cases), drawn after the code was frozen and run once.

**Head-to-head subset.** Both baselines are slow, so the competitor round ran on every second case of each split. Cases are ordered by gold position, so the stride keeps the positions balanced: each split contributes 150 cases, which are 50 questions at all three positions. Across the two splits that makes 300 cases from 100 distinct questions, and every system in the head-to-head table is scored on exactly those cases.

**Reader and metric.** `claude-haiku-5-5` with `effort: low` through the Claude API, up to 1,024 output tokens, and the same prompt for every arm: "Write a high-quality answer for the given question using only the provided search results (some of which might be irrelevant)", then the context and the question. A case scores 1 when a normalised gold answer appears in the normalised response (the `best_subspan_em` rule from the lost-in-the-middle code) and 0 otherwise.

**Budget.** 25% of the full context in LLMSlim's token count. LLMSlim gets `target_ratio=0.25`; the baselines get the same budget as an explicit token target. LLMSlim and llmlingua both count tokens with `cl100k_base`, so every arm had the same nominal target, about 722 tokens, and all of them came in under it. Mean context size over the 300 cases: full 2,889 tokens, LLMSlim with the reranker 668, LongLLMLingua 630, LLMLingua-2 680, LLMSlim lexical 658.

**Dev-only tuning.** Nothing was tuned on test_a or test_b. `devcheck.py` runs free, offline checks on the dev split: did the answer, the supporting sentences or the labelled evidence survive compression? Each change in 0.8.0 (compound matching, the optional reranker and its lexical share, units of about 1/32 of the budget) was kept only if it held up on dev. Neighbour smoothing, entity bridging, per-group diversity and passage titles in headers were dropped because dev did not support them. After LongLLMLingua was measured, two more ideas were tried with the reader on 300 dev cases; neither helped, so neither ships (section 10).

**Prompt integrity.** `verify_built.py` rebuilds a random sample of 25 prompts per arm from the frozen source and checks they are identical to the prompts that were sent.

**Statistics.** Paired bootstrap 95% intervals over the same cases, 4,000 resamples. Section 8 also resamples whole questions and adds a per-question sign test.

**Cost.** About $30 of API calls for every suite in the benchmark README together, with per-suite ledgers in `results/spend-*.json`.

## 5. Exact configurations

These are the calls in `benchmarks/live/bench2.py`, trimmed to the arguments that matter. `docs` is the list of 20 passages, `context` is the same passages joined with newlines, and `question` is the question.

```python
budget = round(0.25 * count_tokens(context))   # the same token target for both baselines

# LLMSlim 0.8.0: lexical (default) and with the optional reranker
compress(context, target_ratio=0.25, query=question)
compress(context, target_ratio=0.25, query=question, relevance="semantic")

# LLMLingua-2, held to the same token count
lingua2 = PromptCompressor(model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
                           use_llmlingua2=True)
lingua2.compress_prompt(docs, target_token=budget,
                        force_tokens=["\n", "?", ".", "!"], drop_consecutive=True)

# LongLLMLingua with the paper's NaturalQuestions settings; LLaMA-2-7B-chat in 4-bit NF4
quant = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                           bnb_4bit_compute_dtype=torch.float16)
longlingua = PromptCompressor(model_name="NousResearch/Llama-2-7b-chat-hf", device_map="auto",
                              model_config={"quantization_config": quant})
longlingua.compress_prompt(docs, question=question, target_token=budget,
                           condition_in_question="after_condition", reorder_context="sort",
                           dynamic_context_compression_ratio=0.3, condition_compare=True,
                           context_budget="+100", rank_method="longllmlingua",
                           concate_question=False)
```

Notes on each arm:

- **Full context:** all 20 passages, uncompressed.
- **Passages in retrieval order:** whole passages in their original order until the token count LLMSlim produced is used up. This is the naive cut.
- **LLMLingua-2:** asked first for `rate=0.25`, it kept 16% of the tokens (469) and scored 64.7 on test_a. Held to the same token count with `target_token`, it scored the same 64.7. The table reports the equal-budget run.
- **LongLLMLingua:** we loaded LLaMA-2-7B-chat in 4-bit NF4 so it fits a 6 GB laptop GPU. The paper does not mention quantization, and llmlingua loads the model unquantized by default. Its key-value cache code predates transformers 5, so it runs in a separate environment with `transformers==4.44.2`, `accelerate==0.34.2` and `bitsandbytes`.
- **Inputs:** both baselines receive the 20 passages as a list, so they can rank whole documents. LLMSlim receives the joined string and finds the passage boundaries itself.

## 6. Accuracy results

Accuracy is the percentage of cases where a gold answer appears in Haiku's response. All rows use the same reader, prompt and cases.

| Compressor | Model it runs | test_a (150 cases) | test_b (150 cases, fresh) | All 300 cases | Mean tokens |
|---|---|---:|---:|---:|---:|
| Full context | none | 87.3 | 90.7 | 89.0 | 2,889 |
| **LLMSlim 0.8 + reranker** | 22M cross-encoder | 74.0 | **83.3** | **78.7** | 668 |
| LongLLMLingua (paper's NQ settings) | LLaMA-2-7B-chat, 4-bit | **80.0** | 76.0 | 78.0 | 630 |
| LLMLingua-2, same token budget | XLM-RoBERTa-large | 64.7 | 73.3 | 69.0 | 680 |
| LLMSlim 0.8, lexical | none | 63.3 | 74.0 | 68.7 | 658 |
| Passages in retrieval order | none | 49.3 | 47.3 | 48.3 | 627 |

Read it in this order. Compression at a quarter of the tokens costs accuracy on this task: the best compressed result is about 10 points under full context. LLMSlim with the reranker and LongLLMLingua finish level, each winning one split. LLMSlim with the reranker is ahead of LLMLingua-2 on both splits. LLMSlim without a model is level with LLMLingua-2, not ahead of it. Every compressor is far ahead of the naive cut.

On the full 300-case splits, where the baselines were not run, LLMSlim with the reranker scored 75.0 (test_a) and 75.7 (test_b), lexical LLMSlim 65.7 and 67.3, and full context 86.3 and 88.0.

### Beyond NaturalQuestions

The two baselines were run on NaturalQuestions only. The other suites in the README compare LLMSlim with full context and with a same-budget cut, under the same reader:

| Suite and cases | LLMSlim budget | Full context | LLMSlim + reranker | LLMSlim lexical | Same-budget cut |
|---|---|---:|---:|---:|---:|
| LongMemEval-S, test_a / test_b (90 each) | 8,000 tokens of ~114k | 82.2 / 72.2 | 76.7 / 76.7 | 68.9 / 67.8 | 14.4 / 14.4 |
| HotpotQA distractor F1, test_a / test_b (200 each) | 40% | 74.9 / 77.5 | 69.4 / 71.8 | 67.5 / 70.1 | 49.5 / 41.1 |
| LongBench v1, macro over 7 tasks, test_a / test_b | 3,000 tokens | 54.9 / 55.1 | 54.2 / 55.4 | 54.9 / 55.0 | 45.2 / 46.8 |
| LongBench v2 short split (180) | 25% | 60.0 | 55.6 | 53.9 | 48.3 |

The cut is the most recent sessions for LongMemEval-S, middle truncation for LongBench v2 and plain truncation elsewhere.

- **LongMemEval-S:** with the reranker, the difference from full history is within noise on both samples (−5.6, CI −15.6 to +4.4; +4.4, CI −6.7 to +15.6). Reader cost per 1,000 questions at Haiku 5.5 list prices was $1.45 against $90.72, 62 times lower, but part of that is the higher rate Haiku charges for prompts over 100K tokens, which every full-history prompt is. By token count the reduction is about 14 times (114,006 to 7,984). LLMSlim Platform 0.9.1 with its defaults scored 63.3 on test_a.
- **HotpotQA:** full context is better on F1 on both samples: −5.5 points for LLMSlim with the reranker on test_a (CI −10.0 to −1.0) and −5.6 on test_b (CI −9.9 to −1.6), on 43% of the input tokens.
- **LongBench v1:** lexical LLMSlim has the same macro score as full context on about a third of the input tokens, 5,232 against 15,532 (+0.0, CI −4.5 to +4.6 on test_a; −0.1, CI −4.2 to +4.1 on test_b). MuSiQue is still below full context.
- **LongBench v2:** with the reranker, −4.4 against full context (CI −11.7 to +2.8); middle truncation to the same budget lost 11.7 (CI −20.6 to −3.3).
- **SWE-bench Verified Mini** (50 issues, a minimal bash-tool agent): with LLMSlim 0.8.0 compressing command outputs over 800 tokens, prompts averaged 51,332 tokens per task, against 58,117 and 48,126 in two uncompressed runs ($0.0029 per task against $0.0032 and $0.0027). That is little saving: Haiku keeps its context near 8k tokens with targeted commands, so there is not much to remove. The README also lists resolved counts, but the grading reports behind them are not committed yet, so they are left out here.
- **InjecAgent (Platform):** Haiku 5.5 resisted all 1,054 attacks without help, so this benchmark shows no defence effect with this model.

## 7. Runtime and hardware tradeoffs

Accuracy is only half of the comparison. What each compressor needs in order to run:

| Compressor | Model it runs | Where it ran | Median time to compress one case |
|---|---|---|---:|
| LLMSlim 0.8, lexical | none | CPU | 25 ms |
| LLMSlim 0.8 + reranker | cross-encoder, 22.7M parameters | GPU | 0.09 s |
| LLMLingua-2 | XLM-RoBERTa-large, about 560M parameters | GPU | 2.3 s |
| LongLLMLingua | LLaMA-2-7B-chat, 6.7B parameters, 4-bit NF4 | GPU | 9.3 s |

These are wall-clock compression times per case, excluding the reader call: the median over the 300 cases of the `build_ms` field in the committed result files, on one machine with a 6 GB laptop GPU. Medians, because the first call of each run also loads the model (up to 31.6 s for the reranker and 71.8 s for LongLLMLingua), which inflates the means. The benchmark README quotes slightly different figures (0.1 s, 9.5 s, 2.9 s and 23 ms), mostly from test_a alone; the medians above cover all 300 cases. One difference in setup: the reranker arm ran as two worker processes sharing the GPU, the baselines as one.

The reader call is the same for every arm. Reader cost per 1,000 cases was $0.32 for LLMSlim with the reranker, $0.31 for LongLLMLingua and $0.40 for LLMLingua-2, against $0.69 for full context.

- **Same accuracy, very different speed.** On the 300 cases where they are level, LongLLMLingua's median was 9.3 s per case and LLMSlim's reranker 0.09 s on the same GPU: about 100 times faster per case in wall-clock time. That is a speed measurement, not a count of operations; by parameters the reranker is about 300 times smaller.
- **A 7B model is a deployment decision.** LongLLMLingua's accuracy comes from LLaMA-2-7B-chat, which we had to load in 4-bit to fit a 6 GB GPU at all. In production that is a GPU per compression worker, or a queue in front of one.
- **LLMLingua-2 on CPU.** The first LLMLingua-2 run, at `rate=0.25` on test_a, ran on CPU and took a median of 41.7 s per case. The equal-budget run in the table used the GPU.
- **No model at all.** Lexical LLMSlim needs no model and no download and took a median of 25 ms per case on CPU. On this task it matched LLMLingua-2; it did not exceed it.

## 8. What the confidence intervals show

Each row is the paired difference in accuracy points, first system minus second, with the bootstrap 95% interval in brackets. The first three columns resample cases, as the benchmark README does. The last column resamples whole questions instead; the method note below explains why.

| Comparison | test_a (150 cases) | test_b (150 cases) | All 300, resampling cases | All 300, resampling questions |
|---|---:|---:|---:|---:|
| LLMSlim + reranker vs LongLLMLingua | −6.0 (−14.7, +2.0) | +7.3 (+0.7, +14.7) | +0.7 (−4.7, +6.0) | +0.7 (−8.0, +9.3) |
| LLMSlim + reranker vs LLMLingua-2 | +9.3 (+0.7, +18.0) | +10.0 (+1.3, +19.3) | +9.7 (+3.7, +15.7) | +9.7 (+0.0, +19.3) |
| LLMSlim lexical vs LLMLingua-2 | −1.3 (−8.7, +6.0) | +0.7 (−9.3, +10.7) | −0.3 (−6.7, +6.0) | −0.3 (−9.7, +9.3) |

- **LLMSlim with the reranker vs LLMLingua-2:** resampling cases, the interval excludes zero on each split and overall. Resampling whole questions, the pooled lower bound falls to about zero. Per question, averaging its three positions, LLMSlim with the reranker did better on 27 questions and worse on 13, with 60 tied; a two-sided sign test gives p = 0.04. The lead holds under both methods, but its size is uncertain: anywhere from about zero to about 19 points.
- **LLMSlim with the reranker vs LongLLMLingua:** −6.0 on test_a and +7.3 on test_b. Pooled, +0.7, with an interval from −4.7 to +6.0 by case and −8.0 to +9.3 by question. Per question it is 15 better, 16 worse and 69 tied. The honest description is "level": the data cannot rule out a gap of several points either way.
- **One split can tell either story.** Each split holds only 50 questions. The test_b interval against LongLLMLingua (+0.7 to +14.7) only just excludes zero, and test_a went the other way. A report built on one split would have favoured whichever system won it, which is why both splits are reported and pooled.
- **Lexical LLMSlim:** level with LLMLingua-2 (−0.3), and 9.3 points behind LongLLMLingua (case-level interval −15.7 to −3.0).

**Method note.** Each case gives a paired difference of −100, 0 or +100 points. The case-level bootstrap draws 300 cases with replacement, takes the mean, repeats that 4,000 times and reads the 2.5th and 97.5th percentiles. But the three cases built from one question are correlated: a question that is easy at position 1 tends to be easy at 10 and 20. The question-level bootstrap draws 100 questions with replacement, each bringing its three cases, which is the more conservative choice and gives wider intervals. Bounds move by a few tenths of a point with the random draws.

The reader adds noise of its own. The benchmark README reports that on 75 cases where two LLMSlim builds sent an identical prompt, Haiku's answers still differed 5 times. That check needs the built prompts, which are not committed, so it cannot be rerun from the repository yet.

## 9. What this does not prove

- **One reader.** Every number uses Claude Haiku 5.5 at `effort: low`. Another model, or more reasoning effort, may read compressed text differently, and section 10 suggests that reading is exactly where these systems differ.
- **One dataset family for the head-to-head.** LLMLingua-2 and LongLLMLingua were run only on 20-document NaturalQuestions: short factual answers, one gold passage. That rewards keeping the right passage and says little about summarisation, multi-hop reasoning, code or chat memory, where the baselines were not measured.
- **We ran the baselines ourselves.** We used published settings (LongLLMLingua's NaturalQuestions settings from the paper, LLMLingua-2's MeetingBank model), and we build one of the systems being compared. Other settings may score differently. The configuration is in section 5 and the code is public so that it can be checked.
- **LongLLMLingua ran in 4-bit.** We used 4-bit NF4 to fit a 6 GB laptop GPU. The paper does not mention quantization, and 4-bit can change the perplexity scores LongLLMLingua ranks with. We did not measure how much, in either direction.
- **Small samples.** 150 cases per split, from 50 questions each, and 100 questions in all. The intervals are several points wide, and wider again when whole questions are resampled (section 8).
- **Cut-off responses.** On the 300 cases, 14 LLMLingua-2 responses hit the 1,024-token output limit, against 10 for full context, 5 for LLMSlim with the reranker, 2 for lexical LLMSlim and 1 for LongLLMLingua. A response cut off before it names the answer scores 0, so this may cost LLMLingua-2 a few points.
- **LLMSlim's lexical mode is not better than LLMLingua-2.** 68.7 against 69.0 over 300 cases. The model-free default is faster, not more accurate, on this task; the accuracy result needs the reranker.
- **Compression still costs accuracy here.** Full context scored 89.0 on the same 300 cases. At a quarter of the tokens, every compressor gave up about 10 points or more.
- **Weak spots elsewhere.** On LongMemEval-S, preference questions are LLMSlim's weakest type: 40.0 with the reranker against 86.7 with the full history on test_a. With 15 questions per type the swings are large; on test_b the same type scored 73.3 against 53.3. On LongBench v1, MuSiQue is still below full context (58.8 lexical, 61.2 with the reranker, 64.9 full).
- **Timings come from one machine.** Absolute times will differ on other hardware and batch sizes; the ratios are the useful part, and they are wall-clock ratios, not operation counts.

## 10. Error analysis

**The split reversal.** LongLLMLingua won test_a by 6.0 points and LLMSlim won test_b by 7.3. With 50 questions per split, a swing of 6 or 7 points is within sampling noise for either side, but the benchmark README also gives a mechanism for test_a: the gap came from reading, not retention. A gold answer was still present in LLMSlim's compressed context more often than in LongLLMLingua's, but the reader turned a present answer into a correct one less often. These figures come from the prompts that were actually sent, which are not committed, so unlike the rest of this article they cannot yet be rerun from the repository:

| test_a, 150 cases (per the benchmark README) | Gold answer present in the compressed context | Reader correct when it was present | Accuracy |
|---|---:|---:|---:|
| LLMSlim 0.8 + reranker | 79.3% | 89.9% | 74.0 |
| LongLLMLingua | 75.3% | 95.6% | 80.0 |

LLMSlim keeps the best units whole and then fills the rest of the budget sentence by sentence, so an answer can arrive as a fragment with little around it. LongLLMLingua reorders documents by relevance and keeps more complete passages, and the reader answered from those more reliably.

**Two fixes tried and dropped.** Both targeted that gap and were tested with the reader on 300 dev cases, where the reranker arm scored 83.0 and full context 88.7:

| Idea, dev split | Accuracy | Change |
|---|---:|---:|
| Ordering documents by relevance, as LongLLMLingua does | 80.3 | −2.7 |
| Sharing a document's best reranker score with its other chunks | 83.3 | +0.3 |

Neither helped enough to ship, so 0.8.0 does not include them, and the test splits were not used to look for a better one.

**Gold position.** Pooling both splits gives 100 cases per position. These figures are computed from `results/nq.jsonl.gz` and `results/nq_b.jsonl.gz`:

| Gold passage position | 1st | 10th | 20th |
|---|---:|---:|---:|
| Full context | 90.0 | 88.0 | 89.0 |
| LLMSlim 0.8 + reranker | 81.0 | 77.0 | 78.0 |
| LongLLMLingua | 81.0 | 75.0 | 78.0 |
| LLMLingua-2, same token budget | 65.0 | 70.0 | 72.0 |
| LLMSlim 0.8, lexical | 69.0 | 69.0 | 68.0 |

With 100 cases per cell, differences of a few points within a row are noise. The useful observation is that none of the compressors collapses when the answer sits 10th or 20th, as the retrieval-order cut does (section 1).

**LLMLingua-2's budget did not limit it.** At `rate=0.25` it kept 469 tokens and scored 64.7 on test_a; held to LLMSlim's budget, about 45% more tokens, it scored exactly the same. On this task, which tokens it keeps without reading the question seems to matter more than how many.

## 11. Reproduce it

Everything runs from the repository root, so the scripts import the checkout's `llmslim`. The API key is read from a file at run time and never printed or written. The datasets are not committed; put them under `benchmarks/live/data/` as the scripts expect:

- NaturalQuestions 20-document files: `nelson-liu/lost-in-the-middle`, the files with the gold passage at index 0, 9 and 19, saved as `data/nq20/gold_at_0.jsonl.gz`, `gold_at_9.jsonl.gz` and `gold_at_19.jsonl.gz`
- HotpotQA distractor validation, rows 0 to 800; LongBench v1: `THUDM/LongBench`; LongBench v2: `zai-org/LongBench-v2`
- LongMemEval: `xiaowu0162/longmemeval-cleaned`, file `longmemeval_s_cleaned.json`
- InjecAgent: the `data/` folder of `uiuc-kang-lab/InjecAgent`; SWE-bench Verified Mini: `MariusHobbhahn/swe-bench-verified-mini`

```bash terminal
git clone https://github.com/Thanatos9404/llmslim
cd llmslim
export PYTHONPATH="$PWD"                                     # the checkout is the package under test
export LLMSLIM_KEY_FILE=/path/to/file/with/CLAUDE_API_KEY   # never printed or written
export HAIKU_CAP_USD=60                                      # hard spend cap across all suites

python benchmarks/live/devcheck.py --suites nq hotpot lb lme   # dev split only, offline, no API calls
bash benchmarks/live/runs.sh                                   # LLMSlim arms on test_a and test_b
LINGUA_PY=/path/to/lingua-env/python LONGLINGUA_PY=/path/to/longlingua-env/python \
  bash benchmarks/live/competitors.sh                          # LLMLingua-2 and LongLLMLingua on NQ
python benchmarks/live/verify_built.py nq:core08f_global nq:core08s_global   # prompts match the source
python benchmarks/live/final_report.py                         # tables and paired CIs, no API calls
```

- **Environments.** The LLMSlim arms need `anthropic`, `tiktoken` and, for the reranker, `sentence-transformers`. LLMLingua-2 runs with `llmlingua` and CUDA PyTorch (`LINGUA_DEVICE=cuda`). LongLLMLingua needs its own environment with `llmlingua`, `transformers==4.44.2`, `accelerate==0.34.2` and `bitsandbytes`. See `competitors.sh`.
- **Without an API key.** Per-case results are committed in `results/*.jsonl.gz`, with predictions truncated to 400 to 600 characters. `final_report.py` prints the NaturalQuestions and HotpotQA tables from them with no data and no API calls; its LongBench part needs the LongBench files under `data/longbench`.
- **The pairwise intervals.** `final_report.py` compares each arm with full context and the same-budget cut; it does not print the arm-against-arm comparisons or the pooled 300-case column. Section 8 uses the same paired bootstrap on the same files. This computes the pooled LLMSlim vs LongLLMLingua row, by case and by question:

```python paired_ci.py
import gzip, json, random
from collections import defaultdict

def head_to_head(path):
    by_id = defaultdict(dict)
    for line in gzip.open(path, "rt", encoding="utf-8"):
        if line.strip():
            row = json.loads(line)
            by_id[row["id"]][row["arm"]] = row["score"]
    return {(path, i): arms for i, arms in by_id.items() if "longllmlingua" in arms}

cases = head_to_head("benchmarks/live/results/nq.jsonl.gz") | head_to_head("benchmarks/live/results/nq_b.jsonl.gz")
diff = {key: 100 * (arms["core08s_global"] - arms["longllmlingua"]) for key, arms in cases.items()}

by_question = defaultdict(list)
for (path, case_id), d in diff.items():
    by_question[(path, case_id.split("-")[0])].append(d)

def interval(groups, draws=4000):
    """Paired bootstrap: resample whole groups with replacement, 95% percentile interval of the mean."""
    rng = random.Random(7)
    means = []
    for _ in range(draws):
        sample = [d for _ in groups for d in rng.choice(groups)]
        means.append(sum(sample) / len(sample))
    means.sort()
    return f"{means[100]:+.1f} to {means[3900]:+.1f}"

print(f"{len(diff)} cases, {len(by_question)} questions, {sum(diff.values()) / len(diff):+.1f} points")
print("resampling cases:    ", interval([[d] for d in diff.values()]))
print("resampling questions:", interval(list(by_question.values())))
```

```text output
300 cases, 100 questions, +0.7 points
resampling cases:     -5.0 to +6.0
resampling questions: -8.0 to +9.3
```

Bootstrap bounds move by a few tenths of a point with the random draws; the README's −4.7 to +6.0 comes from the same case-level procedure. Replace `"longllmlingua"` with `"lingua2_eq"` in the `diff` line for the LLMLingua-2 row, which prints +9.7 points, +3.7 to +15.7 by case and +0.0 to +19.3 by question.

## 12. Try it

Core is on PyPI. The default install has no model dependencies; the reranker is an extra.

```bash terminal
pip install llmslim                  # lexical relevance: no model, no download
pip install "llmslim[semantic]"      # adds the optional reranker (sentence-transformers)
```

`relevance="semantic"` turns the reranker on for `compress`, and the same keyword works with `compress_documents` for a list of retrieved passages:

```python try_it.py
from llmslim import compress

passages = [
    "Document [1](Title: Lake Baikal) Lake Baikal in Siberia is the deepest lake in the world, reaching 1,642 metres. It holds roughly a fifth of the fresh surface water on Earth.",
    "Document [2](Title: Lake Titicaca) Lake Titicaca sits on the border of Peru and Bolivia. At about 3,800 metres it is often described as the highest navigable lake.",
    "Document [3](Title: Caspian Sea) The Caspian Sea is the largest inland body of water by area. It is bordered by five countries and its water is brackish rather than fresh.",
    "Document [4](Title: Lake Superior) Lake Superior is the largest of the Great Lakes of North America. By surface area it is the largest freshwater lake in the world.",
    "Document [5](Title: Lake Victoria) Lake Victoria is the largest lake in Africa. Its basin is shared by Uganda, Kenya and Tanzania, and it feeds the White Nile.",
    "Document [6](Title: Great Salt Lake) The Great Salt Lake in Utah is the largest saltwater lake in the Western Hemisphere. Its level rises and falls with the seasons.",
]
question = "Which freshwater lake has the greatest surface area?"

result = compress("\n".join(passages), target_ratio=0.25, query=question, relevance="semantic")
print(f"{result.original_tokens} -> {result.compressed_tokens} tokens")
print(result.compressed_text)
```

Output, from a clean Python 3.12 virtual environment with `pip install "llmslim[semantic,fast-tokens]==0.8.0"` from PyPI:

```text output
236 -> 35 tokens
Document [4](Title: Lake Superior) Lake Superior is the largest of the Great Lakes of North America. By surface area it is the largest freshwater lake in the world.
```

The `fast-tokens` extra adds `tiktoken` for exact token counts; without it LLMSlim estimates tokens from characters, prints a warning, and may keep a slightly different selection. The first semantic call downloads the cross-encoder from Hugging Face if it is not cached, and the model loads once per process.

The full tables are on the [results page](/results) and in the [benchmark README](https://github.com/Thanatos9404/llmslim/blob/main/benchmarks/live/README.md). If a rerun gives you different numbers, please [open an issue](https://github.com/Thanatos9404/llmslim/issues) with the command and the output.

## Key takeaways

- On 300 paired cases (100 NaturalQuestions questions, each with the gold passage 1st, 10th and 20th) at a 25% token budget, LLMSlim 0.8 with its reranker scored 78.7 and LongLLMLingua 78.0. That is level: the difference is +0.7 points, 95% CI −4.7 to +6.0.
- Against LLMLingua-2 held to the same token count, LLMSlim with the reranker was 9.7 points ahead (95% CI +3.7 to +15.7, or about 0 to +19 when whole questions are resampled). Without any model, LLMSlim was level with LLMLingua-2 (68.7 vs 69.0).
- Median wall-clock compression time per case, on the same 6 GB laptop GPU: 0.09 s for LLMSlim's 22M-parameter reranker, 2.3 s for LLMLingua-2 and 9.3 s for LongLLMLingua's 4-bit LLaMA-2-7B-chat. At equal accuracy, LLMSlim was about 100 times faster per case than LongLLMLingua.
- Full context still scored 89.0 on the same cases. One reader, one dataset family and baselines we ran ourselves limit what these numbers show; section 9 lists the limits.
- Splits, scripts, per-case results and spend ledgers (about $30 of API calls in total) are in benchmarks/live in the LLMSlim repository.

## References

- [Pan et al. (2024)] [LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression (arXiv:2403.12968)](https://arxiv.org/abs/2403.12968)
- [Jiang et al. (2023)] [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (arXiv:2310.06839)](https://arxiv.org/abs/2310.06839)
- [microsoft/LLMLingua] [LLMLingua, LongLLMLingua and LLMLingua-2 source code (Microsoft, GitHub)](https://github.com/microsoft/LLMLingua)
- [Liu et al. (2023)] [Lost in the Middle: How Language Models Use Long Contexts (arXiv:2307.03172), source of the 20-document NaturalQuestions files](https://arxiv.org/abs/2307.03172)
- [ms-marco-MiniLM-L-6-v2] [cross-encoder/ms-marco-MiniLM-L-6-v2, the optional LLMSlim reranker (Hugging Face)](https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-6-v2)
- [LLMSlim benchmarks] [benchmarks/live: README, scripts, splits and per-case results (GitHub)](https://github.com/Thanatos9404/llmslim/tree/main/benchmarks/live)

Canonical page: https://www.llmslim.app/articles/llmslim-vs-llmlingua-2-vs-longllmlingua