LLMSlim vs LLMLingua. An honest comparison.

Both are open source in microsoft/LLMLingua under the MIT license. We ran their public code ourselves; Microsoft and the papers' authors did not run, review or endorse this comparison. Code: github.com/microsoft/LLMLingua.

1. 01

   Selection, not token pruning

   LLMSlim ranks the context's units (documents, chat rounds, paragraphs) against the query, keeps the best ones whole and in their original order, and fills the rest of the budget sentence by sentence. It never deletes words inside a sentence, so the model reads text from your documents, unchanged.

2. 02

   No 7B model

   Relevance is lexical by default: BM25 with light stemming and one round of pseudo-relevance feedback. No model, no download and deterministic output, at a median of 25 ms per question on CPU in our NaturalQuestions run.

3. 03

   An optional 22M reranker

   pip install "llmslim[semantic]" and relevance="semantic" add cross-encoder/ms-marco-MiniLM-L-6-v2 (22.7M parameters, trained on MS MARCO web search). It rescores the 80 strongest lexical candidates, so it works on top of BM25. Any callable scorer(query, texts) works too.

4. 04

   One budget across documents

   With a query, retrieved documents share one budget: the relevant ones stay whole and the rest are dropped, instead of every document losing the same share. Retrieved text keeps untrusted RAG provenance, so imperative wording inside it is never force-kept.

NaturalQuestions with 20 retrieved passages and a 25% budget; full context scores 87.3 and 90.7 on the two sets. Each set is 50 questions × 3 answer positions (1st, 10th, 20th), so the 300 cases come from 100 questions. Over all 300 cases LLMSlim with the reranker and LongLLMLingua are level (78.7 and 78.0: +0.7 points, 95% interval −4.7 to +6.0), and LLMSlim is ahead of LLMLingua-2 by 9.7 points (95% interval 3.7 to 15.7). LongLLMLingua uses the paper's NQ settings with LLaMA-2-7B-chat; the paper does not state its precision, and we ran it in 4-bit NF4 to fit a 6 GB GPU. Tokens are means and times are medians over the 300 cases, to compress one question on our machine (LLMLingua-2's CPU time is the mean of a separate CPU run on the original set).

Paired differences with 95% intervals

Paired differences in accuracy points, first system minus second, with the bootstrap 95% interval in brackets (4,000 resamples). Each question appears at three answer positions, so the three cases are correlated: resampling whole questions is the more conservative interval. Bounds move by a few tenths of a point with the random draws.

What this does not show

One reader

Every number uses Claude Haiku 5.5 at effort low. Another reader may rank these systems differently. The papers' own NaturalQuestions numbers use GPT-3.5-Turbo, so raw scores are not comparable across the two.

One dataset for the competitors

LLMLingua-2 and LongLLMLingua were run on 20-document NaturalQuestions only: short factual answers, one gold passage. That says little about summarisation, multi-hop reasoning, code or chat memory, where they were not measured.

Baselines we ran ourselves

We used published settings (LongLLMLingua's NaturalQuestions settings from its paper, LLMLingua-2's MeetingBank checkpoint), and we build one of the systems compared. Other settings may score differently. The configuration and scripts are public.

LongLLMLingua in 4-bit

LLaMA-2-7B-chat ran in 4-bit NF4 to fit a 6 GB laptop GPU. The paper does not report quantization, and 4-bit can change the perplexity scores LongLLMLingua ranks with. We did not measure the effect, in either direction.

Small samples

100 questions (50 per split), each at three answer positions. A 6 to 7 point swing between splits is within sampling noise, which is why both splits are reported and pooled.

Same budget, slightly different use

Every arm had the same nominal budget, 25% of the context's cl100k tokens (about 722), and all came in under it: a mean of 668 tokens for LLMSlim with the reranker, 630 for LongLLMLingua and 680 for LLMLingua-2.

Cut-off responses

14 LLMLingua-2 responses hit the 1,024-token output limit, against 5 for LLMSlim with the reranker and 1 for LongLLMLingua. A response cut off before it names the answer scores 0, so this may cost LLMLingua-2 a few points.

Timings from one machine

Wall-clock medians per question, excluding the reader call. "About 100× faster" is wall-clock speed, not a count of operations; by parameters the reranker is about 300× smaller than LLaMA-2-7B. The reranker ran as two worker processes sharing the GPU, the baselines as one.

Full methodology and error analysis Every benchmark Per-case results

• You compress one long prompt with no question

  LLMSlim's selection is designed around a query, and it keeps or drops whole sentences. If you compress instructions, few-shot examples or a transcript with nothing to rank against, and want every sentence shortened rather than some sentences dropped, task-agnostic token-level compression such as LLMLingua-2 is built for that.

• Your reader does better with reordered passages

  LongLLMLingua won our original sample by 6.0 points. The benchmark README's error analysis traces that gap to reading, not retention: the reader answered more reliably from LongLLMLingua's reordered, more complete passages. (Those retention figures need the prompts that were sent, which are not committed yet.) Ordering by relevance did not help LLMSlim on our development split (−2.7 points), so LLMSlim keeps the original order.

• You want a method with peer-reviewed papers

  LLMLingua-2 and LongLLMLingua are described and evaluated in peer-reviewed papers at ACL venues. LLMSlim has no paper: its evidence is a public harness with per-case results and spend ledgers.

• LongLLMLingua already works for you

  Over 300 cases the two were level (+0.7, 95% CI −4.7 to +6.0). If you already run a GPU for the 7B model and it meets your latency budget, switching buys speed and a smaller footprint, not a measured accuracy gain.

• Your text is mostly not in English

  LLMSlim's default lexical scoring uses English stemming and stopwords, and we have no published results on other languages. Evaluate both on your own data before choosing.

• Retrieval with a question, where you want the reader to see verbatim passages that match it, inside one shared budget.
• No GPU, or a tight latency budget: the default needs no model, and the reranker is 22M parameters rather than 7B.
• Deterministic output and a small install, with no PyTorch unless you add the reranker.
• LangChain, LlamaIndex or Haystack pipelines, with the same adapter options in each.
• Agents and RAG where retrieved or tool text must stay untrusted when it is compressed.

# pip install langchain-community llmlingua
from langchain_classic.retrievers.contextual_compression import (
    ContextualCompressionRetriever,
)
from langchain_community.document_compressors import LLMLinguaCompressor

compressor = LLMLinguaCompressor(
    model_name="NousResearch/Llama-2-7b-hf",  # the class defaults
    device_map="cuda",
    target_token=300,
)
retriever = ContextualCompressionRetriever(
    base_compressor=compressor, base_retriever=base_retriever
)

# pip install "llmslim[langchain]" langchain-classic
from langchain_classic.retrievers.contextual_compression import (
    ContextualCompressionRetriever,
)
from llmslim.integrations.langchain import LLMSlimCompressor

compressor = LLMSlimCompressor(
    target_ratio=0.25,       # a fraction of the retrieved tokens
    # relevance="semantic",  # optional 22M reranker: llmslim[semantic]
)
retriever = ContextualCompressionRetriever(
    base_compressor=compressor, base_retriever=base_retriever
)

# pip install llama-index-postprocessor-longllmlingua
from llama_index.postprocessor.longllmlingua import LongLLMLinguaPostprocessor

# Loads NousResearch/Llama-2-7b-hf on CUDA unless you pass model_name and device_map.
postprocessor = LongLLMLinguaPostprocessor(
    instruction_str="Given the context, please answer the final question",
    target_token=300,
    rank_method="longllmlingua",
)
query_engine = index.as_query_engine(
    similarity_top_k=10, node_postprocessors=[postprocessor]
)

# pip install "llmslim[llamaindex]"   # Python 3.10+
from llmslim.integrations.llama_index import LLMSlimPostprocessor

postprocessor = LLMSlimPostprocessor(
    target_ratio=0.25,       # a fraction of the retrieved tokens
    # relevance="semantic",  # optional 22M reranker: llmslim[semantic]
)
query_engine = index.as_query_engine(
    similarity_top_k=10, node_postprocessors=[postprocessor]
)

# pip install llmlingua
from llmlingua import PromptCompressor

compressor = PromptCompressor(
    model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
    use_llmlingua2=True,
)
result = compressor.compress_prompt(docs, target_token=500)
context = result["compressed_prompt"]

# pip install llmslim
from llmslim import compress_documents

results = compress_documents(docs, query=question, target_ratio=0.25)
context = "\n\n".join(r.compressed_text for r in results)

# pip install "llmslim[haystack]"
from llmslim.integrations.haystack import LLMSlimDocumentCompressor

pipeline.add_component("compressor", LLMSlimDocumentCompressor(target_ratio=0.25))
pipeline.connect("retriever.documents", "compressor.documents")
pipeline.connect("compressor.documents", "prompt_builder.documents")
# Pass the query to the compressor too: {"compressor": {"query": query}}

• target_ratio is a fraction of the input's tokens, not a token count: 0.25 keeps a quarter.
• Kept documents gain llmslim_original_tokens and llmslim_compressed_tokens in their metadata; record_token_counts=False turns this off.
• Re-run your own evaluation after switching. Our results cover one dataset and one reader.

Sources

• LLMLingua-2 paper (arXiv:2403.12968)
• LongLLMLingua paper (arXiv:2310.06839)
• microsoft/LLMLingua on GitHub
• LLMLingua-2 checkpoint (Hugging Face)
• LangChain LLMLinguaCompressor source
• LlamaIndex LongLLMLinguaPostprocessor source
• LLMSlim reranker model card
• Benchmark README and harness
• Per-case results
• Script that reruns the competitors