Does it work with Haystack 3?

Yes. It is a Haystack 2.x component that also works with Haystack 3. When loading a saved pipeline in Haystack 3, allow the module: Pipeline.loads(yaml_text, allowed_modules=["llmslim.integrations.haystack"]).

Why is the query passed to both the retriever and the compressor?

Pipeline inputs are per component. The retriever needs the query to search; the compressor needs it so the documents share one budget and the irrelevant ones are dropped. Without a query, each document is compressed to target_ratio on its own.

Does it call a model or an API?

No. The default lexical relevance is BM25 in your process, with no model and no network call. relevance="semantic" runs a 22M-parameter cross-encoder locally after pip install "llmslim[semantic]". Your documents are not sent anywhere either way.

Back to integrations

FrameworkPipeline component

LLMSlim + Haystack

A Haystack 2.x and 3 component that compresses retrieved documents between the retriever and the prompt builder.

LLMSlimDocumentCompressor is a Haystack component backed by llmslim.compress_documents. It takes documents and an optional query, returns the kept documents inside a token budget, runs locally with no model and no API call, and keeps each document's id, score and meta.

1. Package installation

The extra installs haystack-ai 2.0 or newer. The component works with Haystack 2.x and Haystack 3.

terminalbash

pip install "llmslim[haystack]"   # haystack-ai; needs llmslim 0.8.1 or newer

2. Architecture & execution flow

Put it between a retriever and the prompt builder: connect "retriever.documents" to "compressor.documents", and "compressor.documents" to "prompt_builder.documents".

run(documents, query=None) calls llmslim.compress_documents on the documents' content and returns {"documents": [...]}.

With a query, the documents share one budget (target_ratio of their combined tokens) and it goes where the query points: the most relevant documents stay whole and irrelevant ones are dropped.

Without a query, each document is compressed to target_ratio on its own.

Kept documents keep their order, id, score and meta. Documents without text, or that compress to nothing, are dropped. The input documents are not modified.

3. Runnable example and its output

haystack_pipeline.pypython

from haystack import Document, Pipeline
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.document_stores.in_memory import InMemoryDocumentStore

from llmslim.integrations.haystack import LLMSlimDocumentCompressor

store = InMemoryDocumentStore()
store.write_documents(
    [
        Document(
            content="Standard shipping takes three to seven business days within the country. "
            "Express shipping arrives in one or two business days and costs fifteen dollars. "
            "We ship to forty countries, and international orders clear customs in about a week. "
            "Tracking numbers are emailed as soon as the parcel leaves the warehouse.",
            meta={"source": "shipping.md"},
        ),
        Document(
            content="You can return any item within 30 days of delivery for a full refund. "
            "Refunds go back to the original payment method within five business days. "
            "Items must be unused and in their original packaging. "
            "Gift cards and final-sale items cannot be returned.",
            meta={"source": "refunds.md"},
        ),
        Document(
            content="The company was founded in 2015 by two engineers in Pune. "
            "Our team of eighty people works across design, engineering and support. "
            "The office has a rooftop garden and a small library of design books. "
            "Every Friday afternoon the team hosts an open demo for customers.",
            meta={"source": "about.md"},
        ),
    ]
)

pipeline = Pipeline()
pipeline.add_component("retriever", InMemoryBM25Retriever(document_store=store, top_k=3))
pipeline.add_component("compressor", LLMSlimDocumentCompressor(target_ratio=0.25))
pipeline.connect("retriever.documents", "compressor.documents")

query = "How many days do I have to return an item for a refund?"
result = pipeline.run({"retriever": {"query": query}, "compressor": {"query": query}})
for doc in result["compressor"]["documents"]:
    print(round(doc.score, 3), doc.meta)
    print(doc.content)

Output (Python 3.12, haystack-ai 3.3.0, token counts from tiktoken):

outputtext

5.838 {'source': 'refunds.md', 'llmslim_original_tokens': 50, 'llmslim_compressed_tokens': 40}
You can return any item within 30 days of delivery for a full refund. Refunds go back to the original payment method within five business days. Items must be unused and in their original packaging.

The three retrieved documents share one budget of 25% of their 158 tokens (40 tokens). The shipping and company pages do not answer the question and are dropped; the refund page keeps the answer and loses its sentence about gift cards. The score is the BM25 retriever's.

4. Options

Keyword arguments, shown as name = default.

target_ratio = 0.25: Fraction of tokens to keep.
relevance = "lexical": "lexical" (BM25, no model), "semantic" (cross-encoder, needs llmslim[semantic]) or a callable scorer(query, texts) -> scores.
selection = "auto": "proportional" gives each document its own share, as in 0.7.
context_role = "rag": Retrieved text is untrusted: imperative wording in it is never force-kept.
record_token_counts = True: Adds llmslim_original_tokens and llmslim_compressed_tokens to meta.
additional_compress_kwargs = {}: Other llmslim.compress_documents arguments, for example mode.

5. Deployment best practices

Place the component between retrieval and the prompt builder, and pass the user's question to it as well as to the retriever. The component serializes with pipeline.dumps() unless relevance is a callable. Haystack 3 loads classes only from allowlisted modules: Pipeline.loads(yaml_text, allowed_modules=["llmslim.integrations.haystack"]). Start at the default target_ratio of 0.25 and check answer quality on your own questions before going lower.

6. Key optimization tips

Pass the user's question as the query. Without one, each text is compressed to target_ratio on its own instead of competing for one budget.
Keep the lexical default when you need deterministic output and no model download; switch to relevance="semantic" when questions and answers share few words.
Keep context_role at "rag" for retrieved text: imperative wording inside untrusted documents is never force-kept.
Set record_token_counts=False if downstream code should not see the llmslim_original_tokens and llmslim_compressed_tokens metadata keys.

7. Limitations

Needs llmslim 0.8.1 or newer and haystack-ai 2.0 or newer. The 0.8.1 release was tested with haystack-ai 2.0.0, 2.21.0 and 3.3.0.
A callable relevance scorer cannot be serialized with pipeline.dumps(). Use "lexical" or "semantic" in pipelines you save as YAML.
Extractive: it keeps or drops whole documents and sentences, copied from the input. It does not shorten or rewrite the sentences it keeps.
Compression costs some accuracy. On NaturalQuestions at a quarter of the tokens, LLMSlim with its optional reranker scored 78.7% against 89.0% for the full context, read by Claude Haiku 5.5. Measure on your own questions before choosing a ratio.
relevance="semantic" needs pip install "llmslim[semantic]" (sentence-transformers) and downloads a 22M-parameter cross-encoder from Hugging Face on first use.

Frequently asked questions

Troubleshooting

ImportError: llmslim.integrations.haystack requires the optional extra

Install the extra in the same environment: pip install "llmslim[haystack]". The component needs llmslim 0.8.1 or newer.

Haystack 3 refuses to load the component from YAML

Pass allowed_modules=["llmslim.integrations.haystack"] to Pipeline.loads().

Every document comes back cut to the same fraction

The query was empty, so each document was compressed to target_ratio on its own. Pass the user's question as the query so the documents share one budget and the irrelevant ones are dropped.