# LLMSlim + LlamaIndex

> A BaseNodePostprocessor for query engines. Use it where you would use LongLLMLinguaPostprocessor.

LLMSlimPostprocessor is a LlamaIndex BaseNodePostprocessor backed by llmslim.compress_documents. It keeps the nodes that answer the query inside a token budget, runs locally with no model and no API call, and keeps each node's score, id and metadata. Python 3.10+.

Category: Framework

## Install

```bash
pip install "llmslim[llamaindex]"   # llama-index-core; Python 3.10+; needs llmslim 0.8.1 or newer
```

The extra installs llama-index-core 0.12 or newer and needs Python 3.10 or newer: llama-index-core dropped Python 3.9 in 0.14.17, and its last 3.9 release does not import on 3.9 (its banks dependency uses 3.10 syntax).

## How it fits

- Add it to a query engine's node_postprocessors, or call postprocess_nodes() on its own.
- _postprocess_nodes(nodes, query_bundle) calls llmslim.compress_documents on the nodes' text, without metadata.
- With a query, the nodes share one budget (target_ratio of their combined tokens) and it goes where the query points: the most relevant nodes stay whole and irrelevant ones are dropped.
- Without a query, each node is compressed to target_ratio on its own.
- Kept nodes keep their order, score, node_id and metadata. Nodes that are empty or compress to nothing are dropped. The input nodes are not modified.
- The token counts added to the metadata are listed in excluded_llm_metadata_keys and excluded_embed_metadata_keys, so they never reach the LLM prompt or the embedder.

## Example

```python
from llama_index.core import Document, VectorStoreIndex
from llama_index.core.embeddings import MockEmbedding
from llama_index.core.llms import MockLLM

from llmslim.integrations.llama_index import LLMSlimPostprocessor

documents = [
    Document(
        text="Standard shipping takes three to seven business days within the country. "
        "Express shipping arrives in one or two business days and costs fifteen dollars. "
        "We ship to forty countries, and international orders clear customs in about a week. "
        "Tracking numbers are emailed as soon as the parcel leaves the warehouse.",
        metadata={"source": "shipping.md"},
    ),
    Document(
        text="You can return any item within 30 days of delivery for a full refund. "
        "Refunds go back to the original payment method within five business days. "
        "Items must be unused and in their original packaging. "
        "Gift cards and final-sale items cannot be returned.",
        metadata={"source": "refunds.md"},
    ),
    Document(
        text="The company was founded in 2015 by two engineers in Pune. "
        "Our team of eighty people works across design, engineering and support. "
        "The office has a rooftop garden and a small library of design books. "
        "Every Friday afternoon the team hosts an open demo for customers.",
        metadata={"source": "about.md"},
    ),
]

# MockEmbedding and MockLLM keep the example offline; use your own models.
index = VectorStoreIndex.from_documents(documents, embed_model=MockEmbedding(embed_dim=8))
query_engine = index.as_query_engine(
    llm=MockLLM(),
    similarity_top_k=3,
    node_postprocessors=[LLMSlimPostprocessor(target_ratio=0.25)],
)

response = query_engine.query("How many days do I have to return an item for a refund?")
for item in response.source_nodes:
    print(round(item.score, 3), item.node.metadata)
    print(item.node.get_content())
```

Output (Python 3.12, llama-index-core 0.14.25, token counts from tiktoken):

```text
1.0 {'source': 'refunds.md', 'llmslim_original_tokens': 50, 'llmslim_compressed_tokens': 40}
You can return any item within 30 days of delivery for a full refund. Refunds go back to the original payment method within five business days. Items must be unused and in their original packaging.
```

The three retrieved nodes share one budget of 25% of their 158 tokens (40 tokens). The shipping and company nodes do not answer the question and are dropped; the refund node keeps the answer and loses its sentence about gift cards. The score is the retriever's (the mock embedding scores every node 1.0).

## Options

| Option | Default | Meaning |
|---|---|---|
| `target_ratio` | `0.25` | Fraction of tokens to keep. |
| `relevance` | `"lexical"` | "lexical" (BM25, no model), "semantic" (cross-encoder, needs llmslim[semantic]) or a callable scorer(query, texts) -> scores. |
| `selection` | `"auto"` | "proportional" gives each node its own share, as in 0.7. |
| `context_role` | `"rag"` | Retrieved text is untrusted: imperative wording in it is never force-kept. |
| `record_token_counts` | `True` | Adds llmslim_original_tokens and llmslim_compressed_tokens to the metadata. |
| `additional_compress_kwargs` | `{}` | Other llmslim.compress_documents arguments, for example mode. |

## Deployment

Place the compressor between retrieval and the prompt, and pass the user's question as the query so the retrieved texts share one budget. Start at the default target_ratio of 0.25 and check answer quality on your own questions before going lower. The default lexical relevance needs no model; when questions and answers use different words, install llmslim[semantic] and pass relevance="semantic" to add the optional 22M-parameter cross-encoder.

## Tips

- Pass the user's question as the query. Without one, each text is compressed to target_ratio on its own instead of competing for one budget.
- Keep the lexical default when you need deterministic output and no model download; switch to relevance="semantic" when questions and answers share few words.
- Keep context_role at "rag" for retrieved text: imperative wording inside untrusted documents is never force-kept.
- Set record_token_counts=False if downstream code should not see the llmslim_original_tokens and llmslim_compressed_tokens metadata keys.

## Limitations

- Needs Python 3.10 or newer. On Python 3.9 the llamaindex extra installs nothing, because llama-index-core no longer supports 3.9.
- Needs llmslim 0.8.1 or newer and llama-index-core 0.12 or newer. The 0.8.1 release was tested with llama-index-core 0.12.0 and 0.14.25.
- Extractive: it keeps or drops whole documents and sentences, copied from the input. It does not shorten or rewrite the sentences it keeps.
- Compression costs some accuracy. On NaturalQuestions at a quarter of the tokens, LLMSlim with its optional reranker scored 78.7% against 89.0% for the full context, read by Claude Haiku 5.5. Measure on your own questions before choosing a ratio.
- relevance="semantic" needs pip install "llmslim[semantic]" (sentence-transformers) and downloads a 22M-parameter cross-encoder from Hugging Face on first use.
- The example uses MockEmbedding and MockLLM to stay offline. Which nodes reach the postprocessor depends on your real embedding model and retriever.

## FAQ

### Can LLMSlimPostprocessor replace LongLLMLinguaPostprocessor?

Yes, in the same node_postprocessors slot. Both are LlamaIndex BaseNodePostprocessor classes. LLMSlimPostprocessor takes a fraction of tokens to keep (target_ratio) rather than a token count, needs no language model by default, and keeps the nodes in their retrieved order. The LLMSlim vs LLMLingua page compares the two approaches on a public benchmark.

### Why does it need Python 3.10?

llama-index-core dropped Python 3.9 in 0.14.17, and its last 3.9 release does not import on 3.9 because its banks dependency uses 3.10 syntax. The rest of LLMSlim still supports Python 3.9.

### Does it call a model or an API?

No. The default lexical relevance is BM25 in your process, with no model and no network call. relevance="semantic" runs a 22M-parameter cross-encoder locally after pip install "llmslim[semantic]". Your documents are not sent anywhere either way.

## Troubleshooting

- **ImportError: llmslim.integrations.llama_index requires the optional extra:** Install the extra on Python 3.10 or newer: pip install "llmslim[llamaindex]". The module is llmslim.integrations.llama_index; the extra is llamaindex.
- **Every document comes back cut to the same fraction:** The query was empty, so each document was compressed to target_ratio on its own. Pass the user's question as the query so the documents share one budget and the irrelevant ones are dropped.

## Further reading

- [LlamaIndex guide on GitHub](https://github.com/Thanatos9404/llmslim/blob/main/docs/integrations/LLAMAINDEX.md)
- [Adapter source: llmslim/integrations/llama_index.py](https://github.com/Thanatos9404/llmslim/blob/main/llmslim/integrations/llama_index.py)
- [LLMSlim vs LLMLingua](https://www.llmslim.app/compare/llmlingua.md)
- [Benchmarks](https://www.llmslim.app/benchmarks.md)

Canonical page: https://www.llmslim.app/integrations/llamaindex