Back to Benchmarks Directory
Target: Multi-Model Evaluation SuiteBaseline: Uncompressed Prompt Baseline (0% Reduction)
LLMSlim Performance Across Dynamic Compression Ratios
Empirical Sweeps Across 20%, 40%, 50%, 60%, and 70% Token Reduction Targets
Detailed parameter sweep measuring trade-offs between target retention ratio (0.3 to 0.8), execution speed, semantic retention, and instruction fidelity.
System Environment & Rig Specification
CPU: AMD EPYC 7763 64-Core Processor @ 2.45GHz
RAM: 64 GB DDR4 ECC RAM
OS: Ubuntu 24.04 LTS (Linux kernel 6.8.0-31-generic)
Runtime: Python 3.12.3
Package: llmslim v0.2.0
Tokenizer: tiktoken v0.7.0 (cl100k_base / o200k_base)
Dataset Size: 500 prompts per evaluation dataset
Iterations: 100 runs per sample (P50/P95/P99 latency recorded)
Empirical Benchmark Matrix
*Costs are projected estimates; latencies and ratios are measured empirical values.| Method Variant | Token Reduction (Measured) | Execution Latency (Measured) | Billed Cost (Projected) | Semantic Retention (Measured) | Instruction Retention (Measured) | Entity Preservation (Measured) |
|---|---|---|---|---|---|---|
| Target Retention 80% (20% Reduction) | 20.1% +/- 0.4% | 18.4 ms +/- 1.2 ms | $20.00 USD (Projected) | 99.2% +/- 0.3% | 100.0% +/- 0.0% | 99.1% +/- 0.4% |
| Target Retention 60% (40% Reduction) | 40.3% +/- 0.6% | 21.6 ms +/- 1.5 ms | $15.00 USD (Projected) | 97.8% +/- 0.5% | 100.0% +/- 0.0% | 97.2% +/- 0.7% |
| Target Retention 50% (50% Reduction) | 50.4% +/- 0.8% | 24.8 ms +/- 2.1 ms | $12.50 USD (Projected) | 96.4% +/- 0.8% | 100.0% +/- 0.0% | 95.1% +/- 1.1% |
| Target Retention 40% (60% Reduction) | 60.2% +/- 0.9% | 27.2 ms +/- 2.3 ms | $10.00 USD (Projected) | 94.1% +/- 1.0% | 100.0% +/- 0.0% | 92.8% +/- 1.3% |
| Target Retention 30% (70% Reduction) | 69.8% +/- 1.2% | 29.8 ms +/- 2.5 ms | $7.50 USD (Projected) | 91.2% +/- 1.4% | 100.0% +/- 0.0% | 89.4% +/- 1.6% |
Key Insights & Analysis
- 01.Target retention of 50% (50% reduction) offers an optimal balance between semantic fidelity (96.4%) and token cost reduction.
- 02.Priority Tier 4 shielding maintains 100% instruction retention even at aggressive 70% token reductions.
- 03.Execution latency remains low (< 30ms) across all compression settings.
Limitations & Non-Recommended Workloads
Honest Engineering Trade-Offs
- •Target ratios below 0.3 (70% reduction) may drop non-critical secondary details if sentence redundancy is low.
- •Inputs consisting entirely of Priority 4 code blocks or directives cannot be pruned below their syntax structural bounds.
Experimental Protocol & Methodology
Executed automated sweeps over 500 standard system prompts, code blocks, and RAG document contexts. Measured exact token reduction via tiktoken tokenizer, semantic cosine similarity, and directive compliance.
Raw Evaluation Dataset Sample (JSON)
Raw evaluation prompt payload sample format used during experimental benchmark runs:
raw_dataset_sample.json
{
"dataset_name": "ratio_sweep_dataset_500",
"ratios_evaluated": [
0.8,
0.6,
0.5,
0.4,
0.3
]
}Reproducible Python Script
Run this exact script on your hardware to reproduce token reduction and execution latency:
benchmark_reproducible.py
from llmslim import compress
prompt = "System: MUST return JSON format.\nContext: " + "Corporate financial result entry sentence... " * 40
for ratio in [0.3, 0.4, 0.5, 0.6, 0.8]:
res = compress(prompt, target_ratio=ratio)
print(f"Target Ratio: {ratio} | Billed Savings: {res.savings_percent:.1f}% | Execution Latency: {res.execution_time_ms:.2f}ms")