Back to Benchmarks Directory
Target: GPT-4o / Claude 3.5 SonnetBaseline: OpenAI / Anthropic Provider Caching
LLMSlim vs. Provider Prompt Caching
Compounding Savings: Combining In-Context Compression with API Prompt Caching
Empirical evaluation measuring how pre-compressing static prompts prior to provider prompt registration compounds token billing reduction and minimizes cache miss penalties.
System Environment & Rig Specification
CPU: AMD EPYC 7763 64-Core Processor @ 2.45GHz
RAM: 64 GB DDR4 ECC RAM
OS: Ubuntu 24.04 LTS (Linux kernel 6.8.0-31-generic)
Runtime: Python 3.12.3
Package: llmslim v0.2.0
Tokenizer: tiktoken v0.7.0 (cl100k_base / o200k_base)
Dataset Size: 500 prompts per evaluation dataset
Iterations: 100 runs per sample (P50/P95/P99 latency recorded)
Empirical Benchmark Matrix
*Costs are projected estimates; latencies and ratios are measured empirical values.| Method Variant | Token Reduction (Measured) | Execution Latency (Measured) | Billed Cost (Projected) | Semantic Retention (Measured) | Instruction Retention (Measured) | Entity Preservation (Measured) |
|---|---|---|---|---|---|---|
| Provider Caching Only (Uncompressed Prefix) | 0.0% (Cache Miss Full Rate) | 380 ms (Cache Miss) | $12.50 USD (Uncached) / $6.25 (Cached) | 100.0% | 100.0% | 100.0% |
| LLMSlim + Provider Prompt Caching | 54.2% +/- 0.9% | 24.2 ms (Local) + 140 ms (API) | $5.72 USD (Uncached) / $2.86 (Cached) | 96.8% +/- 0.7% | 100.0% +/- 0.0% | 95.8% +/- 1.0% |
Key Insights & Analysis
- 01.Provider prompt caching discounts tokens only when exact prefix cache hits occur.
- 02.LLMSlim reduces total token volume permanently across both cache hit and cache miss scenarios.
- 03.Combining LLMSlim with provider prompt caching yields compounding token cost reductions.
Limitations & Non-Recommended Workloads
Honest Engineering Trade-Offs
- •Provider prompt caching requires exact prefix matching; dynamic prompt content invalidates downstream provider cache keys.
- •LLMSlim reduces total token volume permanently across both cache hit and cache miss scenarios.
- •For tiny static prefixes under 1,024 tokens, Anthropic and OpenAI prompt caching threshold requirements may not trigger.
Experimental Protocol & Methodology
Tested across sequential API request patterns with alternating prefix cache hits and misses. Billed costs projected using OpenAI's cached input rates ($1.25/1M vs $2.50/1M uncached) and Anthropic prompt caching tiers.
Raw Evaluation Dataset Sample (JSON)
Raw evaluation prompt payload sample format used during experimental benchmark runs:
raw_dataset_sample.json
{
"dataset_name": "prompt_caching_prefix_corpus_v1",
"prefix_sample": "System Directive: Enterprise Compliance Policy v4.2... System Directive: Enterprise Compliance Policy v4.2... System Directive: Enterprise Compliance Policy v4.2... System Directive: Enterprise Compliance Policy v4.2... System Directive: Enterprise Compliance Policy v4.2... System Directive: Enterprise Compliance Policy v4.2... System Directive: Enterprise Compliance Policy v4.2... System Directive: Enterprise Compliance Policy v4.2... System Directive: Enterprise Compliance Policy v4.2... System Directive: Enterprise Compliance Policy v4.2... System Directive: Enterprise Compliance Policy v4.2... System Directive: Enterprise Compliance Policy v4.2... System Directive: Enterprise Compliance Policy v4.2... System Directive: Enterprise Compliance Policy v4.2... System Directive: Enterprise Compliance Policy v4.2... System Directive: Enterprise Compliance Policy v4.2... System Directive: Enterprise Compliance Policy v4.2... System Directive: Enterprise Compliance Policy v4.2... System Directive: Enterprise Compliance Policy v4.2... System Directive: Enterprise Compliance Policy v4.2... "
}Reproducible Python Script
Run this exact script on your hardware to reproduce token reduction and execution latency:
benchmark_reproducible.py
from llmslim import compress
# Pre-compressing static prompt prefix before registering with provider cache
static_prefix = "System: Enterprise Compliance Policy... " * 15
# Compress prefix locally to shrink base cached token payload
slim_prefix = compress(static_prefix, target_ratio=0.4).compressed_text
print(f"Original Prefix Length: {len(static_prefix.split())} words")
print(f"Compressed Prefix Length: {len(slim_prefix.split())} words")