Target: Python runtimeBaseline: LLMSlim v0.4.0 release gate
v0.4.0 Release Verification
Measured test, coverage, lint, and schema-tax benchmark results
The release gate verifies correctness and schema-tax measurements. It does not make a universal latency, retention, or provider-cost claim.
System Environment & Rig Specification
CPU: Environment-dependent; benchmark locally
RAM: Environment-dependent
OS: Cross-platform Python
Runtime: Python 3.8+
Package: llmslim v0.4.0
Tokenizer: tiktoken when available; otherwise documented heuristic
Dataset Size: 375 generated tool schemas across 18 catalogs
Iterations: See benchmark scripts; pytest results are collected from test reports
Empirical Benchmark Matrix
*Costs are projected estimates; latencies and ratios are measured empirical values.| Method Variant | Token Reduction (Measured) | Execution Latency (Measured) | Billed Cost (Projected) | Semantic Retention (Measured) | Instruction Retention (Measured) | Entity Preservation (Measured) |
|---|---|---|---|---|---|---|
| Pytest | Not measured by this gate | 489 passed / 0 failed | N/A | N/A | N/A | 90.93% coverage |
| Schema-tax benchmark | 0.00% lossless serialization reduction | 18 catalogs | N/A | 375 generated schemas | 0 provenance-boundary violations | Ruff passed |
Key Insights & Analysis
- 01.Stable tool APIs preserve contracts through canonicalization, fingerprints, and verification.
- 02.Tool retrieval and selective exposure remain explicit research-only features.
- 03.Untrusted RAG/tool/assistant content cannot obtain protected priority from imperative wording.
Limitations & Non-Recommended Workloads
Honest Engineering Trade-Offs
- •Compression ratio and latency vary with text, hardware, tokenizer availability, and strategy.
- •The schema-tax benchmark contains 375 generated schemas in 18 catalogs; it is not a broad model-quality evaluation.
- •LLMSlim mitigates compression-induced instruction elevation; it is not a complete prompt-injection defense.
Experimental Protocol & Methodology
Run pytest, branch coverage, Ruff, and python benchmark.py against the checked-in repository. The runner uses pytest_runtest_logreport to collect actual results.
Raw Evaluation Dataset Sample (JSON)
Raw evaluation prompt payload sample format used during experimental benchmark runs:
raw_dataset_sample.json
{
"release": "v0.4.0",
"tests_passed": 489,
"tests_failed": 0,
"coverage_percent": 90.93,
"schema_tax_catalogs": 18,
"generated_tool_schemas": 375
}Reproducible Python Script
Run this exact script on your hardware to reproduce token reduction and execution latency:
benchmark_reproducible.py
pytest tests
pytest --cov=llmslim --cov-branch --cov-report=term-missing
ruff check llmslim benchmarks/benchmark.py tests
python benchmark.py