Target: Python runtimeBaseline: LLMSlim v0.4.0 release gate

v0.4.0 Release Verification

Measured test, coverage, lint, and schema-tax benchmark results

The release gate verifies correctness and schema-tax measurements. It does not make a universal latency, retention, or provider-cost claim.

System environment

cpu

Environment-dependent; benchmark locally

ram

Environment-dependent

Cross-platform Python

python Version

Python 3.8+

package Version

llmslim v0.4.0

tokenizer Version

tiktoken when available; otherwise documented heuristic

sample Size

375 generated tool schemas across 18 catalogs

iterations Per Sample

See benchmark scripts; pytest results are collected from test reports

Empirical benchmark matrix

Method	Token reduction	Latency	Cost	Semantic retention	Instruction retention	Entity preservation
Pytest	Not measured by this gate	489 passed / 0 failed	N/A	N/A	N/A	90.93% coverage
Schema-tax benchmark	0.00% lossless serialization reduction	18 catalogs	N/A	375 generated schemas	0 provenance-boundary violations	Ruff passed

Key insights

Stable tool APIs preserve contracts through canonicalization, fingerprints, and verification.
Tool retrieval and selective exposure remain explicit research-only features.
Untrusted RAG/tool/assistant content cannot obtain protected priority from imperative wording.

Limitations and non-recommended workloads

Compression ratio and latency vary with text, hardware, tokenizer availability, and strategy.
The schema-tax benchmark contains 375 generated schemas in 18 catalogs; it is not a broad model-quality evaluation.
LLMSlim mitigates compression-induced instruction elevation; it is not a complete prompt-injection defense.

Experimental methodology

Run pytest, branch coverage, Ruff, and python benchmark.py against the checked-in repository. The runner uses pytest_runtest_logreport to collect actual results.

Raw evaluation dataset

raw_dataset_sample.jsonjson

{
  "release": "v0.4.0",
  "tests_passed": 489,
  "tests_failed": 0,
  "coverage_percent": 90.93,
  "schema_tax_catalogs": 18,
  "generated_tool_schemas": 375
}

Reproducible Python script

benchmark_reproducible.pypython

pytest tests
pytest --cov=llmslim --cov-branch --cov-report=term-missing
ruff check llmslim benchmarks/benchmark.py tests
python benchmark.py