sqb compile command. They measure complete compiler workflows,
including project discovery, parsing, dependency resolution, SQL expansion and analysis, contracts,
tests, Rules, cache publication, and artifact generation.
The benchmark projects are generated so each run is deterministic and reproducible. Their shape is
not arbitrary: the reference workload was calibrated from a real working SQLBuild project, including
its resource ratios, graph structure, SQL-size distribution, tests, audits, macros, functions,
sources, seeds, and hooks.
The generated projects contain neutral names and data. They do not contain source code or business
data from the reference project.
Production-shaped workload
The reference compiler workload reproduces the measured characteristics of a working SQLBuild project:
The generated dependency graph combines a 54-layer spine with bounded model chains. The workload
also includes path defaults, reusable schemas, enforced contracts, attached audits, SQL hooks,
source and seed references, SQL functions, composed Python macros, CTEs, derived tables, assertions,
repeated test targets, and test fixtures ranging from 40 to 200 rows.
This preserves the characteristics that exercise the compiler without publishing or depending on
one specific project.
Compiler performance
On the production-shaped 976-model workload, a required CI run measured:
Model, test, and macro edits reuse unaffected compiler work. The project-configuration scenario
deliberately changes an input with broad impact and therefore performs wider recomputation.
Model and SQL-volume scaling
A separate compiler profile increases model count and authored SQL volume while retaining the measured model-size distribution:
This profile isolates compiler scaling by model count and SQL volume. The production-shaped profile
above is the broader test of sources, seeds, functions, macros, tests, audits, hooks, contracts, and
artifact generation occurring together.
Rules performance
The Rules benchmark scales the same production-shaped generator to 5,000 models. The resulting project contains:
On this project, 20 custom Rules complete their first compile in just over 10 seconds. An unchanged
compile completes in under six seconds.
Even with 100 custom Rules, an unchanged compile completes in just over seven seconds.
Each measurement executes the complete
sqb compile command. The edit scenario changes 50 of the
5,000 models before recompiling.
Because most Rules in this profile evaluate once per model, the 20- and 100-Rule configurations
perform roughly 100,000 and 500,000 Rule evaluations respectively.
Increasing the custom Rule pack from 20 to 100 multiplies the number of Rule evaluations by five,
while the unchanged compile median increases from 5.81 to 7.24 seconds.
Rule implementation changes
Changing a Rule or one of its imported helpers participates in cache invalidation. In the 5,000-model, 100-Rule profile, editing one custom Rule invalidated exactly 5,001 evaluations while preserving 495,001 cached results. The complete compile finished in a 9.89-second median. CI also verifies exact invalidation for model edits, SQL-test edits, macro edits, project configuration, directly observed project inputs, custom Rule source, and imported custom helpers. A cache result with an unexpected hit or miss count fails the benchmark.Methodology
Compiler and Rules benchmarks run on an 8-vCPU, 32 GB Linux CI runner class using Python 3.12 and Polyglot 0.9.2. The Rules results shown above are the median of five required-CI runs. Allocated CPUs across those runs were AMD EPYC or Intel Xeon processors. The Rules benchmark output records:- complete CLI elapsed time;
- compiler phase timings;
- Rule cache hits and misses;
- Rule cache size;
- peak resident memory;
- workload resource counts;
- SQLBuild, Python, and Polyglot versions;
- every raw timing and memory sample.