Maintaining the library

This is the maintainer runbook for changing the library itself - building the kernel, regenerating fixtures, running the gate, deploying to the reference host, and publishing issues. It is distinct from the rest of the operations manual, which is for users running Tessero in their own applications (see [Deployment(https://github.com/DMJ-CV-91913/tessero/blob/main/docs/operations/deployment.md) and [Upgrading(https://github.com/DMJ-CV-91913/tessero/blob/main/docs/operations/upgrading.md)), and from the day-to-day numerical-parity rules in prompts/numpy-scipy-parity-program.md, CONTRIBUTING and RELEASING.

The two machines#

Development is split across two hosts on purpose:

The workstation has no C toolchain, so the loop is: edit locally → sync to the reference host → build and verify there → commit and push from the workstation → deploy (pull) on the host. Commits only ever originate on the workstation because the reference host's git deploy key is read-only (it can pull, never push - see git access).

One-command update#

On the reference host, a single script does the whole sync-build-test cycle:

bash tools/update.sh              # force-sync to origin/main, build kernel+ext, C tests + FFI parity
bash tools/update.sh --publish    # ... and publish the beta issue list to GitHub (needs gh auth, once)
bash tools/update.sh --quick      # skip the ~4 min FFI parity run

update.sh force-syncs (git reset --hard origin/main), so it discards local changes - it is for a mirror checkout, not a dirty working tree. It wraps tools/pull-build.sh, which can also be run directly.

Deploy: pull-build.sh#

bash tools/pull-build.sh            # git pull --ff-only, build kernel + extension, C tests, FFI parity
bash tools/pull-build.sh --quick   # C tests only (skip FFI parity)
bash tools/pull-build.sh --no-ext  # FFI kernel only (skip the extension build)

The extension is built in a scratch copy ($EXT_BUILD, default ~/ext-build), never in-place, so it can't pollute the tracked ext/ tree. The extension's parallel build has an intermittent dependency-file race; pull-build.sh falls back to a serial make automatically, so a flake self-recovers.

A clean deploy of a pushed batch, from any state:

cd ~/tessero
git fetch origin
git checkout -f main            # -f discards any scp'd work-in-progress
git reset --hard origin/main
bash tools/pull-build.sh

Avoid git clean -fd on the reference host - it would remove .venv-parity and vendor/, which are costly to rebuild and not needed to deploy.

The gate#

tools/parity/gate.sh is the full acceptance gate (prompts/numpy-scipy-parity-program.md section 8): generated-file checks, kernel build + tests + sanitizers, extension build + tests, NumPy/SciPy fixtures on both backends, PHPUnit suites, the runnable cookbook, fuzz + soak + benchmarks, and mkdocs build --strict.

source .venv-parity/bin/activate
EXT_DIR=~/ext-build bash tools/parity/gate.sh            # everything
QUICK=1 EXT_DIR=~/ext-build bash tools/parity/gate.sh    # skip sanitizers, fuzz, soak, benchmarks

Use QUICK=1 on an AVX2 host. The benchmark thresholds are calibrated for an AVX-512 reference host, so the rng normal bench gate reports a spurious failure on AVX2 (tracked as TSR-206); the stress block (fuzz/soak/bench) is pure C-kernel and orthogonal to most changes. QUICK=1 runs the full functional verification and yields a clean GATE_EXIT=0 on AVX2.

Fixtures and the pinned environment#

Reference values are generated by SciPy/NumPy in a pinned environment (ADR 0012) so they do not depend on the host's SIMD dispatch:

source .venv-parity/bin/activate
bash tools/parity/gen-fixtures.sh                 # every group
bash tools/parity/gen-fixtures.sh legacy np       # some groups

gen-fixtures.sh sets NPY_DISABLE_CPU_FEATURES (so NumPy's float64 loops are the C library's on every CPU) and OPENBLAS_CORETYPE + single-threaded BLAS (so reductions through BLAS use one summation order). The generators refuse to write without the pinned environment (TSR_FIXTURE_UNPINNED=1 for a throwaway file).

Host constraint (TSR-207). The canonical fixture host is AVX2 (Haswell). The pin was re-baselined from SkylakeX (AVX-512) to Haswell because the SkylakeX OpenBLAS kernel is an illegal instruction on an AVX2 CPU - any NumPy call segfaults. Consequences:

• Generate fixtures on an AVX2+ host. Single-threaded BLAS keeps the summation order deterministic; Haswell kernels run on any AVX2+ CPU.
• Transcendental fixtures can differ by ~1 ulp from another host's glibc/libm, within the test tolerances - expected, not a regression.
• fixtures.py (scipy.special) is CPU-independent and slow (hours) on this host; don't regenerate the special/stats groups unless they actually changed. The committed fixtures already pass here (the gate's run-tests.php is 777/0).

Adding a function#

New numerical functions go into the kernel function registry (ADR 0011), so both backends get them from one implementation. The per-item loop:

1. Spec/table entry - spec/*.yaml (element-wise), or csrc/src/np_*.c / csrc/cxx/*.cpp (routines), or a tools/parity/scope.yaml row.
2. Fixtures - bash tools/parity/gen-fixtures.sh <group> (pinned env).
3. Implementation + build - make -C csrc; bash tools/sync-ext.sh to mirror the kernel into ext/libtessero.
4. Façades - regenerated by tools/parity/gen-facades.php after a build.
5. Verify both backends - php tools/parity/run-tests.php --backend=ffi and --backend=ext.
6. Ledger - update .work/ledger.md.

A hand-written loop ufunc (e.g. heaviside) is a LOOP2 + a table row in csrc/src/ufunc.c, a wrapper on Tessero\Math, a fixture in tests/fixtures/generate_ufunc.py, and the ufunc-count assertions in csrc/tests/test_main.c, tests/Unit/MathTest.php and tests/docs/examples.php.

Coverage and metrics (regenerate, never hand-edit)#

The coverage page is generated; regenerate it whenever the public API changes:

php -d ffi.enable=1 -d extension=<tessero.so> tools/api-dump.php > tools/parity/api.json
python3 tools/gap_inventory.py tools/parity/api.json > docs/project/numpy-scipy-coverage.md
python3 tools/parity/report.py           # metrics.md, metrics.json, deviations.md, usage.md

Two rules the generators do not enforce on their own (TSR-121):

• tools/api-dump.php must list every generated façade class, or that class's area reads 0 %.
• tools/gap_inventory.py must reference every shipped function. The generator only errors on a reference to a function that does not exist - it never flags a shipped function with no reference, so the page silently understates coverage if a row is left empty. After adding functions, wire their rows.

Publishing issues#

The beta issue list is one source with two outputs (tools/beta-issues.php):

php tools/beta-issues.php render > docs/project/open-issues.md   # the snapshot page (gate checks it is current)
php tools/beta-issues.php check                                  # CI/gate: fail if the page is stale
gh auth login                                                    # once, interactive
php tools/beta-issues.php create --repo=DMJ-CV-91913/tessero     # create one GitHub issue per entry (idempotent)

create uses the GitHub CLI and needs it authenticated once. It is idempotent (skips IDs already present as issues). Edit issues in .github/beta-issues.json, then render to refresh the page.

The website then pulls the issues onto its Contribute → Issues page with php artisan issues:sync (run in the web app, not here). Because the repo is private, the web app needs a GITHUB_TOKEN with issues read access in its .env; the token is server-side only and never reaches visitors.

Git access on the reference host#

The reference host authenticates to GitHub with a read-only deploy key (~/.ssh/github_deploy). Pin it in ~/.ssh/config so git doesn't offer the wrong key (a scopeless or website-scoped key yields Permission denied):

Host github.com
    HostName github.com
    User git
    IdentityFile ~/.ssh/github_deploy
    IdentitiesOnly yes

The key can fetch/pull but never push - all pushes originate from the dev workstation. A reboot that clears a loaded ssh-agent can re-expose a wrong-key problem; the ~/.ssh/config entry above makes the key selection explicit.

Edit this page on GitHub