Advanced Workflows

This section covers advanced automated testing workflows, including closed-loop execution, cluster-based offline testing, and automated failure bisection.

The MAPE-K Self-Adaptive Loop

The --mape flag runs the full self-adaptive testing cycle:

$ spack ai-test openmpi --mape --local

The pipeline executes through four coordinated stages:

Monitor: Retrieves accumulated test history from the Knowledge Base (kb.json) matching the package schema hash.
Analyze: Calculates empirical failure rates and prioritizes high-risk dependency branches.
Plan: Prompts the LLM with multi-source RAG context (live registry version gaps, historical failure patterns, and active GitHub issue signals).
Execute: * Pre-validates generated specs statically against the package schema. * Concretizes valid specs with Spack’s internal ASP solver. * Persists test outcomes (pass, concretize failure, build failure) back to the Knowledge Base.

The terminal prints a single header line summarising the package state (KB entry count, failure rate, active compiler), followed by one result line per spec, then a totals line at the end. For example:

openmpi | KB: 12 entries | failure rate: 0.42 (5/12) | compiler: gcc@13.3.0
[PASS]  openmpi@4.1.5 +pmi %gcc@13.3.0
[FAIL]  openmpi@4.0.7 +cuda %gcc@13.3.0
[PASS]  openmpi@4.1.6 ~shared %gcc@13.3.0
3 tested | 2 concretized | 1 failed -> ~/.spack/cache/ai_test/kb.json

Testing Depth: Build and Regression Testing

By default, --mape validates that specs concretize. To extend testing to compilation and test suites:

Building Concretized Specs

To compile concretized specs via spack install:

$ spack ai-test zlib --mape --local --build

Running Package Test Suites

To compile and execute the package test suite (spack install --test=root):

$ spack ai-test zlib --mape --local --test

Note

Passing --test automatically implies --build.

HPC Cluster Workflows (Offline / Compute Nodes)

In High Performance Computing (HPC) clusters, compute nodes frequently lack external internet access required to reach LLM APIs. spack ai-test provides a decoupled workflow to handle this environment:

Phase 1: Login Node (Online Generation)

On a login or edge node with internet connectivity, generate candidate specs and queue them in the Knowledge Base:

$ spack ai-test openmpi --plan-only --model gemini-2.5-flash

This saves generated specs to kb.json with a status of pending without running concretization or builds.

Phase 2: Compute Node (Offline Execution)

In a compute node batch script (e.g., Slurm job) without internet connectivity:

$ spack ai-test openmpi --execute-queued --test

This reads all pending specs from the Knowledge Base, concretizes them, builds the packages, and records the test outcomes.

Automated Failure Bisection

When a package build or test fails, determining the exact version where the regression occurred can be tedious. Adding --bisect enables automated failure localization:

$ spack ai-test openmpi --mape --test --bisect

When a deterministic failure occurs on package version \(v_{\text{fail}}\):

Exponential Galloping: The system tests predecessor versions in powers of two (\(1, 2, 4, 8, \dots\)) to establish a bounding range between a passing version and a failing version.
Binary Search: Once bounded, it performs a binary search to identify the exact commit or release version where the regression was introduced.

Knowledge Base

The Knowledge Base (kb.json) stores historical test outcomes keyed by package schema hash.

Deduplication: Previously tested configurations are never re-tested, so repeated runs of --mape progressively explore new parts of the configuration space.
Schema Versioning: When a package.py changes (e.g., a new version is added), the schema hash changes and the package’s KB history resets automatically. Prior data for that package is preserved but becomes inactive.
Pattern Mining: Features with high failure rates across accumulated tests (e.g., a specific variant flag or compiler) are automatically surfaced as risk context in future LLM prompts, improving generation quality over time.

Inspecting the Knowledge Base

The KB is a plain JSON file located by default at ~/.spack/cache/ai_test/kb.json. It is human-readable and can be inspected directly:

$ cat ~/.spack/cache/ai_test/kb.json | python -m json.tool | head -60

Resetting the Knowledge Base

To clear the history for a specific package, delete entries with its pkg_name field from the JSON file. To reset the entire KB:

$ rm ~/.spack/cache/ai_test/kb.json

The file is re-created automatically on the next run.