Getting Started

This guide walks you through setting up and running your first test with spack ai-test.

A note on terminology: a leading-edge configuration is what most CI pipelines test — the newest package version, the latest compiler, and all default variants. This tool targets off-leading-edge configurations: older version pins, non-default variants, and dependency combinations that are rarely (if ever) tested but frequently encountered by real users.

Prerequisites

Before installing spack-ai-test, ensure you have:

  • Spack (v0.21 or later) installed and available in your PATH.

  • Python 3.9 or higher.

  • An API key for at least one supported LLM provider:

    • Google Gemini (GEMINI_API_KEY)

    • Anthropic Claude (ANTHROPIC_API_KEY)

    • OpenAI (OPENAI_API_KEY)

Installation

spack-ai-test is packaged as a standard Spack command extension.

1. Clone the Extension Repository

Clone the repository to a convenient location on your system — for example, alongside other Spack extensions or in your home directory:

$ git clone https://github.com/VaishnaviOnPC/spack-ai-test.git /path/to/spack-ai-test

2. Register the Extension in Spack

Add the cloned directory to the extensions list in your Spack configuration (e.g., ~/.spack/config.yaml):

config:
  extensions:
    - /path/to/spack-ai-test

Alternatively, register it directly from the command line:

$ spack config add "config:extensions:[/path/to/spack-ai-test]"

3. Verify Installation

Check that Spack recognizes the extension:

$ spack ai-test --help

You should see the command description and the full list of available flags.

Configuring API Keys

Export your preferred provider’s API key in your shell environment before running any generation commands:

# Google Gemini (default)
$ export GEMINI_API_KEY="your-gemini-api-key"

# Anthropic Claude
$ export ANTHROPIC_API_KEY="your-anthropic-api-key"

# OpenAI
$ export OPENAI_API_KEY="your-openai-api-key"

Note

Default models and settings can also be configured in ~/.spack/ai_test.yaml. See Configuration for details.

Warning

If no API key is set for the selected model, the tool raises an error at the Plan stage before any concretization is attempted. Inspection commands (spack ai-test <package> without --mape or --generate) do not require an API key.

Running Your First Test

To inspect a package’s metadata and computed risk signals without invoking the LLM:

$ spack ai-test zlib

The output lists the extracted versions, variants, declared dependency constraints, and a set of computed risk signals — for example, unbounded version ranges or virtual provider dependencies. No API key is required for this step.

To run the full self-adaptive testing loop against locally installed compilers:

$ spack ai-test zlib --mape --local --model gemini-2.5-flash

The tool prints one result line per generated spec ([PASS], [FAIL], or [BUILD_FAIL]) followed by a totals summary showing how many specs were tested, concretized, and failed.