Benchmarks API

Benchmark registry - single source of truth for available benchmarks.

This module defines all built-in benchmarks and their execution methods, providing a deterministic mapping for CLI discovery and execution.

pgsi_analyzer.benchmarks.registry.list_algorithms() → List[str]

Return list of all available benchmark algorithms in deterministic order.

Returns:

List of algorithm names (sorted alphabetically)

pgsi_analyzer.benchmarks.registry.list_methods(algorithm: str = None) → List[str]

Return list of execution methods for an algorithm, or all methods if algorithm is None.

Parameters:

algorithm – Algorithm name, or None for all methods

Returns:

List of method names

pgsi_analyzer.benchmarks.registry.get_benchmark_path(algorithm: str, method: str) → Path

Get the filesystem path to a benchmark’s main entry point.

Uses importlib.resources to access package resources, falling back to filesystem path resolution for development.

Parameters:

• algorithm – Algorithm name (e.g., ‘hanoi’)

• method – Execution method (e.g., ‘cpython’)

Returns:

Path object to the benchmark directory or main.py file

Raises:

• ValueError – If algorithm or method is invalid

• FileNotFoundError – If benchmark path doesn’t exist

pgsi_analyzer.benchmarks.registry.validate_algorithm(algorithm: str) → bool

Check if algorithm exists in registry.

pgsi_analyzer.benchmarks.registry.validate_method(algorithm: str, method: str) → bool

Check if method exists for algorithm.

Benchmark discovery helpers for built-in and user-defined benchmarks.

User-defined benchmarks are discovered from a folder with this layout:

<benchmarks_dir>/<algorithm>/<method>/main.py

Methods should match pgsi execution methods (cpython, pypy, cython, ctypes, py_compile).

pgsi_analyzer.benchmarks.discovery.discover_user_benchmarks(benchmarks_dir: Path) → Dict[str, Dict[str, str]]

Discover user-defined benchmarks from an external directory.

pgsi_analyzer.benchmarks.discovery.load_user_registry(benchmarks_dir: Path) → Dict[str, Dict[str, str]]

Load optional user registry file from benchmarks_dir/pgsi_registry.json.

Expected JSON structure:

{"benchmarks": {"algo-name": {"cpython": "...", ...}}}

pgsi_analyzer.benchmarks.discovery.build_registry(benchmarks_dir: Path | None = None) → Dict[str, Dict[str, str]]

Build effective benchmark registry.

Built-ins are always included. User-defined algorithms from benchmarks_dir are merged in. If names collide, user-defined entries override built-ins.

pgsi_analyzer.benchmarks.discovery.list_algorithms_from_registry(registry: Dict[str, Dict[str, str]]) → List[str]

pgsi_analyzer.benchmarks.discovery.list_methods_from_registry(registry: Dict[str, Dict[str, str]], algorithm: str | None = None) → List[str]

pgsi_analyzer.benchmarks.discovery.get_benchmark_path_from_registry(registry: Dict[str, Dict[str, str]], algorithm: str, method: str) → Path

Scaffolding helpers for user benchmark template generation.

pgsi_analyzer.benchmarks.template.generate_benchmark_template(output_dir: Path, algorithms: Iterable[str], force: bool = False) → Path

Generate a benchmark project tree for external user projects using pre-implemented built-in source files.

The generator prefers copying shipped benchmark source files so users start from realistic implementations. If a source method does not exist in the package, it falls back to templates that are still runnable and instrumentation-ready.

Parameters:

• output_dir – Target root directory for generated benchmark project.

• algorithms – Algorithm names to include in the scaffold.

• force – Whether to allow generation inside a non-empty directory.

Returns:

Path to the generated benchmark project root.

Return type:

Path

Raises:

• ValueError – If algorithm list is invalid or target directory is non-empty and force is False.

• OSError – If files/directories cannot be created or copied.

Examples

>>> from pathlib import Path
>>> root = generate_benchmark_template(
...     output_dir=Path("my-benchmarks"),
...     algorithms=["hanoi", "fasta"],
...     force=True,
... )
>>> root.name
'my-benchmarks'

pgsi_analyzer.benchmarks.template.create_benchmark_scaffold(benchmarks_dir: Path, benchmark_name: str, force: bool = False, register: bool = True) → Path

Create a single benchmark scaffold under benchmarks_dir and register it.

This function is optimized for incremental project growth: users can add one algorithm at a time while preserving the same folder contract expected by discovery and orchestrator modules.

Parameters:

• benchmarks_dir – Benchmark project root where scaffold will be created.

• benchmark_name – Name of the new benchmark folder/key.

• force – Whether to allow writing into an existing non-empty benchmark folder.

• register – Whether to update pgsi_registry.json with the new benchmark.

Returns:

Path to the created benchmark directory.

Return type:

Path

Raises:

• ValueError – If benchmark name is invalid or target directory exists and is non-empty while force is False.

• OSError – If files/directories cannot be created or written.

Examples

>>> from pathlib import Path
>>> created = create_benchmark_scaffold(
...     benchmarks_dir=Path("benchmarks"),
...     benchmark_name="my_algo",
...     force=True,
...     register=True,
... )
>>> created.name
'my_algo'