Skip to content

Inspect AI Integration

Overview

The toolkit's per-sample detectors are available as Inspect AI scorers, so any Inspect evaluation can attach metacognition analysis to its existing tasks. Add a scorer and every sample's output gets scored for verbalized evaluation awareness or sandbagging indicators, with detector evidence preserved in the score metadata and aggregated as a mean metric.

Installation

Inspect AI requires Python 3.10+ and ships as an optional extra:

pip install "ai-metacognition-toolkit[inspect]"

Without the extra, the rest of the toolkit works normally and the scorers are simply absent from ai_metacognition.integrations.

Usage

from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.solver import generate

from ai_metacognition.integrations import (
    awareness_scorer,
    sandbagging_scorer,
)

@task
def arithmetic_with_metacognition():
    return Task(
        dataset=[
            Sample(input="What is 2+2?", target="4"),
            # ...
        ],
        solver=generate(),
        scorer=[awareness_scorer(), sandbagging_scorer()],
    )

Run it like any Inspect eval:

inspect eval arithmetic_with_metacognition --model anthropic/claude-sonnet-5

Each sample in the eval log then carries both scores:

  • awareness_scorer — verbalized evaluation awareness in [0, 1], with detection and depth component scores in the metadata
  • sandbagging_scorer — text-level sandbagging indicators in [0, 1] (evaluation-context mentions combined with hedging, uncertainty, or refusal markers), with the extracted features in the metadata

Both accept a pre-configured detector if you want non-default settings:

from ai_metacognition.detectors import AwarenessProfiler

scorer = awareness_scorer(profiler=AwarenessProfiler())

Cross-Sample Analyses

These scorers analyze each sample independently. The toolkit's cross-sample analyses need trials grouped by item, so run them over the completed eval log instead:

  • ConsistencyDetector — collect repeated trials per item (shuffled options, paraphrases) across epochs, then build OptionOrderTrial / ParaphraseTrial records from the log.
  • AwarenessProfiler.profile() — run the same dataset under neutral and eval-framed system prompts, then compare the two logs' completions.