Skip to content

evalr.contracts

Contract suites: what every adapter of a port must do.

Each check takes an adapter, exercises it through its port, and raises ContractViolation where it behaves differently from the port's contract. evalr runs them against every adapter it ships; the libraries run them against theirs, such as check_feedback_source against their feedback sources:

async def test_the_feedback_source_meets_the_contract() -> None:
    await check_feedback_source(HelpfulnessFeedback(workspace))

The checks need no test framework.

See Testing with the contracts.

Checks

check_evaluator async 

check_evaluator(
    evaluator: Evaluator[InputT, VerdictT],
    inputs: Sequence[InputT],
) -> None

Check that an evaluator gives well-formed verdicts, stamped and traced.

Each input is judged inside a trace, which its verdict must record. An input the evaluator hands off is skipped, but it must judge at least one.

Parameters:

Name Type Description Default
evaluator Evaluator[InputT, VerdictT]

The evaluator to check.

 required
inputs Sequence[InputT]

Inputs it can judge.

 required

Raises:

Type Description
ContractViolation

The evaluator behaves differently from the Evaluator contract.
UnsupportedField

Its verdict type cannot be judged.

check_optimizer async 

check_optimizer(
    optimizer: Optimizer[InputT, VerdictT, EvaluatorT],
    evaluator: EvaluatorT,
    *,
    train: Dataset[InputT, VerdictT],
    validate: Dataset[InputT, VerdictT],
) -> None

Check that an optimizer returns a working evaluator and leaves the one given alone.

Parameters:

Name Type Description Default
optimizer Optimizer[InputT, VerdictT, EvaluatorT]

The optimizer to check.

 required
evaluator EvaluatorT

An evaluator of the kind it fits.

 required
train Dataset[InputT, VerdictT]

Labelled examples to fit on.

 required
validate Dataset[InputT, VerdictT]

Labelled examples to check the fit on, none of them in train.

 required

Raises:

Type Description
ContractViolation

The optimizer behaves differently from the Optimizer contract.

check_dataset_store async 

check_dataset_store(
    store: DatasetStore, *, name: str = "evalr-contract"
) -> None

Check that a dataset store keeps every revision it saves, and nothing else.

The store must not hold a dataset named name yet.

Raises:

Type Description
ContractViolation

The store behaves differently from the DatasetStore contract.

check_feedback_source async 

check_feedback_source(
    source: FeedbackSource[InputT, VerdictT],
) -> None

Check that a feedback source yields well-formed, stable examples with verdicts.

Raises:

Type Description
ContractViolation

The source behaves differently from the FeedbackSource contract.
UnsupportedField

The source's verdict type cannot be judged.

check_score_sink async 

check_score_sink(
    sink: ScoreSink,
    recorded: Callable[[], Awaitable[Sequence[Score]]],
) -> None

Check that a score sink keeps one score per id: the latest recorded, whole.

A score sink has no reads, so the caller says how to see what it recorded: the in-memory sink's scores, or what a backend's fake received. A score must be kept with everything it holds (its value and type, trace, span, session, time and metadata), except that a score without a time may be given the time it was recorded.

Parameters:

Name Type Description Default
sink ScoreSink

The sink to check, holding no scores yet.

 required
recorded Callable[[], Awaitable[Sequence[Score]]]

Returns the scores the sink holds, as Score values.

 required

Raises:

Type Description
ContractViolation

The sink behaves differently from the ScoreSink contract.

check_score_config_store async 

check_score_config_store(store: ScoreConfigStore) -> None

Check that a score config store lists every config created in it, by name.

Parameters:

Name Type Description Default
store ScoreConfigStore

The store to check, holding no configs yet.

 required

Raises:

Type Description
ContractViolation

The store behaves differently from the ScoreConfigStore contract.

check_experiment_tracker async 

check_experiment_tracker(
    tracker: ExperimentTracker,
) -> None

Check that a tracker runs every example, judges every output, and isolates failures.

Raises:

Type Description
ContractViolation

The tracker behaves differently from the ExperimentTracker contract.

ContractViolation

Bases: AssertionError

An adapter behaves differently from its port's contract.

The contracts' types

The inputs, verdicts and outputs the checks save, judge and run.

ContractInput pydantic-model

Bases: BaseModel

The input type of the examples the contract suites use.

Fields:

ContractVerdict pydantic-model

Bases: BaseModel

A verdict type with a field of every kind, for the contract suites.

Fields:

rating pydantic-field 

rating: int

How good it is

ContractOutput pydantic-model

Bases: BaseModel

What the contract experiment's task produces.

Fields: