evalr.jsonl¶

A dataset store on local JSON Lines files.

Each dataset is a directory under the store's root, named by the dataset (org/helpfulness becomes org/helpfulness/). Every revision is a file of JSON records, one example to a line, named by the dataset's content hash, so saving the same examples again rewrites nothing. dataset.json names the latest revision and holds the description:

root/
  helpfulness/
    dataset.json            {"description": "...", "latest": "3f2a9c0d1e4b5a67"}
    3f2a9c0d1e4b5a67.jsonl  one example per line

See JSON Lines files.

The store¶

JsonlDatasetStore ¶

JsonlDatasetStore(root: str | PathLike[str])

Saves datasets as JSON Lines files under a root directory.

The files are plain JSON Lines, readable by any tool, including Hugging Face's load_dataset("json", data_files=...).

Use a directory, created on the first save.

Parameters:

Name	Type	Description	Default
`root`	`str \| PathLike[str]`	The directory holding the datasets.	required

save `async` ¶

save(dataset: Dataset[InputT, VerdictT]) -> str

Write the dataset's revision and make it the latest.

Returns:

Type	Description
`str`	The dataset's content hash, which names the revision.

Raises:

Type	Description
`ValueError`	The dataset's name cannot be a path under the root.

load `async` ¶

load(
    name: str,
    /,
    *,
    input_type: type[InputT],
    verdict_type: type[VerdictT],
    revision: str | None = None,
) -> Dataset[InputT, VerdictT]

Read the latest revision of a dataset, or the one given.