Rolling Dataset Contribution Guide

This guide is for contributors who want to add a new dataset or trial set to OpenSVBench without changing the frozen current core benchmark.

Rolling contributions start from the leaderboard web form. Contributors upload a small contribution package for maintainer review. This package is normally a reproducible recipe, not the dataset itself: it should explain how to download or place the source audio, how to prepare manifests, and how to build trials. If a proposal is accepted, maintainers can turn it into a code change or ask the contributor for a follow-up patch. Rolling trial sets do not affect the current core ranking unless maintainers explicitly promote them in a later benchmark version.

Web Upload Flow

Open the Submit page and use the rolling dataset form. The uploaded package is stored on the server for maintainer review; it is not evaluated automatically and does not modify the benchmark catalog.

Recommended package contents:

README.md
download_or_access_notes.md
prepare_<dataset_id>.sh
make_trials_<dataset_id>.py
catalog_patch_example.py
stats_<trial_set_id>.json
sample_trials_<trial_set_id>.txt

Large raw audio should usually stay outside the upload package. Prefer public source links, official download instructions, or manual placement notes. Small example files or tiny samples are fine when they make the protocol easier to review.

What To Contribute

A rolling dataset package should contribute reproducible benchmark code and metadata, not raw large audio files.

Required pieces:

• A short dataset README under test_prepare_dataset/<dataset_id>/README.md.
• A download or placement script under local/download_<dataset_id>.sh.
• A prepare script under local/prepare_<dataset_id>.sh.
• Any Python helper scripts needed to build manifests or trials.
• A catalog patch using collection="rolling" for DatasetSpec and ScenarioSpec.
• A trial file in the existing OpenSVBench format.
• A summary of source data, license, split, protocol source, statistics, and limitations.

Do not upload:

• Raw audio archives or extracted large audio.
• Private, non-consented, or non-redistributable data.
• Generated embeddings or model score files.
• Private keys or unreleased evaluation labels.

Trial Format

Trial files must use the existing 3- or 4-column format:

utt1 utt2 label [condition]

Rules:

• label must be target or nontarget.
• condition is recommended and should be stable, lowercase, and parseable.
• Every utt1 and utt2 must exist in wav.scp or wav_all.scp.
• Trial construction must be deterministic, with a documented random seed when sampling is used.
• Target and nontarget construction rules must be described in the dataset README.

Recommended prepared files:

wav.scp
wav_all.scp
utt2spk
spk2utt
trials_<trial_set_id>.txt
stats_<dataset_id>_source.json
stats_<trial_set_id>.json

Add extra utt2* metadata files when they are useful for analysis, for example utt2lang, utt2device, utt2distance, utt2style, or utt2session.

Catalog Rules

Rolling specs must not enter the active core by accident.

Use:

DatasetSpec(
    dataset_id="my_dataset",
    display_name="My Dataset",
    collection="rolling",
    ...
)

ScenarioSpec(
    scenario_id="my_dataset_trial",
    dataset_id="my_dataset",
    display_name="My Dataset Trial",
    variant="clean",
    collection="rolling",
    ...
)

Do not add the rolling trial set to a core CapabilitySpec unless maintainers explicitly approve a core benchmark update.

Review Checklist

Before uploading a contribution package, confirm:

• The data source is public, properly licensed, or clearly documented as requiring manual placement.
• The package does not include large raw audio or generated embeddings.
• ./local/download_<dataset_id>.sh --data <data_root> is safe to rerun.
• ./local/prepare_<dataset_id>.sh --data <data_root> creates all expected prepared files.
• Trial files use utt1 utt2 label condition and contain no duplicate canonical pairs unless intentionally documented.
• Basic statistics are reported: utterances, speakers, target trials, nontarget trials, condition counts.
• The dataset README states whether the protocol is official, an official subset, or self-constructed.
• python3 tools/generate_submission.py --data <data_root> --precheck-only still reports the current core as ready.

Minimal Example

See examples/rolling_dataset_minimal/ for a minimal example package. It shows the expected manifests, trial format, statistics, and catalog patch without adding anything to the current core benchmark.