Provider-backed Run

This tutorial shows the minimum provider-facing pattern without relying on the legacy dataset-loader API.

Core Shape

from themis import (
    BenchmarkSpec,
    DatasetQuerySpec,
    Orchestrator,
    PromptVariantSpec,
    ScoreSpec,
    SliceSpec,
)
from themis.specs import DatasetSpec, GenerationSpec


class RemoteDatasetProvider:
    def scan(self, _slice_spec, query):
        # push the subset or filter into the remote source when possible
        return [{"item_id": "item-1", "question": "2 + 2", "answer": "4"}]


class ProviderEngine:
    def infer(self, _trial, _context, _runtime):
        # Messages are already rendered by orchestration before the engine runs.
        ...

Why It Matters

• DatasetQuerySpec keeps subset and filter intent outside dataset payloads
• engines receive already-rendered benchmark prompts plus preserved prompt metadata
• slice dimensions stay queryable later through BenchmarkResult.aggregate(...)

Full Next Step

Pair this pattern with Build a Dataset Provider and Build a Provider Engine.