Skip to content

Run from Python vs config and CLI

Goal: choose the execution style that best matches how you manage experiments.

When to use this:

Use this guide when you already know Themis concepts but need to decide how to organize real runs in code, config, or shell workflows.

Procedure

Use Python when you want:

  • direct imports and type-checked objects
  • custom components without module-path indirection
  • the shortest path to experiments and local debugging

Use config and CLI when you want:

  • reproducible checked-in experiment definitions
  • shell-friendly automation
  • submission flows such as worker-pool and batch

Config-backed execution details:

  • Experiment.from_config(...) supports YAML (.yaml / .yml) and TOML (.toml)
  • config component fields accept builtin ids or importable module paths such as package.module:factory
  • config files carry strings, not live Python objects; object instances belong in Python authoring only
  • relative storage and runtime paths resolve relative to the config file directory
  • CLI or Python callers can pass dotlist overrides before compile/run time

Use the config-backed external execution example when you want one runnable path from config file to execution:

from __future__ import annotations

from pathlib import Path
from tempfile import TemporaryDirectory

from themis.core.submission import run_worker_once, submit_experiment
from themis.core.experiment import Experiment


CONFIG_TEMPLATE = """
generation:
  generator: builtin/demo_generator
  candidate_policy:
    num_samples: 1
  reducer: builtin/majority_vote
evaluation:
  metrics:
    - builtin/exact_match
  parsers:
    - builtin/json_identity
storage:
  store: sqlite
  parameters:
    path: runs/themis.sqlite3
runtime:
  queue_root: runs/queue
datasets:
  - dataset_id: sample
    cases:
      - case_id: case-1
        input:
          question: 2+2
        expected_output:
          answer: "4"
""".strip()


def run_example() -> dict[str, object]:
    """Submit an experiment to the worker-pool flow and execute one worker cycle."""

    with TemporaryDirectory() as tmp:
        root = Path(tmp)
        config_path = root / "experiment.yaml"
        config_path.write_text(CONFIG_TEMPLATE, encoding="utf-8")
        experiment = Experiment.from_config(config_path)
        manifest = submit_experiment(
            experiment, config_path=str(config_path), mode="worker_pool"
        )
        result = run_worker_once(root / "runs" / "queue")
        assert result is not None
        return {
            "run_id": result.run_id,
            "status": result.status.value,
            "manifest_path": str(manifest.manifest_path),
        }


if __name__ == "__main__":
    print(run_example())

Variants

Variant Best when Tradeoff Related APIs / commands
Ad hoc scripts and notebooks You want direct imports, live objects, and quick local debugging Harder to standardize across repeated runs Python authoring, evaluate(...) for sync scripts, evaluate_async(...) or Experiment.run_async() inside notebooks and async apps
Checked-in experiment specs and automation You want reproducible definitions that work well with shell workflows and deferred execution Component references must be config-loadable rather than live objects Experiment.from_config(...), themis run, themis submit
Mixed approach You want checked-in configs for repeatable runs but still keep custom component logic in Python Requires discipline about what lives in config vs code Config files plus importable module paths

Expected result

You should know whether the next example or guide you follow should be code-first or config-first.

Troubleshooting