Quickstart

This walkthrough builds a tiny two-step pipeline: image a measurement set with WSClean, then make a mask from the resulting image with breizorro. It mirrors examples/simple_selfcal.py in the repository.

Define some cabs

A Cab is a typed, backend-agnostic description of an atomic task. Give it a name, the command to run, an optional container image, and pydantic models for its inputs and outputs.

from pydantic import BaseModel

from shinobi import Cab, Recipe, step
from shinobi.loaders import build_model


class ImageInputs(BaseModel):
    ms: str = "obs.ms"
    prefix: str = "img"


class ImageOutputs(BaseModel):
    restored: str | None = None


wsclean = Cab(
    name="wsclean",
    command="wsclean",
    image="quay.io/stimela/wsclean:latest",
    inputs_model=ImageInputs,
    outputs_model=ImageOutputs,
)

breizorro = Cab(
    name="breizorro",
    command="breizorro",
    image="breizorro:latest",
    inputs_model=build_model("MaskInputs", {"restored_image": ("File", True, None)}),
    outputs_model=build_model("MaskOutputs", {"mask": ("File", False, None)}),
)

You can hand-write the pydantic models, or build them from a compact {name: (dtype, required, default)} spec with shinobi.loaders.build_model() (the same helper the YAML loaders use).

Run a single cab

Wrap a cab in a @shinobi.step function to make it runnable. The body receives an ExecContext (ctx); calling ctx.run() executes the cab on the chosen backend and returns its result.

@step(wsclean, backend="native")
def image(ctx):
    """Image the visibilities. A near-empty body auto-runs the cab."""
    return ctx.run()

Save the file as myrecipe.py and run it straight from the command line – the cab’s input schema becomes the CLI options:

$ ninja run myrecipe.py:image --ms data.ms --prefix out

Compose a recipe

A Recipe wires steps together. Use add_step with the recipe.inputs / recipe.outputs proxies to declare the data flow: each proxy attribute is a reference that the engine resolves at run time.

selfcal = Recipe(
    name="selfcal",
    inputs_model=ImageInputs,
    outputs_model=build_model("Out", {"mask": ("File", False, None)}),
)
selfcal.add_step("image", wsclean, ms=selfcal.inputs.ms, prefix=selfcal.inputs.prefix)
selfcal.add_step("mask", breizorro, restored_image=selfcal.outputs.image.restored)
selfcal.set_output("mask", selfcal.outputs.mask.mask)

Here selfcal.outputs.image.restored is the restored output of the step named image – wiring it into the mask step’s restored_image input creates the dependency edge between the two.

Preview the graph

Before running anything, use --dryrun to see the execution graph the recipe produces for a given set of inputs:

$ ninja run myrecipe.py:selfcal --ms data.ms --dryrun
[ image ]
    |
    v
[ mask ]

Run the recipe

Drop --dryrun to execute it for real:

$ ninja run myrecipe.py:selfcal --ms data.ms

Where to next

• Cabs – defining cabs in Python or loading them from YAML.

• Steps – @shinobi.step vs @shinobi.pystep.

• Recipes – declarative vs orchestration-function wiring.

• Backends – running natively, in containers, or on a cluster.

• Offloading to a cluster – compiling a recipe to Slurm and detaching.