API reference

This reference is generated from the source docstrings. The most useful names are re-exported from the top-level shinobi package and documented here; supporting types live in their home modules below.

Top-level package

class shinobi.Cab(*, name, info=None, inputs_model, outputs_model, backend=None, image=None, venv=None, input_mutability=<factory>, field_meta=<factory>, cache=None, cache_dir=None, sandbox=None, harvest=<factory>, scratch=<factory>, resources=None, command, flavour='binary', policies=<factory>, input_patterns=<factory>, output_patterns=<factory>, wranglers=<factory>)[source]

Bases: Scope

An atomic step backed by a single command.

Parameters:
command: str
flavour: str
policies: Policies
input_patterns: list[ParamPattern]
output_patterns: list[ParamPattern]
wranglers: dict[str, list[str]]
param_name(field)[source]

Resolve the tool-facing name for a declared input field.

Parameters:

field (str) – The cab’s own (shinobi-side) field name.

Returns:

The field’s nom_de_guerre if declared in field_meta, otherwise field unchanged.

Return type:

str

match_pattern(name)[source]

Check name against this cab’s dynamic input patterns.

Parameters:

name (str) – An input name not declared as a literal field.

Returns:

The matched ParamMeta, or None if no input_patterns entry matches.

Return type:

ParamMeta | None

match_output_pattern(name)[source]

Check name against this cab’s dynamic output patterns.

Parameters:

name (str) – An output name not declared as a literal field.

Returns:

The matched ParamMeta, or None if no output_patterns entry matches.

Return type:

ParamMeta | None

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.ExecContext(scope, raw_inputs, *, backend_override=None, recipe_backend=None, config=None, cache_enabled=False, cache_dir='', cache_path='', stream=True, pin=False, sandbox_root=None, clear_outputs=True, input_keys=None, budget=None, run_id='')[source]

Bases: object

Live execution state, created by _dispatch. inputs is a validated snapshot for inspection; the raw caller kwargs are kept separately because MUTABLE fields must reach the backend as the caller’s original objects.

Parameters:

  • scope (Scope)

  • raw_inputs (dict[str, Any])

  • backend_override (str | None)

  • recipe_backend (str | None)

  • config (AppConfig | None)

  • cache_enabled (bool)

  • cache_dir (str)

  • cache_path (str)

  • stream (bool)

  • pin (bool)

  • sandbox_root (str | None)

  • clear_outputs (bool)

  • input_keys (dict[str, Any] | None)

  • budget (Budget | None)

  • run_id (str)

prepare_inputs()[source]

Validated + mutability-processed inputs, with no overrides applied – for a plain-function step’s own function to call the underlying function with (see steps/pyfunc.py’s adapter, and the manual bare-Scope pattern documented on Scope/StepRef). Reuses the already-validated self.inputs snapshot rather than re-validating.

Return type:

dict[str, Any]

resolve_backend_name(override=None)[source]

Resolve the effective backend name using the standard priority chain. Exposed so orchestration functions (e.g. the pystep adapter) can inspect which backend is active without duplicating the precedence logic.

Parameters:

override (str | None)

Return type:

str

import_func(func, module=None)[source]

Import and return a callable by name.

If module is None, looks up func in builtins (e.g. print, len). Otherwise imports module and returns getattr(module, func).

Useful for pysteps that invoke container-only functions (e.g. CASA tasks) without triggering linter warnings about missing imports on the host.

Parameters:

  • func (str)

  • module (str | None)

Return type:

Callable

run(*, backend=None, **overrides)[source]

Run the underlying Cab or Recipe with optional input overrides.

Parameters:

  • backend (str | None) – Backend name to use for this run, taking priority over the scope’s own/recipe-inherited/config default.

  • **overrides (Any) – Input values to override on top of the raw inputs this context was created with.

Returns:

The step’s StepResult. Also stored on self.outputs.

Raises:

TypeError – If self.scope is neither a Cab nor a Recipe (a plain-function step must return its result directly instead of calling ctx.run()).

Return type:

StepResult

class shinobi.InputRef(*, field)[source]

Bases: BaseModel

Wiring source: this sub-step’s input comes from the enclosing Recipe’s own input field field.

Parameters:

field (str)

field: str
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.LoopIteration(*, loop, index, prev_step=None, sentinel_step=None, sentinel_field)[source]

Bases: BaseModel

Marks a step as belonging to iteration index of the unrolled loop loop (see Recipe.add_loop). Carried by every step the unrolling produces; it is bookkeeping for the skip decision only – the dependency edges that make the unrolled chain a real DAG are ordinary wiring plus after, never this.

prev_step is the same body step one iteration earlier (so selfcal.4.image passes through selfcal.3.image), and sentinel_step/sentinel_field name the previous iteration’s output whose existence on disk means “the loop has already converged, do no work”. prev_step and sentinel_step are both None for the first iteration, which can never skip – there is nothing before it to pass through or to have converged. sentinel_field is always set: which output carries the signal is a property of the loop, not of one iteration.

Parameters:

  • loop (str)

  • index (int)

  • prev_step (str | None)

  • sentinel_step (str | None)

  • sentinel_field (str)

loop: str
index: int
prev_step: str | None
sentinel_step: str | None
sentinel_field: str
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.LoopRef(name, steps, final)[source]

Bases: object

What Recipe.add_loop returns: the loop’s step names, and an outputs proxy resolving to the final iteration’s producers.

Not a StepRef – a loop is not a step. It has already been unrolled into the recipe’s steps by the time this is returned; this is just a handle for wiring whatever comes next.

Parameters:
property outputs: _LoopOutputsProxy

Wiring proxy (definition layer) – NOT runtime values.

class shinobi.Mutability(*values)[source]

Bases: str, Enum

Whether a step’s input may be changed in place by the step’s own orchestration function without that change propagating back to the caller’s object.

IMMUTABLE = 'immutable'
MUTABLE = 'mutable'
class shinobi.OutputRef(*, step, field)[source]

Bases: BaseModel

Wiring source: this input (or, in Recipe.output_wiring, the recipe’s own output) comes from step step’s output field field.

Parameters:
step: str
field: str
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.Recipe(*, name, info=None, inputs_model, outputs_model, backend=None, image=None, venv=None, input_mutability=<factory>, field_meta=<factory>, cache=None, cache_dir=None, sandbox=None, harvest=<factory>, scratch=<factory>, resources=None, steps=<factory>, output_wiring=<factory>, max_workers=None)[source]

Bases: Scope

A composite step: declared sub-steps with explicit wiring.

The one deliberately mutable Scope: builder methods (add_step, step, set_output) extend steps/output_wiring before first run.

Parameters:
steps: list[StepRef]
output_wiring: dict[str, OutputRef]
max_workers: int | None
property inputs: _InputsProxy

Wiring proxy (definition layer) – NOT runtime values.

property outputs: _OutputsProxy

Wiring proxy (definition layer) – NOT runtime values.

add_step(name, scope, *, scatter=None, resources=None, **kwargs)[source]

Add a step. scope is usually a bare Scope/Cab/Recipe, but can also be an already-built StepRef (e.g. from @shinobi.pystep or @shinobi.step) – its func is carried over so the step keeps its orchestration function, not just its schema.

Parameters:

  • scatter (list[str] | ScatterSpec | None) – Fields to fan out over. Each must be a field of the step’s inputs_model; at runtime the corresponding value must be a list, and every scattered field must have the same length.

  • resources (Resources | None) – What this step costs to run, for the scheduler’s admission control. An explicit keyword rather than a kwarg, since **kwargs here is split into wiring and step params and would otherwise swallow it as a constant input value.

  • name (str)

  • scope (Scope | StepRef)

  • kwargs (Any)

Return type:

Recipe

add_loop(name, body, *, max_iter, until, carry, index_input=None, **kwargs)[source]

Unroll body max_iter times into this recipe, as a declared chain.

The loop is bounded unrolling with short-circuit pass-through: the body’s steps are flattened into this recipe once per iteration and chained by real carry wiring, so the whole thing is an ordinary statically-inspectable DAG – –dryrun renders every iteration and the offload compiler emits a plain dependency chain. The only runtime decision is whether an already-declared step does any work: once an iteration writes the until sentinel file, every later step passes the corresponding previous iteration’s outputs straight through.

A Recipe body is flattened (selfcal.3.image); any other Scope is one step per iteration (selfcal.3).

Parameters:

  • name (str) – The loop’s name; every step it creates is prefixed with it.

  • body (Scope | StepRef) – The loop body. Its outputs must be re-consumable as its inputs (see carry) – a loop body is a fixed point.

  • max_iter (int) – How many times to unroll. The upper bound on iterations, not the exact count.

  • until (str) – Name of a path-typed output of body. Once that file exists, the loop has converged. A path (rather than a bool) is what lets the identical predicate serve both local execution and an offloaded sbatch script.

  • carry (dict[str, str]) – Body output field -> body input field, the loop-carried dependency. Required and explicit: these pairs are the edges between iterations, and inferring them from matching names would make the graph’s shape depend on name coincidence.

  • index_input (str | None) – Name of a body input to bind to the 1-based iteration number. Without it every iteration resolves to identical values, so a body cannot name its outputs per cycle (a per-cycle image prefix, say) – which real self-calibration does. Steps consuming it must tolerate a changing value: it is the one input that deliberately differs between iterations.

  • **kwargs (Any) – Iteration 1’s inputs, split into wiring and constants exactly as add_step does. Constants apply to every iteration.

Returns:

A LoopRef whose outputs proxy resolves to the final iteration.

Return type:

LoopRef

step(scope, *, backend=None, scatter=None, resources=None, **kwargs)[source]

Decorate a function as a new step appended to this recipe.

Parameters:

  • scope (Scope) – The Cab, Recipe, or bare Scope to bind as this step.

  • backend (str | None) – Backend override for this step.

  • scatter (list[str] | ScatterSpec | None) – Fields to fan out over (see add_step).

  • resources (Resources | None) – What this step costs to run (see add_step).

  • **kwargs (Any) – Split into wiring (InputRef/OutputRef values) and per-step constant params via _split_kwargs.

Returns:

A decorator that binds the given function, appends the resulting StepRef to self.steps, and returns it.

set_output(field, ref)[source]

Wire a recipe output field to an upstream step’s output.

Parameters:

  • field (str) – Name of the recipe’s own output field.

  • ref (OutputRef) – The step output that should populate field.

Returns:

self, for chaining.

Return type:

Recipe

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.ScatterSpec(*, fields)[source]

Bases: BaseModel

Declaration that a step should fan out over one or more list-typed input fields. Each listed field must be a list at runtime and all listed fields must have the same length. The step is executed once per index, with slice i receiving element i of every scattered field.

The step’s own inputs_model/outputs_model describe one slice. A downstream step sees the scattered step’s outputs gathered into lists (one element per slice), so it can scatter over them in turn or consume the whole list as a gathered result.

Parameters:

fields (list[str])

fields: list[str]
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.Scope(*, name, info=None, inputs_model, outputs_model, backend=None, image=None, venv=None, input_mutability=<factory>, field_meta=<factory>, cache=None, cache_dir=None, sandbox=None, harvest=<factory>, scratch=<factory>, resources=None)[source]

Bases: BaseModel

Definition: schema, metadata, backend config. Never carries inputs/outputs/func fields – those live in ExecContext/StepRef.

Cab/Recipe are the two execution-aware subclasses ExecContext.run() knows how to run. A bare Scope is also valid – it’s the manual building block for a plain-Python-function step whose own function returns its StepResult directly rather than calling ctx.run(); see StepRef’s docstring and steps/pyfunc.py’s @shinobi.pystep (which automates this pattern from a function’s own signature).

image is optional: when set on a bare Scope (typically via @shinobi.pystep(image=…)), the step’s Python function can be executed inside a container instead of in-process. Cab inherits this field for the same purpose (container backends need it to wrap argv in a runtime invocation).

Parameters:
name: str
info: str | None
inputs_model: type[BaseModel]
outputs_model: type[BaseModel]
backend: str | None
image: str | None
venv: str | None
input_mutability: dict[str, Mutability]
field_meta: dict[str, ParamMeta]
cache: bool | None
cache_dir: str | None
sandbox: bool | None
harvest: list[str]
scratch: list[str]
resources: Resources | None
mutability_of(field)[source]

Look up the declared mutability of an input field.

Parameters:

field (str) – Name of the input field.

Returns:

The field’s Mutability, defaulting to Mutability.IMMUTABLE if not explicitly declared.

Return type:

Mutability

with_backend(backend)[source]

A copy bound to backend, or self unchanged if backend is None. Shared by @shinobi.step and Recipe.step, which both bind a per-step backend override onto a Scope before wrapping it in a StepRef.

Parameters:

backend (str | None)

Return type:

Scope

with_resources(resources)[source]

A copy bound to resources, or self unchanged if None. Same shape as with_backend, and used the same way: Recipe.add_step, Recipe.step and @shinobi.step bind a per-step footprint onto a Scope before wrapping it in a StepRef, so one shared Cab can be declared cheap in one recipe position and expensive in another without either mutating the Cab itself.

Parameters:

resources (Resources | None)

Return type:

Scope

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.StepRef(*, name, step, func=None, wiring=<factory>, params=<factory>, scatter=None, after=<factory>, loop=None)[source]

Bases: BaseModel

A named, executable binding of a Scope: orchestration function, wiring (meaningful only inside a Recipe), and per-step constants. Returned by @shinobi.step (free-standing) and @recipe.step (appended to recipe.steps). arbitrary_types_allowed is needed only for func.

step is typed as the general Scope (not Cab | Recipe) so it can also hold a bare Scope – the manual, no-magic way to write a plain-Python-function step: build Scope(name=, inputs_model=, outputs_model=) yourself, write a function that always returns its own StepResult (never calls ctx.run(), which only knows how to execute a Cab or Recipe), and wrap it in a StepRef directly. @shinobi.pystep (steps/pyfunc.py) automates exactly this pattern by deriving the Scope’s schema from the function’s own signature. Passing a Cab/Recipe instance here is unaffected – pydantic’s default revalidate_instances=”never” keeps an already-constructed instance’s real subtype, it does not downcast to bare Scope.

Parameters:
model_config = {'arbitrary_types_allowed': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str
step: Scope
func: Callable | None
wiring: dict[str, 'InputRef | OutputRef | list[InputRef | OutputRef]']
params: dict[str, Any]
scatter: ScatterSpec | None
after: list[str]
loop: LoopIteration | None
shinobi.pystep(*, name=None, info=None, image=None, venv=None, backend=None, sandbox=None, harvest=None, write_paths=None, **params)[source]

Decorate (or directly call on an existing function, matching @shinobi.step’s precedent: pystep()(existing_func)) a plain, type-hinted function to turn it into a StepRef. See the module docstring for the schema-derivation and outputs rules.

image enables container execution: when set and a container backend is resolved, the function runs inside the specified container image instead of in-process. The function’s source module is mounted into the container so it can be imported by the runner script.

venv enables virtualenv execution: when set (a path or a key into backend.venv.envs) and the venv backend is resolved, the function runs under that venv’s own interpreter, importing the venv’s real packages (unlike the container path, the target’s own package is not stubbed). A scope may carry both image and venv; the resolved backend name decides which is used.

backend sets the default backend for this step (same as on any Scope). With image, this is typically a container backend name like "docker" or "apptainer"; with venv, "venv".

sandbox/harvest opt this step into sandboxed execution and declare extra keep-globs (see shinobi.sandbox and the fields on Scope). Sandboxing applies to the out-of-process paths (container and venv) – an in-process run ignores it (os.chdir is process-global, and recipes run steps on a thread pool), executing in the caller’s cwd as always.

write_paths names the parameters that are destinations this function creates rather than data it reads (ParamMeta.write_path) – the one thing no signature can express, since outputvis: Path returned as the output and vis: Path rewritten in place are the same annotation. Only a named parameter has its stale product cleared before a re-run (sandbox.clear_stale_outputs); everything else is treated as the caller’s data and left alone.

**params are per-call constants, same as @shinobi.step.

Parameters:
Return type:

Callable[[Callable], StepRef]

shinobi.step(scope, *, backend=None, name=None, scatter=None, resources=None, **params)[source]

Decorate a function with an existing Scope (Cab or Recipe). See the module docstring.

Parameters:
Return type:

Callable[[Callable], StepRef]

Schema helpers

Supporting types used when defining cabs, not re-exported at the top level.

class shinobi.steps.schema.ParamMeta(*, nom_de_guerre=None, implicit=None, info=None, positional=False, positional_head=False, repeat_as_tokens=False, dtype=None, choices=None, abbreviation=None, write_path=False, writable=None)[source]

Bases: BaseModel

Per-field metadata a plain pydantic model can’t express: the name the underlying tool actually expects (nom_de_guerre), a value always supplied by the cab itself rather than the caller (implicit), human-facing help (info), the cab dtype string (dtype, e.g. “File”/”MS”) for a ParamPattern attr – since a dynamically-named param has no declared field/type annotation for path_fields to inspect, this is how backends know to bind-mount its directory – positional: emitted as a bare value (no –flag), in field-declaration order, after every flagged/pattern-matched arg. positional_head: the same, but emitted before every flagged/ pattern-matched arg instead of after – real cult-cargo’s own cubical.yml names this exact policy (parset: {policies: {positional_head: true}}) for a tool whose own CLI only recognises a leading bare token as a parset file to seed defaults from (CubiCal’s main.py: if len(sys.argv) > 1 and not sys.argv[1][0].startswith(‘-‘): custom_parset_file = sys.argv[1]); killMS’s kMS.py has the identical sys.argv[1]-only check. A plain positional field there would always land after every –flag, which these two tools’ own argv[1]-anchored parset detection can’t see – either raising (“Unexpected number of arguments”, CubiCal) or silently not reading the parset at all (killMS, which never validates leftover-arg count). Setting both positional and positional_head on the same field is nonsensical; positional_head wins if both are set. Head positionals, like tail ones, are emitted in field-declaration order. – and repeat_as_tokens: a list/tuple value is emitted as separate bare argv tokens (after the one flag occurrence, or as separate positional tokens) instead of joined into one comma-separated token – real cult-cargo cabs express this as a per-field policies: {repeat: list} (see e.g. wsclean’s -size <w> <h>/-weight briggs <n>, which need two separate argv tokens, not “4096,4096” as one).

write_path: this string-typed input names a filesystem path the tool writes to. Two shapes, both covered: a stem products are built from (wsclean’s prefix, ddfacet’s Output-Name, which never exists as a file itself), and a complete path written directly (a cache directory, a logfile). It is a declaration of intent only, and deliberately changes nothing about how the value is handled. In particular it does not make the field a path: a path dtype here would be rewritten by sandbox.absolutize_path_inputs and the tool would then write outside the sandbox, which is precisely why the convention types these as strings (see declared_output_dirs). Mounting and pre-creation still come from the output side – a path-typed output whose implicit template names this field, a harvest glob, or a scratch glob.

What it buys is enforcement. That convention previously existed only as prose and authorial discipline: a write path nobody declared a target for produces no error, just products that stay inside the container or outside the harvest. Marking the field lets Cab check that something does declare it (see Cab._write_path_declares_a_write_target).

A path-typed input may be marked too, and there it is the only thing that distinguishes the two identical-looking spellings of an output field that echoes a same-named input. mstransform takes outputvis and creates it; flagdata takes vis and rewrites the caller’s own data; both declare that one name on inputs_model and on outputs_model, so mutated_path_fields – and any other structural test – sees one shape. Marking outputvis says which it is, and sandbox.clear_stale_outputs then clears the stale product before a re-run instead of leaving the tool to trip over it. The default (unmarked) is the safe reading: an echoed path is the caller’s data and is never deleted. Note the marker still changes nothing about how the value is handled – a path-typed one is anchored at the workspace by sandbox.absolutize_path_inputs exactly as before, which is what a complete destination path (as opposed to a stem) wants.

writable: this path-typed input is one the tool must not modify (writable: false, cult-cargo/scabha’s own key). The container backends bind-mount the directories a read-only input contributes :ro, and re-assert the input itself :ro at its own path when something writable shares that directory (backends.container.bind_dir_modes). None – the default, and every Python-typed pystep input – means unmarked, which is read as writable. Carried onto a declared field’s json_schema_extra by the loaders (readonly_path_fields reads it from there, the same channel abbreviation rides); kept here as well because a ParamPattern attr has no declared model field for readonly_path_fields to inspect, and a dynamically-named input is exactly the case that wants the marker – QuartiCal’s <term>.load_from names a previous run’s gain store, which the step reads and must not write back into.

choices: the field’s allowed values (cult-cargo/classic’s choices key). A loader that sets this also narrows the field’s real annotation on inputs_model/outputs_model to typing.Literal[*choices] (see loaders._modelgen.narrow_choices), so an out-of-set value fails pydantic validation the same way a wrong dtype would – not merely documented in info. Kept here too (rather than only inferred from the model’s own annotation) so a ParamPattern attr – which has no declared model field for a dynamically-matched name – can still carry it.

abbreviation: a short single-dash CLI alias for the field’s generated –long-flag (cult-cargo/classic’s abbreviation key, e.g. simms’ ascii-sky -> -as). Purely a ninja run convenience – carried onto the field’s json_schema_extra by the loaders so clickutil.build_options can emit the alias; it never affects the argv the tool actually receives (that still uses nom_de_guerre).

On an output field, a string implicit containing {name} placeholders is resolved by steps.dispatch._fill_outputs as a str.format template against the step’s prepared (validated) input values – e.g. implicit=”{prefix}-MFS-image.fits” derives a tool’s output path from its own prefix input, without shinobi ever importing/executing the tool’s own schema-generation code. A plain string with no {…} is used as a literal constant, same as on an input field.

Parameters:

  • nom_de_guerre (str | None)

  • implicit (Any)

  • info (str | None)

  • positional (bool)

  • positional_head (bool)

  • repeat_as_tokens (bool)

  • dtype (str | None)

  • choices (list[Any] | None)

  • abbreviation (str | None)

  • write_path (bool)

  • writable (bool | None)

nom_de_guerre: str | None
implicit: Any
info: str | None
positional: bool
positional_head: bool
repeat_as_tokens: bool
dtype: str | None
choices: list[Any] | None
abbreviation: str | None
write_path: bool
writable: bool | None
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.steps.schema.ParamPattern(*, separator='.', segments)[source]

Bases: BaseModel

A family of inputs whose names are <segment><separator><segment>…, e.g. QuartiCal’s K.type/G.time_interval or cubical’s g1-solvable/ g-time-int. Matched as one anchored regex assembled from segments: exactly one segment is attrs (the known, enumerable part, each value with its own ParamMeta – dtype/nom_de_guerre/info); every other segment is a regex (soft shape-validation of a level that can’t be enumerated ahead of time). See AGENTS.md for the motivating tools.

The attrs segment is usually last (cubical/QuartiCal: an unenumerable term name followed by a known attribute, g1.solvable), but doesn’t have to be – wsclean’s dynamic output names are the opposite shape, a known/enumerable image type followed by an open-ended qualifier tail (dirty.per-band, restored.i.per-interval.mfs), so attrs there is the first segment. Only one segment may carry attrs; the rest must all be regex.

A segment regex that should behave as an unconstrained “match anything” level (the old design’s prefix) should be written lazily (.+?, not .+): with more than one registered attr, an eager .+ prefers the shortest attr that completes an overall match, which is wrong when one attr is itself a suffix of another (e.g. “int” vs “time-int” with separator “-”) – .+? tries the shortest prefix first, which is exactly “prefer the longest/most specific attr”.

Parameters:
separator: str
segments: list[ParamSegment]
matches(name)[source]

Check whether name matches this pattern and look up its metadata.

Parameters:

name (str) – A dynamic parameter name to test, e.g. “g1.solvable”.

Returns:

The ParamMeta for the matched attrs segment value, or None if name doesn’t match the compiled pattern.

Return type:

ParamMeta | None

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

class shinobi.steps.schema.ParamSegment(*, regex=None, attrs=None)[source]

Bases: BaseModel

One level of a dotted/dashed dynamic-parameter name. A “shape” segment carries only regex – soft validation, no metadata, for a level whose actual values can’t be enumerated at cab-authoring time (e.g. a solver term name like QuartiCal’s K/G). The “meta” segment – always the last one in a ParamPattern – carries attrs: the known, enumerable part, each value with its own ParamMeta.

Parameters:
regex: str | None
attrs: dict[str, ParamMeta] | None
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.steps.schema.Policies(*, prefix='--', replace=<factory>, list_sep=', ', repeat_list=False, key_value=False, repeat=None, explicit_true=False, explicit_false=False, true_token='true', false_token='false')[source]

Bases: BaseModel

How a cab’s parameters are turned into command-line arguments.

key_value/repeat mirror real cult-cargo cab-level policy keys verbatim (e.g. QuartiCal’s policies: {key_value: true, repeat: ‘[]’, prefix: ‘’}): key_value=True means a hydra-style single name=value argv token instead of two tokens (–name, value); repeat=”[]” means a list value formats as one bracketed-literal token (solver.terms=[K,G]) instead of list_sep-joining. Distinct from a per-field ParamMeta.repeat_as_tokens (real per-field policies: {repeat: list}, e.g. wsclean’s bare -size 4096 4096), which is a field-level override and takes precedence when set.

explicit_true/explicit_false also mirror real cult-cargo cab-level policy keys verbatim (e.g. CubiCal’s policies: {explicit_true: true, explicit_false: false}): by default a True boolean value emits as a bare flag (–flag, argparse store_true-style) and False is omitted entirely. Some real CLIs (CubiCal’s own optparse-derived parser among them) instead expect every boolean option to always take an explicit value token – passing a bare flag with no value corrupts parsing of everything after it, since the parser consumes the next token as that flag’s value. explicit_true=True emits –flag true (two tokens, “true”/”false” lowercase) instead of a bare flag when the value is True; explicit_false=True does the same instead of omitting the flag when the value is False. Each direction is independent (CubiCal only needs explicit_true, never explicit_false), and this applies uniformly to declared fields and ParamPattern-matched dynamic ones (e.g. CubiCal’s own per-Jones-term g-solvable).

true_token/false_token are what those two tokens actually say. Lowercase “true”/”false” (the default) is what CubiCal’s parser reads, but it is not universal: DDFacet and killMS share a parset reader (DDFacet.Parset.ReadCFG) that parses “0”/”1” as ints and “True”/”False” as bools, and leaves an unrecognised “false” as a string – which, being non-empty, is truthy. –Mask-Auto false there switches the mask on. So the pair of tokens is a cab-level policy of its own (policies: {explicit_true: true, explicit_false: true, true_token: ‘1’, false_token: ‘0’} for those two), alongside the explicit_* switches rather than instead of them: explicit_* decides whether a value token is emitted, these decide what it reads.

They spell a boolean in every value position, not only after a flag: a key_value cab’s single name=value token, a positional, an element of a joined or bracketed list, one occurrence of a repeated flag. Python’s own str(True) is “True”, so a path that missed this emitted a spelling no policy ever asked for (policies._scalar_token is the one place that decides).

Parameters:
prefix: str
replace: dict[str, str]
list_sep: str
repeat_list: bool
key_value: bool
repeat: str | None
explicit_true: bool
explicit_false: bool
true_token: str
false_token: str
arg_name(name)[source]

Build the CLI flag name for a parameter name.

Parameters:

name (str) – The parameter’s declared/matched name.

Returns:

name with each replace substitution applied, prefixed by prefix (e.g. “–”).

Return type:

str

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.steps.schema.ScatterSpec(*, fields)[source]

Bases: BaseModel

Declaration that a step should fan out over one or more list-typed input fields. Each listed field must be a list at runtime and all listed fields must have the same length. The step is executed once per index, with slice i receiving element i of every scattered field.

The step’s own inputs_model/outputs_model describe one slice. A downstream step sees the scattered step’s outputs gathered into lists (one element per slice), so it can scatter over them in turn or consume the whole list as a gathered result.

Parameters:

fields (list[str])

fields: list[str]
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

shinobi.steps.schema.path_fields(model)[source]

Names of every field of model whose (Optional/list-unwrapped) type is a filesystem path (pathlib.Path). File-like cab dtypes (File/MS/Directory/URI) map to Path, so this drives both container bind-mounting and the CLI’s click.Path() mapping.

Parameters:

model (type[BaseModel])

Return type:

set[str]

shinobi.steps.schema.declared_output_dirs(scope, prepared)[source]

(directory, source) for every directory the step is declared to write into, resolved before the run from the declarations alone – never from the shape of an input’s value.

This is the answer to “where does this tool put its products”, which a field’s dtype cannot express: the established way to declare an output stem is a string-typed input (wsclean’s prefix, ddfacet’s Output-Name), deliberately so, because a path dtype would be rewritten by sandbox.absolutize_path_inputs and the tool would then write outside the sandbox. Such an input can say so with ParamMeta.write_path, which changes nothing here – this function reads the declarations either way – but lets Cab reject a stem that no declaration names, which is the authoring mistake this convention is otherwise silent about. The write target is instead declared by the output side: a path-typed output field whose implicit template names the input ("{prefix}-MFS-image.fits"), a harvest glob, or a scratch glob for a write target that is not a product.

Knowable pre-run means the parent of each declared_output_paths entry (per path-typed output field, its value taken from a same-named input, its resolved implicit template, or its field default – the same priority as _fill_outputs), then the literal (glob-free) directory prefix of each harvest and scratch pattern. source describes the declaration (output 'restored_image', harvest pattern '{prefix}-*.fits', scratch pattern '{cache_dir}/*') so callers can name it in an error.

A pattern that references a field whose value is None is skipped, not resolved: substituting it would produce a literal None path segment ("{cache_dir}/*" -> "None/*"), which is a real directory name and would have the sandbox create one. An unset optional write target declares nothing, which is the same answer a path-typed output whose value is None already gives.

Both relative and absolute directories are returned, order-preserving and de-duplicated; . and anything containing .. is dropped. The two consumers filter complementary halves: sandbox pre-creates the relative ones inside the scratch dir, the container backend bind-mounts the absolute ones so a product declared outside the workdir still reaches the host. Both consumers treat a scratch directory exactly like a product’s: it is mounted so the tool’s write lands on the host, and pre-created so the tool doesn’t have to. Only harvest differs – it never rescues a scratch path out of a sandbox, which is the whole point of the distinction. Best-effort by design – a template that fails to resolve is skipped here, not raised; output filling and harvest report those errors with full context.

Parameters:
Return type:

list[tuple[Path, str]]

Execution

class shinobi.results.BackendRun(returncode, stdout='', stderr='', image_digest=None, containerized=False, venv=None, venv_digest=None, stdout_dropped=0, stderr_dropped=0, wrangler_lines_dropped=False)[source]

Bases: object

What a backend returns after running a command – just the raw run outcome. Wrangling stdout/stderr into structured outputs and filling an outputs_model is the dispatch layer’s job, not the backend’s.

Parameters:

  • returncode (int)

  • stdout (str)

  • stderr (str)

  • image_digest (str | None)

  • containerized (bool)

  • venv (str | None)

  • venv_digest (str | None)

  • stdout_dropped (int)

  • stderr_dropped (int)

  • wrangler_lines_dropped (bool)

returncode: int
stdout: str = ''
stderr: str = ''
image_digest: str | None = None
containerized: bool = False
venv: str | None = None
venv_digest: str | None = None
stdout_dropped: int = 0
stderr_dropped: int = 0
wrangler_lines_dropped: bool = False
property success: bool

Whether the run exited with return code 0.

class shinobi.results.StepResult(name, returncode, outputs, inputs, stdout='', stderr='', cached=False, skipped=False, kind='cab', backend=None, image=None, image_digest=None, containerized=False, venv=None, venv_digest=None, sandboxed=False, resources=None, sub_results=None, cache_key=None, output_keys=None)[source]

Bases: object

The outcome of running a step (a Cab or a Recipe).

outputs is a validated instance of the step’s outputs_model; inputs is a validated instance of the effective (post-override) inputs the step actually ran with. For a Recipe these aggregate from its sub-steps.

Parameters:

  • name (str)

  • returncode (int)

  • outputs (BaseModel)

  • inputs (BaseModel)

  • stdout (str)

  • stderr (str)

  • cached (bool)

  • skipped (bool)

  • kind (str)

  • backend (str | None)

  • image (str | None)

  • image_digest (str | None)

  • containerized (bool)

  • venv (str | None)

  • venv_digest (str | None)

  • sandboxed (bool)

  • resources (Resources | None)

  • sub_results (dict[str, StepResult] | None)

  • cache_key (str | None)

  • output_keys (dict[str, ProvenanceKey] | None)

name: str
returncode: int
outputs: BaseModel
inputs: BaseModel
stdout: str = ''
stderr: str = ''
cached: bool = False
skipped: bool = False
kind: str = 'cab'
backend: str | None = None
image: str | None = None
image_digest: str | None = None
containerized: bool = False
venv: str | None = None
venv_digest: str | None = None
sandboxed: bool = False
resources: Resources | None = None
sub_results: dict[str, StepResult] | None = None
cache_key: str | None = None
output_keys: dict[str, ProvenanceKey] | None = None
property success: bool

Whether the step exited with return code 0.

provenance_key(field)[source]

The cache key identifying whatever produced output field.

A leaf step produces all its outputs in one run, so every field resolves to that step’s own cache_key. A Recipe fans out to output_keys instead – each declared output comes from a distinct sub-step, and invalidating a downstream consumer because some unrelated sub-step re-ran would throw away most of the cache’s value.

None means “no provenance available” (caching disabled, or a step that isn’t cacheable) – callers must treat that as “contribute nothing”, not as a key in its own right.

Parameters:

field (str)

Return type:

Any

shinobi.steps.dispatch.register_step_backend(name, backend)[source]

Register a backend instance under name, overriding the real class-based registry. Mainly for tests.

Parameters:
Return type:

None

shinobi.steps.dispatch.get_step_backend(name)[source]

Resolve a backend instance by name, checking test overrides first.

Parameters:

name (str) – Backend name, e.g. “native”, “slurm”, or a name registered via register_step_backend.

Returns:

The backend instance registered under name in _STEP_BACKENDS, else a fresh instance from shinobi.backends.get_backend.

Return type:

Any

Short-circuit semantics for an unrolled loop (see Recipe.add_loop).

A loop is declared, not interpreted: add_loop flattens its body into the recipe max_iter times and chains the copies with real wiring, so the graph is fully inspectable before anything runs. What stays a run-time decision is narrow – whether an already-declared step does any work. This module is that decision, and nothing else.

The rule lives here rather than in the scheduler because two tiers evaluate it: _run_recipe calls should_skip in-process, and the Slurm offload compiler emits the same predicate as a shell [ -f … ] test at the top of each iteration’s script. A convergence signal that is a path is what makes that possible – a bool would have no way to cross a node boundary, and the two tiers would need separate definitions that could drift into running a different number of cycles for the same recipe.

shinobi.steps.loops.sentinel_exists(value)[source]

Whether a convergence sentinel has actually been produced.

Parameters:

value (Any) – The sentinel output’s value – a path, or None if the producing step never set it.

Returns:

True if value names an existing file or directory.

Return type:

bool

shinobi.steps.loops.sentinel_value(ref, results)[source]

The sentinel value this step’s skip decision reads, or None if it cannot skip (the first iteration, or a step outside any loop).

Parameters:

  • ref (StepRef) – The step about to be scheduled.

  • results (dict[str, StepResult]) – Completed steps by name. The sentinel producer is guaranteed to be present: add_loop gives every iteration an edge to it, so the scheduler cannot reach this step first.

Returns:

The previous iteration’s sentinel output value, or None.

Return type:

Any

shinobi.steps.loops.should_skip(ref, results)[source]

Whether ref should pass its predecessor’s outputs through instead of running, because an earlier iteration already converged.

Parameters:
Returns:

True if the previous iteration’s sentinel exists on disk.

Return type:

bool

shinobi.steps.loops.passthrough_result(ref, prev, inputs)[source]

The result of a step that was skipped: the same body step’s outputs from one iteration earlier, handed on unchanged.

Passing the previous outputs object through (rather than re-deriving field by field) is what makes convergence sticky without extra bookkeeping: the sentinel is itself one of those outputs, so every later iteration sees it and skips in turn.

cache_key/output_keys are carried over too. A skipped step produced no new data, so contributing nothing would drop the upstream term from every downstream cache key (see shinobi.cache) and needlessly invalidate work that is genuinely unchanged.

kind deliberately keeps the scope’s real kind – skipped is a separate flag. shinobi.provenance.apply_manifest_pins asserts that a record’s kind still matches the scope’s type, so inventing a “skipped” kind would make any early-converging run unreplayable.

Parameters:

  • ref (StepRef) – The skipped step.

  • prev (StepResult) – The corresponding step’s result from the previous iteration.

  • inputs (Any) – The validated inputs this step would have run with.

Returns:

A successful StepResult marked skipped.

Return type:

StepResult

Backends

Backend abstraction: a backend takes a cab and a resolved argv and runs it somewhere – natively, in a container, on Slurm, on Kubernetes, …

A backend knows nothing about recipes or output schemas beyond the argv it’s handed and the cab’s image/command metadata; it only knows how to execute and how to capture output. Wrangling that output into structured results is the dispatch layer’s job, so a backend returns a raw BackendRun (returncode/stdout/stderr), nothing schema-aware.

class shinobi.backends.Backend[source]

Bases: ABC

Abstract base class for execution backends.

Variables:

name (str) – Registry key used to look up this backend (e.g. "native", "container", "slurm").

name: str
abstractmethod run(cab, argv, inputs, *, label='', stream=True, pin=False, cwd=None)[source]

Execute argv (as built by shinobi.policies.build_argv) and return a BackendRun. Must not raise on a non-zero exit – that’s reported via BackendRun.returncode / BackendRun.success.

inputs is the prepared inputs dict argv was built from (the one _prepare_inputs produces, so MUTABLE fields are the caller’s own objects by reference). Most backends ignore it, but container backends need it to know which File/MS-valued params to bind-mount.

label/stream control live stdout/stderr echo (see shinobi.backends._stream.run_streaming) – only native and container act on them today; slurm/kubernetes accept and ignore both (neither has any log-tailing infrastructure yet, so they keep reading output once after the job/pod finishes).

cwd is the working directory to run in (shinobi.sandbox passes the step’s sandbox here), defaulting to the process cwd. slurm/ kubernetes accept and ignore it too: they run in a remote/pod cwd the dispatch layer can’t scope, so a sandboxed step on those backends degrades gracefully to an unsandboxed run (harvest finds the outputs already in the workspace and moves nothing).

Parameters:
Return type:

BackendRun

shinobi.backends.register(backend_cls)[source]

Register a backend class under its name attribute.

Intended for use as a class decorator on Backend subclasses.

Parameters:

backend_cls (type[Backend]) – The backend class to register.

Returns:

The same class, unmodified, so it can be used as a decorator.

Return type:

type[Backend]

shinobi.backends.get_backend(name, **opts)[source]

Instantiate a registered backend by name.

Parameters:

  • name (str) – Registry key of the backend (e.g. "native", "slurm").

  • **opts – Keyword arguments forwarded to the backend’s constructor.

Returns:

A new instance of the requested backend.

Raises:

ValueError – If no backend is registered under name.

Return type:

Backend

shinobi.backends.registered_backend_names()[source]

Every name a backend is registered under, sorted.

The authoritative answer to “is this a real backend?”, for callers that need to check a name rather than instantiate it – the CLI validating –backend before a run starts (shinobi.cli). Not every consumer of a backend name reaches get_backend: the pystep adapter matches the resolved name against CONTAINER_RUNTIMES and otherwise runs the function in-process, so a typo there would go unnoticed rather than raise.

Returns:

The registry keys, sorted.

Return type:

list[str]

shinobi.backends.registered_backend_classes()[source]

Return all backend classes currently registered.

Returns:

A list of registered Backend subclasses.

Return type:

list[type[Backend]]

Building argv

Turn a cab’s schema + resolved parameter values into a command line.

Operates on the step-model Cab: the parameter values come from an already-validated inputs_model instance (or the prepared dict dispatch builds from it), while per-field naming/implicit metadata comes from the cab’s field_meta, dynamically-named params from input_patterns, and arg formatting from policies.

shinobi.policies.build_argv(cab, resolved)[source]

Build a full argv (starting with the cab’s command) from a resolved parameter dict, according to the cab’s policies and field metadata.

Rejects any non-“binary” flavour before building argv – so a non-executable command can never reach subprocess as argv[0] (see SECURITY.md).

Parameters:
Return type:

list[str]

Loaders

Load YAML cab definitions in the scabha dialect into shinobi Cab objects.

Lineage. shinobi’s cab schema is borrowed from scabha, the schema library underneath Stimela 2.0, and the vocabulary here is deliberately scabha’s: inputs/outputs with dtype/required/default/info/choices, policies, management.wranglers, image, flavour, command. Reusing it was a design decision, not an accident of history – the cab schema is the part of stimela2 that got it right, and shinobi’s own Cab mirrors it closely enough that loading a scabha cab is a translation rather than an interpretation. What shinobi drops is the layer above the cab: stimela2’s recipe, alias and expression machinery (see stimela-ninja’s AGENTS.md).

cult-cargo is the largest published library of cabs written in this dialect and is what this loader is usually pointed at, but the dialect is scabha’s and nothing here is specific to that project. shinobi.loaders.worker_schema reads a scabha-derived config dialect through the same shared helpers.

shinobi-native keys. shinobi’s own Cab carries a few things scabha has no vocabulary for, so the dialect accepts them as an extension rather than inventing a second format for cabs authored against shinobi directly. A document using none of them is a plain scabha document, and cult-cargo’s own files remain a readable subset.

Per field, alongside the scabha keys: write_path: true marks a string-typed input naming a filesystem path the tool writes to – a stem products are built from, or a complete path written directly (see ParamMeta.write_path) – and mutable: true marks an input the step may change in place (Mutability.MUTABLE). Both are registered in _LEAF_SPEC_KEYS, which matters more than it looks: _is_section tells a leaf param from a nested CLI section by whether the mapping has any known param-spec key, so a spec carrying only an unregistered key is read as a section and the field disappears without a word.

Per cab: sandbox, harvest and scratch, which mirror the Scope fields of the same names.

image: may also name a key rather than a reference, resolved through the caller-supplied images mapping (see loads) – the same shape as package_roots, and for the same reason: shinobi has no manifest and does not go looking for one. It lets a document say image: WSCLEAN and leave which reference that is to the deployment that loads it.

Also per cab: input_patterns/output_patterns, families of dynamically-named params (ParamPattern). A pattern is a separator plus ordered segments; each segment is either a regex (a level that cannot be enumerated ahead of time) or attrs (the known level, each attr a param spec in its own right). ParamPattern’s own validator enforces the real rule – exactly one segment carries attrs – so the loader only produces the shape and lets it object, naming the cab and key when it does.

An attr spec is read by the same _param_meta as a declared field, with one asymmetry: an attr keeps its dtype, a field does not. A declared field’s dtype is already its model annotation, and repeating it on field_meta would make every field of every cab differ from its Python-authored equivalent; a pattern attr has no model field at all, which is the reason ParamMeta.dtype exists.

Support is deliberately partial. This reads the static, declarative subset and refuses the parts that are a programming language wearing YAML. The boundary is drawn once, here and in SECURITY.md, and the sections below say exactly where it falls: composition mechanisms this implements, then the scabha features it does not.

Composition mechanisms, implemented in a deliberately minimal form – real scabha cab files are not self-contained and rely on stimela2’s config system for these:

  • _include: [file, ...] – merges other YAML files in (relative to the including file), most often to pull in a shared vars:/lib: namespace. Merging is a plain deep-merge; the including file’s own keys win over included ones. Resolved wherever it appears in the document (top level, or nested under inputs:/outputs:, as real cult-cargo’s cubical.yml/quartical.yml do) via the same tree-walking resolve_directive helper _use already relies on.

  • _use: dotted.path – deep-merges a dict looked up by dotted path in the fully-merged document (post-_include) into the dict it appears in, with that dict’s own sibling keys taking precedence. Used both for small things (image: {_use: vars.cult-cargo.images, name: breizorro}) and to inherit a cab’s entire command/flavour block.

  • The package-scoped include form (_include: (pkg.dotted.path)file.yaml or _include: [{(pkg.dotted.path): [file, ...]}]) – searches an installed package’s data directory rather than a relative path. Resolving a dotted package name to a filesystem directory would normally mean importing the package (importlib), but that risks executing arbitrary code from any __init__.py on the path – shinobi never imports a cab package for any reason (see SECURITY.md’s “never eval()/exec() a cab’s command” boundary, which this extends to “never import a cab package”). Instead, callers pass package_roots={"cultcargo": Path(...)} to load_file()/loads(): an explicit, caller-supplied mapping from a dotted package prefix to its filesystem directory. A dotted name is resolved against the longest registered prefix, descending the remainder as subdirectories (cultcargo.genesis.cubical against {"cultcargo": Path("/.../cultcargo")} -> Path("/.../cultcargo/genesis/cubical")) – the normal package/subpackage-is-a-subdirectory convention, without ever asking Python’s import machinery to confirm it. A package-scoped _include naming a package with no registered root raises a clear CabLoadError.

Deliberately NOT implemented (this is the boundary – see SECURITY.md). Each of these is a place where scabha stops describing a tool and starts computing something, which is the line shinobi does not cross in a cab:

  • Expressions and substitutions. The =config.x.y / =recipe.ms / ${...} / =IFSET(...) language scabha values can contain. Left as literal strings, so a cab carrying one loads with that value verbatim rather than resolved – visible in the built Cab, not silently dropped. ParamMeta.implicit is the one templating shinobi does resolve, and it is plain str.format against the step’s own validated inputs: no name resolution across steps, no function calls, no conditionals.

  • Conditionals and control flow. Anything whose value depends on evaluating a predicate at load or run time. A cab is a parameter table; branching over it belongs in the Python that calls the step, where it is visible to the reader and to the DAG.

  • Aliases and propagation. stimela2 propagates parameter values up and down between recipe and step level, which is what forces its expression language to exist. shinobi wires steps with typed InputRef/OutputRef objects instead, so there is nothing to propagate.

  • dynamic_schema: dotted.path – a reference to a Python function that would need importing and calling to get a cab’s real schema (real cult-cargo’s wsclean.yml/cubical.yml/quartical.yml use this). Resolving it for real is not just a parsing gap like the above: it means executing arbitrary code named by a cab file at load time. Not implemented, and not worked around here either: a cab using dynamic_schema always loads with a warning and whatever static inputs:/outputs: are present – silently incomplete unless you notice the warning. The hand-authored, cross-checked static schemas for the three real cabs that need this (wsclean, cubical, quartical) live in dosho (the native shinobi cab repository, a sibling project) instead of as a stopgap table in this loader – this loader used to carry one (a small per-cab ParamPattern table read from each cab’s own static data files, e.g. cubical’s schema_JONES_TEMPLATE.yaml), removed once dosho’s real ports superseded it. See dosho/cabs/wsclean.py/ cubical.py/quartical.py for that knowledge now, and prefer porting a cab there over reintroducing a table here.

Building the expression language out, or actually executing a cab’s own dynamic_schema, would mean re-deriving stimela2’s config engine (or its code-execution trust model) – exactly what this project exists to avoid unless a real cab actually needs it.

shinobi.loaders.yaml_cab.load_file(path, *, package_roots=None, images=None)[source]

Load a YAML cab definition file into Cab instances.

Parameters:

  • path (str | Path) – Path to the YAML cab definition file.

  • package_roots (dict[str, Path] | None) – Mapping of package name to filesystem root, used to resolve _include directives that reference other packages.

  • images (dict[str, str] | None) – Mapping of image key to full reference. See loads.

Returns:

A dict mapping cab name to its built Cab instance.

Return type:

dict[str, Cab]

shinobi.loaders.yaml_cab.loads(text, *, package_roots=None, images=None)[source]

Parse cab defs from a YAML string. Supports _use (resolved against the document itself) and package-scoped _include (resolved against package_roots), but not a plain relative-path _include, since there’s no base directory to resolve a relative file path against.

images maps an image key to its full reference, for a document that names images symbolically (image: WSCLEAN) rather than by a baked-in reference. Caller-supplied for the same reason package_roots is: shinobi has no manifest of its own and will not go looking for one. A cab repository passes its own – dosho’s images.yaml is exactly this – so a deployment’s overrides still decide the reference at load time instead of it being fixed when the document was written.

An image string absent from the mapping is left alone, because a literal reference is the older and still-valid form (cult-cargo’s files carry quay.io/stimela2/... directly). A key that is simply misspelled therefore reaches the runtime as an image name and fails there – loudly, at pull time, which is the safe direction: the alternative rejects every legitimate bare name (ubuntu) to catch a typo.

Parameters:
Return type:

dict[str, Cab]

Load stimela-classic style parameters.json cab definitions into shinobi Cab objects. This is a different cab schema format from scabha-dialect YAML (see shinobi.loaders.yaml_cab) – useful for exactly the tools cult-cargo doesn’t have a loadable definition for (several CASA tasks, msutils – see AGENTS.md/examples/ninja_selfcal.py for which ones and why).

Classic’s format: one JSON file per cab (e.g. stimela/cargo/cab/casa_mstransform/parameters.json), a top-level task/binary/base/prefix/msdir plus a flat parameters list – unlike cult-cargo, there’s no _include/_use composition to resolve; each file is fully self-contained.

Field mapping (into a generated inputs_model + field_meta):

  • name -> the model field name, sanitised to a valid identifier if needed (the original kept as a nom_de_guerre).

  • dtype -> a Python type on the generated model. A param can declare dtype as a list of alternatives (e.g. [“int”, “str”]) for a genuine type union; the first alternative is used and the rest are dropped – narrowing a real union to shinobi’s simpler model, not a bug.

  • io: “msfile” forces dtype to “MS” (matching shinobi/cult-cargo convention for the main measurement-set parameter), regardless of whatever the raw dtype said (almost always “file” anyway). io: “input”/”output” have no separate shinobi concept – a file-like type alone already drives bind-mounting via path_fields – so they’re otherwise dropped.

  • required, default, info -> the model field / its ParamMeta.

  • mapping -> ParamMeta.nom_de_guerre (classic’s own name for the same concept: what the underlying tool actually calls this parameter).

  • choices -> ParamMeta.choices, and the generated model field’s real annotation is narrowed to typing.Literal[*choices] (see _modelgen.narrow_choices) – an out-of-set value fails pydantic validation, not just a note in info. Also still appended to info as a human-readable parenthetical, for callers that only look at info.

flavour: classic’s CASA-task cabs (base containing “casa”) are not real standalone executables – binary there is a CASA task name (mstransform/listobs/flagdata/…), invoked by wrapping it in a CASA script, not subprocess.run([“mstransform”, …]). These load with flavour=”casa-task” (shinobi’s existing non-executable flavour, UnsupportedFlavourError-guarded in shinobi.policies – see SECURITY.md’s “Never eval()/exec() a cab’s command” section), not “binary”, so they can’t be silently misrun as if they were real binaries. Cabs with any other base (msutils, wsclean, cubical, …) are real CLI tools and load as flavour=”binary”.

image: classic’s base (e.g. “stimela/casa”) is a base-image family name, not a concrete pullable reference – the real tag/version lives in separate tag/version fields (arrays of compatible versions, no single “the” version). base is used as a best-effort image default; override it on the loaded Cab if you need a specific real image.

shinobi.loaders.stimela_classic.load_file(path)[source]

Load a stimela-classic cab definition (JSON) file into a Cab.

Parameters:

path (str | Path) – Path to the JSON cab definition file.

Returns:

The built Cab instance.

Return type:

Cab

shinobi.loaders.stimela_classic.loads(text)[source]

Parse a stimela-classic cab definition from a JSON string.

Parameters:

text (str) – JSON text of the cab definition.

Returns:

The built Cab instance.

Return type:

Cab

Cab loaders, plus the public helpers for building a cab’s schema by hand.

The shinobi.loaders.yaml_cab and shinobi.loaders.stimela_classic submodules load cab definitions from their respective on-disk formats. build_model (and sanitize_unique) are the same helpers those loaders use to turn a flat {name: (dtype, required, default)} spec into the pydantic inputs_model/outputs_model a Cab needs – re-exported here as the supported way to build those models directly, without hand-writing a pydantic class. The implementation lives in the internal _modelgen module.

shinobi.loaders.build_model(name, fields, *, allow_extra=False, choices=None, extras=None)[source]

Create a pydantic model class named name.

fields maps a field name to (dtype, required, default). See required_field_spec for the required/default rule applied to each. choices maps a field name to its allowed values (see narrow_choices) – omitted or absent for a field means its plain dtype-derived type applies unchanged. extras maps a field name to a json_schema_extra dict carried onto that field (e.g. abbreviation for the CLI); a field absent from extras gets none. Mirrors what worker_schema._leaf_field builds per field, so both scabha-dialect loaders attach field-level hints the same way.

Parameters:
Return type:

type

shinobi.loaders.merge_field_meta(base, override)[source]

base updated with override, merging (not replacing) the entry of any field name both declare – see merge_param_meta. The one way a loader is allowed to compose a cab’s input-side and output-side metas, so a cab built from a document and the same cab built in Python cannot disagree about what its fields declare.

Parameters:
Return type:

dict[str, Any]

shinobi.loaders.merge_param_meta(base, override)[source]

Merge two ParamMeta`s describing one field name, attribute by attribute: `override wins wherever it says something, and base survives wherever override left the default.

This exists for the dual declaration – one name on inputs and on outputs, the echo-the-input-back idiom every MS-producing cab uses. Each side declares what it knows: the output side carries implicit (how the product’s path is derived), the input side carries write_path (whether the tool creates that path or rewrites the caller’s data). Replacing the whole object, as the loaders used to, keeps only the last one collected – so the input’s write_path was silently dropped for precisely the fields it exists to mark, and sandbox.clear_stale_outputs then read every such cab as “the caller’s data, leave it alone”. Silent, and on the half of the schema that decides whether a re-run works at all.

Attribute-wise rather than “the input wins” so the output’s implicit still overrides – that is what the whole-object merge got right, and what every loaded cab already depends on.

Parameters:
Return type:

Any

shinobi.loaders.sanitize_unique(name, seen)[source]

Like sanitize, but raises if two distinct raw names collide on the same sanitized identifier. seen maps a sanitized field name to the first raw name that produced it – share one seen dict across a single cab’s parameter list.

Parameters:
Return type:

str

Configuration

Application configuration: layered defaults < config file < env vars < CLI overrides, all validated by the same pydantic models used everywhere else in shinobi. No OmegaConf/scabha/munch/benedict stack – just pydantic-settings, reusing the validation library the cab schemas already depend on.

class shinobi.config.VenvConfig(*, default=None, envs=<factory>)[source]

Bases: BaseModel

Settings for the venv backend (shinobi.backends.venv).

default is the venv used when a step declares none of its own (a path, or a key into envs); None means “no default”, so a venv-backend step with nothing declared falls back to native. envs maps short names to venv paths so recipes/config can refer to a venv by name rather than a machine-specific absolute path. A venv path is a deployment concern, so these live here (or on a Scope in Python), never in a shared cab repo.

Parameters:
default: str | None
envs: dict[str, str]
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.config.BackendConfig(*, default='native', run_as_host_user=True, venv=<factory>)[source]

Bases: BaseModel

Settings controlling which execution backend cabs run under.

Parameters:
default: str
run_as_host_user: bool
venv: VenvConfig
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.config.ExecutionConfig(*, max_workers=1, resources=<factory>, enforce_resources='auto', clear_stale_outputs=True)[source]

Bases: BaseModel

Settings controlling recipe step scheduling.

Parameters:

  • max_workers (int)

  • resources (ResourceBudget)

  • enforce_resources (Literal['auto', 'always', 'never'])

  • clear_stale_outputs (bool)

max_workers: int
resources: ResourceBudget
enforce_resources: Literal['auto', 'always', 'never']
clear_stale_outputs: bool
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.config.SnapshotConfig(*, mode='auto')[source]

Bases: BaseModel

Settings controlling mutation-chain snapshots (shinobi.snapshots).

Parameters:

mode (Literal['auto', 'copy', 'off'])

mode: Literal['auto', 'copy', 'off']
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.config.CacheConfig(*, enabled=False, dir='.shinobi/cache', snapshots=<factory>, content_sample=False)[source]

Bases: BaseModel

Settings controlling step-level skip-if-unchanged caching.

Parameters:
enabled: bool
dir: str
snapshots: SnapshotConfig
content_sample: bool
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.config.LogConfig(*, dir='.', file=None, level='INFO', stream=True, capture_head_lines=5000, capture_tail_lines=5000)[source]

Bases: BaseModel

Settings controlling logging and live output streaming.

Parameters:

  • dir (str)

  • file (str | None)

  • level (str)

  • stream (bool)

  • capture_head_lines (int)

  • capture_tail_lines (int)

dir: str
file: str | None
level: str
stream: bool
capture_head_lines: int
capture_tail_lines: int
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.config.SandboxConfig(*, enabled=False, dir='.shinobi/work')[source]

Bases: BaseModel

Settings controlling per-step sandbox execution (shinobi.sandbox).

Parameters:
enabled: bool
dir: str
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.config.ProvenanceConfig(*, enabled=False, dir='.shinobi/runs')[source]

Bases: BaseModel

Settings controlling reproducible-run provenance (shinobi.provenance).

Parameters:
enabled: bool
dir: str
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.config.AppConfig(_case_sensitive=None, _nested_model_default_partial_update=None, _env_prefix=None, _env_prefix_target=None, _env_file=PosixPath('.'), _env_file_encoding=None, _env_ignore_empty=None, _env_nested_delimiter=None, _env_nested_max_split=None, _env_parse_none_str=None, _env_parse_enums=None, _cli_prog_name=None, _cli_parse_args=None, _cli_settings_source=None, _cli_parse_none_str=None, _cli_hide_none_type=None, _cli_avoid_json=None, _cli_enforce_required=None, _cli_use_class_docs_for_groups=None, _cli_show_env_vars=None, _cli_exit_on_error=None, _cli_prefix=None, _cli_flag_prefix_char=None, _cli_implicit_flags=None, _cli_ignore_unknown_args=None, _cli_kebab_case=None, _cli_shortcuts=None, _secrets_dir=None, _build_sources=None, *, backend=<factory>, execution=<factory>, log=<factory>, cache=<factory>, provenance=<factory>, sandbox=<factory>)[source]

Bases: BaseSettings

Precedence, highest to lowest: CLI overrides > env vars (SHINOBI_*) > config file > built-in defaults.

Parameters:

  • _case_sensitive (bool | None)

  • _nested_model_default_partial_update (bool | None)

  • _env_prefix (str | None)

  • _env_prefix_target (EnvPrefixTarget | None)

  • _env_file (DotenvType | None)

  • _env_file_encoding (str | None)

  • _env_ignore_empty (bool | None)

  • _env_nested_delimiter (str | None)

  • _env_nested_max_split (int | None)

  • _env_parse_none_str (str | None)

  • _env_parse_enums (bool | None)

  • _cli_prog_name (str | None)

  • _cli_parse_args (bool | list[str] | tuple[str, ...] | None)

  • _cli_settings_source (CliSettingsSource[Any] | None)

  • _cli_parse_none_str (str | None)

  • _cli_hide_none_type (bool | None)

  • _cli_avoid_json (bool | None)

  • _cli_enforce_required (bool | None)

  • _cli_use_class_docs_for_groups (bool | None)

  • _cli_show_env_vars (bool | None)

  • _cli_exit_on_error (bool | None)

  • _cli_prefix (str | None)

  • _cli_flag_prefix_char (str | None)

  • _cli_implicit_flags (bool | Literal['dual', 'toggle'] | None)

  • _cli_ignore_unknown_args (bool | None)

  • _cli_kebab_case (bool | Literal['all', 'no_enums'] | None)

  • _cli_shortcuts (Mapping[str, str | list[str]] | None)

  • _secrets_dir (PathType | None)

  • _build_sources (tuple[tuple[PydanticBaseSettingsSource, ...], dict[str, Any]] | None)

  • backend (BackendConfig)

  • execution (ExecutionConfig)

  • log (LogConfig)

  • cache (CacheConfig)

  • provenance (ProvenanceConfig)

  • sandbox (SandboxConfig)

model_config = {'arbitrary_types_allowed': True, 'case_sensitive': False, 'cli_avoid_json': False, 'cli_enforce_required': False, 'cli_exit_on_error': True, 'cli_flag_prefix_char': '-', 'cli_hide_none_type': False, 'cli_ignore_unknown_args': False, 'cli_implicit_flags': False, 'cli_kebab_case': False, 'cli_parse_args': None, 'cli_parse_none_str': None, 'cli_prefix': '', 'cli_prog_name': None, 'cli_shortcuts': None, 'cli_show_env_vars': False, 'cli_use_class_docs_for_groups': False, 'enable_decoding': True, 'env_file': None, 'env_file_encoding': None, 'env_ignore_empty': False, 'env_nested_delimiter': '__', 'env_nested_max_split': None, 'env_parse_enums': None, 'env_parse_none_str': None, 'env_prefix': 'SHINOBI_', 'env_prefix_target': 'variable', 'extra': 'forbid', 'json_file': None, 'json_file_encoding': None, 'nested_model_default_partial_update': False, 'protected_namespaces': ('model_validate', 'model_dump', 'settings_customise_sources'), 'secrets_dir': None, 'toml_file': None, 'validate_default': True, 'yaml_config_section': None, 'yaml_file': None, 'yaml_file_encoding': None}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

backend: BackendConfig
execution: ExecutionConfig
log: LogConfig
cache: CacheConfig
provenance: ProvenanceConfig
sandbox: SandboxConfig
classmethod settings_customise_sources(settings_cls, init_settings, env_settings, dotenv_settings, file_secret_settings)[source]

Set the settings source precedence: init > env vars > YAML file.

Parameters:

  • settings_cls – The BaseSettings subclass being configured.

  • init_settings – Source for values passed directly to __init__.

  • env_settings – Source for SHINOBI_* environment variables.

  • dotenv_settings – Unused; .env files are not supported.

  • file_secret_settings – Unused; Docker/Kubernetes secret files are not supported.

Returns:

The ordered tuple of settings sources pydantic-settings should consult, highest precedence first.

classmethod load(config_file=None, **cli_overrides)[source]

Build an AppConfig, layering defaults, config file, env, and overrides.

Parameters:

  • config_file (str | Path | None) – Path to a YAML config file. Defaults to DEFAULT_CONFIG_FILE (~/.shinobi/config.yml) if not given.

  • **cli_overrides (Any) – Explicit values that take precedence over the config file and environment variables.

Returns:

A fully-resolved AppConfig instance.

Return type:

AppConfig