API reference

This reference is generated from the source docstrings. The most useful names are re-exported from the top-level shinobi package and documented here; supporting types live in their home modules below.

Top-level package

class shinobi.Cab(*, name, info=None, inputs_model, outputs_model, backend=None, image=None, input_mutability=<factory>, cache=None, cache_dir=None, command, flavour='binary', policies=<factory>, field_meta=<factory>, input_patterns=<factory>, output_patterns=<factory>, wranglers=<factory>)[source]

Bases: Scope

An atomic step backed by a single command.

Parameters:
command: str
flavour: str
policies: Policies
field_meta: dict[str, ParamMeta]
input_patterns: list[ParamPattern]
output_patterns: list[ParamPattern]
wranglers: dict[str, list[str]]
param_name(field)[source]

Resolve the tool-facing name for a declared input field.

Parameters:

field (str) – The cab’s own (shinobi-side) field name.

Returns:

The field’s nom_de_guerre if declared in field_meta, otherwise field unchanged.

Return type:

str

match_pattern(name)[source]

Check name against this cab’s dynamic input patterns.

Parameters:

name (str) – An input name not declared as a literal field.

Returns:

The matched ParamMeta, or None if no input_patterns entry matches.

Return type:

ParamMeta | None

match_output_pattern(name)[source]

Check name against this cab’s dynamic output patterns.

Parameters:

name (str) – An output name not declared as a literal field.

Returns:

The matched ParamMeta, or None if no output_patterns entry matches.

Return type:

ParamMeta | None

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.ExecContext(scope, raw_inputs, *, backend_override=None, recipe_backend=None, config=None, cache_enabled=False, cache_dir='', cache_path='', stream=True)[source]

Bases: object

Live execution state, created by _dispatch. inputs is a validated snapshot for inspection; the raw caller kwargs are kept separately because MUTABLE fields must reach the backend as the caller’s original objects.

Parameters:
prepare_inputs()[source]

Validated + mutability-processed inputs, with no overrides applied – for a plain-function step’s own function to call the underlying function with (see steps/pyfunc.py’s adapter, and the manual bare-Scope pattern documented on Scope/StepRef). Reuses the already-validated self.inputs snapshot rather than re-validating.

Return type:

dict[str, Any]

resolve_backend_name(override=None)[source]

Resolve the effective backend name using the standard priority chain. Exposed so orchestration functions (e.g. the pystep adapter) can inspect which backend is active without duplicating the precedence logic.

Parameters:

override (str | None)

Return type:

str

import_func(func, module=None)[source]

Import and return a callable by name.

If module is None, looks up func in builtins (e.g. print, len). Otherwise imports module and returns getattr(module, func).

Useful for pysteps that invoke container-only functions (e.g. CASA tasks) without triggering linter warnings about missing imports on the host.

Parameters:

  • func (str)

  • module (str | None)

Return type:

Callable

run(*, backend=None, **overrides)[source]

Run the underlying Cab or Recipe with optional input overrides.

Parameters:

  • backend (str | None) – Backend name to use for this run, taking priority over the scope’s own/recipe-inherited/config default.

  • **overrides (Any) – Input values to override on top of the raw inputs this context was created with.

Returns:

The step’s StepResult. Also stored on self.outputs.

Raises:

TypeError – If self.scope is neither a Cab nor a Recipe (a plain-function step must return its result directly instead of calling ctx.run()).

Return type:

StepResult

class shinobi.InputRef(*, field)[source]

Bases: BaseModel

Wiring source: this sub-step’s input comes from the enclosing Recipe’s own input field field.

Parameters:

field (str)

field: str
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.Mutability(*values)[source]

Bases: str, Enum

Whether a step’s input may be changed in place by the step’s own orchestration function without that change propagating back to the caller’s object.

IMMUTABLE = 'immutable'
MUTABLE = 'mutable'
class shinobi.OutputRef(*, step, field)[source]

Bases: BaseModel

Wiring source: this input (or, in Recipe.output_wiring, the recipe’s own output) comes from step step’s output field field.

Parameters:
step: str
field: str
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.Recipe(*, name, info=None, inputs_model, outputs_model, backend=None, image=None, input_mutability=<factory>, cache=None, cache_dir=None, steps=<factory>, output_wiring=<factory>, max_workers=None)[source]

Bases: Scope

A composite step: declared sub-steps with explicit wiring.

The one deliberately mutable Scope: builder methods (add_step, step, set_output) extend steps/output_wiring before first run.

Parameters:
steps: list[StepRef]
output_wiring: dict[str, OutputRef]
max_workers: int | None
property inputs: _InputsProxy

Wiring proxy (definition layer) – NOT runtime values.

property outputs: _OutputsProxy

Wiring proxy (definition layer) – NOT runtime values.

add_step(name, scope, **kwargs)[source]

Add a step. scope is usually a bare Scope/Cab/Recipe, but can also be an already-built StepRef (e.g. from @shinobi.pystep or @shinobi.step) – its func is carried over so the step keeps its orchestration function, not just its schema.

Parameters:
Return type:

Recipe

step(scope, *, backend=None, **kwargs)[source]

Decorate a function as a new step appended to this recipe.

Parameters:

  • scope (Scope) – The Cab, Recipe, or bare Scope to bind as this step.

  • backend (str | None) – Backend override for this step.

  • **kwargs (Any) – Split into wiring (InputRef/OutputRef values) and per-step constant params via _split_kwargs.

Returns:

A decorator that binds the given function, appends the resulting StepRef to self.steps, and returns it.

set_output(field, ref)[source]

Wire a recipe output field to an upstream step’s output.

Parameters:

  • field (str) – Name of the recipe’s own output field.

  • ref (OutputRef) – The step output that should populate field.

Returns:

self, for chaining.

Return type:

Recipe

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.Scope(*, name, info=None, inputs_model, outputs_model, backend=None, image=None, input_mutability=<factory>, cache=None, cache_dir=None)[source]

Bases: BaseModel

Definition: schema, metadata, backend config. Never carries inputs/outputs/func fields – those live in ExecContext/StepRef.

Cab/Recipe are the two execution-aware subclasses ExecContext.run() knows how to run. A bare Scope is also valid – it’s the manual building block for a plain-Python-function step whose own function returns its StepResult directly rather than calling ctx.run(); see StepRef’s docstring and steps/pyfunc.py’s @shinobi.pystep (which automates this pattern from a function’s own signature).

image is optional: when set on a bare Scope (typically via @shinobi.pystep(image=…)), the step’s Python function can be executed inside a container instead of in-process. Cab inherits this field for the same purpose (container backends need it to wrap argv in a runtime invocation).

Parameters:
name: str
info: str | None
inputs_model: type[BaseModel]
outputs_model: type[BaseModel]
backend: str | None
image: str | None
input_mutability: dict[str, Mutability]
cache: bool | None
cache_dir: str | None
mutability_of(field)[source]

Look up the declared mutability of an input field.

Parameters:

field (str) – Name of the input field.

Returns:

The field’s Mutability, defaulting to Mutability.IMMUTABLE if not explicitly declared.

Return type:

Mutability

with_backend(backend)[source]

A copy bound to backend, or self unchanged if backend is None. Shared by @shinobi.step and Recipe.step, which both bind a per-step backend override onto a Scope before wrapping it in a StepRef.

Parameters:

backend (str | None)

Return type:

Scope

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.StepRef(*, name, step, func=None, wiring=<factory>, params=<factory>)[source]

Bases: BaseModel

A named, executable binding of a Scope: orchestration function, wiring (meaningful only inside a Recipe), and per-step constants. Returned by @shinobi.step (free-standing) and @recipe.step (appended to recipe.steps). arbitrary_types_allowed is needed only for func.

step is typed as the general Scope (not Cab | Recipe) so it can also hold a bare Scope – the manual, no-magic way to write a plain-Python-function step: build Scope(name=, inputs_model=, outputs_model=) yourself, write a function that always returns its own StepResult (never calls ctx.run(), which only knows how to execute a Cab or Recipe), and wrap it in a StepRef directly. @shinobi.pystep (steps/pyfunc.py) automates exactly this pattern by deriving the Scope’s schema from the function’s own signature. Passing a Cab/Recipe instance here is unaffected – pydantic’s default revalidate_instances=”never” keeps an already-constructed instance’s real subtype, it does not downcast to bare Scope.

Parameters:
model_config = {'arbitrary_types_allowed': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str
step: Scope
func: Callable | None
wiring: dict[str, 'InputRef | OutputRef | list[InputRef | OutputRef]']
params: dict[str, Any]
shinobi.pystep(*, name=None, info=None, image=None, backend=None, **params)[source]

Decorate (or directly call on an existing function, matching @shinobi.step’s precedent: pystep()(existing_func)) a plain, type-hinted function to turn it into a StepRef. See the module docstring for the schema-derivation and outputs rules.

image enables container execution: when set and a container backend is resolved, the function runs inside the specified container image instead of in-process. The function’s source module is mounted into the container so it can be imported by the runner script.

backend sets the default backend for this step (same as on any Scope). With image, this is typically a container backend name like "docker" or "apptainer".

**params are per-call constants, same as @shinobi.step.

Parameters:

  • name (str | None)

  • info (str | None)

  • image (str | None)

  • backend (str | None)

  • params (Any)

Return type:

Callable[[Callable], StepRef]

shinobi.step(scope, *, backend=None, name=None, **params)[source]

Decorate a function with an existing Scope (Cab or Recipe). See the module docstring.

Parameters:
Return type:

Callable[[Callable], StepRef]

Schema helpers

Supporting types used when defining cabs, not re-exported at the top level.

class shinobi.steps.schema.ParamMeta(*, nom_de_guerre=None, implicit=None, info=None, positional=False, repeat_as_tokens=False, dtype=None, choices=None)[source]

Bases: BaseModel

Per-field metadata a plain pydantic model can’t express: the name the underlying tool actually expects (nom_de_guerre), a value always supplied by the cab itself rather than the caller (implicit), human-facing help (info), the cab dtype string (dtype, e.g. “File”/”MS”) for a ParamPattern attr – since a dynamically-named param has no declared field/type annotation for path_fields to inspect, this is how backends know to bind-mount its directory – positional: emitted as a bare value (no –flag), in field-declaration order, after every flagged/pattern-matched arg – and repeat_as_tokens: a list/tuple value is emitted as separate bare argv tokens (after the one flag occurrence, or as separate positional tokens) instead of joined into one comma-separated token – real cult-cargo cabs express this as a per-field policies: {repeat: list} (see e.g. wsclean’s -size <w> <h>/-weight briggs <n>, which need two separate argv tokens, not “4096,4096” as one).

choices: the field’s allowed values (cult-cargo/classic’s choices key). A loader that sets this also narrows the field’s real annotation on inputs_model/outputs_model to typing.Literal[*choices] (see loaders._modelgen.narrow_choices), so an out-of-set value fails pydantic validation the same way a wrong dtype would – not merely documented in info. Kept here too (rather than only inferred from the model’s own annotation) so a ParamPattern attr – which has no declared model field for a dynamically-matched name – can still carry it.

On an output field, a string implicit containing {name} placeholders is resolved by steps.dispatch._fill_outputs as a str.format template against the step’s prepared (validated) input values – e.g. implicit=”{prefix}-MFS-image.fits” derives a tool’s output path from its own prefix input, without shinobi ever importing/executing the tool’s own schema-generation code. A plain string with no {…} is used as a literal constant, same as on an input field.

Parameters:

  • nom_de_guerre (str | None)

  • implicit (Any)

  • info (str | None)

  • positional (bool)

  • repeat_as_tokens (bool)

  • dtype (str | None)

  • choices (list[Any] | None)

nom_de_guerre: str | None
implicit: Any
info: str | None
positional: bool
repeat_as_tokens: bool
dtype: str | None
choices: list[Any] | None
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.steps.schema.ParamPattern(*, separator='.', segments)[source]

Bases: BaseModel

A family of inputs whose names are <segment><separator><segment>…, e.g. QuartiCal’s K.type/G.time_interval or cubical’s g1-solvable/ g-time-int. Matched as one anchored regex assembled from segments: exactly one segment is attrs (the known, enumerable part, each value with its own ParamMeta – dtype/nom_de_guerre/info); every other segment is a regex (soft shape-validation of a level that can’t be enumerated ahead of time). See AGENTS.md for the motivating tools.

The attrs segment is usually last (cubical/QuartiCal: an unenumerable term name followed by a known attribute, g1.solvable), but doesn’t have to be – wsclean’s dynamic output names are the opposite shape, a known/enumerable image type followed by an open-ended qualifier tail (dirty.per-band, restored.i.per-interval.mfs), so attrs there is the first segment. Only one segment may carry attrs; the rest must all be regex.

A segment regex that should behave as an unconstrained “match anything” level (the old design’s prefix) should be written lazily (.+?, not .+): with more than one registered attr, an eager .+ prefers the shortest attr that completes an overall match, which is wrong when one attr is itself a suffix of another (e.g. “int” vs “time-int” with separator “-”) – .+? tries the shortest prefix first, which is exactly “prefer the longest/most specific attr”.

Parameters:
separator: str
segments: list[ParamSegment]
matches(name)[source]

Check whether name matches this pattern and look up its metadata.

Parameters:

name (str) – A dynamic parameter name to test, e.g. “g1.solvable”.

Returns:

The ParamMeta for the matched attrs segment value, or None if name doesn’t match the compiled pattern.

Return type:

ParamMeta | None

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

class shinobi.steps.schema.ParamSegment(*, regex=None, attrs=None)[source]

Bases: BaseModel

One level of a dotted/dashed dynamic-parameter name. A “shape” segment carries only regex – soft validation, no metadata, for a level whose actual values can’t be enumerated at cab-authoring time (e.g. a solver term name like QuartiCal’s K/G). The “meta” segment – always the last one in a ParamPattern – carries attrs: the known, enumerable part, each value with its own ParamMeta.

Parameters:
regex: str | None
attrs: dict[str, ParamMeta] | None
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.steps.schema.Policies(*, prefix='--', replace=<factory>, list_sep=', ', repeat_list=False, key_value=False, repeat=None, explicit_true=False, explicit_false=False)[source]

Bases: BaseModel

How a cab’s parameters are turned into command-line arguments.

key_value/repeat mirror real cult-cargo cab-level policy keys verbatim (e.g. QuartiCal’s policies: {key_value: true, repeat: ‘[]’, prefix: ‘’}): key_value=True means a hydra-style single name=value argv token instead of two tokens (–name, value); repeat=”[]” means a list value formats as one bracketed-literal token (solver.terms=[K,G]) instead of list_sep-joining. Distinct from a per-field ParamMeta.repeat_as_tokens (real per-field policies: {repeat: list}, e.g. wsclean’s bare -size 4096 4096), which is a field-level override and takes precedence when set.

explicit_true/explicit_false also mirror real cult-cargo cab-level policy keys verbatim (e.g. CubiCal’s policies: {explicit_true: true, explicit_false: false}): by default a True boolean value emits as a bare flag (–flag, argparse store_true-style) and False is omitted entirely. Some real CLIs (CubiCal’s own optparse-derived parser among them) instead expect every boolean option to always take an explicit value token – passing a bare flag with no value corrupts parsing of everything after it, since the parser consumes the next token as that flag’s value. explicit_true=True emits –flag true (two tokens, “true”/”false” lowercase) instead of a bare flag when the value is True; explicit_false=True does the same instead of omitting the flag when the value is False. Each direction is independent (CubiCal only needs explicit_true, never explicit_false), and this applies uniformly to declared fields and ParamPattern-matched dynamic ones (e.g. CubiCal’s own per-Jones-term g-solvable).

Parameters:
prefix: str
replace: dict[str, str]
list_sep: str
repeat_list: bool
key_value: bool
repeat: str | None
explicit_true: bool
explicit_false: bool
arg_name(name)[source]

Build the CLI flag name for a parameter name.

Parameters:

name (str) – The parameter’s declared/matched name.

Returns:

name with each replace substitution applied, prefixed by prefix (e.g. “–”).

Return type:

str

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

shinobi.steps.schema.path_fields(model)[source]

Names of every field of model whose (Optional/list-unwrapped) type is a filesystem path (pathlib.Path). File-like cab dtypes (File/MS/Directory/URI) map to Path, so this drives both container bind-mounting and the CLI’s click.Path() mapping.

Parameters:

model (type[BaseModel])

Return type:

set[str]

Execution

class shinobi.results.BackendRun(returncode, stdout='', stderr='')[source]

Bases: object

What a backend returns after running a command – just the raw run outcome. Wrangling stdout/stderr into structured outputs and filling an outputs_model is the dispatch layer’s job, not the backend’s.

Parameters:
returncode: int
stdout: str = ''
stderr: str = ''
property success: bool

Whether the run exited with return code 0.

class shinobi.results.StepResult(name, returncode, outputs, inputs, stdout='', stderr='', cached=False)[source]

Bases: object

The outcome of running a step (a Cab or a Recipe).

outputs is a validated instance of the step’s outputs_model; inputs is a validated instance of the effective (post-override) inputs the step actually ran with. For a Recipe these aggregate from its sub-steps.

Parameters:
name: str
returncode: int
outputs: BaseModel
inputs: BaseModel
stdout: str = ''
stderr: str = ''
cached: bool = False
property success: bool

Whether the step exited with return code 0.

shinobi.steps.dispatch.register_step_backend(name, backend)[source]

Register a backend instance under name, overriding the real class-based registry. Mainly for tests.

Parameters:
Return type:

None

shinobi.steps.dispatch.get_step_backend(name)[source]

Resolve a backend instance by name, checking test overrides first.

Parameters:

name (str) – Backend name, e.g. “native”, “slurm”, or a name registered via register_step_backend.

Returns:

The backend instance registered under name in _STEP_BACKENDS, else a fresh instance from shinobi.backends.get_backend.

Return type:

Any

Backends

Backend abstraction: a backend takes a cab and a resolved argv and runs it somewhere – natively, in a container, on Slurm, on Kubernetes, …

A backend knows nothing about recipes or output schemas beyond the argv it’s handed and the cab’s image/command metadata; it only knows how to execute and how to capture output. Wrangling that output into structured results is the dispatch layer’s job, so a backend returns a raw BackendRun (returncode/stdout/stderr), nothing schema-aware.

class shinobi.backends.Backend[source]

Bases: ABC

Abstract base class for execution backends.

name

Registry key used to look up this backend (e.g. "native", "container", "slurm").

Type:

str

name: str
abstractmethod run(cab, argv, inputs, *, label='', stream=True)[source]

Execute argv (as built by shinobi.policies.build_argv) and return a BackendRun. Must not raise on a non-zero exit – that’s reported via BackendRun.returncode / BackendRun.success.

inputs is the prepared inputs dict argv was built from (the one _prepare_inputs produces, so MUTABLE fields are the caller’s own objects by reference). Most backends ignore it, but container backends need it to know which File/MS-valued params to bind-mount.

label/stream control live stdout/stderr echo (see shinobi.backends._stream.run_streaming) – only native and container act on them today; slurm/kubernetes accept and ignore both (neither has any log-tailing infrastructure yet, so they keep reading output once after the job/pod finishes).

Parameters:
Return type:

BackendRun

shinobi.backends.register(backend_cls)[source]

Register a backend class under its name attribute.

Intended for use as a class decorator on Backend subclasses.

Parameters:

backend_cls (type[Backend]) – The backend class to register.

Returns:

The same class, unmodified, so it can be used as a decorator.

Return type:

type[Backend]

shinobi.backends.get_backend(name, **opts)[source]

Instantiate a registered backend by name.

Parameters:

  • name (str) – Registry key of the backend (e.g. "native", "slurm").

  • **opts – Keyword arguments forwarded to the backend’s constructor.

Returns:

A new instance of the requested backend.

Raises:

ValueError – If no backend is registered under name.

Return type:

Backend

shinobi.backends.registered_backend_classes()[source]

Return all backend classes currently registered.

Returns:

A list of registered Backend subclasses.

Return type:

list[type[Backend]]

Building argv

Turn a cab’s schema + resolved parameter values into a command line.

Operates on the step-model Cab: the parameter values come from an already-validated inputs_model instance (or the prepared dict dispatch builds from it), while per-field naming/implicit metadata comes from the cab’s field_meta, dynamically-named params from input_patterns, and arg formatting from policies.

shinobi.policies.build_argv(cab, resolved)[source]

Build a full argv (starting with the cab’s command) from a resolved parameter dict, according to the cab’s policies and field metadata.

Rejects any non-“binary” flavour before building argv – so a non-executable command can never reach subprocess as argv[0] (see AGENTS.md, “Never eval()/exec() a cab’s command”).

Parameters:
Return type:

list[str]

Loaders

Load cult-cargo style YAML cab definitions into shinobi Cab objects.

The cult-cargo cab schema (inputs/outputs/policies/wranglers) is a good design and is reused as-is here – it’s stimela2’s recipe/alias layer that shinobi drops, not this. This loader lets shinobi use the existing library of cult-cargo tool wrappers without anyone having to rewrite them.

Real cult-cargo cab files, however, are not self-contained: they rely on composition mechanisms from stimela2’s config system, which this loader implements a deliberately minimal version of:

  • _include: [file, ...] – merges other YAML files in (relative to the including file), most often to pull in a shared vars:/lib: namespace. Merging is a plain deep-merge; the including file’s own keys win over included ones. Resolved wherever it appears in the document (top level, or nested under inputs:/outputs:, as real cult-cargo’s cubical.yml/quartical.yml do) via the same tree-walking resolve_directive helper _use already relies on.

  • _use: dotted.path – deep-merges a dict looked up by dotted path in the fully-merged document (post-_include) into the dict it appears in, with that dict’s own sibling keys taking precedence. Used both for small things (image: {_use: vars.cult-cargo.images, name: breizorro}) and to inherit a cab’s entire command/flavour block.

  • The package-scoped include form (_include: (pkg.dotted.path)file.yaml or _include: [{(pkg.dotted.path): [file, ...]}]) – searches an installed package’s data directory rather than a relative path. Resolving a dotted package name to a filesystem directory would normally mean importing the package (importlib), but that risks executing arbitrary code from any __init__.py on the path – shinobi never imports a cab package for any reason (see AGENTS.md’s “never eval()/exec() a cab’s command” boundary, which this extends to “never import a cab package”). Instead, callers pass package_roots={"cultcargo": Path(...)} to load_file()/loads(): an explicit, caller-supplied mapping from a dotted package prefix to its filesystem directory. A dotted name is resolved against the longest registered prefix, descending the remainder as subdirectories (cultcargo.genesis.cubical against {"cultcargo": Path("/.../cultcargo")} -> Path("/.../cultcargo/genesis/cubical")) – the normal package/subpackage-is-a-subdirectory convention, without ever asking Python’s import machinery to confirm it. A package-scoped _include naming a package with no registered root raises a clear CabLoadError.

Deliberately NOT implemented (this is the boundary – see AGENTS.md):

  • The =config.x.y/${...} expression language cult-cargo values can contain – left as literal strings.

  • dynamic_schema: dotted.path – a reference to a Python function that would need importing and calling to get a cab’s real schema (real cult-cargo’s wsclean.yml/cubical.yml/quartical.yml use this). Resolving it for real is not just a parsing gap like the above: it means executing arbitrary code named by a cab file at load time. Not implemented, and not worked around here either: a cab using dynamic_schema always loads with a warning and whatever static inputs:/outputs: are present – silently incomplete unless you notice the warning. The hand-authored, cross-checked static schemas for the three real cabs that need this (wsclean, cubical, quartical) live in dosho (the native shinobi cab repository, a sibling project) instead of as a stopgap table in this loader – this loader used to carry one (a small per-cab ParamPattern table read from each cab’s own static data files, e.g. cubical’s schema_JONES_TEMPLATE.yaml), removed once dosho’s real ports superseded it. See dosho/cabs/wsclean.py/ cubical.py/quartical.py for that knowledge now, and prefer porting a cab there over reintroducing a table here.

Building the expression language out, or actually executing a cab’s own dynamic_schema, would mean re-deriving stimela2’s config engine (or its code-execution trust model) – exactly what this project exists to avoid unless a real cab actually needs it.

shinobi.loaders.cultcargo.load_file(path, *, package_roots=None)[source]

Load a cult-cargo cab definition file into Cab instances.

Parameters:

  • path (str | Path) – Path to the YAML cab definition file.

  • package_roots (dict[str, Path] | None) – Mapping of package name to filesystem root, used to resolve _include directives that reference other packages.

Returns:

A dict mapping cab name to its built Cab instance.

Return type:

dict[str, Cab]

shinobi.loaders.cultcargo.loads(text, *, package_roots=None)[source]

Parse cab defs from a YAML string. Supports _use (resolved against the document itself) and package-scoped _include (resolved against package_roots), but not a plain relative-path _include, since there’s no base directory to resolve a relative file path against.

Parameters:
Return type:

dict[str, Cab]

Load stimela-classic style parameters.json cab definitions into shinobi Cab objects. This is a different cab schema format from cult-cargo’s YAML (see shinobi.loaders.cultcargo) – useful for exactly the tools cult-cargo doesn’t have a loadable definition for (several CASA tasks, msutils – see AGENTS.md/examples/ninja_selfcal.py for which ones and why).

Classic’s format: one JSON file per cab (e.g. stimela/cargo/cab/casa_mstransform/parameters.json), a top-level task/binary/base/prefix/msdir plus a flat parameters list – unlike cult-cargo, there’s no _include/_use composition to resolve; each file is fully self-contained.

Field mapping (into a generated inputs_model + field_meta):

  • name -> the model field name, sanitised to a valid identifier if needed (the original kept as a nom_de_guerre).

  • dtype -> a Python type on the generated model. A param can declare dtype as a list of alternatives (e.g. [“int”, “str”]) for a genuine type union; the first alternative is used and the rest are dropped – narrowing a real union to shinobi’s simpler model, not a bug.

  • io: “msfile” forces dtype to “MS” (matching shinobi/cult-cargo convention for the main measurement-set parameter), regardless of whatever the raw dtype said (almost always “file” anyway). io: “input”/”output” have no separate shinobi concept – a file-like type alone already drives bind-mounting via path_fields – so they’re otherwise dropped.

  • required, default, info -> the model field / its ParamMeta.

  • mapping -> ParamMeta.nom_de_guerre (classic’s own name for the same concept: what the underlying tool actually calls this parameter).

  • choices -> ParamMeta.choices, and the generated model field’s real annotation is narrowed to typing.Literal[*choices] (see _modelgen.narrow_choices) – an out-of-set value fails pydantic validation, not just a note in info. Also still appended to info as a human-readable parenthetical, for callers that only look at info.

flavour: classic’s CASA-task cabs (base containing “casa”) are not real standalone executables – binary there is a CASA task name (mstransform/listobs/flagdata/…), invoked by wrapping it in a CASA script, not subprocess.run([“mstransform”, …]). These load with flavour=”casa-task” (shinobi’s existing non-executable flavour, UnsupportedFlavourError-guarded in shinobi.policies – see AGENTS.md’s “Never eval()/exec() a cab’s command” section), not “binary”, so they can’t be silently misrun as if they were real binaries. Cabs with any other base (msutils, wsclean, cubical, …) are real CLI tools and load as flavour=”binary”.

image: classic’s base (e.g. “stimela/casa”) is a base-image family name, not a concrete pullable reference – the real tag/version lives in separate tag/version fields (arrays of compatible versions, no single “the” version). base is used as a best-effort image default; override it on the loaded Cab if you need a specific real image.

shinobi.loaders.stimela_classic.load_file(path)[source]

Load a stimela-classic cab definition (JSON) file into a Cab.

Parameters:

path (str | Path) – Path to the JSON cab definition file.

Returns:

The built Cab instance.

Return type:

Cab

shinobi.loaders.stimela_classic.loads(text)[source]

Parse a stimela-classic cab definition from a JSON string.

Parameters:

text (str) – JSON text of the cab definition.

Returns:

The built Cab instance.

Return type:

Cab

Cab loaders, plus the public helpers for building a cab’s schema by hand.

The shinobi.loaders.cultcargo and shinobi.loaders.stimela_classic submodules load cab definitions from their respective on-disk formats. build_model (and sanitize_unique) are the same helpers those loaders use to turn a flat {name: (dtype, required, default)} spec into the pydantic inputs_model/outputs_model a Cab needs – re-exported here as the supported way to build those models directly, without hand-writing a pydantic class. The implementation lives in the internal _modelgen module.

shinobi.loaders.build_model(name, fields, *, allow_extra=False, choices=None)[source]

Create a pydantic model class named name.

fields maps a field name to (dtype, required, default). See required_field_spec for the required/default rule applied to each. choices maps a field name to its allowed values (see narrow_choices) – omitted or absent for a field means its plain dtype-derived type applies unchanged.

Parameters:
Return type:

type

shinobi.loaders.sanitize_unique(name, seen)[source]

Like sanitize, but raises if two distinct raw names collide on the same sanitized identifier. seen maps a sanitized field name to the first raw name that produced it – share one seen dict across a single cab’s parameter list.

Parameters:
Return type:

str

Configuration

Application configuration: layered defaults < config file < env vars < CLI overrides, all validated by the same pydantic models used everywhere else in shinobi. No OmegaConf/scabha/munch/benedict stack – just pydantic-settings, reusing the validation library the cab schemas already depend on.

class shinobi.config.BackendConfig(*, default='native', run_as_host_user=True)[source]

Bases: BaseModel

Settings controlling which execution backend cabs run under.

Parameters:

  • default (str)

  • run_as_host_user (bool)

default: str
run_as_host_user: bool
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.config.ExecutionConfig(*, max_workers=1)[source]

Bases: BaseModel

Settings controlling recipe step scheduling.

Parameters:

max_workers (int)

max_workers: int
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.config.CacheConfig(*, enabled=False, dir='.shinobi/cache')[source]

Bases: BaseModel

Settings controlling step-level skip-if-unchanged caching.

Parameters:
enabled: bool
dir: str
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.config.LogConfig(*, dir='.', level='INFO', stream=True)[source]

Bases: BaseModel

Settings controlling logging and live output streaming.

Parameters:
dir: str
level: str
stream: bool
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.config.AppConfig(_case_sensitive=None, _nested_model_default_partial_update=None, _env_prefix=None, _env_prefix_target=None, _env_file=PosixPath('.'), _env_file_encoding=None, _env_ignore_empty=None, _env_nested_delimiter=None, _env_nested_max_split=None, _env_parse_none_str=None, _env_parse_enums=None, _cli_prog_name=None, _cli_parse_args=None, _cli_settings_source=None, _cli_parse_none_str=None, _cli_hide_none_type=None, _cli_avoid_json=None, _cli_enforce_required=None, _cli_use_class_docs_for_groups=None, _cli_exit_on_error=None, _cli_prefix=None, _cli_flag_prefix_char=None, _cli_implicit_flags=None, _cli_ignore_unknown_args=None, _cli_kebab_case=None, _cli_shortcuts=None, _secrets_dir=None, _build_sources=None, *, backend=<factory>, execution=<factory>, log=<factory>, cache=<factory>)[source]

Bases: BaseSettings

Precedence, highest to lowest: CLI overrides > env vars (SHINOBI_*) > config file > built-in defaults.

Parameters:

  • _case_sensitive (bool | None)

  • _nested_model_default_partial_update (bool | None)

  • _env_prefix (str | None)

  • _env_prefix_target (EnvPrefixTarget | None)

  • _env_file (DotenvType | None)

  • _env_file_encoding (str | None)

  • _env_ignore_empty (bool | None)

  • _env_nested_delimiter (str | None)

  • _env_nested_max_split (int | None)

  • _env_parse_none_str (str | None)

  • _env_parse_enums (bool | None)

  • _cli_prog_name (str | None)

  • _cli_parse_args (bool | list[str] | tuple[str, ...] | None)

  • _cli_settings_source (CliSettingsSource[Any] | None)

  • _cli_parse_none_str (str | None)

  • _cli_hide_none_type (bool | None)

  • _cli_avoid_json (bool | None)

  • _cli_enforce_required (bool | None)

  • _cli_use_class_docs_for_groups (bool | None)

  • _cli_exit_on_error (bool | None)

  • _cli_prefix (str | None)

  • _cli_flag_prefix_char (str | None)

  • _cli_implicit_flags (bool | Literal['dual', 'toggle'] | None)

  • _cli_ignore_unknown_args (bool | None)

  • _cli_kebab_case (bool | Literal['all', 'no_enums'] | None)

  • _cli_shortcuts (Mapping[str, str | list[str]] | None)

  • _secrets_dir (PathType | None)

  • _build_sources (tuple[tuple[PydanticBaseSettingsSource, ...], dict[str, Any]] | None)

  • backend (BackendConfig)

  • execution (ExecutionConfig)

  • log (LogConfig)

  • cache (CacheConfig)

model_config = {'arbitrary_types_allowed': True, 'case_sensitive': False, 'cli_avoid_json': False, 'cli_enforce_required': False, 'cli_exit_on_error': True, 'cli_flag_prefix_char': '-', 'cli_hide_none_type': False, 'cli_ignore_unknown_args': False, 'cli_implicit_flags': False, 'cli_kebab_case': False, 'cli_parse_args': None, 'cli_parse_none_str': None, 'cli_prefix': '', 'cli_prog_name': None, 'cli_shortcuts': None, 'cli_use_class_docs_for_groups': False, 'enable_decoding': True, 'env_file': None, 'env_file_encoding': None, 'env_ignore_empty': False, 'env_nested_delimiter': '__', 'env_nested_max_split': None, 'env_parse_enums': None, 'env_parse_none_str': None, 'env_prefix': 'SHINOBI_', 'env_prefix_target': 'variable', 'extra': 'forbid', 'json_file': None, 'json_file_encoding': None, 'nested_model_default_partial_update': False, 'protected_namespaces': ('model_validate', 'model_dump', 'settings_customise_sources'), 'secrets_dir': None, 'toml_file': None, 'validate_default': True, 'yaml_config_section': None, 'yaml_file': None, 'yaml_file_encoding': None}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

backend: BackendConfig
execution: ExecutionConfig
log: LogConfig
cache: CacheConfig
classmethod settings_customise_sources(settings_cls, init_settings, env_settings, dotenv_settings, file_secret_settings)[source]

Set the settings source precedence: init > env vars > YAML file.

Parameters:

  • settings_cls – The BaseSettings subclass being configured.

  • init_settings – Source for values passed directly to __init__.

  • env_settings – Source for SHINOBI_* environment variables.

  • dotenv_settings – Unused; .env files are not supported.

  • file_secret_settings – Unused; Docker/Kubernetes secret files are not supported.

Returns:

The ordered tuple of settings sources pydantic-settings should consult, highest precedence first.

classmethod load(config_file=None, **cli_overrides)[source]

Build an AppConfig, layering defaults, config file, env, and overrides.

Parameters:

  • config_file (str | Path | None) – Path to a YAML config file. Defaults to DEFAULT_CONFIG_FILE (~/.shinobi/config.yml) if not given.

  • **cli_overrides (Any) – Explicit values that take precedence over the config file and environment variables.

Returns:

A fully-resolved AppConfig instance.

Return type:

AppConfig