API reference

This reference is generated from the source docstrings. The most useful names are re-exported from the top-level shinobi package and documented here; supporting types live in their home modules below.

Top-level package

class shinobi.DerivedRead(*, member, family)[source]

Bases: ProductModel

Parameters:
member: Literal['file', 'directory']
family: FamilySpec
validate_inputs(fields)[source]

Reject attribute/index traversal and unknown inputs at construction.

model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.DirectoryBundle(*, path)[source]

Bases: ProductModel

One owned local directory; its recursive inventory is execution evidence.

Parameters:

path (Path)

path: Path
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.ProductFamily(*, resolved=False, members=())[source]

Bases: ProductModel, Generic[T]

Parameters:
resolved: bool
members: tuple[ProductMember[T], ...]
select(**coordinates)[source]
Return type:

T

model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.ProductMember(*, coordinates, value)[source]

Bases: ProductModel, Generic[T]

Parameters:
coordinates: dict[str, Coordinate]
value: T
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.FamilySpec(*, root='.', coordinates, rules, min_members=0, max_members=10000)[source]

Bases: ProductModel

Parameters:
root: str
coordinates: dict[str, Literal['int', 'str'] | tuple[Literal['int', 'str'], ...]]
rules: tuple[MemberRule, ...]
min_members: int
max_members: int
validate_coordinate(key, value)[source]
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.AxisSpec(*, count_input=None, values_input=None, values=None, tokens=None, uppercase=False, split=None, include=None, suffix='{value}', singleton_suffix=None, aggregate=None)[source]

Bases: ProductModel

Parameters:
count_input: str | None
values_input: str | None
values: tuple[Coordinate, ...] | None
tokens: tuple[str, ...] | None
uppercase: bool
split: Literal['csv_or_compact'] | None
include: tuple[Coordinate, ...] | None
suffix: str
singleton_suffix: str | None
aggregate: AggregateSpec | None
expand(inputs, limit)[source]
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.MemberRule(*, path, coordinates=<factory>, axes=<factory>, captures=<factory>, when=<factory>, when_set=<factory>, required=False, accept_existing=False, min_members=0)[source]

Bases: ProductModel

Parameters:
path: str
coordinates: dict[str, Coordinate]
axes: dict[str, AxisSpec]
captures: dict[str, Literal['int', 'str']]
when: dict[str, tuple[str | int | bool, ...]]
when_set: dict[str, bool]
required: bool
accept_existing: bool
min_members: int
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.Cab(*, name, info=None, inputs_model, outputs_model, backend=None, image=None, venv=None, input_mutability=<factory>, field_meta=<factory>, dataset_accesses=<factory>, derived_reads=<factory>, cache=None, cache_dir=None, sandbox=None, harvest=<factory>, scratch=<factory>, resources=None, command, flavour='binary', policies=<factory>, input_patterns=<factory>, output_patterns=<factory>, wranglers=<factory>)[source]

Bases: Scope

An atomic step backed by a single command.

Parameters:
command: str
flavour: str
policies: Policies
input_patterns: list[ParamPattern]
output_patterns: list[ParamPattern]
wranglers: dict[str, list[str]]
param_name(field)[source]

Resolve the tool-facing name for a declared input field.

Parameters:

field (str) – The cab’s own (shinobi-side) field name.

Returns:

The field’s nom_de_guerre if declared in field_meta, otherwise field unchanged.

Return type:

str

match_pattern(name)[source]

Check name against this cab’s dynamic input patterns.

Parameters:

name (str) – An input name not declared as a literal field.

Returns:

The matched ParamMeta, or None if no input_patterns entry matches.

Return type:

ParamMeta | None

match_output_pattern(name)[source]

Check name against this cab’s dynamic output patterns.

Parameters:

name (str) – An output name not declared as a literal field.

Returns:

The matched ParamMeta, or None if no output_patterns entry matches.

Return type:

ParamMeta | None

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.DatasetClosure(*, profile='msv2-dataset-closure/v2', requested_root, storage_namespace, root=None, status, message, resources=(), capabilities=None, unsupported_features=())[source]

Bases: BaseModel

Serializable observation of an MSv2’s physical CASA-table resources.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

profile: str
requested_root: Path
storage_namespace: Path
root: Path | None
status: ClosureStatus
message: str
resources: tuple[ClosureResource, ...]
capabilities: ClosureCapabilities | None
unsupported_features: tuple[str, ...]
property valid: bool
class shinobi.DatasetAccess(*, field, mode, table=DatasetTable.MAIN, columns=None, selection=None, allow_row_count_change=False, allow_schema_change=False, allow_keyword_change=False, allow_subtable_change=(), allow_present_subtable_rewrite=False, root_field=None, reservation=None)[source]

Bases: BaseModel

One field-local MSv2 access declaration attached to a Scope.

field names an input or output field. root_field is needed only when field itself names a referenced subtable whose parent MS cannot be inferred from its path (notably an externally stored subtable). reservation is a canonicalizable path envelope used when the dataset path is produced dynamically and is therefore unknown during planning. allow_present_subtable_rewrite grants an explicit MAIN writer backing- file rewrites of supported subtables present before execution and still linked at the same path afterward; table schema authority stays explicit.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

field: str
mode: DatasetMode
table: DatasetTable
columns: DatasetColumns | None
selection: DatasetSelection | None
allow_row_count_change: bool
allow_schema_change: bool
allow_keyword_change: bool
allow_subtable_change: tuple[str, ...]
allow_present_subtable_rewrite: bool
root_field: str | None
reservation: Path | None
exception shinobi.DatasetAccessError[source]

Bases: ShinobiError, ValueError

A dataset access declaration cannot be resolved safely.

class shinobi.DatasetBackendCapability(*, schema_version=1, profile='msv2-backend-route/v1', backend, lifecycle, status, namespace_mode, execution_location, inspection_location, recovery_location, reason)[source]

Bases: BaseModel

Versioned capability decision for one backend and lifecycle kind.

inspection_location and recovery_location are explicit because a scheduler accepting a job on one host does not prove that its compute node sees the same path or that restoring the submit host repairs what the task changed.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

schema_version: Literal[1]
profile: Literal['msv2-backend-route/v1']
backend: str
lifecycle: Literal['contained-native-msv2-read/v1', 'contained-native-msv2-mutation/v1']
status: DatasetBackendStatus
namespace_mode: DatasetNamespaceMode
execution_location: str
inspection_location: str
recovery_location: str
reason: str
class shinobi.DatasetBackendMount(*, source, target, writable, reason)[source]

Bases: BaseModel

One identity bind required in addition to schema-derived mounts.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

source: Path
target: Path
writable: bool
reason: str
class shinobi.DatasetBackendPlan(*, schema_version=1, capability, storage_namespace, mounts=())[source]

Bases: BaseModel

Closed, data-only plan handed to a local backend adapter.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

schema_version: Literal[1]
capability: DatasetBackendCapability
storage_namespace: Path
mounts: tuple[DatasetBackendMount, ...]
class shinobi.DatasetBackendStatus(*values)[source]

Bases: str, Enum

Evidence state for one concrete backend/lifecycle route.

TESTED = 'tested'
UNAVAILABLE = 'unavailable'
UNSUPPORTED = 'unsupported'
class shinobi.DatasetColumns(*, read=(), write=(), create=(), remove=())[source]

Bases: BaseModel

Column-level intent retained for validation and provenance.

Column detail deliberately does not relax scheduling: any writer excludes every concurrent access to the associated resolved dataset closure. None on DatasetAccess means the columns are unknown and therefore requests the conservative whole-dataset fallback. Entries may format direct string inputs of their own scope, for example "{column}" or "MODEL_{suffix}". Only undecorated field names are allowed; a missing or None input makes the entire access’s columns unknown.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

read: tuple[str, ...]
write: tuple[str, ...]
create: tuple[str, ...]
remove: tuple[str, ...]
class shinobi.DatasetCacheDecision(*, decision, cache_key, identity, dataset_coverage, reason)[source]

Bases: BaseModel

Which identity and fingerprint coverage one leaf’s cache decision used.

Parameters:

  • decision (Literal['hit', 'miss', 'rejected-hit', 'disabled'])

  • cache_key (str | None)

  • identity (str)

  • dataset_coverage (tuple[str, ...])

  • reason (str)

model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

decision: Literal['hit', 'miss', 'rejected-hit', 'disabled']
cache_key: str | None
identity: str
dataset_coverage: tuple[str, ...]
reason: str
class shinobi.DatasetFallback(*values)[source]

Bases: str, Enum

Why planning conservatively reserves a whole dataset identity.

UNDECLARED = 'undeclared'
UNKNOWN_COLUMNS = 'unknown-columns'
UNKNOWN_PATH = 'unknown-path'
class shinobi.DatasetLeafAttempt(*, step_path, accesses, cache=None, pre_observations=(), absent_before=(), post_observations=(), mutations=(), auxiliary_mutations=(), outcome='pending', reason='')[source]

Bases: BaseModel

One atomic step’s strict observations under the workflow’s claim.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

step_path: str
accesses: tuple[ResolvedDatasetAccess, ...]
cache: DatasetCacheDecision | None
pre_observations: tuple[DatasetObservation, ...]
absent_before: tuple[Path, ...]
post_observations: tuple[DatasetObservation, ...]
mutations: tuple[DatasetMutationRecord, ...]
auxiliary_mutations: tuple[AuxiliaryMutationRecord, ...]
outcome: Literal['pending', 'validated', 'committed', 'reused', 'failed', 'refused']
reason: str
class shinobi.DatasetLifecycleAttempt(*, schema_version=1, attempt_id, scope, workspace, phase, events, backends, backend_capabilities=(), capability='contained-native-msv2-read/v1', capability_supported, planned_accesses=(), claim=None, pre_observations=(), post_observations=(), absent_roots=(), leaves=(), recovery=(), overwrites=(), outcome='pending', reason)[source]

Bases: BaseModel

Versioned durable provenance for one contained strict workflow.

Schema 1 is the historical read-only baseline. Schema 2 adds per-leaf records for readers and writers, and is required for mutation capability. backend_capabilities records the versioned namespace/lifecycle decision for every effective backend in the claimed flat workflow.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

schema_version: Literal[1, 2]
attempt_id: str
scope: str
workspace: Path
phase: DatasetLifecyclePhase
events: tuple[DatasetLifecycleEvent, ...]
backends: tuple[str, ...]
backend_capabilities: tuple[DatasetBackendCapability, ...]
capability: Literal['contained-native-msv2-read/v1', 'contained-native-msv2-mutation/v1']
capability_supported: bool
planned_accesses: tuple[ResolvedDatasetAccess, ...]
claim: WorkspaceOwner | None
pre_observations: tuple[DatasetObservation, ...]
post_observations: tuple[DatasetObservation, ...]
absent_roots: tuple[Path, ...]
leaves: tuple[DatasetLeafAttempt, ...]
recovery: tuple[str, ...]
overwrites: tuple[DatasetOverwrite, ...]
outcome: Literal['pending', 'committed', 'refused', 'failed']
reason: str
class shinobi.DatasetLifecyclePhase(*values)[source]

Bases: str, Enum

Durable phases of one strict dataset attempt.

PLANNED = 'planned'
CLAIMED = 'claimed'
REVALIDATED = 'revalidated'
EXECUTING = 'executing'
VALIDATED = 'validated'
COMMITTED = 'committed'
REFUSED = 'refused'
FAILED = 'failed'
class shinobi.DatasetLifecycleSnapshot(*, accesses, observations, absent_roots=())[source]

Bases: BaseModel

One complete access-plan and on-disk observation.

absent_roots are CREATE targets, observed as absent: their predecessor is absence by contract.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

accesses: tuple[ResolvedDatasetAccess, ...]
observations: tuple[DatasetObservation, ...]
absent_roots: tuple[Path, ...]
exception shinobi.DatasetLifecycleUnavailableError[source]

Bases: ShinobiError

Execution was refused for a strict dataset annotation.

Only the contained native MSv2 read and mutation lifecycles are executable. This exception keeps every unsupported route from looking enforced while an annotation is only being carried as a path by that execution stack, and it is also the refusal raised before a strict mutation launches when exact predecessor recovery cannot be promised.

exception shinobi.DatasetLifecycleViolationError[source]

Bases: ShinobiError

A strict dataset broke its declared postcondition under its claim.

For a reader, the dataset changed; for a writer, the change exceeded what its access contract permits. A writer’s dataset has been rolled back to its exact predecessor when this is raised, unless a note says otherwise.

class shinobi.DatasetMode(*values)[source]

Bases: str, Enum

How a step uses an MSv2 dataset.

READ = 'read'
WRITE = 'write'
CREATE = 'create'
class shinobi.DatasetMutationOutcome(*values)[source]

Bases: str, Enum

What finally happened to one strict dataset a leaf wrote or created.

PENDING = 'pending'
COMMITTED = 'committed'
ROLLED_BACK = 'rolled-back'
PRE_RUN_RESTORED = 'pre-run-restored'
ABSENT_RESTORED = 'absent-restored'
UNTRUSTED = 'untrusted'
REFUSED = 'refused'
class shinobi.DatasetMutationRecord(*, field, element_index=None, mode, root, predecessor_state, predecessor_signature, predecessor_snapshot, successor_state, predecessor_fingerprint=None, successor_signature=None, successor_fingerprint=None, successor_snapshot=None, signature_coverage='MAIN row count; MAIN column, keyword and subtable names; closure membership, table files and storage managers. Cell values, keyword values and visibility data are not examined.', fingerprint_coverage="Relative path, size and mtime of every regular file in the closure's tables. Detects writes through the filesystem, including in-place cell updates; not a content hash, so a deliberately restored mtime is not detected.", outcome=DatasetMutationOutcome.PENDING, restored_state=None)[source]

Bases: BaseModel

Predecessor/successor evidence for one strict write or create.

The four identities in the module docstring are separate fields here on purpose; none of them stands in for another.

Parameters:

  • field (str)

  • element_index (int | None)

  • mode (DatasetMode)

  • root (Path)

  • predecessor_state (str | None)

  • predecessor_signature (str | None)

  • predecessor_snapshot (Path | None)

  • successor_state (str)

  • predecessor_fingerprint (str | None)

  • successor_signature (str | None)

  • successor_fingerprint (str | None)

  • successor_snapshot (Path | None)

  • signature_coverage (str)

  • fingerprint_coverage (str)

  • outcome (DatasetMutationOutcome)

  • restored_state (str | None)

model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

field: str
element_index: int | None
mode: DatasetMode
root: Path
predecessor_state: str | None
predecessor_signature: str | None
predecessor_snapshot: Path | None
successor_state: str
predecessor_fingerprint: str | None
successor_signature: str | None
successor_fingerprint: str | None
successor_snapshot: Path | None
signature_coverage: str
fingerprint_coverage: str
outcome: DatasetMutationOutcome
restored_state: str | None
class shinobi.AuxiliaryMutationRecord(*, schema_version=1, profile='regular-file/v1', address, path, predecessor_state, predecessor_signature, predecessor_fingerprint, predecessor_snapshot, successor_state, successor_signature=None, successor_fingerprint=None, successor_snapshot=None, outcome=DatasetMutationOutcome.PENDING, restored_state=None)[source]

Bases: BaseModel

Regular-file participants in the existing strict mutation group.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

schema_version: Literal[1]
profile: Literal['regular-file/v1']
address: DerivedAddress
path: Path
predecessor_state: str
predecessor_signature: str
predecessor_fingerprint: str
predecessor_snapshot: Path
successor_state: str
successor_signature: str | None
successor_fingerprint: str | None
successor_snapshot: Path | None
outcome: DatasetMutationOutcome
restored_state: str | None
class shinobi.DatasetNamespaceMode(*values)[source]

Bases: str, Enum

How host paths relate to paths observed by the executor.

LOCAL = 'local'
IDENTITY_BIND = 'identity-bind'
SHARED_IDENTITY = 'shared-identity'
REMOTE_DELEGATED = 'remote-delegated'
UNPROVEN = 'unproven'
class shinobi.DatasetOverwrite(*, step, field, path, existed, invalidated)[source]

Bases: BaseModel

One CREATE target an explicit overwrite deleted before planning.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

step: str
field: str
path: Path
existed: bool
invalidated: tuple[str, ...]
class shinobi.DatasetSelection(*, row_ranges=(), data_description_ids=(), field_ids=(), observation_ids=(), scan_numbers=(), array_ids=())[source]

Bases: BaseModel

A bounded, data-only row selection.

Half-open row_ranges and finite ID sets cover the common static selection shapes without admitting TaQL, Python callables, or another expression language. Scheduling remains conservative at dataset scope.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

row_ranges: tuple[tuple[int, int], ...]
data_description_ids: tuple[int, ...]
field_ids: tuple[int, ...]
observation_ids: tuple[int, ...]
scan_numbers: tuple[int, ...]
array_ids: tuple[int, ...]
class shinobi.DatasetTable(*values)[source]

Bases: str, Enum

MSv2 tables whose closure membership is understood by Shinobi.

MAIN = 'MAIN'
ANTENNA = 'ANTENNA'
DATA_DESCRIPTION = 'DATA_DESCRIPTION'
DOPPLER = 'DOPPLER'
FEED = 'FEED'
FIELD = 'FIELD'
FLAG_CMD = 'FLAG_CMD'
FREQ_OFFSET = 'FREQ_OFFSET'
HISTORY = 'HISTORY'
OBSERVATION = 'OBSERVATION'
POINTING = 'POINTING'
POLARIZATION = 'POLARIZATION'
PROCESSOR = 'PROCESSOR'
SOURCE = 'SOURCE'
SPECTRAL_WINDOW = 'SPECTRAL_WINDOW'
STATE = 'STATE'
SYSCAL = 'SYSCAL'
WEATHER = 'WEATHER'
class shinobi.ExecContext(scope, raw_inputs, *, backend_override=None, recipe_backend=None, config=None, cache_enabled=False, cache_forced_off=False, cache_dir='', cache_path='', stream=True, pin=False, sandbox_root=None, clear_outputs=True, input_keys=None, budget=None, run_id='', validated_inputs=None, planned_inputs=None, planned_leaf_inputs=None, leaf_inputs=None, dataset_lifecycle=None, publication_gate=None)[source]

Bases: object

Live execution state, created by _dispatch. inputs is a validated snapshot for inspection; the raw caller kwargs are kept separately because MUTABLE fields must reach the backend as the caller’s original objects.

Parameters:

  • scope (Scope)

  • raw_inputs (dict[str, Any])

  • backend_override (str | None)

  • recipe_backend (str | None)

  • config (AppConfig | None)

  • cache_enabled (bool)

  • cache_forced_off (bool)

  • cache_dir (str)

  • cache_path (str)

  • stream (bool)

  • pin (bool)

  • sandbox_root (str | None)

  • clear_outputs (bool)

  • input_keys (dict[str, Any] | None)

  • budget (Budget | None)

  • run_id (str)

  • validated_inputs (BaseModel | None)

  • planned_inputs (BaseModel | dict[str, Any] | _ProductReservations | None)

  • planned_leaf_inputs (dict[int, _ProductReservations] | None)

  • leaf_inputs (dict[int, tuple[BaseModel, bool]] | None)

  • dataset_lifecycle (Any | None)

  • publication_gate (Callable[[Callable[[], None]], None] | None)

prepare_inputs()[source]

Validated + mutability-processed inputs, with no overrides applied – for a plain-function step’s own function to call the underlying function with (see steps/pyfunc.py’s adapter, and the manual bare-Scope pattern documented on Scope/StepRef). Reuses the already-validated self.inputs snapshot rather than re-validating.

Return type:

dict[str, Any]

resolve_backend_name(override=None)[source]

Resolve the effective backend name using the standard priority chain. Exposed so orchestration functions (e.g. the pystep adapter) can inspect which backend is active without duplicating the precedence logic.

Parameters:

override (str | None)

Return type:

str

validate_contained_inputs(prepared=None)[source]

Check every leaf launcher, including native and manual functions.

Parameters:

prepared (dict[str, Any] | None)

Return type:

None

workflow_dataset_resources()[source]

Every protected resource, including other leaves and planned creates.

Return type:

set[Path]

dataset_backend_plan(backend_name, prepared=None)[source]

Prepared namespace plan for an atomic leaf under a strict lifecycle.

The outer lifecycle has already resolved and claimed the workflow’s authoritative access set. Resolving this leaf again from the exact prepared values checks that an annotated leaf receives only paths covered by that claim. Every workflow root is then re-asserted read-only for the container, upgraded only for roots this leaf declares write/create access to; an unannotated leaf gets no upgrade. The strict-leaf wrapper uses the same resolution and claim helper for its bookkeeping. Keeping plan construction here gives cab and pystep container launchers one shared adapter.

Parameters:
Return type:

Any | None

import_callable(name, module=None)[source]

Import and return a callable by name.

If module is None, looks up name in builtins (e.g. print, len). Otherwise imports module – which may be a dotted path of any depth ("astropy.io.fits") – and returns getattr(module, name).

Useful for pysteps that invoke container-only callables (e.g. CASA tasks) without triggering linter warnings about missing imports on the host.

Callable, not function, and the distinction is not pedantry: what a pystep asks for here is a class about as often as it is a function (casacore.tables.table, astropy.wcs.WCS, astropy.coordinates.SkyCoord), and numpy hands back neither – numpy.log10 is a ufunc and numpy.linspace an _ArrayFunctionDispatcher. The one property all of them share is the one the caller is about to rely on, so that is the one checked.

The check is what makes the split point safe. This always returns an attribute of a module, never a module: name the full dotted path in module and the attribute in name. Splitting it at the wrong point asks for a submodule as though it were an attribute of its parent, and whether that raises depends on the package – ("ndimage", "scipy") quietly returns the module on any scipy new enough to expose its subpackages lazily, while ("tables", "casacore") and ("fits", "astropy.io") raise AttributeError. Version-dependent luck is a bad way to learn the path was wrong, so a non-callable result is a TypeError here rather than a mystery at the first call site. Use import_module when the step wants the module.

Parameters:

  • name (str)

  • module (str | None)

Return type:

Callable

import_func(func, module=None)[source]

Deprecated alias for import_callable.

Kept because it is what every existing pystep calls, and renaming a method a container shim lifts by source is not a change worth breaking a caller over. The name was wrong – half of what real pysteps ask for is a class, not a function – but the behaviour is import_callable’s, including its callability check, so there is exactly one implementation to reason about.

Parameters:

  • func (str)

  • module (str | None)

Return type:

Callable

import_module(module)[source]

Import and return a module by its full dotted path.

The companion to import_callable for the case where a pystep wants the module itself rather than one of its attributes – np = ctx.import_module("numpy"), fits = ctx.import_module("astropy.io.fits"). import_callable cannot express this: its one-argument form is a builtins lookup, and its two-argument form ends in a getattr, which does not reach a submodule (import_module does not bind one onto its parent package) – and now rejects the module it would occasionally have returned by accident.

Kept as a separate name rather than folded into import_callable’s one-argument form so the return type stays predictable per method instead of varying with whatever the name happens to resolve to.

Parameters:

module (str)

Return type:

ModuleType

run(*, backend=None, **overrides)[source]

Run the underlying Cab or Recipe with optional input overrides.

Parameters:

  • backend (str | None) – Backend name to use for this run, taking priority over the scope’s own/recipe-inherited/config default.

  • **overrides (Any) – Input values to override on top of the raw inputs this context was created with.

Returns:

The step’s StepResult. Also stored on self.outputs.

Raises:

TypeError – If self.scope is neither a Cab nor a Recipe (a plain-function step must return its result directly instead of calling ctx.run()).

Return type:

StepResult

class shinobi.InputRef(*, field)[source]

Bases: BaseModel

Wiring source: this sub-step’s input comes from the enclosing Recipe’s own input field field.

Parameters:

field (str)

field: str
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.LoopIteration(*, loop, index, prev_step=None, sentinel_step=None, sentinel_field)[source]

Bases: BaseModel

Marks a step as belonging to iteration index of the unrolled loop loop (see Recipe.add_loop). Carried by every step the unrolling produces; it is bookkeeping for the skip decision only – the dependency edges that make the unrolled chain a real DAG are ordinary wiring plus after, never this.

prev_step is the same body step one iteration earlier (so selfcal.4.image passes through selfcal.3.image), and sentinel_step/sentinel_field name the previous iteration’s output whose existence on disk means “the loop has already converged, do no work”. prev_step and sentinel_step are both None for the first iteration, which can never skip – there is nothing before it to pass through or to have converged. sentinel_field is always set: which output carries the signal is a property of the loop, not of one iteration.

Parameters:

  • loop (str)

  • index (int)

  • prev_step (str | None)

  • sentinel_step (str | None)

  • sentinel_field (str)

loop: str
index: int
prev_step: str | None
sentinel_step: str | None
sentinel_field: str
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.LoopRef(name, steps, final)[source]

Bases: object

What Recipe.add_loop returns: the loop’s step names, and an outputs proxy resolving to the final iteration’s producers.

Not a StepRef – a loop is not a step. It has already been unrolled into the recipe’s steps by the time this is returned; this is just a handle for wiring whatever comes next.

Parameters:
property outputs: _LoopOutputsProxy

Wiring proxy (definition layer) – NOT runtime values.

class shinobi.Mutability(*values)[source]

Bases: str, Enum

Whether a step’s input may be changed in place by the step’s own orchestration function without that change propagating back to the caller’s object.

IMMUTABLE = 'immutable'
MUTABLE = 'mutable'
class shinobi.OutputRef(*, step, field, selection=None)[source]

Bases: BaseModel

Wiring source: this input (or, in Recipe.output_wiring, the recipe’s own output) comes from step step’s output field field.

Parameters:
step: str
field: str
selection: dict[str, Coordinate] | None
select(**coordinates)[source]
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.Recipe(*, name, info=None, inputs_model, outputs_model, backend=None, image=None, venv=None, input_mutability=<factory>, field_meta=<factory>, dataset_accesses=<factory>, derived_reads=<factory>, cache=None, cache_dir=None, sandbox=None, harvest=<factory>, scratch=<factory>, resources=None, steps=<factory>, output_wiring=<factory>, max_workers=None)[source]

Bases: Scope

A composite step: declared sub-steps with explicit wiring.

The one deliberately mutable Scope: builder methods (add_step, step, set_output) extend steps/output_wiring before first run.

Parameters:
steps: list[StepRef]
output_wiring: dict[str, OutputRef]
max_workers: int | None
property inputs: _InputsProxy

Wiring proxy (definition layer) – NOT runtime values.

property outputs: _OutputsProxy

Wiring proxy (definition layer) – NOT runtime values.

add_step(name, scope, *, scatter=None, resources=None, **kwargs)[source]

Add a step. scope is usually a bare Scope/Cab/Recipe, but can also be an already-built StepRef (e.g. from @shinobi.pystep or @shinobi.step) – its func is carried over so the step keeps its orchestration function, not just its schema.

Parameters:

  • scatter (list[str] | ScatterSpec | None) – Fields to fan out over. Each must be a field of the step’s inputs_model; at runtime the corresponding value must be a list, and every scattered field must have the same length.

  • resources (Resources | None) – What this step costs to run, for the scheduler’s admission control. An explicit keyword rather than a kwarg, since **kwargs here is split into wiring and step params and would otherwise swallow it as a constant input value.

  • name (str)

  • scope (Scope | StepRef)

  • kwargs (Any)

Return type:

Recipe

add_loop(name, body, *, max_iter, until, carry, index_input=None, **kwargs)[source]

Unroll body max_iter times into this recipe, as a declared chain.

The loop is bounded unrolling with short-circuit pass-through: the body’s steps are flattened into this recipe once per iteration and chained by real carry wiring, so the whole thing is an ordinary statically-inspectable DAG – –dryrun renders every iteration and the offload compiler emits a plain dependency chain. The only runtime decision is whether an already-declared step does any work: once an iteration writes the until sentinel file, every later step passes the corresponding previous iteration’s outputs straight through.

A Recipe body is flattened (selfcal.3.image); any other Scope is one step per iteration (selfcal.3).

Parameters:

  • name (str) – The loop’s name; every step it creates is prefixed with it.

  • body (Scope | StepRef) – The loop body. Its outputs must be re-consumable as its inputs (see carry) – a loop body is a fixed point.

  • max_iter (int) – How many times to unroll. The upper bound on iterations, not the exact count.

  • until (str) – Name of a path-typed output of body. Once that file exists, the loop has converged. A path (rather than a bool) is what lets the identical predicate serve both local execution and an offloaded sbatch script.

  • carry (dict[str, str]) – Body output field -> body input field, the loop-carried dependency. Required and explicit: these pairs are the edges between iterations, and inferring them from matching names would make the graph’s shape depend on name coincidence.

  • index_input (str | None) – Name of a body input to bind to the 1-based iteration number. Without it every iteration resolves to identical values, so a body cannot name its outputs per cycle (a per-cycle image prefix, say) – which real self-calibration does. Steps consuming it must tolerate a changing value: it is the one input that deliberately differs between iterations.

  • **kwargs (Any) – Iteration 1’s inputs, split into wiring and constants exactly as add_step does. Constants apply to every iteration.

Returns:

A LoopRef whose outputs proxy resolves to the final iteration.

Return type:

LoopRef

step(scope, *, backend=None, scatter=None, resources=None, **kwargs)[source]

Decorate a function as a new step appended to this recipe.

Parameters:

  • scope (Scope) – The Cab, Recipe, or bare Scope to bind as this step.

  • backend (str | None) – Backend override for this step.

  • scatter (list[str] | ScatterSpec | None) – Fields to fan out over (see add_step).

  • resources (Resources | None) – What this step costs to run (see add_step).

  • **kwargs (Any) – Split into wiring (InputRef/OutputRef values) and per-step constant params via _split_kwargs.

Returns:

A decorator that binds the given function, appends the resulting StepRef to self.steps, and returns it.

set_output(field, ref)[source]

Wire a recipe output field to an upstream step’s output.

Parameters:

  • field (str) – Name of the recipe’s own output field.

  • ref (OutputRef) – The step output that should populate field.

Returns:

self, for chaining.

Return type:

Recipe

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.RecipeAccessPlan(graph, accesses, reasons)[source]

Bases: object

A graph augmented with deterministic access-hazard edges.

Parameters:
graph: Any
accesses: dict[str, tuple[ResolvedDatasetAccess, ...]]
reasons: dict[tuple[str, str], tuple[str, ...]]
class shinobi.ResolvedDatasetAccess(*, field, element_index=None, declaration, requested_path, root, resources, mode, whole_dataset, path_known, columns_known, fallback=None, closure_status=None, reason)[source]

Bases: BaseModel

Serializable resolution of one dataset access declaration.

declaration is the effective copy with concrete column names, or columns=None when input values are unavailable. The scope retains its original templates; existing literal-column records are unchanged.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

field: str
element_index: int | None
declaration: DatasetAccess
requested_path: Path | None
root: Path | None
resources: tuple[Path, ...]
mode: DatasetMode
whole_dataset: bool
path_known: bool
columns_known: bool
fallback: DatasetFallback | None
closure_status: ClosureStatus | None
reason: str
property address: DatasetAddress
property writes: bool
class shinobi.ScatterSpec(*, fields)[source]

Bases: BaseModel

Declaration that a step should fan out over one or more list-typed input fields. Each listed field must be a list at runtime and all listed fields must have the same length. The step is executed once per index, with slice i receiving element i of every scattered field.

The step’s own inputs_model/outputs_model describe one slice. A downstream step sees the scattered step’s outputs gathered into lists (one element per slice), so it can scatter over them in turn or consume the whole list as a gathered result.

Parameters:

fields (list[str])

fields: list[str]
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.Scope(*, name, info=None, inputs_model, outputs_model, backend=None, image=None, venv=None, input_mutability=<factory>, field_meta=<factory>, dataset_accesses=<factory>, derived_reads=<factory>, cache=None, cache_dir=None, sandbox=None, harvest=<factory>, scratch=<factory>, resources=None)[source]

Bases: BaseModel

Definition: schema, metadata, backend config. Never carries inputs/outputs/func fields – those live in ExecContext/StepRef.

Cab/Recipe are the two execution-aware subclasses ExecContext.run() knows how to run. A bare Scope is also valid – it’s the manual building block for a plain-Python-function step whose own function returns its StepResult directly rather than calling ctx.run(); see StepRef’s docstring and steps/pyfunc.py’s @shinobi.pystep (which automates this pattern from a function’s own signature).

image is optional: when set on a bare Scope (typically via @shinobi.pystep(image=…)), the step’s Python function can be executed inside a container instead of in-process. Cab inherits this field for the same purpose (container backends need it to wrap argv in a runtime invocation).

Parameters:
name: str
info: str | None
inputs_model: type[BaseModel]
outputs_model: type[BaseModel]
backend: str | None
image: str | None
venv: str | None
input_mutability: dict[str, Mutability]
field_meta: dict[str, ParamMeta]
dataset_accesses: list[DatasetAccess]
derived_reads: dict[str, DerivedRead]
cache: bool | None
cache_dir: str | None
sandbox: bool | None
harvest: list[str]
scratch: list[str]
resources: Resources | None
mutability_of(field)[source]

Look up the declared mutability of an input field.

Parameters:

field (str) – Name of the input field.

Returns:

The field’s Mutability, defaulting to Mutability.IMMUTABLE if not explicitly declared.

Return type:

Mutability

with_backend(backend)[source]

A copy bound to backend, or self unchanged if backend is None. Shared by @shinobi.step and Recipe.step, which both bind a per-step backend override onto a Scope before wrapping it in a StepRef.

Parameters:

backend (str | None)

Return type:

Scope

with_resources(resources)[source]

A copy bound to resources, or self unchanged if None. Same shape as with_backend, and used the same way: Recipe.add_step, Recipe.step and @shinobi.step bind a per-step footprint onto a Scope before wrapping it in a StepRef, so one shared Cab can be declared cheap in one recipe position and expensive in another without either mutating the Cab itself.

Parameters:

resources (Resources | None)

Return type:

Scope

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.StepRef(*, name, step, func=None, wiring=<factory>, params=<factory>, scatter=None, after=<factory>, loop=None)[source]

Bases: BaseModel

A named, executable binding of a Scope: orchestration function, wiring (meaningful only inside a Recipe), and per-step constants. Returned by @shinobi.step (free-standing) and @recipe.step (appended to recipe.steps). arbitrary_types_allowed is needed only for func.

step is typed as the general Scope (not Cab | Recipe) so it can also hold a bare Scope – the manual, no-magic way to write a plain-Python-function step: build Scope(name=, inputs_model=, outputs_model=) yourself, write a function that always returns its own StepResult (never calls ctx.run(), which only knows how to execute a Cab or Recipe), and wrap it in a StepRef directly. @shinobi.pystep (steps/pyfunc.py) automates exactly this pattern by deriving the Scope’s schema from the function’s own signature. Passing a Cab/Recipe instance here is unaffected – pydantic’s default revalidate_instances=”never” keeps an already-constructed instance’s real subtype, it does not downcast to bare Scope.

Parameters:
model_config = {'arbitrary_types_allowed': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str
step: Scope
func: Callable | None
wiring: dict[str, 'InputRef | OutputRef | list[InputRef | OutputRef]']
params: dict[str, Any]
scatter: ScatterSpec | None
after: list[str]
loop: LoopIteration | None
shinobi.dataset_backend_capability(backend, *, mutation)[source]

Return the explicit capability decision for backend.

Local processes and virtualenvs share the lifecycle controller’s namespace directly. The four local container runtimes use identity bind mounts augmented from the resolved closure plan; daemon-backed runtimes are unavailable when their configuration does not prove a local daemon. The blocking Slurm backend deliberately remains unavailable: it has no validated submit-host-to-executor namespace mapping and does not run observation and recovery at the execution authority. remote is a launch surface, not a local backend; the remote ninja run must make its own decision.

Parameters:
Return type:

DatasetBackendCapability

shinobi.dataset_attempt_path(workspace, attempt_id)[source]

Return the durable local attempt-record path.

Parameters:

  • workspace (Path)

  • attempt_id (str)

Return type:

Path

shinobi.pystep(*, name=None, info=None, image=None, venv=None, backend=None, cache=None, cache_dir=None, sandbox=None, harvest=None, write_paths=None, dataset_accesses=None, **params)[source]

Decorate (or directly call on an existing function, matching @shinobi.step’s precedent: pystep()(existing_func)) a plain, type-hinted function to turn it into a StepRef. See the module docstring for the schema-derivation and outputs rules.

image enables container execution: when set and a container backend is resolved, the function runs inside the specified container image instead of in-process. The function’s source module is mounted into the container so it can be imported by the runner script.

venv enables virtualenv execution: when set (a path or a key into backend.venv.envs) and the venv backend is resolved, the function runs under that venv’s own interpreter, importing the venv’s real packages (unlike the container path, the target’s own package is not stubbed). A scope may carry both image and venv; the resolved backend name decides which is used.

backend sets the default backend for this step (same as on any Scope). With image, this is typically a container backend name like "docker" or "apptainer"; with venv, "venv".

cache/cache_dir set this step’s cache policy, with the same precedence as the corresponding fields on any Scope. In particular, cache=False lets a function whose products depend on run-time choices opt out when one static output schema cannot describe them safely.

sandbox/harvest opt this step into sandboxed execution and declare extra keep-globs (see shinobi.sandbox and the fields on Scope). Sandboxing applies to the out-of-process paths (container and venv) – an in-process run ignores it (os.chdir is process-global, and recipes run steps on a thread pool), executing in the caller’s cwd as always.

write_paths names the parameters that are destinations this function creates rather than data it reads (ParamMeta.write_path) – the one thing no signature can express, since outputvis: Path returned as the output and vis: Path rewritten in place are the same annotation. Only a named parameter has its stale product cleared before a re-run (sandbox.clear_stale_outputs); everything else is treated as the caller’s data and left alone.

dataset_accesses attaches the same serializable MSv2 access contracts a binary Cab carries on Scope; it does not inspect a path while the function is decorated.

**params are per-call constants, same as @shinobi.step.

Parameters:
Return type:

Callable[[Callable], StepRef]

shinobi.plan_recipe_accesses(recipe, inputs, *, workspace=None, validated_steps=None, validated_inputs=None, overwrite_steps=frozenset({}))[source]

Resolve dataset contracts and add backward hazard edges before dispatch.

Explicit wiring/after edges determine the stable topological order. Otherwise-unordered access conflicts are then oriented backwards in that order, so the result is deterministic and cannot invent data lineage. Nested recipes and dataset-bearing scatter are refused explicitly in this first bounded planner rather than partially planned.

Parameters:
Return type:

RecipeAccessPlan

shinobi.resolve_dataset_closure(path, *, storage_namespace, max_resources=32, max_entries_per_resource=100000)[source]

Resolve an MSv2 into canonical physical CASA-table resources.

Resolution detects changes across its reads, but filesystem observation is not an atomic snapshot. Consumers must use a cooperative immutable or snapshot boundary and revalidate the observation before acting on it.

Parameters:

  • path (str | Path)

  • storage_namespace (str | Path)

  • max_resources (int)

  • max_entries_per_resource (int)

Return type:

DatasetClosure

shinobi.read_dataset_attempt(path)[source]

Read and validate one durable lifecycle attempt record.

Parameters:

path (Path)

Return type:

DatasetLifecycleAttempt

shinobi.step(scope, *, backend=None, name=None, scatter=None, resources=None, **params)[source]

Decorate a function with an existing Scope (Cab or Recipe). See the module docstring.

Parameters:
Return type:

Callable[[Callable], StepRef]

Schema helpers

Supporting types used when defining cabs, not re-exported at the top level.

class shinobi.derived.DerivedAddress(*, name, coordinates)[source]

Bases: ProductModel

Parameters:
name: str
coordinates: dict[str, Coordinate]
property field
property element_index
property label
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.steps.schema.ParamMeta(*, nom_de_guerre=None, implicit=None, info=None, positional=False, positional_head=False, repeat_as_tokens=False, dtype=None, choices=None, nullable=None, string_pattern=None, abbreviation=None, write_path=False, writable=None, family=None)[source]

Bases: BaseModel

Per-field metadata a plain pydantic model can’t express: the name the underlying tool actually expects (nom_de_guerre), a value always supplied by the cab itself rather than the caller (implicit), human-facing help (info), the cab dtype string (dtype, e.g. “File”/”MS”) for a ParamPattern attr – since a dynamically-named param has no declared field/type annotation for path_fields to inspect, this is how backends know to bind-mount its directory – positional: emitted as a bare value (no –flag), in field-declaration order, after every flagged/pattern-matched arg. positional_head: the same, but emitted before every flagged/ pattern-matched arg instead of after – real cult-cargo’s own cubical.yml names this exact policy (parset: {policies: {positional_head: true}}) for a tool whose own CLI only recognises a leading bare token as a parset file to seed defaults from (CubiCal’s main.py: if len(sys.argv) > 1 and not sys.argv[1][0].startswith(‘-‘): custom_parset_file = sys.argv[1]); killMS’s kMS.py has the identical sys.argv[1]-only check. A plain positional field there would always land after every –flag, which these two tools’ own argv[1]-anchored parset detection can’t see – either raising (“Unexpected number of arguments”, CubiCal) or silently not reading the parset at all (killMS, which never validates leftover-arg count). Setting both positional and positional_head on the same field is nonsensical; positional_head wins if both are set. Head positionals, like tail ones, are emitted in field-declaration order. – and repeat_as_tokens: a list/tuple value is emitted as separate bare argv tokens (after the one flag occurrence, or as separate positional tokens) instead of joined into one comma-separated token – real cult-cargo cabs express this as a per-field policies: {repeat: list} (see e.g. wsclean’s -size <w> <h>/-weight briggs <n>, which need two separate argv tokens, not “4096,4096” as one).

write_path: this string-typed input names a filesystem path the tool writes to. Two shapes, both covered: a stem products are built from (wsclean’s prefix, ddfacet’s Output-Name, which never exists as a file itself), and a complete path written directly (a cache directory, a logfile). It is a declaration of intent only, and deliberately changes nothing about how the value is handled. In particular it does not make the field a path: a path dtype here would be rewritten by sandbox.absolutize_path_inputs and the tool would then write outside the sandbox, which is precisely why the convention types these as strings (see declared_output_dirs). Mounting and pre-creation still come from the output side – a path-typed output whose implicit template names this field, a harvest glob, or a scratch glob.

What it buys is enforcement. That convention previously existed only as prose and authorial discipline: a write path nobody declared a target for produces no error, just products that stay inside the container or outside the harvest. Marking the field lets Cab check that something does declare it (see Cab._write_path_declares_a_write_target).

A path-typed input may be marked too, and there it is the only thing that distinguishes the two identical-looking spellings of an output field that echoes a same-named input. mstransform takes outputvis and creates it; flagdata takes vis and rewrites the caller’s own data; both declare that one name on inputs_model and on outputs_model, so mutated_path_fields – and any other structural test – sees one shape. Marking outputvis says which it is, and sandbox.clear_stale_outputs then clears the stale product before a re-run instead of leaving the tool to trip over it. The default (unmarked) is the safe reading: an echoed path is the caller’s data and is never deleted. Note the marker still changes nothing about how the value is handled – a path-typed one is anchored at the workspace by sandbox.absolutize_path_inputs exactly as before, which is what a complete destination path (as opposed to a stem) wants.

writable: this path-typed input is one the tool must not modify (writable: false, cult-cargo/scabha’s own key). The container backends bind-mount the directories a read-only input contributes :ro, and re-assert the input itself :ro at its own path when something writable shares that directory (backends.container.bind_dir_modes). None – the default, and every Python-typed pystep input – means unmarked, which is read as writable. Carried onto a declared field’s json_schema_extra by the loaders (readonly_path_fields reads it from there, the same channel abbreviation rides); kept here as well because a ParamPattern attr has no declared model field for readonly_path_fields to inspect, and a dynamically-named input is exactly the case that wants the marker – QuartiCal’s <term>.load_from names a previous run’s gain store, which the step reads and must not write back into.

choices: the field’s allowed values (cult-cargo/classic’s choices key). A loader that sets this also narrows the field’s real annotation on inputs_model/outputs_model to typing.Literal[*choices] (see loaders._modelgen.narrow_choices), so an out-of-set value fails pydantic validation the same way a wrong dtype would – not merely documented in info. Kept here too (rather than only inferred from the model’s own annotation) so a ParamPattern attr – which has no declared model field for a dynamically-matched name – can still carry it.

string_pattern: a literal scalar string field’s validation regex. Static loaders attach it to the generated pydantic annotation, preserving its JSON-schema pattern. Python authors use a constrained model field. It is not a path classifier, a dynamic attr validator, or executable cab code.

abbreviation: a short single-dash CLI alias for the field’s generated –long-flag (cult-cargo/classic’s abbreviation key, e.g. simms’ ascii-sky -> -as). Purely a ninja run convenience – carried onto the field’s json_schema_extra by the loaders so clickutil.build_options can emit the alias; it never affects the argv the tool actually receives (that still uses nom_de_guerre).

On an output field, a string implicit containing {name} placeholders is resolved by steps.dispatch._fill_outputs as a str.format template against the step’s prepared (validated) input values – e.g. implicit=”{prefix}-MFS-image.fits” derives a tool’s output path from its own prefix input, without shinobi ever importing/executing the tool’s own schema-generation code. A plain string with no {…} is used as a literal constant, same as on an input field.

Parameters:

  • nom_de_guerre (str | None)

  • implicit (Any)

  • info (str | None)

  • positional (bool)

  • positional_head (bool)

  • repeat_as_tokens (bool)

  • dtype (str | None)

  • choices (list[Any] | None)

  • nullable (bool | None)

  • string_pattern (str | None)

  • abbreviation (str | None)

  • write_path (bool)

  • writable (bool | None)

  • family (FamilySpec | None)

nom_de_guerre: str | None
implicit: Any
info: str | None
positional: bool
positional_head: bool
repeat_as_tokens: bool
dtype: str | None
choices: list[Any] | None
nullable: bool | None
string_pattern: str | None
abbreviation: str | None
write_path: bool
writable: bool | None
family: FamilySpec | None
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.steps.schema.ParamPattern(*, separator='.', segments)[source]

Bases: BaseModel

A family of inputs whose names are <segment><separator><segment>…, e.g. QuartiCal’s K.type/G.time_interval or cubical’s g1-solvable/ g-time-int. Matched as one anchored regex assembled from segments: exactly one segment is attrs (the known, enumerable part, each value with its own ParamMeta – dtype/nom_de_guerre/info); every other segment is a regex (soft shape-validation of a level that can’t be enumerated ahead of time). See AGENTS.md for the motivating tools.

The attrs segment is usually last (cubical/QuartiCal: an unenumerable term name followed by a known attribute, g1.solvable), but doesn’t have to be – wsclean’s dynamic output names are the opposite shape, a known/enumerable image type followed by an open-ended qualifier tail (dirty.per-band, restored.i.per-interval.mfs), so attrs there is the first segment. Only one segment may carry attrs; the rest must all be regex.

A segment regex that should behave as an unconstrained “match anything” level (the old design’s prefix) should be written lazily (.+?, not .+): with more than one registered attr, an eager .+ prefers the shortest attr that completes an overall match, which is wrong when one attr is itself a suffix of another (e.g. “int” vs “time-int” with separator “-”) – .+? tries the shortest prefix first, which is exactly “prefer the longest/most specific attr”.

Parameters:
separator: str
segments: list[ParamSegment]
matches(name)[source]

Check whether name matches this pattern and look up its metadata.

Parameters:

name (str) – A dynamic parameter name to test, e.g. “g1.solvable”.

Returns:

The ParamMeta for the matched attrs segment value, or None if name doesn’t match the compiled pattern.

Return type:

ParamMeta | None

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

  • self (BaseModel) – The BaseModel instance.

  • context (Any) – The context.

Return type:

None

class shinobi.steps.schema.ParamSegment(*, regex=None, attrs=None)[source]

Bases: BaseModel

One level of a dotted/dashed dynamic-parameter name. A “shape” segment carries only regex – soft validation, no metadata, for a level whose actual values can’t be enumerated at cab-authoring time (e.g. a solver term name like QuartiCal’s K/G). The “meta” segment – always the last one in a ParamPattern – carries attrs: the known, enumerable part, each value with its own ParamMeta.

Parameters:
regex: str | None
attrs: dict[str, ParamMeta] | None
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.steps.schema.Policies(*, prefix='--', replace=<factory>, list_sep=', ', repeat_list=False, key_value=False, repeat=None, explicit_true=False, explicit_false=False, true_token='true', false_token='false')[source]

Bases: BaseModel

How a cab’s parameters are turned into command-line arguments.

key_value/repeat mirror real cult-cargo cab-level policy keys verbatim (e.g. QuartiCal’s policies: {key_value: true, repeat: ‘[]’, prefix: ‘’}): key_value=True means a hydra-style single name=value argv token instead of two tokens (–name, value); repeat=”[]” means a list value formats as one bracketed-literal token (solver.terms=[K,G]) instead of list_sep-joining. Distinct from a per-field ParamMeta.repeat_as_tokens (real per-field policies: {repeat: list}, e.g. wsclean’s bare -size 4096 4096), which is a field-level override and takes precedence when set.

explicit_true/explicit_false also mirror real cult-cargo cab-level policy keys verbatim (e.g. CubiCal’s policies: {explicit_true: true, explicit_false: false}): by default a True boolean value emits as a bare flag (–flag, argparse store_true-style) and False is omitted entirely. Some real CLIs (CubiCal’s own optparse-derived parser among them) instead expect every boolean option to always take an explicit value token – passing a bare flag with no value corrupts parsing of everything after it, since the parser consumes the next token as that flag’s value. explicit_true=True emits –flag true (two tokens, “true”/”false” lowercase) instead of a bare flag when the value is True; explicit_false=True does the same instead of omitting the flag when the value is False. Each direction is independent (CubiCal only needs explicit_true, never explicit_false), and this applies uniformly to declared fields and ParamPattern-matched dynamic ones (e.g. CubiCal’s own per-Jones-term g-solvable).

true_token/false_token are what those two tokens actually say. Lowercase “true”/”false” (the default) is what CubiCal’s parser reads, but it is not universal: DDFacet and killMS share a parset reader (DDFacet.Parset.ReadCFG) that parses “0”/”1” as ints and “True”/”False” as bools, and leaves an unrecognised “false” as a string – which, being non-empty, is truthy. –Mask-Auto false there switches the mask on. So the pair of tokens is a cab-level policy of its own (policies: {explicit_true: true, explicit_false: true, true_token: ‘1’, false_token: ‘0’} for those two), alongside the explicit_* switches rather than instead of them: explicit_* decides whether a value token is emitted, these decide what it reads.

They spell a boolean in every value position, not only after a flag: a key_value cab’s single name=value token, a positional, an element of a joined or bracketed list, one occurrence of a repeated flag. Python’s own str(True) is “True”, so a path that missed this emitted a spelling no policy ever asked for (policies._scalar_token is the one place that decides).

Parameters:
prefix: str
replace: dict[str, str]
list_sep: str
repeat_list: bool
key_value: bool
repeat: str | None
explicit_true: bool
explicit_false: bool
true_token: str
false_token: str
arg_name(name)[source]

Build the CLI flag name for a parameter name.

Parameters:

name (str) – The parameter’s declared/matched name.

Returns:

name with each replace substitution applied, prefixed by prefix (e.g. “–”).

Return type:

str

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.steps.schema.ScatterSpec(*, fields)[source]

Bases: BaseModel

Declaration that a step should fan out over one or more list-typed input fields. Each listed field must be a list at runtime and all listed fields must have the same length. The step is executed once per index, with slice i receiving element i of every scattered field.

The step’s own inputs_model/outputs_model describe one slice. A downstream step sees the scattered step’s outputs gathered into lists (one element per slice), so it can scatter over them in turn or consume the whole list as a gathered result.

Parameters:

fields (list[str])

fields: list[str]
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

shinobi.steps.schema.path_fields(model)[source]

Names of every field of model whose direct-value leaf type is a filesystem path (pathlib.Path). File-like cab dtypes (File/MS/Directory/URI) map to Path, so this drives both container bind-mounting and the CLI’s click.Path() mapping. Direct-value containers include concrete and abstract sequences and sets plus tuples.

Parameters:

model (type[BaseModel])

Return type:

set[str]

shinobi.steps.schema.declared_output_dirs(scope, prepared, *, workspace=None)[source]

(directory, source) for every directory the step is declared to write into, resolved before the run from the declarations alone – never from the shape of an input’s value.

This is the answer to “where does this tool put its products”, which a field’s dtype cannot express: the established way to declare an output stem is a string-typed input (wsclean’s prefix, ddfacet’s Output-Name), deliberately so, because a path dtype would be rewritten by sandbox.absolutize_path_inputs and the tool would then write outside the sandbox. Such an input can say so with ParamMeta.write_path, which changes nothing here – this function reads the declarations either way – but lets Cab reject a stem that no declaration names, which is the authoring mistake this convention is otherwise silent about. The write target is instead declared by the output side: a path-typed output field whose implicit template names the input ("{prefix}-MFS-image.fits"), a harvest glob, or a scratch glob for a write target that is not a product.

Knowable pre-run means the parent of each declared_output_paths entry (per path-typed output field, its value taken from a same-named input, its resolved implicit template, or its field default – the same priority as _fill_outputs), then the literal (glob-free) directory prefix of each harvest and scratch pattern. source describes the declaration (output 'restored_image', harvest pattern '{prefix}-*.fits', scratch pattern '{cache_dir}/*') so callers can name it in an error.

A pattern that references a field whose value is None is skipped, not resolved: substituting it would produce a literal None path segment ("{cache_dir}/*" -> "None/*"), which is a real directory name and would have the sandbox create one. An unset optional write target declares nothing, which is the same answer a path-typed output whose value is None already gives.

Both relative and absolute directories are returned, order-preserving and de-duplicated; . and anything containing .. is dropped. Execution pre-creates these directories at the tool’s actual cwd, including absolute destinations. The container backend bind-mounts the absolute ones so a product declared outside the workdir reaches the host. Both consumers treat a scratch directory exactly like a product’s: it is mounted so the tool’s write lands on the host, and pre-created so the tool doesn’t have to. Only harvest differs – it never rescues a scratch path out of a sandbox, which is the whole point of the distinction. Best-effort by design – a template that fails to resolve is skipped here, not raised; output filling and harvest report those errors with full context.

Parameters:
Return type:

list[tuple[Path, str]]

shinobi.steps.schema.declared_output_paths(scope, prepared, *, workspace=None)[source]

(path, source) for every path-typed output field the step is declared to produce, resolved before the run from the declarations alone – the products themselves, where declared_output_dirs (built on this) reports only the directories they land in.

Knowable pre-run means, per path-typed output field and in declaration order: its value taken from a same-named input, its resolved implicit template, or its field default – the same priority as _fill_outputs. A list-valued field contributes each element. A field that resolves to None contributes nothing (an unset optional product declares no path). source names the declaration (output 'restored_image') so callers can report it.

Deliberately not included: harvest/scratch glob patterns. Those describe a family whose member names the tool chooses at run time, rather than concrete products. Bounded families can be proved disjoint from protected datasets before execution by product_pattern_issue, while harvesting still distinguishes a declared destination from a glob-matched one.

Best-effort by design – a template that fails to resolve is skipped here, not raised; output filling reports those errors with full context.

Parameters:
Return type:

list[tuple[Path, str]]

shinobi.steps.schema.readonly_path_fields(model, field_meta=None)[source]

Names of the path_fields explicitly marked writable: false in their schema. The container backend bind-mounts the directories these contribute read-only (backends.container.bind_dir_modes). A path field with no writable marker (the default, including every Python-typed pystep input) is treated as writable.

Two spellings of the same marker, because a field has two places to carry one and a cab author should not have to know which: the loaders (yaml_cab, worker_schema) put it on the field’s json_schema_extra, and a Python-authored cab sets ParamMeta.writable in field_meta. Either says read-only; pass field_meta (a Scope always has one) so both are seen. Reading only the model would accept-and-drop the Python spelling, which is the failure this marker exists to prevent.

Declared fields only. A ParamPattern-matched input has no model field at all, so it declares the marker on its attr’s ParamMeta.writable, which bind_dir_modes reads directly.

Parameters:
Return type:

set[str]

shinobi.steps.schema.mutated_path_fields(scope)[source]

Names of the path inputs scope rewrites in place.

Three spellings, all meaning the same thing: the field is declared on outputs_model as well as inputs_model, its input is declared Mutability.MUTABLE (the flag/gaincal/applycal shape, where a cab rewrites its MS without re-declaring it as an output), or a DatasetAccess declares mode="write" on the input. The last is the strict MSv2 spelling of the second; leaving it out would let the cache key fingerprint a dataset its own step rewrites while the snapshotter ignored it.

This lives here, in schema, rather than in either of its two callers, because they must never disagree. cache.compute_cache_key drops these fields from the key; shinobi.snapshots decides what to snapshot from exactly the same set. If one grew a case the other lacked, the cache would key a step on content the snapshotter had already rolled back – the drift is silent and the failure is wrong science, so there is one computation and both import it.

Parameters:

scope (Scope)

Return type:

set[str]

shinobi.steps.schema.paths_overlap(a, b)[source]

Whether two canonical paths can name the same bytes: equal, or one inside the other. Containment is not a nicety here – a Measurement Set is a directory, so /data/obs.ms and /data/obs.ms/CORRECTED name the same data while comparing unequal.

Shared by the two places that compare resolved path values rather than the way each step happened to spell them: offload.slurm’s MutationOrder (which steps must be ordered against each other) and sandbox.clear_stale_outputs (whether a declared output is really an input this step reads). Both are wrong in the same expensive way if they disagree, so there is one predicate.

Parameters:
Return type:

bool

Datasets

class shinobi.dataset_access.DatasetAddress(field, element_index=None)[source]

Bases: object

Typed field/element identity; labels are for diagnostics only.

Parameters:

  • field (str)

  • element_index (int | None)

field: str
element_index: int | None = None
property label: str
class shinobi.dataset_state.DatasetStateStore(directory='.shinobi/states', *, cache_dir='.shinobi/cache')[source]

Bases: object

A local immutable state store with explicit, recoverable operations.

Paths are local filesystem paths. Source and destination parent directories must exist. Stored state references are deliberately not accepted by Recipe. cache_dir is used only to refuse an unfinished Tier 1 mutation on export.

Parameters:
list()[source]

List committed manifest identities; this does not verify stored data.

Return type:

list[StateResult]

verify(state_id, representation_id=None)[source]

Fully verify contracts, physical bytes and both public logical roots.

Parameters:

  • state_id (str)

  • representation_id (str | None)

Return type:

StateResult

export(source, *, block_rows=None, lossless_profile=None)[source]

Capture a supported contained MS under a shared READ ownership claim.

Parameters:
Return type:

StateResult

materialize(state_id, destination, *, representation_id=None, fidelity='exact-logical')[source]

Create a fresh native MS after independent validation, never overwrite.

Parameters:

  • state_id (str)

  • destination (str | Path)

  • representation_id (str | None)

  • fidelity (str)

Return type:

StateResult

derive_precision(state_id, policy, *, representation_id=None)[source]

Create a separate approximate MSv4 artifact from a verified exact state.

Parameters:

  • state_id (str)

  • representation_id (str | None)

verify_derived(derived_id, representation_id=None)[source]
Parameters:

  • derived_id (str)

  • representation_id (str | None)

list_derived()[source]
recover_derived()[source]
recover(destination=None)[source]

Settle dead attempts and sweep only their recorded private stage trees.

A sweep is all-or-nothing: any live, uncertain or invalid attempt aborts it. Use destination to recover one target independently.

Parameters:

destination (str | Path | None)

Return type:

list[StateAttempt]

class shinobi.state_codecs.LosslessProfile

Closed, versioned MSv4 lossless codec and chunk layout selection.

class shinobi.dataset_state.LogicalState

Strict shinobi-logical-state/v1 manifest: native state ID, native model, hash/profile/structural/closure contracts, and exact-logical fidelity.

class shinobi.dataset_state.StateRepresentation

Strict shinobi-state-representation/v1 physical inventory, logical bindings, source observation, software versions and export provenance.

class shinobi.dataset_state.StateResult

State and representation IDs, verified flag, fidelity, optional native destination and attempt-record path. Provides Pydantic JSON serialization.

class shinobi.dataset_state.StateAttempt

Versioned durable operation record: ownership, phases, native state, physical representation, staging identities and recovery outcome.

class shinobi.dataset_state.StateProvenance

Closed logical-state, representation, mapping, reconstruction, fidelity and decision evidence embedded by state-attempt schema v2.

shinobi.dataset_state.read_state_attempt(path)[source]

Read and strictly validate a durable reusable-state attempt record.

Parameters:

path (str | Path)

Return type:

StateAttempt

class shinobi.dataset_state.StateError(code, message, *, details=None)[source]

Bases: ShinobiError

Stable refusal code plus any upstream table/column/row diagnostics.

Parameters:
shinobi.datasets.CasaTab

A path annotated with the versioned casa-table/v1 declaration.

shinobi.datasets.MSv2

A path annotated with the versioned msv2-structural/v1 declaration.

shinobi.datasets.CasaTable

Object-identical compatibility alias for CasaTab.

shinobi.datasets.MeasurementSetV2

Object-identical compatibility alias for MSv2.

class shinobi.datasets.DatasetType(kind, profile)[source]

Bases: object

Serializable declaration attached to a path annotation.

profile versions the exact structural contract. New requirements must get a new profile rather than silently changing the meaning of an existing annotation.

Parameters:
kind: DatasetKind
profile: str
class shinobi.datasets.InspectionLimits(max_columns=4096, max_keywords=4096)[source]

Bases: object

Bounds on metadata retained by one inspection.

Casacore supplies column and keyword names as metadata. The inspector refuses an object whose metadata exceeds these bounds instead of silently truncating it; column cells are never read at any limit.

Parameters:

  • max_columns (int)

  • max_keywords (int)

max_columns: int = 4096
max_keywords: int = 4096
class shinobi.datasets.DatasetDescriptor(*, path, expected, status, message, nrows=None, columns=(), keywords=(), subtables=(), missing_columns=(), missing_primary_data_columns=(), missing_subtables=(), unreadable_subtables=(), missing_subtable_columns=<factory>, unsupported_features=())[source]

Bases: BaseModel

Serializable observation produced by explicit structural inspection.

The complete column and keyword name sets are retained (within InspectionLimits), including optional and custom columns. The descriptor is an observation, not a parameter annotation and not a cache identity.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

path: Path
expected: DatasetType
status: DatasetStatus
message: str
nrows: int | None
columns: tuple[str, ...]
keywords: tuple[str, ...]
subtables: tuple[str, ...]
missing_columns: tuple[str, ...]
missing_primary_data_columns: tuple[str, ...]
missing_subtables: tuple[str, ...]
unreadable_subtables: tuple[str, ...]
missing_subtable_columns: dict[str, tuple[str, ...]]
unsupported_features: tuple[str, ...]
property valid: bool

Whether the observed path satisfies the requested profile.

class shinobi.datasets.DatasetKind(*values)[source]

Bases: str, Enum

Dataset families understood by the structural inspector.

CASA_TABLE = 'casa-table'
MEASUREMENT_SET_V2 = 'measurement-set-v2'
class shinobi.datasets.DatasetStatus(*values)[source]

Bases: str, Enum

Outcome categories returned by inspect_dataset().

MISSING_PATH = 'missing-path'
NOT_A_DIRECTORY = 'not-a-directory'
INSPECTOR_UNAVAILABLE = 'inspector-unavailable'
NOT_A_CASA_TABLE = 'not-a-casa-table'
NOT_A_MEASUREMENT_SET = 'not-a-measurement-set'
INCOMPLETE_MEASUREMENT_SET_V2 = 'incomplete-measurement-set-v2'
UNSUPPORTED = 'unsupported'
VALID = 'valid'
class shinobi.dataset_closure.DatasetClosure(*, profile='msv2-dataset-closure/v2', requested_root, storage_namespace, root=None, status, message, resources=(), capabilities=None, unsupported_features=())[source]

Bases: BaseModel

Serializable observation of an MSv2’s physical CASA-table resources.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

profile: str
requested_root: Path
storage_namespace: Path
root: Path | None
status: ClosureStatus
message: str
resources: tuple[ClosureResource, ...]
capabilities: ClosureCapabilities | None
unsupported_features: tuple[str, ...]
property valid: bool
class shinobi.dataset_closure.ClosureRequirement(*values)[source]

Bases: str, Enum

Physical requirements of the accepted ordinary-directory profile.

TABLE_MEMBERS = 'table-members'
PRESERVE_NAMESPACE_PATHS = 'preserve-namespace-paths'
REWRITE_KEYWORD_REFERENCES = 'rewrite-keyword-references'
class shinobi.dataset_closure.ClosureCapabilities(*, copy_requirements, mount_requirements, stage_requirements, materialize_requirements, restore_requirements)[source]

Bases: BaseModel

Operation requirements established by the accepted closure profile.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

copy_requirements: tuple[ClosureRequirement, ...]
mount_requirements: tuple[ClosureRequirement, ...]
stage_requirements: tuple[ClosureRequirement, ...]
materialize_requirements: tuple[ClosureRequirement, ...]
restore_requirements: tuple[ClosureRequirement, ...]
class shinobi.dataset_closure.ClosureResource(*, path, namespace_path, members, table_files, external_to_root, storage_managers)[source]

Bases: BaseModel

One canonical physical CASA table in a dataset closure.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

path: Path
namespace_path: Path
members: tuple[str, ...]
table_files: tuple[str, ...]
external_to_root: bool
storage_managers: tuple[str, ...]
class shinobi.dataset_closure.ClosureStatus(*values)[source]

Bases: str, Enum

Outcome of MSv2 physical closure resolution.

VALID = 'valid'
INVALID_ROOT = 'invalid-root'
MISSING_RESOURCE = 'missing-resource'
UNREADABLE_RESOURCE = 'unreadable-resource'
DANGLING_REFERENCE = 'dangling-reference'
CYCLIC_REFERENCE = 'cyclic-reference'
ESCAPED_NAMESPACE = 'escaped-namespace'
CHANGED_DURING_RESOLUTION = 'changed-during-resolution'
UNSUPPORTED = 'unsupported'
shinobi.dataset_closure.resolve_dataset_closure(path, *, storage_namespace, max_resources=32, max_entries_per_resource=100000)[source]

Resolve an MSv2 into canonical physical CASA-table resources.

Resolution detects changes across its reads, but filesystem observation is not an atomic snapshot. Consumers must use a cooperative immutable or snapshot boundary and revalidate the observation before acting on it.

Parameters:

  • path (str | Path)

  • storage_namespace (str | Path)

  • max_resources (int)

  • max_entries_per_resource (int)

Return type:

DatasetClosure

class shinobi.dataset_access.DatasetAccess(*, field, mode, table=DatasetTable.MAIN, columns=None, selection=None, allow_row_count_change=False, allow_schema_change=False, allow_keyword_change=False, allow_subtable_change=(), allow_present_subtable_rewrite=False, root_field=None, reservation=None)[source]

Bases: BaseModel

One field-local MSv2 access declaration attached to a Scope.

field names an input or output field. root_field is needed only when field itself names a referenced subtable whose parent MS cannot be inferred from its path (notably an externally stored subtable). reservation is a canonicalizable path envelope used when the dataset path is produced dynamically and is therefore unknown during planning. allow_present_subtable_rewrite grants an explicit MAIN writer backing- file rewrites of supported subtables present before execution and still linked at the same path afterward; table schema authority stays explicit.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

field: str
mode: DatasetMode
table: DatasetTable
columns: DatasetColumns | None
selection: DatasetSelection | None
allow_row_count_change: bool
allow_schema_change: bool
allow_keyword_change: bool
allow_subtable_change: tuple[str, ...]
allow_present_subtable_rewrite: bool
root_field: str | None
reservation: Path | None
class shinobi.dataset_access.DatasetMode(*values)[source]

Bases: str, Enum

How a step uses an MSv2 dataset.

READ = 'read'
WRITE = 'write'
CREATE = 'create'
class shinobi.dataset_access.DatasetFallback(*values)[source]

Bases: str, Enum

Why planning conservatively reserves a whole dataset identity.

UNDECLARED = 'undeclared'
UNKNOWN_COLUMNS = 'unknown-columns'
UNKNOWN_PATH = 'unknown-path'
class shinobi.dataset_access.DatasetTable(*values)[source]

Bases: str, Enum

MSv2 tables whose closure membership is understood by Shinobi.

MAIN = 'MAIN'
ANTENNA = 'ANTENNA'
DATA_DESCRIPTION = 'DATA_DESCRIPTION'
DOPPLER = 'DOPPLER'
FEED = 'FEED'
FIELD = 'FIELD'
FLAG_CMD = 'FLAG_CMD'
FREQ_OFFSET = 'FREQ_OFFSET'
HISTORY = 'HISTORY'
OBSERVATION = 'OBSERVATION'
POINTING = 'POINTING'
POLARIZATION = 'POLARIZATION'
PROCESSOR = 'PROCESSOR'
SOURCE = 'SOURCE'
SPECTRAL_WINDOW = 'SPECTRAL_WINDOW'
STATE = 'STATE'
SYSCAL = 'SYSCAL'
WEATHER = 'WEATHER'
class shinobi.dataset_access.DatasetColumns(*, read=(), write=(), create=(), remove=())[source]

Bases: BaseModel

Column-level intent retained for validation and provenance.

Column detail deliberately does not relax scheduling: any writer excludes every concurrent access to the associated resolved dataset closure. None on DatasetAccess means the columns are unknown and therefore requests the conservative whole-dataset fallback. Entries may format direct string inputs of their own scope, for example "{column}" or "MODEL_{suffix}". Only undecorated field names are allowed; a missing or None input makes the entire access’s columns unknown.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

read: tuple[str, ...]
write: tuple[str, ...]
create: tuple[str, ...]
remove: tuple[str, ...]
class shinobi.dataset_access.DatasetSelection(*, row_ranges=(), data_description_ids=(), field_ids=(), observation_ids=(), scan_numbers=(), array_ids=())[source]

Bases: BaseModel

A bounded, data-only row selection.

Half-open row_ranges and finite ID sets cover the common static selection shapes without admitting TaQL, Python callables, or another expression language. Scheduling remains conservative at dataset scope.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

row_ranges: tuple[tuple[int, int], ...]
data_description_ids: tuple[int, ...]
field_ids: tuple[int, ...]
observation_ids: tuple[int, ...]
scan_numbers: tuple[int, ...]
array_ids: tuple[int, ...]
class shinobi.dataset_access.ResolvedDatasetAccess(*, field, element_index=None, declaration, requested_path, root, resources, mode, whole_dataset, path_known, columns_known, fallback=None, closure_status=None, reason)[source]

Bases: BaseModel

Serializable resolution of one dataset access declaration.

declaration is the effective copy with concrete column names, or columns=None when input values are unavailable. The scope retains its original templates; existing literal-column records are unchanged.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

field: str
element_index: int | None
declaration: DatasetAccess
requested_path: Path | None
root: Path | None
resources: tuple[Path, ...]
mode: DatasetMode
whole_dataset: bool
path_known: bool
columns_known: bool
fallback: DatasetFallback | None
closure_status: ClosureStatus | None
reason: str
property address: DatasetAddress
property writes: bool
class shinobi.dataset_access.RecipeAccessPlan(graph, accesses, reasons)[source]

Bases: object

A graph augmented with deterministic access-hazard edges.

Parameters:
graph: Any
accesses: dict[str, tuple[ResolvedDatasetAccess, ...]]
reasons: dict[tuple[str, str], tuple[str, ...]]
shinobi.dataset_access.resolve_scope_dataset_accesses(scope, values, *, workspace, planned_roots=None, unresolved_inputs=frozenset({}), allow_existing_create=False)[source]

Resolve one leaf scope’s MSv2 declarations against concrete values.

Unknown columns and omitted declarations degrade to whole-dataset access. An unknown path is refused unless its declaration supplies a reservation envelope, because otherwise neither ordering nor ownership can cover it. A CREATE target that already exists is refused unless allow_existing_create is set, which only the explicit overwrite path does, to locate the target it is about to delete.

Parameters:
Return type:

tuple[ResolvedDatasetAccess, …]

shinobi.dataset_access.plan_recipe_accesses(recipe, inputs, *, workspace=None, validated_steps=None, validated_inputs=None, overwrite_steps=frozenset({}))[source]

Resolve dataset contracts and add backward hazard edges before dispatch.

Explicit wiring/after edges determine the stable topological order. Otherwise-unordered access conflicts are then oriented backwards in that order, so the result is deterministic and cannot invent data lineage. Nested recipes and dataset-bearing scatter are refused explicitly in this first bounded planner rather than partially planned.

Parameters:
Return type:

RecipeAccessPlan

class shinobi.dataset_backends.DatasetBackendCapability(*, schema_version=1, profile='msv2-backend-route/v1', backend, lifecycle, status, namespace_mode, execution_location, inspection_location, recovery_location, reason)[source]

Bases: BaseModel

Versioned capability decision for one backend and lifecycle kind.

inspection_location and recovery_location are explicit because a scheduler accepting a job on one host does not prove that its compute node sees the same path or that restoring the submit host repairs what the task changed.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

schema_version: Literal[1]
profile: Literal['msv2-backend-route/v1']
backend: str
lifecycle: Literal['contained-native-msv2-read/v1', 'contained-native-msv2-mutation/v1']
status: DatasetBackendStatus
namespace_mode: DatasetNamespaceMode
execution_location: str
inspection_location: str
recovery_location: str
reason: str
class shinobi.dataset_backends.DatasetBackendMount(*, source, target, writable, reason)[source]

Bases: BaseModel

One identity bind required in addition to schema-derived mounts.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

source: Path
target: Path
writable: bool
reason: str
class shinobi.dataset_backends.DatasetBackendPlan(*, schema_version=1, capability, storage_namespace, mounts=())[source]

Bases: BaseModel

Closed, data-only plan handed to a local backend adapter.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

schema_version: Literal[1]
capability: DatasetBackendCapability
storage_namespace: Path
mounts: tuple[DatasetBackendMount, ...]
class shinobi.dataset_backends.DatasetBackendStatus(*values)[source]

Bases: str, Enum

Evidence state for one concrete backend/lifecycle route.

TESTED = 'tested'
UNAVAILABLE = 'unavailable'
UNSUPPORTED = 'unsupported'
class shinobi.dataset_backends.DatasetNamespaceMode(*values)[source]

Bases: str, Enum

How host paths relate to paths observed by the executor.

LOCAL = 'local'
IDENTITY_BIND = 'identity-bind'
SHARED_IDENTITY = 'shared-identity'
REMOTE_DELEGATED = 'remote-delegated'
UNPROVEN = 'unproven'
shinobi.dataset_backends.dataset_backend_capability(backend, *, mutation)[source]

Return the explicit capability decision for backend.

Local processes and virtualenvs share the lifecycle controller’s namespace directly. The four local container runtimes use identity bind mounts augmented from the resolved closure plan; daemon-backed runtimes are unavailable when their configuration does not prove a local daemon. The blocking Slurm backend deliberately remains unavailable: it has no validated submit-host-to-executor namespace mapping and does not run observation and recovery at the execution authority. remote is a launch surface, not a local backend; the remote ninja run must make its own decision.

Parameters:
Return type:

DatasetBackendCapability

shinobi.dataset_backends.plan_dataset_backend(backend, accesses, *, workspace, mutation, leaf_accesses=None)[source]

Build the backend-specific namespace plan for resolved accesses.

accesses is the shared workflow closure plan. leaf_accesses is the current leaf’s exact declared subset (and defaults to accesses for direct callers). Every existing workflow root is mounted read-only, then only roots written or created by this leaf are upgraded. The lifecycle has already restricted execution to contained pairwise-disjoint closures, but this function validates that invariant again at the adapter boundary: mounting an incomplete closure would turn a policy bug into a successful tool run against the wrong dataset.

Parameters:
Return type:

DatasetBackendPlan

shinobi.datasets.dataset_fields(model)[source]

Backward-compatible name for dataset_declarations().

Unlike its original implementation, this returns qualified paths for declarations in nested models and containers as well as top-level fields.

Parameters:

model (type[BaseModel])

Return type:

dict[str, DatasetType]

shinobi.datasets.dataset_declarations(model, *, error=<class 'TypeError'>)[source]

Find every dataset declaration reachable from a Pydantic model.

Returned keys are qualified field paths. [] denotes a sequence item and .* a mapping value, for example groups.*.members[].ms. Nested models and unions are traversed recursively. An ancestor stack prevents self-referential models from recursing forever without suppressing the same model used independently by two fields.

error selects the exception for declaration conflicts at a loader boundary; ordinary Python callers retain the default TypeError.

Raises:

TypeError – If one qualified path contains conflicting declarations.

Parameters:
Return type:

dict[str, DatasetType]

shinobi.datasets.inspect_dataset(path, expected=DatasetType(kind=<DatasetKind.MEASUREMENT_SET_V2: 'measurement-set-v2'>, profile='msv2-structural/v1'), *, limits=None)[source]

Inspect bounded CASA-table metadata against a named profile.

Parameters:

  • path (str | Path) – Candidate on-disk table directory.

  • expected (DatasetType) – Dataset declaration/profile to check.

  • limits (InspectionLimits | None) – Maximum metadata-name counts retained in the descriptor.

Returns:

A serializable descriptor with a precise status. Ordinary invalid datasets are results rather than exceptions; programmer errors such as an unknown profile are reported as UNSUPPORTED.

Return type:

DatasetDescriptor

shinobi.datasets.inspect_casa_table(path, *, limits=None)[source]

Inspect path against CasaTab’s current profile.

Parameters:
Return type:

DatasetDescriptor

shinobi.datasets.inspect_measurement_set_v2(path, *, limits=None)[source]

Inspect path against MSv2’s current profile.

Parameters:
Return type:

DatasetDescriptor

class shinobi.dataset_lifecycle.DatasetLifecycleEvent(*, phase, observed_at, reason)[source]

Bases: BaseModel

One append-only phase observation in an attempt record.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

phase: DatasetLifecyclePhase
observed_at: float
reason: str
class shinobi.dataset_lifecycle.DatasetFileObservation(*, path, device, inode, mode, size, mtime_ns, ctime_ns)[source]

Bases: BaseModel

Cheap metadata identity for one declared CASA table member.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

path: Path
device: int
inode: int
mode: int
size: int
mtime_ns: int
ctime_ns: int
class shinobi.dataset_lifecycle.DatasetObservation(*, structural_profile='msv2-structural/v1', closure_profile='msv2-dataset-closure/v2', root, descriptor, closure, files, table_columns=<factory>)[source]

Bases: BaseModel

Serializable structural, closure and member observation of one MSv2.

Supported subtable columns are supplemental postcondition evidence; historical observations may omit them, and signature v1 excludes them.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

structural_profile: Literal['msv2-structural/v1']
closure_profile: Literal['msv2-dataset-closure/v1', 'msv2-dataset-closure/v2']
root: Path
descriptor: DatasetDescriptor
closure: DatasetClosure
files: tuple[DatasetFileObservation, ...]
table_columns: dict[str, tuple[str, ...]]
shinobi.dataset_lifecycle.STRUCTURAL_SIGNATURE_COVERAGE = 'MAIN row count; MAIN column, keyword and subtable names; closure membership, table files and storage managers. Cell values, keyword values and visibility data are not examined.'

str(object=’’) -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.__str__() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to ‘strict’.

shinobi.dataset_lifecycle.structural_signature(observation)[source]

The structural identity of one observed MSv2 closure.

Paths are recorded relative to the root and stat metadata is excluded, so an exact snapshot restored at the same path, or the same state observed through an alias, has the same signature. See STRUCTURAL_SIGNATURE_COVERAGE for what it does not cover.

Parameters:

observation (DatasetObservation)

Return type:

str

shinobi.dataset_lifecycle.leaf_postcondition_issues(accesses, pre, post)[source]

Every way one leaf’s observed change exceeds its declared contract.

Per dataset root: a pure reader must leave the observation identical; a CREATE must produce a valid contained MSv2 carrying its declared columns; a writer may change row count, schema (columns, subtables, closure membership) and MAIN keyword names only where its declarations permit, may add/remove only declared columns when its columns are known, and may touch only the tables it declares or stable present subtables covered by its rewrite grant – unless it was an undeclared whole-dataset writer, for which any table is in scope.

These are structural checks. Which cells a writer changed is not observable here, so declared column writes are recorded, not verified.

Parameters:
Return type:

list[str]

class shinobi.ownership.WorkspaceAccess(*, path, writes)[source]

Bases: BaseModel

One canonical path covered by a workflow ownership claim.

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

path: str
writes: bool
class shinobi.ownership.WorkspaceOwner(*, schema_version=1, workflow_id, kind, workspace, acquired_at, submission=None, accesses=(), registry)[source]

Bases: BaseModel

Parameters:
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

schema_version: Literal[1]
workflow_id: str
kind: Literal['local', 'slurm']
workspace: str
acquired_at: float
submission: str | None
accesses: tuple[WorkspaceAccess, ...]
registry: str

Execution

class shinobi.results.BackendRun(returncode, stdout='', stderr='', image_digest=None, containerized=False, venv=None, venv_digest=None, stdout_dropped=0, stderr_dropped=0, wrangler_lines_dropped=False)[source]

Bases: object

What a backend returns after running a command – just the raw run outcome. Wrangling stdout/stderr into structured outputs and filling an outputs_model is the dispatch layer’s job, not the backend’s.

Parameters:

  • returncode (int)

  • stdout (str)

  • stderr (str)

  • image_digest (str | None)

  • containerized (bool)

  • venv (str | None)

  • venv_digest (str | None)

  • stdout_dropped (int)

  • stderr_dropped (int)

  • wrangler_lines_dropped (bool)

returncode: int
stdout: str = ''
stderr: str = ''
image_digest: str | None = None
containerized: bool = False
venv: str | None = None
venv_digest: str | None = None
stdout_dropped: int = 0
stderr_dropped: int = 0
wrangler_lines_dropped: bool = False
property success: bool

Whether the run exited with return code 0.

class shinobi.results.StepResult(name, returncode, outputs, inputs, stdout='', stderr='', cached=False, skipped=False, kind='cab', backend=None, image=None, image_digest=None, containerized=False, venv=None, venv_digest=None, sandboxed=False, sandbox_path=None, code_digest=None, worker_digest=None, job_id=None, scheduler_state=None, resources=None, sub_results=None, bundle_inventories=None, cache_key=None, output_keys=None)[source]

Bases: object

The outcome of running a step (a Cab or a Recipe).

outputs is a validated instance of the step’s outputs_model; inputs is a validated instance of the effective (post-override) inputs the step actually ran with. For a Recipe these aggregate from its sub-steps.

Parameters:
name: str
returncode: int
outputs: BaseModel
inputs: BaseModel
stdout: str = ''
stderr: str = ''
cached: bool = False
skipped: bool = False
kind: str = 'cab'
backend: str | None = None
image: str | None = None
image_digest: str | None = None
containerized: bool = False
venv: str | None = None
venv_digest: str | None = None
sandboxed: bool = False
sandbox_path: str | None = None
code_digest: str | None = None
worker_digest: str | None = None
job_id: str | None = None
scheduler_state: str | None = None
resources: Resources | None = None
sub_results: dict[str, StepResult] | None = None
bundle_inventories: list[dict[str, Any]] | None = None
cache_key: str | None = None
output_keys: dict[str, ProvenanceKey | list[ProvenanceKey | None]] | None = None
property success: bool

Whether the step exited with return code 0.

provenance_key(field)[source]

The cache key identifying whatever produced output field.

A leaf step produces all its outputs in one run, so every field resolves to that step’s own cache_key. A Recipe fans out to output_keys instead – each declared output comes from a distinct sub-step, and invalidating a downstream consumer because some unrelated sub-step re-ran would throw away most of the cache’s value.

None means “no provenance available” (caching disabled, or a step that isn’t cacheable) – callers must treat that as “contribute nothing”, not as a key in its own right.

Parameters:

field (str)

Return type:

Any

shinobi.steps.dispatch.register_step_backend(name, backend)[source]

Register a backend instance under name, overriding the real class-based registry. Mainly for tests.

Parameters:
Return type:

None

shinobi.steps.dispatch.get_step_backend(name)[source]

Resolve a backend instance by name, checking test overrides first.

Parameters:

name (str) – Backend name, e.g. “native”, “slurm”, or a name registered via register_step_backend.

Returns:

The backend instance registered under name in _STEP_BACKENDS, else a fresh instance from shinobi.backends.get_backend.

Return type:

Any

Output wranglers: extract structured outputs from a cab’s stdout/stderr by matching each line against the cab’s configured regexes.

Only the PARSE_OUTPUT action is implemented for now – enough to pull named values out of a tool’s console output. Display-oriented actions from Stimela 2 (HIGHLIGHT, SUPPRESS, SEVERITY, …) are currently unsupported.

shinobi.wranglers.apply_wranglers(wranglers, lines)[source]

Extract structured output values from a cab’s console output.

Parameters:

  • wranglers (dict[str, list[str]]) – Mapping of regex pattern to the list of wrangler action strings to apply on each matching line (e.g. “PARSE_OUTPUT:<groupname>:<typename>”).

  • lines (list[str]) – The cab’s combined stdout/stderr lines, in order.

Returns:

A dict of extracted output field name to typed value, built by applying each matched line’s actions in order.

Return type:

dict[str, Any]

shinobi.wranglers.parse_output_action(action)[source]

Parse one wrangler action string: (groupname, typename) for a PARSE_OUTPUT:<groupname>:<typename> action, or None for any other (unimplemented) action kind. The one place both apply_wranglers (which populates the field at run time, below) and graph._wrangler_output_fields (which needs the field name statically, for check_offloadable, without running anything) agree on this grammar – so the two can’t silently drift on what a wrangler action means the way they briefly did (one required exactly 3 colon-separated parts, the other tolerated 2+).

Parameters:

action (str)

Return type:

tuple[str, str] | None

Short-circuit semantics for an unrolled loop (see Recipe.add_loop).

A loop is declared, not interpreted: add_loop flattens its body into the recipe max_iter times and chains the copies with real wiring, so the graph is fully inspectable before anything runs. What stays a run-time decision is narrow – whether an already-declared step does any work. This module is that decision, and nothing else.

The rule lives here rather than in the scheduler because two tiers evaluate it: _run_recipe calls should_skip in-process, and the Slurm offload compiler emits the same predicate as a shell [ -f … ] test at the top of each iteration’s script. A convergence signal that is a path is what makes that possible – a bool would have no way to cross a node boundary, and the two tiers would need separate definitions that could drift into running a different number of cycles for the same recipe.

shinobi.steps.loops.sentinel_exists(value)[source]

Whether a convergence sentinel has actually been produced.

Parameters:

value (Any) – The sentinel output’s value – a path, or None if the producing step never set it.

Returns:

True if value names an existing file or directory.

Return type:

bool

shinobi.steps.loops.sentinel_value(ref, results)[source]

The sentinel value this step’s skip decision reads, or None if it cannot skip (the first iteration, or a step outside any loop).

Parameters:

  • ref (StepRef) – The step about to be scheduled.

  • results (dict[str, StepResult]) – Completed steps by name. The sentinel producer is guaranteed to be present: add_loop gives every iteration an edge to it, so the scheduler cannot reach this step first.

Returns:

The previous iteration’s sentinel output value, or None.

Return type:

Any

shinobi.steps.loops.should_skip(ref, results)[source]

Whether ref should pass its predecessor’s outputs through instead of running, because an earlier iteration already converged.

Parameters:
Returns:

True if the previous iteration’s sentinel exists on disk.

Return type:

bool

shinobi.steps.loops.passthrough_result(ref, prev, inputs)[source]

The result of a step that was skipped: the same body step’s outputs from one iteration earlier, handed on unchanged.

Passing the previous outputs object through (rather than re-deriving field by field) is what makes convergence sticky without extra bookkeeping: the sentinel is itself one of those outputs, so every later iteration sees it and skips in turn.

cache_key/output_keys are carried over too. A skipped step produced no new data, so contributing nothing would drop the upstream term from every downstream cache key (see shinobi.cache) and needlessly invalidate work that is genuinely unchanged.

kind deliberately keeps the scope’s real kind – skipped is a separate flag. shinobi.provenance.apply_manifest_pins asserts that a record’s kind still matches the scope’s type, so inventing a “skipped” kind would make any early-converging run unreplayable.

Parameters:

  • ref (StepRef) – The skipped step.

  • prev (StepResult) – The corresponding step’s result from the previous iteration.

  • inputs (Any) – The validated inputs this step would have run with.

Returns:

A successful StepResult marked skipped.

Return type:

StepResult

Graphs and resources

The shared dependency graph of a Recipe’s declared steps.

A Recipe is a declared DAG (see AGENTS.md): its steps list plus the OutputRef wiring between them is the graph. build_graph reads that graph once – validating every wiring reference and rejecting cycles – and returns a RecipeGraph that BOTH consumers depend on:

  • the executor (shinobi.steps.dispatch._run_recipe) schedules a topological wavefront over these true dependency edges;

  • the –dryrun renderer (shinobi.dag.graph_nodes) builds its display view on top of the same edges (re-adding a display-only sequential chain between otherwise-independent steps).

Because both go through build_graph, they can never disagree on what depends on what, or on whether the graph is even valid. Validation runs at run time and at dryrun time – not at build time, since a Recipe is deliberately mutable through its builder methods and a forward OutputRef is legitimate mid-construction.

exception shinobi.graph.RecipeGraphError[source]

Bases: ValueError

A recipe’s declared graph is invalid: a wiring reference points at a field/step that doesn’t exist, a step name is duplicated, or the dependency edges form a cycle.

exception shinobi.graph.RecipeNotOffloadableError[source]

Bases: ValueError

A recipe is a valid graph but cannot be compiled to an external workflow engine and detached: it relies on shinobi’s live, in-process execution semantics. See check_offloadable for the exact rules and AGENTS.md’s “DAG offload” section for why. Such a recipe should run locally (the default ctx.run() path) instead – it is never silently degraded.

class shinobi.graph.RecipeGraph(names, deps, dependents)[source]

Bases: object

The validated dependency graph. Node ids are indices into the recipe’s steps list (so id == declaration order).

deps[i] are the ids step i truly depends on (via OutputRef wiring) – no artificial sequential chaining; independent steps have empty deps. dependents is the reverse edge set, precomputed for the scheduler’s in-degree bookkeeping.

Parameters:
names: list[str]
deps: list[set[int]]
dependents: list[set[int]]
topological_indices()[source]

A stable topological order, preferring declaration order.

build_graph has already established acyclicity. The heap makes a newly-ready lower declaration index outrank an unrelated later node, matching the local scheduler’s max_workers=1 order rather than batching whole topological generations.

Return type:

list[int]

shinobi.graph.build_graph(recipe)[source]

Validate a Recipe’s wiring and build its dependency graph.

Raises RecipeGraphError on: a duplicate step name; an InputRef to a field not on the recipe’s inputs_model; an OutputRef (in a step’s wiring or in output_wiring) to an unknown step or to a field that is not an output of that step; a wiring field that is not an input of the consuming step; a recipe output field that is not in the recipe’s outputs model; a dependency cycle; or non-Cab derived reads anywhere in the declared recipe tree.

Parameters:

recipe (Recipe)

Return type:

RecipeGraph

shinobi.graph.check_offloadable(recipe, *, worker=False, allow_dataset_planning=False)[source]

Raise RecipeNotOffloadableError (with all disqualifying reasons) unless recipe is a purely declarative DAG that can be compiled to an external engine and detached. A valid graph is a precondition, so this calls build_graph first (which may raise RecipeGraphError) and returns it, so a caller that needs the graph right after checking eligibility (e.g. offload.slurm.compile_slurm) doesn’t have to call build_graph a second time.

worker=True checks the experimental worker-bundle graph shape: explicitly identified pysteps and venvs are admitted, and structured outputs can cross nodes in result records. The bundle compiler must additionally validate the serializable schema, code and environment. It does not enable worker submission in the legacy Slurm compiler.

allow_dataset_planning=True is reserved for a compiler that resolves and records dataset hazards but marks the resulting workflow as non-submittable. It never enables dataset execution.

The default rules follow directly from “the cluster runs the graph, shinobi is not in the loop per step” (see AGENTS.md / the design note):

  • No orchestration function on any step – arbitrary Python run against live upstream outputs can’t be statically compiled, and shipping it would be a code-execution hazard.

  • Every step is a `Cab` – a bare Scope/nested Recipe step has no single argv to emit.

  • A MUTABLE input must be a path on a Cab. MUTABLE means pass-by-reference, which for a live Python object is a single-heap concept with no meaning across nodes – but a Cab’s inputs are paths, and a shared filesystem is exactly that reference. The hazard in-place mutation actually poses is ordering, not representation: two steps rewriting the same MS get no graph edge unless one wires from the other’s output, so a cluster would run them concurrently. That is resolved where the real path values are known, by offload.slurm.MutationOrder, which emits the missing edges – so this check no longer has to refuse the whole class. (A MUTABLE input that is not path-typed still has no cross-node meaning and is refused; so is any MUTABLE input on a non-Cab step.)

  • Only `binary`-flavour cabs – a code-carrying flavour has no argv to compile, and build_argv would refuse it later anyway. Reported here so an unofloadable recipe says so up front rather than failing mid-compile.

  • Every inter-step `OutputRef` (one step’s input wired from another step’s output) carries a path output that is not wrangler-derived – across nodes, data flows only as shared-filesystem paths, and a wrangler output is never populated when shinobi doesn’t see the step’s stdout. output_wiring (the recipe’s own external outputs) is deliberately not checked: it is a reporting boundary, not inter-node data flow – under detach the caller gets a handle, and a downstream ninja status reconstructs what it can (paths yes, dynamic values best-effort).

Parameters:
Return type:

RecipeGraph

Declared resource footprints, the machine’s real budget, and the admission control that keeps the two consistent.

_run_recipe (steps/dispatch.py) schedules by counting slots: a step is “running” or “not running”, and max_workers bounds how many run at once. That is fine when steps are cheap and interchangeable. It is not fine for radio-astronomy pipelines, where a single wsclean/DDFacet/QuartiCal step is routinely sized to use most of one machine on its own – five of those running concurrently because five slots were free will oversubscribe the CPU and blow through a memory cap, and the step that gets OOM-killed takes its innocent siblings’ shared memory down with it.

So a Scope may declare a Resources footprint, and the scheduler admits work against a Budget instead of a slot count. Three properties are worth stating up front, because they are what make this honest rather than decorative:

An undeclared step is free. Nothing declares a footprint by default, so a recipe that declares nothing is admitted exactly as before. A declaration is only ever as good as its coverage – this models what you tell it, and tells you nothing you did not.

The budget comes from the cgroup, not from `/proc/meminfo`. This is the whole point. A tool that sizes itself from /proc/meminfo sees the host’s memory, not the slice quota it is actually confined to, and cannot self-limit even if it wants to – DDFacet on a 251.7 GiB-capped user slice of a much larger box is the motivating case. detect_budget walks the full cgroup ancestor chain, because the cap that matters is almost never on the leaf cgroup a process is in (see its docstring).

The declaration is not enforcement. Locally, shinobi decides whether to start a step; it cannot hold a running process to its word. Container and cluster backends do enforce it (–memory/–cpus, #SBATCH –mem), which is a real difference – a containerised runaway dies in its own cgroup instead of eating the shared slice. A native or venv step has no such backstop at all. Enforcement is also per dimension: a rootless runtime can only apply the controllers systemd delegated to the session, so see delegated_controllers for what is actually enforceable here, and note that it is a strictly weaker question than what the budget above says you may use.

shinobi.resources.parse_size(value)[source]

Parse a byte size: a bare number, or a number with a unit suffix.

Parameters:

value (float | str) – An int/float count of bytes, or a string like “200GB”, “250 GiB”, “2T”, “1024”.

Returns:

The size in bytes.

Raises:

ValueError – If the string is not a number with an optional known unit suffix, or the size is negative.

Return type:

int

shinobi.resources.format_size(size)[source]

Render a byte count for humans, in binary units.

For log lines and error messages only – every backend wants its own grammar (docker takes raw bytes, Slurm megabytes), so this is deliberately not the shared emitter for those.

Parameters:

size (int) – Size in bytes.

Returns:

A string like “251.7GiB”.

Return type:

str

class shinobi.resources.Resources(*, cpus=None, memory=None)[source]

Bases: BaseModel

A declared resource footprint: what one step needs to run.

Both fields default to None, meaning “undeclared” – which the scheduler reads as free, not as unknown-so-assume-the-worst. Assuming the worst would serialise every undeclared recipe, i.e. punish everyone who has not opted in; free keeps the feature genuinely additive at the cost of being only as complete as your declarations.

cpus also takes “all”, for the very common tool that is simply told to use every core it can see (wsclean’s default, -j 0, nproc). That step is the opposite of free and must not read as undeclared, but the number it stands for is not knowable where recipes are built: a recipe is a static, machine-independent declaration, and probing the builder’s core count would bake one host’s answer into a job that may run on another (and, for a caller whose builder is not allowed to touch the filesystem at all, is not available even in principle). So it stays symbolic until Budget resolves it against the live pool, on the host that is actually admitting the work. See resolved_against.

Shaped so a third dimension (gpus) can be added later without a schema break – but not added speculatively.

Parameters:

  • cpus (Literal['all'] | float | None)

  • memory (int | None)

cpus: Literal['all'] | float | None
memory: int | None
is_empty()[source]

True when nothing is declared, i.e. this step costs the scheduler nothing and is admitted on slot count alone.

Return type:

bool

property enforceable_cpus: float | None

The core count a backend can cap this step at, or None.

None for “all” as well as for undeclared: a runtime limit of “the whole machine” is not a limit, and no backend should invent a number for it – a container flag would be a no-op at best and the building host’s core count at worst, while a batch script would be asking a scheduler for a node size it cannot know. “all” is a claim on the local pool, and the local scheduler is where it is honoured.

resolved_against(total)[source]

This footprint with cpus=”all” replaced by total’s own count.

Resolution has to happen against the very total that admission is decided against, or the two disagree: resolving to anything larger would trip _exceeds_total and be waved through unconstrained – turning the strongest declaration into the weakest, which is the opposite of what it says.

An unconstrained dimension resolves to None (undeclared): if the pool has no CPU total then nothing can exhaust it, so there is nothing for this step to be held back by.

Parameters:

total (Resources) – The pool’s resolved totals.

Returns:

self unchanged unless cpus is “all”.

Return type:

Resources

describe()[source]

A short human rendering for log lines (“cpus=4, memory=200.0GiB”).

Return type:

str

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.resources.ResourceBudget(*, cpus='auto', memory='auto')[source]

Bases: BaseModel

Config-side total: how much of the machine a recipe may occupy.

“auto” (the default) detects the real limit, cgroup-aware; “unbounded” disables that dimension; anything else is an explicit value. Note None is deliberately not the “unbounded” spelling – everywhere else in shinobi.config None means “unset / fall back / off”, and quietly inverting that here would be a trap.

Parameters:

  • cpus (Literal['auto', 'unbounded'] | float | int)

  • memory (Literal['auto', 'unbounded'] | int | str)

cpus: Literal['auto', 'unbounded'] | float | int
memory: Literal['auto', 'unbounded'] | int | str
resolve(root='/')[source]

Resolve to concrete totals, detecting only if something asks for it.

Parameters:

root (str | Path) – Filesystem root to detect against. A test seam; “/” in real use.

Returns:

(totals, source) – the resolved budget and a short description of where each value came from, for the one log line the scheduler emits. Whether the number came from the cgroup or from /proc/meminfo is exactly the thing worth being able to check.

Return type:

tuple[Resources, str]

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

shinobi.resources.delegated_controllers(root='/')[source]

Which cgroup v2 controllers this user session may actually apply limits with – i.e. what a rootless container runtime can enforce here.

Detection is the other half of enforcement. detect_budget asks “how much of this machine am I allowed to use”; this asks “which of those dimensions can I hold a child process to”. They are different questions, and on a shared box the answer to the second is routinely some of them: systemd delegates only what Delegate= names, so a session can easily have memory pids and no cpu. A runtime handed –cpus there cannot create cpu.max in its new scope and the container never starts – taking the memory limit, which was perfectly enforceable, down with it.

What is read, and why that one file. A rootless runtime does not put its scope next to the calling process – it asks the systemd user manager to create one, so the scope lands under user@<uid>.service and inherits whatever that cgroup makes available. That is the delegation boundary, and its cgroup.controllers is the honest answer.

The obvious cheaper proxy – this process’s own cgroup – is measurably wrong: on the machine this was developed on, user@1000.service has cpu memory pids and apptainer applies –cpus fine, while the calling shell sits several levels down under app.slice with only memory pids, because that intermediate slice never enabled cpu in its subtree_control. Reading the caller’s cgroup would have dropped a CPU limit that works – silently losing enforcement, which is the one error direction worth engineering against here. Over-estimating merely fails loudly, exactly as it did before this probe existed.

Parameters:

root (str) – Filesystem root to probe against. A test seam; “/” in real use. Cached per root – delegation does not change within a run.

Return type:

frozenset[str] | None

The caller is not always inside the manager it will ask. An ssh login lands in /user.slice/user-<uid>.slice/session-<n>.scope – a sibling of user@<uid>.service, not a descendant – so the path walk below finds no manager even though one is running and its delegation is sitting there to be read. Returning “couldn’t tell” for that shape means emitting –cpus on a host known not to delegate cpu, which is the abort this probe exists to prevent, and ssh is how batch runs are started. So a session that names its user slice is resolved through _user_manager_beside.

Parameters:

root (str) – Filesystem root to probe against. A test seam; “/” in real use. Cached per root – delegation does not change within a run.

Returns:

The delegated controller names, or None when this cannot be determined – no cgroup v2, no unified entry in /proc/self/cgroup, or neither a user manager in this process’s cgroup path nor one beside it (a container, or a session systemd never took charge of). Callers must read None as “assume everything works” and keep failing loudly, never as “nothing works”.

Return type:

frozenset[str] | None

shinobi.resources.detect_budget(root='/')[source]

Detect how much CPU and memory this process may actually use.

Tried in order, most authoritative first: cgroup v2, cgroup v1, then the host’s own totals. The cgroup tiers walk the full ancestor chain (see _walk_limits) and take the tightest limit at any level.

Parameters:

root (str | Path) – Filesystem root to detect against. A test seam; “/” in real use.

Returns:

(budget, source), where an unset field means “nothing constrains this dimension”. source names the tier that answered, so the scheduler’s log line can say whether the number is a real cgroup quota or just the host’s size.

Return type:

tuple[Resources, str]

class shinobi.resources.Budget(total)[source]

Bases: object

Admission control over a shared pool of CPU and memory.

One Budget is created per top-level run and shared by every nested recipe’s scheduler (threaded down like AppConfig). That sharing is the point: caracal-shaped pipelines nest each worker as its own Recipe, so a budget scoped to one _run_recipe invocation would leave every branch free to independently decide it owns the machine – exactly the failure this is meant to stop.

Not a pydantic model: this is live mutable state guarded by one threading.Condition. Every method below takes that lock; nothing reads the totals outside it.

## The wake protocol

A scheduler that is refused must park until something changes, but it cannot park while holding the lock inside try_acquire (it still has futures to reap). So refusal and parking are two separate lock acquisitions – and a release landing in the window between them would be a lost wakeup and a permanent hang. _generation, bumped under the lock by every release, closes that window: try_acquire hands back the generation it decided at, and wait_for_change returns immediately if anything has happened since. This is reachable in the ordinary case (one recipe is refused at the instant a sibling finishes), not a theoretical race.

## Fairness

Admission is strictly FIFO by ticket. Within one recipe, draining in declaration order already prevents starvation; across sibling recipes sharing this budget it does not – four branches running a steady stream of small steps can hold the reserved total above zero indefinitely, and a branch needing 200 of 250 GiB would never be admitted. A refused scheduler takes a ticket, and no later arrival is admitted ahead of it, so current holders drain and the large step runs. This costs some utilisation, and is the same trade “block in declaration order” already makes, extended across recipes.

## Backfill

Blocking on the head costs more than “some” utilisation when the head is large: a 200 GiB step at the front of the queue idles a machine with room for the 2-core steps behind it. try_backfill recovers that without giving the guarantee up, by admitting a trailing unit only when the parked head would still fit alongside it – and alongside every earlier backfill still running – once the work reserved now drains. The head’s start time is therefore unchanged: it waits for exactly the work it was already waiting for, and nothing new.

That test has to be cumulative. Bounding each grant against the total on its own admits two 30 GiB units in a 100 GiB budget behind a 70 GiB head, and the head then cannot start when the original holder finishes. See _leaves_room_for.

## Why this cannot deadlock

  • A reservation is only ever held by a step that is running; a parked scheduler holds a ticket, which reserves nothing.

  • A step holding a reservation never re-enters _run_recipe (the scheduler skips demand for nested-Recipe steps), so no reservation can be waiting on itself.

  • Every running step terminates, and _run_recipe releases its reservation on every exit path including exceptions.

Given those, some reservation always eventually releases and bumps the generation, and FIFO guarantees the oldest waiter is the one served. A demand larger than the whole budget never waits at all – it is granted immediately, since no amount of waiting could ever make it fit.

Parameters:

total (Resources)

property reserved: Resources

What is currently reserved. For tests and diagnostics.

try_acquire(demand, label='')[source]

Reserve demand if it is admissible right now. Never blocks.

Parameters:

  • demand (Resources) – What the step declared. An empty Resources always succeeds and reserves nothing.

  • label (str) – Step name, used only in the over-budget warning.

Returns:

(granted, generation). On refusal the caller should reap any in-flight work and, if it has nothing else to do, park in wait_for_change(generation) – passing back the generation returned here is what makes the wait race-free.

Return type:

tuple[bool, int]

can_backfill(demand, behind)[source]

Whether try_backfill would grant demand right now, without reserving anything.

For deciding whether a candidate is worth the work of preparing – the scheduler resolves a step’s wiring before it has a unit to admit, and that is wasted (and surfaces errors early) for a step that could not be admitted anyway. Advisory by nature: capacity can change before the real try_backfill, which simply refuses then.

Parameters:
Return type:

bool

try_backfill(demand, behind, label='')[source]

Reserve demand only if doing so cannot delay a unit needing behind. Never blocks, never queues.

Admission blocks on the head of the queue so a large step is never starved by a stream of small ones – but a head that does not fit also stops everything behind it, idling a machine that has room for work the head will not use anyway. This is the escape hatch, and it is deliberately not try_acquire with a flag:

  • try_acquire refuses on FIFO fairness as well as capacity, and a backfill must keep honouring that (jumping a sibling recipe’s older ticket is exactly the cross-recipe starvation the tickets prevent);

  • but a try_acquire grant pops the caller’s ticket, and the caller here is the very scheduler thread whose head is parked. It would give up the queue position it is holding on the head’s behalf.

So this grants without ever touching _waiters: no ticket is taken, and none is dropped.

Parameters:

  • demand (Resources) – What the candidate unit reserves while it runs.

  • behind (Resources) – What the parked head unit needs. The grant must leave room for it – see _leaves_room_for.

  • label (str) – Step name, for logging.

Returns:

Whether demand was reserved. On True the caller must later release(demand, backfilled=True).

Return type:

bool

Note

Only correct when called after this thread’s own try_acquire refusal in the same pass: that refusal is what registers the ticket which makes the fairness check below read as “we are the oldest waiter” rather than “someone is ahead of us”.

release(demand, *, backfilled=False)[source]

Return demand to the budget and wake every parked scheduler.

Parameters:

  • demand (Resources) – Exactly what try_acquire or try_backfill was granted for this unit – passed as declared, “all” and all; it is resolved here the same way, against the same total, so what is returned is exactly what was reserved.

  • backfilled (bool) – Whether this reservation came from try_backfill. Must match how it was granted – the outstanding-backfill total it feeds is what keeps the cumulative admission test honest, and a missed decrement makes backfill quietly refuse everything for the rest of the run.

Return type:

None

wait_for_change(since, timeout=None)[source]

Park until the budget changes.

Parameters:

  • since (int) – The generation try_acquire refused at. If anything has been released since then this returns immediately, which is what makes refusal-then-park race-free.

  • timeout (float | None) – Optional cap on the wait, in seconds.

Return type:

None

abandon()[source]

Give up this thread’s place in the queue.

A scheduler that stops waiting – because it failed, or finished – must not leave a ticket behind: strict FIFO means a stale ticket from a dead scheduler would block every later arrival forever. Called from _run_recipe’s exit path.

Return type:

None

Backends

Backend abstraction: a backend takes a cab and a resolved argv and runs it somewhere – natively, in a container, on Slurm, …

A backend knows nothing about recipes or output schemas beyond the argv it’s handed and the cab’s image/command metadata; it only knows how to execute and how to capture output. Wrangling that output into structured results is the dispatch layer’s job, so a backend returns a raw BackendRun (returncode/stdout/stderr), nothing schema-aware.

class shinobi.backends.Backend[source]

Bases: ABC

Abstract base class for execution backends.

Variables:

name (str) – Registry key used to look up this backend (e.g. "native", "container", "slurm").

name: str
abstractmethod run(cab, argv, inputs, *, label='', stream=True, pin=False, cwd=None, dataset_plan=None)[source]

Execute argv (as built by shinobi.policies.build_argv) and return a BackendRun. Must not raise on a non-zero exit – that’s reported via BackendRun.returncode / BackendRun.success.

inputs is the prepared inputs dict argv was built from (the one _prepare_inputs produces, so MUTABLE fields are the caller’s own objects by reference). Most backends ignore it, but container backends need it to know which File/MS-valued params to bind-mount.

label/stream control live stdout/stderr echo (see shinobi.backends._stream.run_streaming) – only native and container act on them today; slurm accepts and ignores both (it reads output once after the job finishes).

cwd is the working directory to run in (shinobi.sandbox passes the step’s sandbox here), defaulting to the process cwd. slurm accepts and ignores it too: it runs in a remote cwd the dispatch layer can’t scope, so a sandboxed Slurm step degrades gracefully to an unsandboxed run (harvest finds the outputs already in the workspace and moves nothing).

dataset_plan is a closed, data-only strict-dataset namespace plan produced after claim acquisition. Local-process backends need no mounts and ignore it; container adapters consume its identity binds. Scheduler backends currently refuse strict execution before run.

Parameters:
Return type:

BackendRun

shinobi.backends.register(backend_cls)[source]

Register a backend class under its name attribute.

Intended for use as a class decorator on Backend subclasses.

Parameters:

backend_cls (type[Backend]) – The backend class to register.

Returns:

The same class, unmodified, so it can be used as a decorator.

Return type:

type[Backend]

shinobi.backends.get_backend(name, **opts)[source]

Instantiate a registered backend by name.

Parameters:

  • name (str) – Registry key of the backend (e.g. "native", "slurm").

  • **opts – Keyword arguments forwarded to the backend’s constructor.

Returns:

A new instance of the requested backend.

Raises:

ValueError – If no backend is registered under name.

Return type:

Backend

shinobi.backends.registered_backend_names()[source]

Every name a backend is registered under, sorted.

The authoritative answer to “is this a real backend?”, for callers that need to check a name rather than instantiate it – the CLI validating –backend before a run starts (shinobi.cli). Not every consumer of a backend name reaches get_backend: the pystep adapter matches the resolved name against CONTAINER_RUNTIMES and otherwise runs the function in-process, so a typo there would go unnoticed rather than raise.

Returns:

The registry keys, sorted.

Return type:

list[str]

shinobi.backends.registered_backend_classes()[source]

Return all backend classes currently registered.

Returns:

A list of registered Backend subclasses.

Return type:

list[type[Backend]]

Building argv

Turn a cab’s schema + resolved parameter values into a command line.

Operates on the step-model Cab: the parameter values come from an already-validated inputs_model instance (or the prepared dict dispatch builds from it), while per-field naming/implicit metadata comes from the cab’s field_meta, dynamically-named params from input_patterns, and arg formatting from policies.

shinobi.policies.build_argv(cab, resolved)[source]

Build a full argv (starting with the cab’s command) from a resolved parameter dict, according to the cab’s policies and field metadata.

Rejects any non-“binary” flavour before building argv – so a non-executable command can never reach subprocess as argv[0] (see SECURITY.md).

Parameters:
Return type:

list[str]

Loaders

Load YAML cab definitions in the scabha dialect into shinobi Cab objects.

Lineage. shinobi’s cab schema uses the scabha dialect, also used by Stimela 2.0. It supports the same vocabulary: inputs/outputs with dtype/required/default/info/choices, policies, management.wranglers, image, flavour, command. The loader translates static cab definitions into shinobi Cab objects; recipes are declared separately in Python (see stimela-ninja’s AGENTS.md).

cult-cargo is the largest published library of cabs written in this dialect and is what this loader is usually pointed at, but the dialect is scabha’s and nothing here is specific to that project. shinobi.loaders.worker_schema reads a scabha-derived config dialect through the same shared helpers.

shinobi-native keys. shinobi’s own Cab carries a few things scabha has no vocabulary for, so the dialect accepts them as an extension rather than inventing a second format for cabs authored against shinobi directly. A document using none of them is a plain scabha document, and cult-cargo’s own files remain a readable subset.

Per field, alongside the scabha keys: write_path: true marks a string-typed input naming a filesystem path the tool writes to – a stem products are built from, or a complete path written directly (see ParamMeta.write_path) – and mutable: true marks an input the step may change in place (Mutability.MUTABLE). Both are registered in _LEAF_SPEC_KEYS, which matters more than it looks: _is_section tells a leaf param from a nested CLI section by whether the mapping has any known param-spec key, so a spec carrying only an unregistered key is read as a section and the field disappears without a word.

Per cab: cache, sandbox, harvest, scratch and dataset_accesses, which mirror the Scope fields of the same names. Access mappings are validated by DatasetAccess; field references use literal sanitized model names (data.ms becomes data_ms). Strict MSv2/CasaTab dtypes retain dataset metadata; legacy MS remains Path. Direct List[MSv2] inputs support reads and explicit joint WRITE declarations, with compatible same-name ordered passthrough outputs. Other strict containers, mixed unions, choices and dynamic patterns are refused for executable cabs, and MSv4 is reserved. Current dispatch supports the bounded MSv2 lifecycle only; CasaTab remains available for declaration and inspection.

image: may also name a key rather than a reference, resolved through the caller-supplied images mapping (see loads) – the same shape as package_roots, and for the same reason: shinobi has no manifest and does not go looking for one. It lets a document say image: WSCLEAN and leave which reference that is to the deployment that loads it.

Also per cab: input_patterns/output_patterns, families of dynamically-named params (ParamPattern). A pattern is a separator plus ordered segments; each segment is either a regex (a level that cannot be enumerated ahead of time) or attrs (the known level, each attr a param spec in its own right). ParamPattern’s own validator enforces the real rule – exactly one segment carries attrs – so the loader only produces the shape and lets it object, naming the cab and key when it does.

An attr spec is read by the same _param_meta as a declared field, with one asymmetry: an attr keeps its dtype, a field does not. A declared field’s dtype is already its model annotation, and repeating it on field_meta would make every field of every cab differ from its Python-authored equivalent; a pattern attr has no model field at all, which is the reason ParamMeta.dtype exists.

Supported scope. This reads static cab definitions and resolves the composition mechanisms below. Expression evaluation and executable schema generation are outside its supported scope; see SECURITY.md for the execution boundary.

Composition mechanisms for sharing and combining scabha cab definitions:

  • _include: [file, ...] – merges other YAML files in (relative to the including file), most often to pull in a shared vars:/lib: namespace. Merging is a plain deep-merge; the including file’s own keys win over included ones. Resolved wherever it appears in the document (top level, or nested under inputs:/outputs:, as real cult-cargo’s cubical.yml/quartical.yml do) via the same tree-walking resolve_directive helper _use already relies on.

  • _use: dotted.path – deep-merges a dict looked up by dotted path in the fully-merged document (post-_include) into the dict it appears in, with that dict’s own sibling keys taking precedence. Used both for small things (image: {_use: vars.cult-cargo.images, name: breizorro}) and to inherit a cab’s entire command/flavour block.

  • The package-scoped include form (_include: (pkg.dotted.path)file.yaml or _include: [{(pkg.dotted.path): [file, ...]}]) – searches an installed package’s data directory rather than a relative path. Resolving a dotted package name to a filesystem directory would normally mean importing the package (importlib), but that risks executing arbitrary code from any __init__.py on the path – shinobi never imports a cab package for any reason (see SECURITY.md’s “never eval()/exec() a cab’s command” boundary, which this extends to “never import a cab package”). Instead, callers pass package_roots={"cultcargo": Path(...)} to load_file()/loads(): an explicit, caller-supplied mapping from a dotted package prefix to its filesystem directory. A dotted name is resolved against the longest registered prefix, descending the remainder as subdirectories (cultcargo.genesis.cubical against {"cultcargo": Path("/.../cultcargo")} -> Path("/.../cultcargo/genesis/cubical")) – the normal package/subpackage-is-a-subdirectory convention, without ever asking Python’s import machinery to confirm it. A package-scoped _include naming a package with no registered root raises a clear CabLoadError.

The following features are not evaluated by the loader (see SECURITY.md):

  • Expressions and substitutions. The =config.x.y / =recipe.ms / ${...} / =IFSET(...) language scabha values can contain. Left as literal strings, so a cab carrying one loads with that value verbatim rather than resolved – visible in the built Cab, not silently dropped. ParamMeta.implicit is the one templating shinobi does resolve, and it is plain str.format against the step’s own validated inputs: no name resolution across steps, no function calls, no conditionals.

  • Conditionals and control flow. Anything whose value depends on evaluating a predicate at load or run time. A cab is a parameter table; branching over it belongs in the Python that calls the step, where it is visible to the reader and to the DAG.

  • Aliases and propagation. Recipe-to-step parameter propagation is not implemented. shinobi wires steps with typed InputRef/OutputRef objects to declare parameter sources and data dependencies explicitly.

  • dynamic_schema: dotted.path – a reference to a Python function that would need importing and calling to get a cab’s real schema (real cult-cargo’s wsclean.yml/cubical.yml/quartical.yml use this). Resolving it for real is not just a parsing gap like the above: it means executing arbitrary code named by a cab file at load time. Not implemented, and not worked around here either: a cab using dynamic_schema always loads with a warning and whatever static inputs:/outputs: are present – silently incomplete unless you notice the warning. The hand-authored, cross-checked static schemas for the three real cabs that need this (wsclean, cubical, quartical) live in dosho (the native shinobi cab repository, a sibling project) instead of as a stopgap table in this loader – this loader used to carry one (a small per-cab ParamPattern table read from each cab’s own static data files, e.g. cubical’s schema_JONES_TEMPLATE.yaml), removed once dosho’s real ports superseded it. See dosho/cabs/wsclean.py/ cubical.py/quartical.py for that knowledge now, and prefer porting a cab there over reintroducing a table here.

The loader reads static schemas. It does not evaluate recipe-level expressions or execute dynamic_schema.

shinobi.loaders.yaml_cab.load_file(path, *, package_roots=None, images=None)[source]

Load a YAML cab definition file into Cab instances.

Parameters:

  • path (str | Path) – Path to the YAML cab definition file.

  • package_roots (dict[str, Path] | None) – Mapping of package name to filesystem root, used to resolve _include directives that reference other packages.

  • images (dict[str, str] | None) – Mapping of image key to full reference. See loads.

Returns:

A dict mapping cab name to its built Cab instance.

Raises:

CabLoadError – If composition cannot be resolved, including an _include or _use cycle. Cycle errors name the closed path.

Return type:

dict[str, Cab]

shinobi.loaders.yaml_cab.loads(text, *, package_roots=None, images=None)[source]

Parse cab defs from a YAML string. Supports _use (resolved against the document itself) and package-scoped _include (resolved against package_roots), but not a plain relative-path _include, since there’s no base directory to resolve a relative file path against.

images maps an image key to its full reference, for a document that names images symbolically (image: WSCLEAN) rather than by a baked-in reference. Caller-supplied for the same reason package_roots is: shinobi has no manifest of its own and will not go looking for one. A cab repository passes its own – dosho’s images.yaml is exactly this – so a deployment’s overrides still decide the reference at load time instead of it being fixed when the document was written.

An image string absent from the mapping is left alone, because a literal reference is the older and still-valid form (cult-cargo’s files carry quay.io/stimela2/... directly). A key that is simply misspelled therefore reaches the runtime as an image name and fails there – loudly, at pull time, which is the safe direction: the alternative rejects every legitimate bare name (ubuntu) to catch a typo.

Raises:

CabLoadError – If composition cannot be resolved, including an _include or _use cycle. Cycle errors name the closed path.

Parameters:
Return type:

dict[str, Cab]

Load scabha-dialect worker/config schema YAML (as used by caracal2’s caracal/schemas/*_schema.yaml) into a plain pydantic model – without depending on scabha itself.

This is a config schema, not a cab schema: there’s no command, policies, or image here, just nested inputs:/outputs: parameter groups describing what a pipeline worker accepts in its config file. See shinobi.loaders.yaml_cab for the sibling loader that builds executable Cab`s from scabha-dialect YAML – this module deliberately does not reuse `Scope/Cab for the result, since a worker config is never dispatched as a step.

Dialect, as actually used by caracal2 (see its caracal/schemas/):

  • A param node is a dict. If it has a dtype key, it’s a leaf parameter. Otherwise it’s a group whose values are themselves leaves/groups, nested arbitrarily deep (e.g. crosscal’s rewind_flags.mode) – one rule, no special-casing per file.

  • dtypes are str/int/float/bool/File and List[<inner>] (bracket syntax; see _modelgen.dtype_to_type).

  • choices (a list) maps to typing.Literal.

  • implicit is a template/expression string (“{current.x}-y.json” or “=IFSET(…)”) – left as a raw, unevaluated string, matching loaders.yaml_cab’s policy on cult-cargo’s own expression language. A field with implicit set is never required from the caller, same rule as loaders.yaml_cab._collect.

  • _include: “(module.path)filename.yaml” – a single package-scoped string (different from cult-cargo’s list-of-plain-paths form), or a plain relative-path string, or a list of either. Resolved recursively (an included file’s own _include resolves relative to its directory). The package-scoped form is resolved against a caller-supplied package_roots={“module.path”: Path(…)} mapping, never by importing the named module – importing it would execute an arbitrary __init__.py named by a config file. Same mechanism, and the same resolve_package_root helper, as loaders.yaml_cab; see SECURITY.md’s “never import a cab package”.

  • _use: dotted.path or _use: [dotted.path, …] – deep-merges one or more dotted lookups (against the fully _include-resolved document) into the dict it appears in, with that dict’s own sibling keys winning – same convention as loaders.yaml_cab, extended to accept a list.

  • _each: {<sub-schema>} – a group whose keys are supplied by the config, each value validated against one declared sub-schema, giving dict[str, <SubModel>]. Every other group declares its keys in the schema; this is for the case where the names are the user’s own data (caracal2’s calibration chains: chains: {primary: …, secondary: …, <whatever the user calls the next one>: …}). _key_pattern (a regex, optional) constrains those names – schemas whose keys become step names or output-path components want ‘^[A-Za-z][A-Za-z0-9_]*$’. A default: alongside _each is a mapping of pre-declared entries, validated at load time and rebuilt per instance. _key_pattern, info and default are the only keys an _each group may carry – anything else raises (a misspelt _key_pattern must not silently switch off the validation).

writable (seen in caracal2’s caracal_base.yaml) is carried onto the generated field’s json_schema_extra: a writable: false directory input is bind-mounted read-only by the container backend (see _leaf_field and backends.container.bind_dir_modes). must_exist/path_policies are still dropped (path-behaviour hints with no consumer yet).

class shinobi.loaders.worker_schema.ConfigSchema(*, name, info=None, inputs_model, outputs_model)[source]

Bases: BaseModel

A loaded worker/config schema: just enough to validate and introspect a config section – name, human info, and the pydantic models for its inputs/outputs.

Parameters:
name: str
info: str | None
inputs_model: type[BaseModel]
outputs_model: type[BaseModel]
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

shinobi.loaders.worker_schema.load_worker_schema(path, *, package_roots=None)[source]

Load a stimela-classic worker schema (YAML) file into a ConfigSchema.

Parameters:

  • path (str | Path) – Path to the YAML worker schema file.

  • package_roots (dict[str, Path] | None) – Mapping of dotted package name to filesystem root, used to resolve package-scoped _include: (module.path)file directives. A package-scoped include naming a package with no registered root raises rather than being resolved by importing it – see resolve_package_root.

Returns:

The built ConfigSchema, with inputs_model/outputs_model pydantic models generated from the schema’s inputs/outputs.

Raises:

ConfigLoadError – If the file’s top-level content isn’t a mapping, it has no top-level name, or a package-scoped _include names a package with no entry in package_roots, or composition contains an _include/_use cycle. Cycle errors name the closed path.

Return type:

ConfigSchema

Load stimela-classic style parameters.json cab definitions into shinobi Cab objects. This is a different cab schema format from scabha-dialect YAML (see shinobi.loaders.yaml_cab) – useful for exactly the tools cult-cargo doesn’t have a loadable definition for (several CASA tasks, msutils – see AGENTS.md/examples/ninja_selfcal.py for which ones and why).

Classic’s format: one JSON file per cab (e.g. stimela/cargo/cab/casa_mstransform/parameters.json), a top-level task/binary/base/prefix/msdir plus a flat parameters list – unlike cult-cargo, there’s no _include/_use composition to resolve; each file is fully self-contained.

Field mapping (into a generated inputs_model + field_meta):

  • name -> the model field name, sanitised to a valid identifier if needed (the original kept as a nom_de_guerre).

  • dtype -> a Python type on the generated model. A param can declare dtype as a list of alternatives (e.g. [“int”, “str”]) for a genuine type union; the first alternative is used and the rest are dropped – narrowing a real union to shinobi’s simpler model, not a bug.

  • io: “msfile” forces dtype to “MS” (matching shinobi/cult-cargo convention for the main measurement-set parameter), regardless of whatever the raw dtype said (almost always “file” anyway). io: “input”/”output” have no separate shinobi concept – a file-like type alone already drives bind-mounting via path_fields – so they’re otherwise dropped.

  • required, default, info -> the model field / its ParamMeta.

  • mapping -> ParamMeta.nom_de_guerre (classic’s own name for the same concept: what the underlying tool actually calls this parameter).

  • choices -> ParamMeta.choices, and the generated model field’s real annotation is narrowed to typing.Literal[*choices] (see _modelgen.narrow_choices) – an out-of-set value fails pydantic validation, not just a note in info. Also still appended to info as a human-readable parenthetical, for callers that only look at info.

flavour: classic’s CASA-task cabs (base containing “casa”) are not real standalone executables – binary there is a CASA task name (mstransform/listobs/flagdata/…), invoked by wrapping it in a CASA script, not subprocess.run([“mstransform”, …]). These load with flavour=”casa-task” (shinobi’s existing non-executable flavour, UnsupportedFlavourError-guarded in shinobi.policies – see SECURITY.md’s “Never eval()/exec() a cab’s command” section), not “binary”, so they can’t be silently misrun as if they were real binaries. Cabs with any other base (msutils, wsclean, cubical, …) are real CLI tools and load as flavour=”binary”.

image: classic’s base (e.g. “stimela/casa”) is a base-image family name, not a concrete pullable reference – the real tag/version lives in separate tag/version fields (arrays of compatible versions, no single “the” version). base is used as a best-effort image default; override it on the loaded Cab if you need a specific real image.

shinobi.loaders.stimela_classic.load_file(path)[source]

Load a stimela-classic cab definition (JSON) file into a Cab.

Parameters:

path (str | Path) – Path to the JSON cab definition file.

Returns:

The built Cab instance.

Return type:

Cab

shinobi.loaders.stimela_classic.loads(text)[source]

Parse a stimela-classic cab definition from a JSON string.

Parameters:

text (str) – JSON text of the cab definition.

Returns:

The built Cab instance.

Return type:

Cab

Cab loaders, plus the public helpers for building a cab’s schema by hand.

The shinobi.loaders.yaml_cab and shinobi.loaders.stimela_classic submodules load cab definitions from their respective on-disk formats. build_model (and sanitize_unique) are the same helpers those loaders use to turn a flat {name: (dtype, required, default)} spec into the pydantic inputs_model/outputs_model a Cab needs – re-exported here as the supported way to build those models directly, without hand-writing a pydantic class. The implementation lives in the internal _modelgen module.

shinobi.loaders.build_model(name, fields, *, allow_extra=False, choices=None, extras=None, string_patterns=None, nullable=None, error=<class 'ValueError'>)[source]

Create a pydantic model class named name.

fields maps a field name to (dtype, required, default). See required_field_spec for the required/default rule applied to each. choices maps a field name to its allowed values (see narrow_choices) – omitted or absent for a field means its plain dtype-derived type applies unchanged. extras maps a field name to a json_schema_extra dict carried onto that field (e.g. abbreviation for the CLI); a field absent from extras gets none. Mirrors what worker_schema._leaf_field builds per field, so both scabha-dialect loaders attach field-level hints the same way. string_patterns adds pydantic regex constraints to literal scalar string fields; constrained defaults are validated when the model is instantiated. nullable supplies explicit nullability independently of omission; non-nullable defaults also receive Pydantic default validation.

Parameters:
Return type:

type

shinobi.loaders.merge_field_meta(base, override)[source]

base updated with override, merging (not replacing) the entry of any field name both declare – see merge_param_meta. The one way a loader is allowed to compose a cab’s input-side and output-side metas, so a cab built from a document and the same cab built in Python cannot disagree about what its fields declare.

Parameters:
Return type:

dict[str, Any]

shinobi.loaders.merge_param_meta(base, override)[source]

Merge two ParamMeta`s describing one field name, attribute by attribute: `override wins wherever it says something, and base survives wherever override left the default.

This exists for the dual declaration – one name on inputs and on outputs, the echo-the-input-back idiom every MS-producing cab uses. Each side declares what it knows: the output side carries implicit (how the product’s path is derived), the input side carries write_path (whether the tool creates that path or rewrites the caller’s data). Replacing the whole object, as the loaders used to, keeps only the last one collected – so the input’s write_path was silently dropped for precisely the fields it exists to mark, and sandbox.clear_stale_outputs then read every such cab as “the caller’s data, leave it alone”. Silent, and on the half of the schema that decides whether a re-run works at all.

Attribute-wise rather than “the input wins” so the output’s implicit still overrides – that is what the whole-object merge got right, and what every loaded cab already depends on.

Parameters:
Return type:

Any

shinobi.loaders.sanitize_unique(name, seen)[source]

Like sanitize, but raises if two distinct raw names collide on the same sanitized identifier. seen maps a sanitized field name to the first raw name that produced it – share one seen dict across a single cab’s parameter list.

Parameters:
Return type:

str

Configuration

Application configuration: layered defaults < config file < env vars < CLI overrides, all validated by the same pydantic models used everywhere else in shinobi. No OmegaConf/scabha/munch/benedict stack – just pydantic-settings, reusing the validation library the cab schemas already depend on.

class shinobi.config.VenvConfig(*, default=None, envs=<factory>)[source]

Bases: BaseModel

Settings for the venv backend (shinobi.backends.venv).

default is the venv used when a step declares none of its own (a path, or a key into envs); None means “no default”, so a venv-backend step with nothing declared falls back to native. envs maps short names to venv paths so recipes/config can refer to a venv by name rather than a machine-specific absolute path. A venv path is a deployment concern, so these live here (or on a Scope in Python), never in a shared cab repo.

Parameters:
default: str | None
envs: dict[str, str]
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.config.BackendConfig(*, default='native', run_as_host_user=True, venv=<factory>)[source]

Bases: BaseModel

Settings controlling which execution backend cabs run under.

Parameters:
default: str
run_as_host_user: bool
venv: VenvConfig
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.config.ExecutionConfig(*, max_workers=1, resources=<factory>, enforce_resources='auto', clear_stale_outputs=True)[source]

Bases: BaseModel

Settings controlling recipe step scheduling.

Parameters:
max_workers: int
resources: ResourceBudget
enforce_resources: Literal['auto', 'always', 'never']
clear_stale_outputs: bool
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.config.SnapshotConfig(*, mode='auto')[source]

Bases: BaseModel

Settings controlling mutation-chain snapshots (shinobi.snapshots).

Parameters:

mode (Literal['auto', 'copy', 'off'])

mode: Literal['auto', 'copy', 'off']
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.config.CacheConfig(*, enabled=False, dir='.shinobi/cache', snapshots=<factory>, content_sample=False)[source]

Bases: BaseModel

Settings controlling step-level skip-if-unchanged caching.

Parameters:
enabled: bool
dir: str
snapshots: SnapshotConfig
content_sample: bool
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.config.StateConfig(*, dir='.shinobi/states')[source]

Bases: BaseModel

Explicit reusable dataset state storage, independent of step caching.

Parameters:

dir (str)

dir: str
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.config.LogConfig(*, dir='.', file=None, level='INFO', stream=True, capture_head_lines=5000, capture_tail_lines=5000)[source]

Bases: BaseModel

Settings controlling logging and live output streaming.

Parameters:

  • dir (str)

  • file (str | None)

  • level (str)

  • stream (bool)

  • capture_head_lines (int)

  • capture_tail_lines (int)

dir: str
file: str | None
level: str
stream: bool
capture_head_lines: int
capture_tail_lines: int
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.config.SandboxConfig(*, enabled=False, dir='.shinobi/work')[source]

Bases: BaseModel

Settings controlling per-step sandbox execution (shinobi.sandbox).

Parameters:
enabled: bool
dir: str
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.config.ProvenanceConfig(*, enabled=False, dir='.shinobi/runs')[source]

Bases: BaseModel

Settings controlling reproducible-run provenance (shinobi.provenance).

Parameters:
enabled: bool
dir: str
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.config.AppConfig(_case_sensitive=None, _nested_model_default_partial_update=None, _env_prefix=None, _env_prefix_target=None, _env_file=PosixPath('.'), _env_file_encoding=None, _env_ignore_empty=None, _env_nested_delimiter=None, _env_nested_max_split=None, _env_parse_none_str=None, _env_parse_enums=None, _cli_prog_name=None, _cli_parse_args=None, _cli_settings_source=None, _cli_parse_none_str=None, _cli_hide_none_type=None, _cli_avoid_json=None, _cli_enforce_required=None, _cli_use_class_docs_for_groups=None, _cli_show_env_vars=None, _cli_exit_on_error=None, _cli_prefix=None, _cli_flag_prefix_char=None, _cli_implicit_flags=None, _cli_ignore_unknown_args=None, _cli_kebab_case=None, _cli_shortcuts=None, _secrets_dir=None, _build_sources=None, *, backend=<factory>, execution=<factory>, log=<factory>, cache=<factory>, state=<factory>, provenance=<factory>, sandbox=<factory>)[source]

Bases: BaseSettings

Precedence, highest to lowest: CLI overrides > env vars (SHINOBI_*) > config file > built-in defaults.

Parameters:

  • _case_sensitive (bool | None)

  • _nested_model_default_partial_update (bool | None)

  • _env_prefix (str | None)

  • _env_prefix_target (EnvPrefixTarget | None)

  • _env_file (DotenvType | None)

  • _env_file_encoding (str | None)

  • _env_ignore_empty (bool | None)

  • _env_nested_delimiter (str | None)

  • _env_nested_max_split (int | None)

  • _env_parse_none_str (str | None)

  • _env_parse_enums (bool | None)

  • _cli_prog_name (str | None)

  • _cli_parse_args (bool | list[str] | tuple[str, ...] | None)

  • _cli_settings_source (CliSettingsSource[Any] | None)

  • _cli_parse_none_str (str | None)

  • _cli_hide_none_type (bool | None)

  • _cli_avoid_json (bool | None)

  • _cli_enforce_required (bool | None)

  • _cli_use_class_docs_for_groups (bool | None)

  • _cli_show_env_vars (bool | None)

  • _cli_exit_on_error (bool | None)

  • _cli_prefix (str | None)

  • _cli_flag_prefix_char (str | None)

  • _cli_implicit_flags (bool | Literal['dual', 'toggle'] | None)

  • _cli_ignore_unknown_args (bool | None)

  • _cli_kebab_case (bool | Literal['all', 'no_enums'] | None)

  • _cli_shortcuts (Mapping[str, str | list[str]] | None)

  • _secrets_dir (PathType | None)

  • _build_sources (tuple[tuple[PydanticBaseSettingsSource, ...], dict[str, Any]] | None)

  • backend (BackendConfig)

  • execution (ExecutionConfig)

  • log (LogConfig)

  • cache (CacheConfig)

  • state (StateConfig)

  • provenance (ProvenanceConfig)

  • sandbox (SandboxConfig)

model_config = {'arbitrary_types_allowed': True, 'case_sensitive': False, 'cli_avoid_json': False, 'cli_enforce_required': False, 'cli_exit_on_error': True, 'cli_flag_prefix_char': '-', 'cli_hide_none_type': False, 'cli_ignore_unknown_args': False, 'cli_implicit_flags': False, 'cli_kebab_case': False, 'cli_parse_args': None, 'cli_parse_none_str': None, 'cli_prefix': '', 'cli_prog_name': None, 'cli_shortcuts': None, 'cli_show_env_vars': False, 'cli_use_class_docs_for_groups': False, 'enable_decoding': True, 'env_file': None, 'env_file_encoding': None, 'env_ignore_empty': False, 'env_nested_delimiter': '__', 'env_nested_max_split': None, 'env_parse_enums': None, 'env_parse_none_str': None, 'env_prefix': 'SHINOBI_', 'env_prefix_target': 'variable', 'extra': 'forbid', 'json_file': None, 'json_file_encoding': None, 'nested_model_default_partial_update': False, 'protected_namespaces': ('model_validate', 'model_dump', 'settings_customise_sources'), 'secrets_dir': None, 'toml_file': None, 'validate_default': True, 'yaml_config_section': None, 'yaml_file': None, 'yaml_file_encoding': None}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

backend: BackendConfig
execution: ExecutionConfig
log: LogConfig
cache: CacheConfig
state: StateConfig
provenance: ProvenanceConfig
sandbox: SandboxConfig
classmethod settings_customise_sources(settings_cls, init_settings, env_settings, dotenv_settings, file_secret_settings)[source]

Set the settings source precedence: init > env vars > YAML file.

Parameters:

  • settings_cls – The BaseSettings subclass being configured.

  • init_settings – Source for values passed directly to __init__.

  • env_settings – Source for SHINOBI_* environment variables.

  • dotenv_settings – Unused; .env files are not supported.

  • file_secret_settings – Unused; secret files are not supported.

Returns:

The ordered tuple of settings sources pydantic-settings should consult, highest precedence first.

classmethod load(config_file=None, **cli_overrides)[source]

Build an AppConfig, layering defaults, config file, env, and overrides.

Parameters:

  • config_file (str | Path | None) – Path to a YAML config file. Defaults to DEFAULT_CONFIG_FILE (~/.shinobi/config.yml) if not given.

  • **cli_overrides (Any) – Explicit values that take precedence over the config file and environment variables.

Returns:

A fully-resolved AppConfig instance.

Return type:

AppConfig

Caching

class shinobi.cache.ProvenanceKey(value, producer_field=None, element_index=None, coordinates=None)[source]

Bases: str

A provenance key that also remembers which output field of the producing step it names.

The key alone answers “which state of this path”, which is all the skip cache ever needed. Snapshotting needs one thing more: a step with two mutated outputs produces two states in one run, and both resolve to that step’s single cache_key (see StepResult.provenance_key), so the key by itself cannot tell them apart. (key, producer field) can.

It is a str subclass rather than a pair because these values are hashed into cache keys, in __upstream__ and through combine_keys, and any change to their shape would rewrite every key in every existing manifest. json.dumps serializes a str subclass as the plain string – as value, as list element, and as dict key, under sort_keys and default=str alike – and equality, ordering and hashing against plain str are inherited unchanged. So the field rides along in memory and vanishes at the point of hashing, which is what lets this land without invalidating anything.

producer_field is a class attribute as well as an instance one: copy.deepcopy and pickle reconstruct a str subclass through str.__new__ and then restore state, so an instance can briefly exist without it, and reading it must not raise.

Parameters:

  • value (str)

  • producer_field (str | None)

  • element_index (int | None)

  • coordinates (dict[str, Any] | None)

Return type:

ProvenanceKey

producer_field: str | None = None
element_index: int | None = None
coordinates: dict[str, Any] | None = None

Exceptions

exception shinobi.exceptions.ShinobiError[source]

Bases: Exception

Base class for all shinobi errors.

exception shinobi.exceptions.ParameterError[source]

Bases: ShinobiError

A step or cab was called with invalid, missing, or unknown parameters, or its outputs could not be validated against the declared schema.

exception shinobi.exceptions.BackendError[source]

Bases: ShinobiError

A backend failed to run a cab.

exception shinobi.exceptions.CabRunError[source]

Bases: ShinobiError

A step’s underlying command or function exited with a non-zero/failure status.

exception shinobi.exceptions.StepError[source]

Bases: ShinobiError

A step failed during execution for a reason not covered by a more specific exception class. The message carries the step/recipe path so the caller can tell which step failed and why.

exception shinobi.exceptions.CabLoadError[source]

Bases: ShinobiError

A cab definition file could not be loaded or resolved.

exception shinobi.exceptions.ConfigLoadError[source]

Bases: ShinobiError

A worker/config schema file could not be loaded or resolved.

exception shinobi.exceptions.ReplayError[source]

Bases: ShinobiError

A run manifest cannot be replayed against the current target.

exception shinobi.exceptions.DatasetLifecycleUnavailableError[source]

Bases: ShinobiError

Execution was refused for a strict dataset annotation.

Only the contained native MSv2 read and mutation lifecycles are executable. This exception keeps every unsupported route from looking enforced while an annotation is only being carried as a path by that execution stack, and it is also the refusal raised before a strict mutation launches when exact predecessor recovery cannot be promised.

exception shinobi.exceptions.DatasetLifecycleViolationError[source]

Bases: ShinobiError

A strict dataset broke its declared postcondition under its claim.

For a reader, the dataset changed; for a writer, the change exceeded what its access contract permits. A writer’s dataset has been rolled back to its exact predecessor when this is raised, unless a note says otherwise.

exception shinobi.exceptions.UnsupportedFlavourError[source]

Bases: ShinobiError

A cab’s flavour isn’t one shinobi knows how to execute.

Cabs with a non-“binary” flavour (cult-cargo’s “python-code”/ “casa-task”/etc.) have a command that is not an executable name – it may be inline Python/shell source, or a dotted reference to a function to import and call. shinobi never treats that as code to eval()/exec(): every backend shells out via subprocess with a list argv, so a non-executable command would otherwise just fail obscurely (subprocess trying to exec a multi-line string as a program path). This is raised deliberately, before that happens.

Product declaration helpers

class shinobi.products.AggregateSpec(*, value, suffix, only_multiple=False)[source]

Bases: ProductModel

Parameters:
value: Coordinate
suffix: str
only_multiple: bool
model_config = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class shinobi.products.ProductModel[source]

Internal closed, frozen base for framework product values and declarations.

shinobi.products.T

Type parameter for File and DirectoryBundle members.

shinobi.products.Coordinate

Strict integer or string coordinate; bool values are rejected.