`export run`

Export data from a Flywheel project to an external storage through a connector that's hosted and scaled within a cluster. Storages need to be registered by site-admins on the UI / Interfaces menu / External Storage tab or using flyw admin storage create in order to make them available for hosted exports.

Usage

Snapshots

Before transferring files, a snapshot is taken from the project's contents. A snapshot fully captures all container and file metadata - versioned - at a point in time, enabling reproducible exports. If no snapshot ID is specified using the --snapshot option, running an export will automatically create a new snapshot of the given project.

Rules

Selecting the files to be exported and configuring how they are stored on the destination can be defined with export rules. At least one rule is required for matching any file in the snapshot. Additional rules may be specified to achieve complex export behaviors.

Rules can be specified using the --rule-set option, which accepts either a rule-set ID or a file path to a YAML file containing adhoc rules. For more information on managing rule-sets, see export rule-set.

Each rule is tied to a Flywheel hierarchy level and can optionally have a list of include and/or exclude filters. Rules are evaluated in order and for every file, the first rule is going to be used where

the rule level matches the file level and
any on the include filters matches (if given) and
none of the exclude filters match (if given)

Files not matching any of the rules are going to be skipped.

Filters

Include and exclude filters are strings in the form <field> <operator> <value>.

Supported filter fields include most Flywheel metadata fields:

project._id
project.label
subject._id
subject.label
subject.firstname
subject.lastname
subject.sex
subject.mlset
subject.info.*
subject.tags
session._id
session.uid
session.label
session.age
session.weight
session.operator
session.timestamp
session.info.*
session.tags
acquisition._id
acquisition.uid
acquisition.label
acquisition.timestamp
acquisition.info.*
acquisition.tags
file.name
file.type
file.modality
file.size
file.info.*
file.tags
file.classification
file.classification.*

Field name components can be abbreviated and are automatically expanded as long as they uniquely identify one of the available fields. In addition, referencing a level only is equivalent to using that level's label (or name) field:

subj.first -> subject.firstname
sess       -> session.label

Supported filter operators depending on the value type:

Operator	Description	Types
`=~`	regex match	`str`
`!~`	regex not match	`str`
`=`	equal	`str,int,float,datetime`
`!=`	not equal	`str,int,float,datetime`
`<`	less	`int,float,datetime`
`>`	greater	`int,float,datetime`
`<=`	less or equal	`int,float,datetime`
`>=`	greater or equal	`int,float,datetime`

Paths

Export path templates provide a syntax similar to python f-strings for formatting metadata associated with a file in Flywheel as a single string.

Syntax	Description
`{field}`	Curly braces for referencing metadata fields
`{field/pat/sub}`	`re.sub` pattern for substituting parts of the value
`{field:format}`	`f-string` format spec (`strftime` for timestamps)
`{field\\|default}`	Default to use instead of `"UNKNOWN"` (for `""`/`None`)

Combining modifiers is allowed in the order /pat/sub >> :format >> |default.

Supported template fields are the same as the filter fields except for

*.tags
file.classification

Defaults

If no --rule-set is specified, a default rule is applied to facilitate simple exports.

Rule-Set Usage

To specify custom rules, use the --rule-set option with either:

A rule-set ID (for pre-configured rule-sets):

flyw export run ... --rule-set <rule-set-id>

A file path to a YAML file with adhoc rules:

flyw export run ... --rule-set /path/to/rules.yaml

For more information on creating and managing rule-sets, see the export rule-set documentation.

Deprecated Options

The following command-line options are deprecated and --rule-set should be used instead: --level, --include, --exclude, --path, --unzip, --unzip_path, --metadata, --rules

Settings

Some options affect the export operation behavior globally:

Option	Value	Description
`--overwrite`	`auto`	Overwrite existing files if source changed
	`never`	Do not overwrite existing files even if changed
	`always`	Overwrite existing files even if unchanged
`--delete-extra`	(flag)	Remove files not part of the export from storage
`--dry-run`	(flag)	Run without actually storing data (for testing)
`--limit`	`N`	Stop after processing `N` files (for testing)