Skip to content

import run

Import data from an external storage to a Flywheel project through a connector that's hosted and scaled within a cluster. Storages need to be registered by site-admins on the UI / Interfaces menu / External Storage tab or using flyw admin storage create in order to make them available for imports.

Usage

Rules

Selecting the files to be imported and configuring how they are stored in Flywheel can be defined with import rules. At least one rule is required for matching any file in the source storage. Additional rules may be specified to achieve complex import behaviors.

Rules can be specified using the --rule-set option, which accepts either a rule-set ID or a file path to a YAML file containing adhoc rules. For more information on managing rule-sets, see import rule-set.

Each rule is tied to a Flywheel hierarchy level where the matching files will be imported and can optionally have a list of include and/or exclude filters. Currently only acquisition level file imports are supported.

Rules are evaluated in order and for every file, the first rule is going to be used where:

  • any on the include filters matches (if given) and
  • none of the exclude filters match (if given)

Files not matching any of the rules are going to be skipped.

Filters

Include and exclude filters are strings in the form <field><operator><value>.

Supported filter fields:

Field Type Description
path str File path (relative)
size int File size
ctime datetime Created timestamp
mtime datetime Modified timestamp

Supported filter operators depending on the value type:

Operator Description Types
=~ regex match str
!~ regex not match str
= equal str,int,float,datetime
!= not equal str,int,float,datetime
< less int,float,datetime
> greater int,float,datetime
<= less or equal int,float,datetime
>= greater or equal int,float,datetime

Mappings

Imports require metadata in order to place files correctly within a Flywheel project's subject/session/acquisition hierarchy. Mappings are strings in the form <template>=<pattern> that allow extracting information from the source file's fields (eg.: path) into one or more Flywheel metadata fields (eg.: session.label and acquisition.label).

The default mapping allows extracting all required metadata fields from each file's path, assuming that files are stored in a compatible folder hierarchy:

{path}={subject.label}/{session.label}/{acquisition.label}/*

If any of the required fields are missing after extracting with one or more mappings, the file will be marked as failed, but the import will continue to process the remaining data on storage.

Use --missing-meta skip to skip these files instead and in turn allow the overall import operation to complete without any failures for this reason.

Alternatively, use --fail-fast to halt the entire import when encountering an error.

Templates

Templates are similar to python f-strings for formatting metadata associated with a file as a single string. Currently only the path source field is available but it will be extended in the future.

Syntax Description
{field} Curly braces for referencing metadata fields
{field/pat/sub} re.sub pattern for substituting parts of the value
{field:format} f-string format spec (strftime for timestamps)
{field\|default} Default to use instead of "UNKNOWN" (for ""/None)

Combining modifiers is allowed in the order /pat/sub >> :format >> |default.

Patterns

Patterns are simplified python regexes tailored for scraping Flywheel metadata fields like acquisition.label from a string with capture groups.

Syntax Description
{field} Curly braces for capturing (dot-notated/nested) fields
[opt] Brackets for making parts of the match optional
* Star to match any string of characters (like glob)
. Dot to match a literal dot (like glob)

File Types

Running the import with the type: <type> option allows setting the file.type metadata field in Flywheel to the specified value. Populating the type is useful for searching and for automatically running gears that are tied to data-types.

Import has additional features when importing DICOM data with type: dicom:

  • files are parsed using pydicom
    (invalid DICOMs are treated as errors)
  • series are grouped by directory and SeriesInstanceUID
    (multiple series per directory are treated as errors)
  • series are uploaded az a single zipped file
    (except on single files, eg.: enhanced)
  • metadata fields have tag-based default mappings
  • mappings in rule-sets can reference DICOM tags

Rule-Set Usage

To specify custom rules, use the --rule-set option with either:

  1. A rule-set ID (for pre-configured rule-sets):
flyw import run ... --rule-set <rule-set-id>
  1. A file path to a YAML file with adhoc rules:
flyw import run ... --rule-set /path/to/rules.yaml

For more information on creating and managing rule-sets, see the import rule-set documentation.

Deprecated Options

The following command-line options are deprecated and --rule-set should be used instead: --level, --include, --exclude, --mapping, --type, --zip, --zip-single, --defaults, --overrides, --dicom-instance-name, --dicom-group-by, --dicom-split-localizer, --rules

DICOM

When using type: dicom, import defaults the metadata fields based on DICOM tags. Default mappings (eg.: setting subject.label to the value of PatientID) are only applied if:

  • the field (subject.label) is not yet populated via a custom mapping
  • the value (PatientID) is not empty

The default mappings for DICOM:

  • subject.label - PatientID
  • subject.firstname - split from PatientName
  • subject.lastname - split from PatientName
  • subject.sex - PatientSex
  • session.uid - StudyInstanceUID
  • session.label - StudyDescription
    fallback to session.timestamp
    fallback to StudyInstanceUID
  • session.age - from PatientAge (converted to seconds)
    fallback to delta between acquisition.timestamp and PatientBirthDate
  • session.weight - PatientWeight
  • session.operator - OperatorsName
  • session.timestamp - from StudyDate & StudyTime
    fallback to SeriesDate & SeriesTime
    fallback to AcquisitionDateTime
    fallback to AcquisitionDate & AcquisitionTime
    with respect to TimezoneOffsetFromUTC
  • acquisition.uid - SeriesInstanceUID
  • acquisition.label - SeriesNumber - SeriesDescription
    fallback to SeriesNumber - ProtocolName
    fallback to acquisition.timestamp (formatted as %Y-%m-%dT%H:%M:%S)
    fallback to SeriesInstanceUID
    only prefixed if SeriesNumber is set
  • acquisition.timestamp - AcquisitionDateTime
    fallback to AcquisitionDate & AcquisitionTime
    fallback to SeriesDate & SeriesTime
    fallback to StudyDate & StudyTime
    with respect to TimezoneOffsetFromUTC

Settings

Option Value Description
--dry-run (flag) Run without actually uploading data (for testing)
--limit N Stop after processing N files (for testing)
--fail-fast N[%] Stop processing after reaching a failure threshold
--missing-meta fail Fail items with missing metadata
skip Skip items with missing metadata
--storage-config (YAML) Override default storage config

Some storage settings may be overridden when running an import, which is useful for getting data from the same bucket, but from a different prefix, for example:

flyw import run ... --storage-config "prefix: other/prefix"

Output options

The same output options are available as in flyw import get, the only difference being that run follows the progress of the import until it completes. Use the --no-wait option to exit immediately instead.

Pressing CTRL+C stops the progress monitoring in the CLI, but the import operation will continue running on the cluster. Use flyw import get to check the import's status or resume monitoring it's progress later.

Referencing files in place

Importing large datasets into Flywheel can incur substantial network costs while transferring and storage costs later for keeping a copy of the data.

Referencing files in place allows importing data into Flywheel without transferring any bytes from one cloud storage bucket to another, saving time, network and storage costs.

The ref-in-place workflow requires that the bucket is registered in Flywheel Core-API as a storage provider and that this provider is used for creating a storage via flyw admin storage create.