Title: Batch Run Flywheel Gears with SDK

Date: May 13th 2020

Description:
This notebook provides an overview of the Flywheel Gears and how to run gears as a batch with SDK. Topics that will be covered:

• Overview of Flywheel Gears
• Batch Run Gears
• Gear Output
• Gear Rules VS Batch Run Gears

### Requirements:¶

2. No Gear Rules applied in the Test Project.
3. A Flywheel Project with ideally the dataset used in the upload-data notebook.

NOTE: This notebook is using a test dataset provided by the upload-data notebook. If you have not uploaded this test dataset yet, we strongly recommend you do so now following steps in here before proceeding because this notebook is based on a specific project structure.

WARNING: The metadata of the acquisitions in your test project will be updated and new files will be created after running the scripts below.

# An Overview of Flywheel Gears¶

Flywheel categorizes the gears into Utility Gears and Analysis Gears.

• Utility Gear is typically a basic pipeline that generates another representation of the data (e.g. convert DICOM to NifTI), perform QA (e.g. A QA tool that generates a reqport) or a Classifier (e.g. extracts the data/header info of a file).
• Analysis Gear is a pipeline which processes the data with and algorithm, such as signal processing algorithm, and generates one or more files to be used for statistical analysis and/or machine learning. See below for a note on using analysis gears in gear rules.

In this notebook, we will be mainly focusing on Utility Gears but the same principle can be applied to Analysis Gears.

# Flywheel API Key and Client¶

Get a API_KEY. More on this at in the Flywheel SDK doc here.

Instantiate the Flywheel API client

Show Flywheel logging information

# Initialize a few values¶

Define your test Project's Label and let's look for it on your Flywheel instance.

# Requirements¶

Before starting off, we want to check your permission on the Flywheel Instance in order to proceed in this notebook.

Tip: Group ID and Project Label can be found on top of the Project page on the Flywheel Instance as shown in the snippet below.

check_user_permission will return True if both the group and project meet the minimum requirement, else a compatible list will be printed.

# Set Up Flywheel CLI¶

Flywheel Gears can be uploaded via the Flywheel CLI, which is a tool that allows us to interact with Flywheel and out data from the command line.

TIP: If you are curious what you can do with Flywheel CLI, see our article here.

Here we will be showing you how you can install the CLI for Linux Operating System, such as the one used by Binder or Google Collab environments.

First we will be getting your instance specific CLI version.

The Flywheel CLI builds are hosted on google storage at the following URL:

NOTE: If you are interested for more detailed instruction and installation guide for other Operating system, please refer to our docs.

To install the Flywheel CLI we need to donwload, unzip and add the fw to somewhere in your \$PATH (here we will be using your current working directory).

You can use fw -h to view all the commands available on Flywheel CLI

If you successfully logged in with you API key, then you should see the message:

You are now logged in as <username>!

## Upload Flywheel Gears to Flywheel Instance¶

Now, we will upload the following Gears to your Flywheel Instance using the Flywheel CLI:

• DCM2NIIX (a Converter Gear).
• MRIQC (a Quality Assessment of MRI).

You can find a list of available Gears at our Flywheel Gear Exchange page.

Now we will be uploading the MRIQC gear into the flywheel instance using the fw gear upload command.

We will be repeating the same process for DCM2NIIX Gear

# Batch Run Gears¶

## Useful Function¶

This run_gear function will be used to run gears in this notebook.

## Main Scripts¶

### Run DICOM-2-NifTI Gear¶

We first retrieve the gear by looking it up:

Then, for each acquisition container in each session container we:

1. Get the dicom file and define it as inputs.
2. Get the destination container (here defined as the parent container of the file, i.e. the Acquisition Container the file is in in this example).
3. Submit the job.
TIP: The format for the inputs for a given gear is a dictionary of key-value pairs, where the key corresponds to the manifest input label (which can be found on the Flywheel Instance, see the figure below) and the value is the value of the item.

For dicom-2-nifti gear, the required manifest input label is dcm2niix_input and the value of the input is the file_obj.

### Run MRIQC Gear¶

We can repeat the same operation for the MRIQC gear.

NOTE: This gear might takes more than 5 minutes to execute.

## Checking Job Status¶

You can check your job status by using the get_job() method.

You can also check the job status on the Flywheel Instance as well.

This can be done on the Provenance tab as shown on the snippets below:

You can also find other gears' status that have been completed earlier.

# Gear Outputs¶

## Output on the Flywheel Instance¶

On the Provenance tab, you can also find out the output name that generated after the job has been processed. These output files can be downloaded or viewed from the UI as shown below:

In this section, we will be demonstrating how to download all of the outputs that were generated from the mriqc Gears by our batch run.

You can also generate a summary table of the QC values from each output file.

# Gear Rules VS Batch Run Gears¶

With Gear Rules, you can automatically run the gears above when new data is added to a project.

TIP: A default Gear Rules might have already set up for the gears above, this can be found in the Gear Rules tab in your project container.

However, Gear Rules can not be applied to gears that use Flywheel SDK unless these gears have the read-only key added to them. Click here to read more about this topic.