Title: Batch Run Flywheel Gears with SDK

Date: May 13th 2020

Description:
This notebook provides an overview of the Flywheel Gears and how to run gears as a batch with SDK. Topics that will be covered:

Requirements:

  1. Access to a Flywheel instance.
  2. No Gear Rules applied in the Test Project.
  3. A Flywheel Project with ideally the dataset used in the upload-data notebook.

NOTE: This notebook is using a test dataset provided by the upload-data notebook. If you have not uploaded this test dataset yet, we strongly recommend you do so now following steps in here before proceeding because this notebook is based on a specific project structure.

WARNING: The metadata of the acquisitions in your test project will be updated and new files will be created after running the scripts below.

An Overview of Flywheel Gears

Flywheel categorizes the gears into Utility Gears and Analysis Gears.

In this notebook, we will be mainly focusing on Utility Gears but the same principle can be applied to Analysis Gears.


Install and Import Dependencies

Flywheel API Key and Client

Get a API_KEY. More on this at in the Flywheel SDK doc here.

Instantiate the Flywheel API client

Show Flywheel logging information


Initialize a few values

Define your test Project's Label and let's look for it on your Flywheel instance.


Requirements

Before starting off, we want to check your permission on the Flywheel Instance in order to proceed in this notebook.

Tip: Group ID and Project Label can be found on top of the Project page on the Flywheel Instance as shown in the snippet below.

check_user_permission will return True if both the group and project meet the minimum requirement, else a compatible list will be printed.


Set Up Flywheel CLI

Flywheel Gears can be uploaded via the Flywheel CLI, which is a tool that allows us to interact with Flywheel and out data from the command line.

TIP: If you are curious what you can do with Flywheel CLI, see our article here.

Here we will be showing you how you can install the CLI for Linux Operating System, such as the one used by Binder or Google Collab environments.

First we will be getting your instance specific CLI version.

The Flywheel CLI builds are hosted on google storage at the following URL:

NOTE: If you are interested for more detailed instruction and installation guide for other Operating system, please refer to our docs.

To install the Flywheel CLI we need to donwload, unzip and add the fw to somewhere in your $PATH (here we will be using your current working directory).

You can use fw -h to view all the commands available on Flywheel CLI

You should be able to log in with your API key with the command below.

If you successfully logged in with you API key, then you should see the message:

You are now logged in as <username>!


Upload Flywheel Gears to Flywheel Instance

Now, we will upload the following Gears to your Flywheel Instance using the Flywheel CLI:

You can find a list of available Gears at our Flywheel Gear Exchange page.

Now we will be uploading the MRIQC gear into the flywheel instance using the fw gear upload command.

NOTE: These gears might be alrady installed on your Flywheel instance. So, you might receive a message saying that the gears exist when you are uploading the gear to your Flywheel instance.

We will be repeating the same process for DCM2NIIX Gear


Batch Run Gears

Useful Function

This run_gear function will be used to run gears in this notebook.

Main Scripts

Run DICOM-2-NifTI Gear

We first retrieve the gear by looking it up:

Then, for each acquisition container in each session container we:

  1. Get the dicom file and define it as inputs.
  2. Get the destination container (here defined as the parent container of the file, i.e. the Acquisition Container the file is in in this example).
  3. Submit the job.
TIP: The format for the inputs for a given gear is a dictionary of key-value pairs, where the key corresponds to the manifest input label (which can be found on the Flywheel Instance, see the figure below) and the value is the value of the item.

For dicom-2-nifti gear, the required manifest input label is dcm2niix_input and the value of the input is the file_obj.

Run MRIQC Gear

We can repeat the same operation for the MRIQC gear.

NOTE: This gear might takes more than 5 minutes to execute.

Checking Job Status

You can check your job status by using the get_job() method.

You can also check the job status on the Flywheel Instance as well.

This can be done on the Provenance tab as shown on the snippets below:

You can also find other gears' status that have been completed earlier.


Gear Outputs

Output on the Flywheel Instance

On the Provenance tab, you can also find out the output name that generated after the job has been processed. These output files can be downloaded or viewed from the UI as shown below:

Download/View Output with SDK

You can also view or download all of the outputs to your destination path

In this section, we will be demonstrating how to download all of the outputs that were generated from the mriqc Gears by our batch run.

You can also generate a summary table of the QC values from each output file.


Gear Rules VS Batch Run Gears

With Gear Rules, you can automatically run the gears above when new data is added to a project.

TIP: A default Gear Rules might have already set up for the gears above, this can be found in the Gear Rules tab in your project container.

However, Gear Rules can not be applied to gears that use Flywheel SDK unless these gears have the `read-only` key added to them. Click here to read more about this topic.

TIP: To learn more about gear rules and how to use them, please visit our documentations here.