Title: Read DICOM in Memory and Download Zip Member
Date: 10/22/2020
Description:
To illustrate how to get archive Zip File from Flywheel and read DICOM file in memory.

Requirements

• Have a DICOM Zip Archive stored in at least one Acquisition container.

Install and Import Dependencies

# Install specific packages required for this notebook
!pip install flywheel-sdk pydicom

# Import packages
from getpass import getpass
import logging
import os
from pathlib import Path
import time

import flywheel
from permission import check_user_permission
import pydicom
import matplotlib.pyplot as plt

# Instantiate a logger
logging.basicConfig(level=logging.INFO, format='%(asctime)s %(levelname)s %(message)s')
log = logging.getLogger('root')

Flywheel API Key and Client

Get your API_KEY. More on this in the Flywheel SDK doc here.

API_KEY = getpass('Enter API_KEY here: ')

Instantiate the Flywheel API client

fw = flywheel.Client(API_KEY if 'API_KEY' in locals() else os.environ.get('FW_KEY'))

Show Flywheel logging information

log.info('You are now logged in as %s to %s', fw.get_current_user()['email'], fw.get_config()['site']['api_url'])

---

Initialize a few values

Define the Group's ID and Project's Label that contain DICOM zip archive files and let's look for it on your Flywheel instance.

Tip: Group ID and Project Label can be found on top of the Project page on the Flywheel Instance as shown in the snippet below.

GROUP_ID = input('Please enter your Group ID here:')

PROJECT_LABEL = input('Please enter your Project Label here: ')

project = fw.lookup(f'{GROUP_ID}/{PROJECT_LABEL}')

---

Requirements

Before starting off, we want to check your permission on the Flywheel Instance in order to proceed in this notebook.

min_reqs = {
"site": "user",
"group": "ro",
"project": ['analyses_view_metadata','files_view_metadata','files_view_contents','files_download']
}

check_user_permission will return True if both the group and project meet the minimum requirement, else a compatible list will be printed.

check_user_permission(fw, min_reqs, group=GROUP_ID, project=PROJECT_LABEL)

---

Helpful Function

def read_dcm_from_archive(file_entry, acq):
    """
    For a given file entry, read a dicom file from the archive and return a pydicom object.
    Function here will get the second zip member from the Dicom zip archive and stream it to memory as a pydicom.DataSet instance
    """
    import pydicom
    from pydicom.filebase import DicomBytesIO
    
    # Check that this file is valid
    if file_entry['type'] != 'dicom':
        raise TypeError('Must be a DICOM file')
    
    log.info(f'Loading {file_entry["name"]}')
    
    # Using the `get_file_zip_info` method to retrieve the files information within the zip file.
    zip_info = acq.get_file_zip_info(file_entry['name'])
    log.info(f'Reading DICOM file from zip archive')
    
    # Read the 2nd member from the path
    raw_dcm = DicomBytesIO(acq.read_file_zip_member(file_entry['name'], zip_info.members[1].path))
    dcm = pydicom.dcmread(raw_dcm, force=True)
    
    return dcm

def show_dcm_info(dcm_list):
    """
    Print out DICOM header and metadata.
    """
    for ds in dcm_list: 
        if ds.SOPClassUID:
            # Normal mode:
            print()
            print(f"SOP Class........: {ds.SOPClassUID} ({ds.SOPClassUID.name})")
            print()

            pat_name = ds.PatientName
            display_name = pat_name.family_name + ", " + pat_name.given_name
            print(f"Patient's Name...: {display_name}")
            print(f"Patient ID.......: {ds.PatientID}")
            print(f"Modality.........: {ds.Modality}")
            print(f"Study Date.......: {ds.StudyDate}")
            print(f"Image size.......: {ds.Rows} x {ds.Columns}")
            print(f"Pixel Spacing....: {ds.PixelSpacing}")

            # use .get() if not sure the item exists, and want a default value if missing
            print(f"Slice location...: {ds.get('SliceLocation', '(missing)')}")

            # plot the image using matplotlib
            # tell matplotlib to display our image, using a gray-scale lookup table.
            plt.imshow(ds.pixel_array, cmap=plt.cm.gray)
            plt.show()
            print('')
        else:
            print('Invalid DICOM image')
            print('')

---

Main Script

For the first part of the tutorial, we will go through how to read the DICOM file from the zip archive and read the data with pydicom. To illustrate we will be reading the 2nd DICOM image in the zip file.

dcm_list = list()

for sess in project.sessions():
    for acq in sess.acquisitions():
        for files in acq['files']:
            if files['type'] == 'dicom':
                dcm = read_dcm_from_archive(files,acq)
                dcm_list.append(dcm)

View DICOM Image

Here we will be using matplotlib to plot the DICOM Image that we have selected for each Subject and some DICOM information.

show_dcm_info(dcm_list)

---

Read DICOM images of a specific Subject

Helpful Function

def read_all_dcm_from_archive(file_entry, acq):
    """
    For a given file entry, read a dicom file from the archive.
    Method here will read the dicom file from the archive as raw data and read that raw data into pydicom.
    It will return a list of pydicom objects for the Subject.
    """
    
    import pydicom
    from pydicom.filebase import DicomBytesIO
    
    subj_dcm = list()
    
    # Check that this file is valid
    if file_entry['type'] != 'dicom':
        raise TypeError('Must be a DICOM file')
        
    log.info(f'Loading {file_entry["name"]}')
    
    # Using the `get_file_zip_info` method to retrieve the files information within the zip file.
    zip_info = acq.get_file_zip_info(file_entry['name'])
    
    for member in zip_info.members:
        if member.size == 0:
            continue
        subj_dcm.append(read_dcm_from_archive(file_entry, acq, i))
        raw_dcm = DicomBytesIO(acq.read_file_zip_member(file_entry['name'], member.path))
        dcm = pydicom.dcmread(raw_dcm, force=True)
        if 'SOPClassUID' in dcm:
            subj_dcm.append(dcm)

    log.info(f'Processed {len(subj_dcm)} DICOM files.')
    
    return subj_dcm

Let's say you are only interested in a specific Subject and you would like to do some analysis with the Subject's DICOM image.

Get the anx_s4 Subject container

subj_04 = project.subjects.find_first('label=anx_s4')

Get the DICOM zip archive for the Subject anx_s4.

for sess in subj_04.sessions():
    for acq in sess.acquisitions():
        for files in acq['files']:
            if files['type'] == 'dicom':
                subj_dcm = read_all_dcm_from_archive(files,acq)

Helpful function that get the zip file member and store all the zip member dicom file for one subject.

show_dcm_info(subj_dcm)

Interactive DICOM viewer

We will be using the ipywidgets.interactive() method to build a DICOM Viewer from the DICOM image that we have pulled from the Subject.

from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets

# slide through dicom images using a slide bar 
plt.figure(10)
def dicom_animation(x):
    plt.imshow(subj_dcm[x].pixel_array, cmap=plt.cm.gray)
    
interact(dicom_animation, x=(0, len(subj_dcm)-1))

---

Download a Zip member from a Zip Archive in Analysis container via SDK

Often a gear will store its output into a large zip file in an analysis container and it can be useful to extract specific file from it across different gear runs. In this section, you will learn how to get a zip file member from Flywheel and download the files into your local directory

Get analyses container

First, we get the Subject container first. In this tutorial we will be working with sub-04.

subj = project.subjects.find_first('label=sub-04')

We will be working with the first Session for Subject sub-4 and the first Analyses in the Session container.

session_container = subj.sessions()[0]

analysis = session_container.reload()['analyses'][0]

Info: Use reload() method here to retrieve a complete view of the data within the Session container .

Now, we will loop through the files within the analyses container in the Session container.

for file in analysis['files']:
    print(file.name)

After looking at the available files in the Analysis container, we decided to work with the second file in the Analysis container.

zip_file_name = analysis['files'][1].name

analysis_id = analysis.id
analysis_id

To retrieve the files information within the zip file, you can use get_file_zip_info which will take the zip_file_name as argument.

zip_info = analysis.get_file_zip_info(zip_file_name)

len(zip_info.members)

Let's see what the archive contains and pinpoint the location of the .nifti file.

for ii, file in enumerate(zip_info.members):
    if file.path[-1] != '/':
        if file.path.endswith('nii.gz'):
            print(f'{ii} {Path(file.path).name}')
    else:
        print(f'dir {file.path}')

Download File

To download the file from the zip output file, we will be using download_file_zip_member method. It requires three parameters, which is file_name: The name of the zip file, member_path: the choosen member in the zip file, and dest_file: The name of the file you would like to download as.

In this example, we are going to download the 86th member in the zip file and save as vectorRGB.nii.gz on your local directory.

member_path = zip_info.members[86].path
dest_file = Path(zip_info.members[86].path).name

analysis.download_file_zip_member(zip_file_name, member_path, dest_file)