An AAPM Grand Challenge
Overview
The American Association of Physicists in Medicine (AAPM) is facilitating a “Grand Challenge” on CT ventilation imaging leading up to the 2019 AAPM Annual Meeting. Computed tomography ventilation imaging evaluation 2019 (CTVIE19) will provide a unique opportunity for participants to compare their algorithms with those of other groups in a structured, direct way using the same datasets. A session at the 2019 AAPM Annual Meeting will focus on the CTVIE19 Challenge; an individual from each of the two top-performing teams will receive a waiver of the meeting registration fee in order to present their methods during this session. Following the Annual Meeting, challenge participants will be invited to co-author a challenge report.
Objective
The overall objective of CTVIE19 is to determine which CT ventilation imaging algorithms best correlate with reference measures across a range of pulmonary pathologies. To this end, we will provide a unique and diverse patient dataset of PFTs and paired multi-inflation CT and reference ventilation images collated from data acquired prospectively by leading functional lung imaging institutions worldwide.
Get Started
- Register to get access via the Challenge website
- Download the data after approval
- Submit preliminary cases as a pre-submission check
- Submit your results
Important Dates
- April 5, 2019: Grand challenge website launch
- April 8, 2019: Training and test data release
- May 17, 2019: Pre-submission check
- May 31, 2019: Final submission of offline test results
- June 24, 2019: Top winners invited to present at challenge symposium
- July 17, 2019: Grand Challenge Symposium, AAPM 2019 Annual Meeting
- August 2019: Participants invited to co-author challenge report
- February 2020: Datasets made publicly available
Challenge Data
Data consist of PFTs, multi-inflation non-contrast CT (4D or breath-hold) and contrast-based ventilation images (nuclear imaging or hyperpolarised gas MRI) for patients with lung cancer and several non-oncological obstructive respiratory diseases including cystic fibrosis, asthma and COPD. The corresponding reference ventilation images for 50 CT scans will be provided for training. Several repeat CT datasets are also available for reproducibility analysis. In addition to the CT data, binary lung masks are provided for each inspiratory and expiratory scan. These may be used by participants in registering the scans if they wish to do so.
Table 1: Summary of CT and reference ventilation imaging training data included in grand challenge.
| Reference Ventilation Modality | Patients and disease | CT breathing manoeuvre | Reference breathing manoeuvre | Data contributors |
|---|---|---|---|---|
| 99mTc-DTPA SPECT ventilation | 21 lung cancer | 4D-CT | Free breathing | Dr Tokihiro Yamamoto (University of California-Davis, Davis, CA, USA) |
| 68Ga-aerosol PET ventilation | 25 lung cancer | 4D-CT | Free breathing | Dr Shankar Siva and Prof. Michael Hofman (Peter MacCallum Cancer Centre, Melbourne, Australia) |
| Xenon CT | 4 sheep | 4D-CT | Mechanical ventilation | Prof Joseph Reinhardt and Gary Christensen (University of Iowa, Iowa, USA) |
Abbreviations: DTPA = diethylenetriamine-pentaacetic acid; SPECT = single photon emission computerized tomography; PET = positron emission tomography; FRC = functional residual capacity; TLC = total lung capacity.
Table 2: Summary of CT and reference ventilation test data included in grand challenge.
| Reference Ventilation Modality | Patients and disease | CT breathing manoeuvre | Reference breathing manoeuvre | Data contributors |
|---|---|---|---|---|
| PFTs | 50 lung cancer | 4DCT | Dr Yevgeniy Vinogradskiy (University of Colorado, Denver, USA) |
|
| 68Ga-aerosol PET ventilation | 18 lung cancer | Breath-hold (End-expiration and inhalation) | Free breathing | Dr Enid Eslick (Royal North Shore Hospital, Sydney, Australia) |
| 68Ga-aerosol PET ventilation | 22 lung cancer (mid-RT) | 4D-CT | Free breathing | Dr Shankar Siva and Prof. Michael Hofman (Peter MacCallum Cancer Centre, Melbourne, Australia) |
| 68Ga-aerosol PET ventilation | 18 lung cancer (post-RT) | 4D-CT | Free breathing | Dr Shankar Siva and Prof. Michael Hofman (Peter MacCallum Cancer Centre, Melbourne, Australia) |
| 3He-MRI static ventilation | 32 lung cancer | 4D-CT | Breath-hold (FRC+1L) | Prof Grace Parraga, Dr Doug Hoover, Dr Brian Yaremko, Dr David Palma, Dr Stewart Gaede (Robarts Research Institute and London Health Sciences Centre, London, ON, Canada) |
| 3He-MRI static ventilation | 32 COPD | Breath-hold (End-expiration and inhalation) | Breath-hold (FRC+1L) | Prof Grace Parraga (Robarts Research Institute, London, ON, Canada) |
| 3He-MRI static ventilation | 26 ex-smokers | Breath-hold (End-expiration and inhalation) | Breath-hold (FRC+1L) | Prof Grace Parraga (Robarts Research Institute, London, ON, Canada) |
| 3He-MRI static ventilation | 12 asthma | Breath-hold (FRC and TLC) | Breath-hold (FRC+1L) | Prof Jim Wild and Prof Chris Brightling (The University of Sheffield, Sheffield, UK and The University of Leicester, Leicester, UK) |
| 3He-MRI static ventilation | 20 lung cancer | Breath-hold (FRC and FRC+1L) | Breath-hold (FRC+1L) | Dr Bilal Tahir, Prof Jim Wild and Prof Matthew Hatton (The University of Sheffield and Weston Park Cancer Hospital, Sheffield, UK) |
| 3He-MRI static ventilation | 19 paediatric cystic fibrosis | Breath-hold (End-expiration and inhalation) | Breath-hold (FRC+1L) | Prof Jim Wild and Dr David Hughes (The University of Sheffield and Sheffield Children's Hospital, Sheffield, UK) |
Abbreviations: DTPA = diethylenetriamine-pentaacetic acid; SPECT = single photon emission computerized tomography; PET = positron emission tomography; RT = radiation therapy; FRC = functional residual capacity; TLC = total lung capacity.
Table 3: Summary of CT reproducibility imaging data included in grand challenge.
| Patients and disease | CT breathing manoeuvre | Time Interval | Data contributors |
|---|---|---|---|
| 10 lung cancer | 4D-CT | Different session (23 mins – 7 days) | Prof Joe Reinhardt (University of Iowa, Iowa City, USA) |
| 18 lung cancer | 4D-CT | Same (n = 10) and different (n = 8) day | Dr Tokihiro Yamamoto (University of California-Davis, Davis, CA, USA) |
| 36 normal smokers and non-smokers | Breath-hold (TLC and FRC) | Same-session | Prof Eric Hoffman (University of Iowa, Iowa City, USA) |
Quantitative evaluation
CT ventilation images will be compared against the corresponding reference ventilation images or repeat CT ventilation images for all test datasets using the following evaluation metrics:
- Dice overlap coefficient
- Voxel-wise and ROI-based Spearman correlation
Moreover, the following global metrics will be computed from the CT ventilation images and correlated against PFTs:
- Percent ventilated volume
- Ventilation defect percent
- Coefficient of variation
Results, prizes and publication plan
At the conclusion of the challenge, the following information will be provided to each participant:
- The evaluation results for the submitted cases
- The overall ranking among the participants
The top 2 participants:
- Will present their algorithm and results at the annual AAPM meeting
- Will receive complimentary registration to the AAPM meeting
- Will receive a Certificate of Merit from AAPM
A manuscript summarizing the challenge results will be submitted for publication following completion of the challenge.
Terms and Conditions
The CTVIE19 challenge is organised in the spirit of cooperative scientific progress. The following rules apply to those who register a team and download the data:
- Anonymous participation is not allowed.
- Entry by commercial entities is permitted, but should be disclosed.
- Once participants submit their outputs to the CTVIE19 challenge organizers, they will be considered fully vested in the challenge, so that their performance results will become part of any presentations, publications, or subsequent analyses derived from the Challenge at the discretion of the organizers.
- The downloaded datasets or any data derived from these datasets, may not be given or redistributed under any circumstances to persons not belonging to the registered team.
- The full data, including reference data associated with the test set cases, is expected to be made publicly available after the publication of the CTVIE19 Challenge. Until the official public release of the data, data downloaded from this site may only be used for the purpose of preparing an entry to be submitted for the CTVIE19 challenge. The data may not be used for other purposes in scientific studies and may not be used to train or develop other algorithms, including but not limited to algorithms used in commercial products.
Organizers and Major Contributors
- Bilal Tahir (Lead organizer) (University of Sheffield and Weston Park Cancer Hospital, UK)
- John Kipriditis and Enid Eslick (Northern Sydney Cancer Centre Royal Sydney, Australia)
- Paul Keall (The University of Sydney, Australia)
- Grace Parraga (Robarts Research Institute, Canada)
- Alberto Biancardi, Joshua Astley, Michael Walker and Jim Wild (University of Sheffield)
- Dante Capaldi (Stanford University School of Medicine, USA)
- Doug Hoover, Brian Yaremko, and Stewart Gaede (London Health Sciences Centre, Canada)
- Joe Reinhardt and Eric Hoffman (University of Iowa, USA)
- Shankar Siva and Michael Hofman (Peter MacCallum Cancer Centre, Australia)
- Tokihiro Yamamoto (UC Davis School Medicine, USA);
- Yevgeniy Vinogradskiy (University of Colorado Denver, USA)
- Matthew Hatton (Weston Park Cancer Hospital, UK)
- Edward Castillo (Beaumont Health System, USA)
- Richard Castillo (Emory University, USA)
- John Bayouth (University of Wisconsin-Madison, USA)
- Samuel Armato and the AAPM Working Group on Grand Challenges
Contact
For further information, please contact the lead organizer, Dr Bilal Tahir.