The American Association of Physicists in Medicine (AAPM) will host a two part “Grand Challenge” on the auto-contouring of MRI for radiotherapy (RT-MRI Auto-Contouring). The RT-MAC Challenge will provide a unique opportunity for participants to compare their algorithms with those of others from academia, industry, and government in a structured, direct way using the same data sets.
A session at the 2019 AAPM Annual Meeting will focus on the RT-MAC Challenge; an individual from each of the two top-performing teams will receive a waiver of the meeting registration fee in order to present their methods during this session. Challenge participants are encouraged to submit their work for peer review to the AAPM’s scientific journal Medical Physics.
Release date of training set cases with truth: May 03, 2019
Release date of test set cases without truth: May 03, 2019
AAPM Early-bird registration ends: May 29, 2019
Submission opens for participants’ test set: June 01, 2019
Submission close for participants’ test set: June 21, 2019
Challenge results released to participants: June 28, 2019
Onsite challenge during AAPM Annual Meeting: July 14 – 18, 2019
To participate in the RT-MAC Challenge, please go the Challenge website.
RT-MAC Challenge Format
The database for this challenge contains a total of 55 MRI cases, each from a single examination from a distinct patient, with each case consisting of a T2-weighted images in DICOM format. 35 of these will be provided as training cases, with the parotid glands, submandibular glands, level 2 and level 3 lymph nodes contoured. Contours will be provided in RTSS format:
Participants will create automatic segmentation results for these structures on the test sets (10 for pre-AAPM challenge and 10 for online challenge)
Participants may use the training set cases in any manner they would like for the purpose of training their systems; there will be no restrictions on the use of the data or the advice sought from local experts for training purposes. The test set cases, however, are to be manipulated, processed, and analyzed without human intervention.
Participants are free to download the training set and, subsequently, the test set when these datasets become available. It is important to note that once participants submit their contours on the test set to the challenge organizers via the challenge website, they will be considered fully vested in the Challenge, so that their performance results (without identifying the participant unless permission is granted) will become part of any presentations, publications, or subsequent analyses derived from the Challenge at the discretion of the organizers.
Data Availability
Release of the training set (with truth): Apr 15, 2019 (tentative)
The training set will consist of 35 MRI with corresponding RT structures sets. This dataset will be representative of the technical properties (scanner type, acquisition parameters, file format) and the contours of the test set. Data will be in DICOM format. Contours will be provided for parotid glands, submandibular glands, level 2 and level 3 lymph nodes.
Release of the test set (without truth): Apr 15, 2019 (tentative)
The off-line test set will consist of 10 MRI cases with no ground truth contours. The test sets must be contoured by the participants for all of the structures in the test set. Contours will need to be submitted in DICOM RTSS format.
Participants should submit their contours for the off-line test set through the challenge web interface by 11:59 PM Pacific Standard Time on May 15, 2019.
Scoring for the Challenge test set results:
Auto-segmented contours will be compared against the manual contours for all test datasets using the following evaluation metrics as implemented in Plastimatch. RTSS will be voxelised to CT resolution for all calculations. Evaluation will be performed in 3D using DICE similarity coefficient, Average Distance, and 95% Hausdorff distance.
Different organs and measures will have different ranges of scores, therefore it is not possible to simply average them to get an overall score. Therefore to be able to normalise the scores with respect to expected values 3 cases have been contoured by multiple observers. The mean score of these observers will be used as a reference score against which submitted contours will be compared. For any organ/measure a perfect value (Dice = 1, AD/HD =0) will be scored 100. A value equivalent to the average inter-observer reference will be given a score of 50. A linear scale will be used to interpolate between these values, and extrapolate beyond them, such that a score of 0 will be given to any result below the reference by more than the perfect score is above the reference.
Score = max ( 50 + ( (T-R)/(P-R) * 50 ), 0 )
Where T is the test contour measure, P is the perfect measure, and R is the reference measure for that organ/measure.
For example, given a reference Dice of 0.85; a test contour with a Dice of 0.9 against the "ground truth" will score 66.6, where as a test contour with a Dice of 0.72 against the "ground truth" would score 7.
The normalized scores for all organs, measure and test cases will be averaged (mean) to give a final score. The overall winner will be the team with the highest final score.
Conflict of Interest:
Participants affiliated with the labs of any of the RT-MAC organizers or major contributors must participate independently of the organizer or contributor. Participants belonging to an affiliated lab of a major contributor or organizer are required to disclose their affiliation.
Organizers and Major Contributors:
- Greg Sharp, Massachusetts General Hospital
- Jinzhong Yang, MD Anderson
- Mark Gooding, Mirada Medical Ltd
- Carlos Cardenas, MD Anderson
- Abdallah Mohamed, MD Anderson
- Harini Veeraraghavan, MSKCC
- Jayashree Kalpathy-Cramer, Harvard University
- Andrew Beers, Harvard University