Low Dose CT Grand Challenge

NIH Logo AAPM Logo Mayo Clinic Logo CT Clinical Innovation Center Logo

Objectives

The overall objective of this Low Dose CT Grand Challenge is to quantitatively assess the diagnostic performance of denoising and iterative reconstruction techniques on common low-dose patient CT datasets using a detection task, allowing the direct comparison of the various algorithms. The results will provide an indication of the range of performances achieved using different classes of denoising or iterative reconstruction techniques.

The specific objective for each participant is to reduce image noise using either image-domain or projection-domain techniques in order to maximize detectability of liver lesions in low-dose patient CT datasets. This will entail performing image-based denoising on low-dose patient CT datasets using images reconstructed using filtered backprojection methods, or performing image reconstruction on low-dose patient CT datasets using fully-preprocessed projection data and a noise map. The resultant noise-reduced images will be read in a blinded and randomized manner at the host institution. The reading task will be limited to detection of liver lesions. The three participants having the highest percentage correct scores, as defined below, will be recognized as winners of this grand challenge.

Patient Data

Patient images in the data library were retrospectively obtained from clinically-indicated examinations after approval by the institutional review board of the host institution (Mayo Clinic). All data that are shared will be fully anonymized.

The patient data library consists of contrast-enhanced abdominal CT examinations selected by the host institution. All data were obtained on similar scanner models (Somatom Definition AS+, or Somatom Definition Flash operated in single-source mode, Siemens Healthcare, Forchheim, Germany). These data were acquired using a flying focal spot technique, which will need to be handled in the reconstruction rebinning process. For information on the flying focal spot technique, see:

Following the routine clinical protocol of the host institution, data were acquired with use of automated exposure control (CareDose 4D, Siemens Healthcare) and automated tube potential selection (CarekV, Siemens Healthcare). The reference tube potential and quality reference effective mAs used by the automated tube potential and exposure control system were, respectively, 120 kVp and 200 effective mAs. Using a weight-based dosage and injection rate, iodinated contrast material was delivered according to the practice of the host institution. Scans were obtained 70 s after contrast injection (portal venous phase).

Cases included in the data library were either negative for findings in the liver or had focal liver lesions. Both benign and metastatic liver lesions were included. Potential cases were reviewed by a board-certified abdominal radiologist and excluded from the data library if there were more than 5 lesions or the lesion was greater than 3 cm. Reference data were gathered for each patient to provide a definitive diagnosis and establish the accuracy of host center’s diagnosis.

Noise Insertion to Simulate Reduced Dose Levels

Poisson noise was inserted into the projection data for each case in the library to reach a noise level that corresponded to 25% of the full dose (i.e. "quarter-dose" data were simulated). The projection data are from right before image reconstruction, i.e. after all preprocessing and taking the logarithm. For reconstruction algorithms requiring statistical information, a noise map, expressed as the noise-equivalent incident number of quanta for each data frame, will be provided.

Training Data

  • The same ten patient cases will be provided to each participant. These cases will be excluded from further analysis in the testing phase.
  • Training cases will include subtle and typical lesions, and at least one negative case. A range of patient sizes will be included in the training data set.
  • Data from the scan of an ACR CT accreditation phantom also will be provided. The phantom will be scanned and reconstructed using the same parameters given above, except that automated exposure control and tube potential systems will be shut off. The phantom data will be collected at 120 kV and 200 effective mAs and 50 effective mAs.
  • For the patient and phantom training data, full-dose and quarter-dose filtered back projection images with lesion locations marked will be provided, as will the corresponding projection data.
  • The DICOM data dictionary for the manufacturer-neutral projection data format (DICOM-CT-PD) developed at Mayo Clinic, and various Matlab reading tools, will be provided.

DICOM-CT-PD User Manual and Dictionary File


The DICOM-CT-PD user manual and DICOM dictionary file can be downloaded here. These files will assist participants in correctly reading and interpreting the provided projection data.

ACR Phantom Data (Projection and Image Data)


Download projection and image data for a helical scan of the ACR CT Accreditation Phantom here. These files will allow participants to prepare their code to read in projection data using the DICOM-CT-PD format. Images are provided to assist in verifying correct operation of participants’ reconstruction code.

Update (Dec 30, 2015): DICOM-CT-PD files have been updated. The Tag (0018,0061) in the header of the DICOM-CT-PD files now has a value of 0.0192 instead of 0.192.

Test Datasets

  • The same twenty test data sets will be provided to each participant and will include a range of patient sizes.
  • Normal cases and cases with 1 or more metastatic lesions will be included in the lesion enriched case cohort. The number of normal cases will not be provided.
  • Test data will consist of either the quarter-dose image data (in 1 or 3 mm image thicknesses) or projection data; no full dose data will be provided, and only 1- or 3-mm thick image data or the projection data will be provided, according to the preference of the participant. Both projection and image data will not be provided to the same participant.

Instructions to Participants

  • Participants will be provided the test data upon completion of the enrollment forms, but no earlier than January 4, 2016, and instructed to return the noise-reduced images to the host institution by midnight of April 30, 2016.
  • Data exchange will be supported by AAPM.
  • Collaboration or discussion among participants is not allowed.
  • The time required to reconstruct each case is to be recorded and returned in a provided data summary spreadsheet. The specifications of the computer(s) used to perform the reconstruction shall be reported in the provided spreadsheet.
  • Participants will be asked to describe their technique, or to provide a publication that describes it, and to answer provided questions about their methodology to allow proper categorization of the technique.
  • Final reconstructed axial images must be 3 mm in thickness and 2 mm in increment.
  • The reconstruction kernel sharpness should be appropriate for detection of soft tissue lesions in the liver.
  • Each site will return all images from all 20 test cases, as well as images from the quarter-dose ACR CT Accreditation Phantom data set (reconstructed using a 5 mm slice thickness and interval) to the host site via an AAPM server.

Registration

Registration for the Low Dose CT Grand Challenge is now closed.

Data Sharing Agreement


Please download the Data Sharing Agreement. This must be signed and a PDF copy returned to mccollough.cynthia@mayo.edu before the patient data sets can be released to a participating site.

Radiologist Interpretation at Host Site

  • The host site will provide radiologist interpretation of the twenty test cases (see Statistical Considerations, below).
  • The pool of readers will be comprised of senior residents, fellows, and faculty.
  • No reader will read the same case twice.
  • Cases from any given participant will be dispersed among readers so as to minimize the impact of reader bias on any one participant.
  • A standardized reading tool will be used for marking of the lesions.
  • Rigorous reader training will be performed to ensure consistent marking between readers.
  • For each case, the radiologist will be required to mark the location of any detected lesion, or to grade the case as normal if no lesions are detected.

Results and Prizes

  • At the conclusion of the challenge, the following information will be provided to each participant in appreciation for their involvement:
    • The reference data for each case (lesion type and location)
    • The per lesion and per case reader score for each case
    • The overall per lesion and per case percent correct
  • The three highest performing participants (see data analysis, below):
    • will be invited to present their algorithm and results at the 2016 AAPM Annual Meeting in a Low Dose CT Grand Challenge Scientific Symposium
    • will receive one complimentary registration to the Annual Meeting
    • will be invited to include an overview of their technique and results in a manuscript to be written after the completion of the Grand Challenge

Statistical Considerations

Reading design

  • Latin squares assignment: A full reading schedule for this study would have r readers, p participant submissions, and c cases (or r*p*c total reader impressions). Given the time constraints and the high potential of recall (e.g., c=20 cases shown with multiple, likely similar, denoising strategies or reconstruction techniques and read with limited washout time), we have designed a Latin squares reading framework. Figure 1 illustrates this framework. Each reader will see every case and every participant’s technique only once. If 20 participants enter the challenge, the total number of reader impressions for each reader will be 20. Should fewer than 20 entries be received, the readers will review less than 20 datasets while preserving the restriction that each submission and case is reviewed only once. Should more than 20 submissions be received, the reading list will be distributed equally over 2 or more readers (i.e., a “pseudo” reader will be formed by distributing the tasks over two or more real readers). Example Image of Latin SquareFigure 1: Latin Square example
  • The design assumes that readers will be exchangeable in performance. Differences in individual reader performance is assumed to be distributed uniformly across the participants’ submissions given the blocking imposed on the design.

Reader marks

  • Readers will be instructed to mark ROIs suspicious for lesions and assign a lesion level confidence for the primary task (detection of a hepatic metastasis). After reviewing the dataset, readers will assign an overall confidence for the task. These ratings will be preserved in the database and used for exploratory analyses and tie-breaking (see below). A standardized manual for marking and scoring will be reviewed with all readers and a rigorous training session held with readers to minimize reader variability.

Scoring plan

  • Reader lesion markings (or notation of the case as normal) will be compared to the reference standard for each patient and the data scored on a per lesion and per patient basis.
  • Reader markings will be considered correct if the location marked as center of the lesion falls anywhere within the true lesion’s boundaries.
  • Scores will be tabulated as follows:
    • Per lesion scoring (includes penalty for false positive and negative markings):
      • +1 for true positive marking of a lesion (correctly marking a lesion)
      • -1 for false positive marking of a lesion (no lesion exists at that location)
      • -1 for false negative (a lesion exists that was not marked)
    • Per case scoring (includes penalty for false positive and negative markings):
      • +1 for true negative case (no lesions marked in a case with no lesions)
      • +1 for true positive case (at least one lesion was correctly marked in a case with lesions)
      • -1 for false negative (no lesions marked in a case that had lesions)
      • -1 for false positive (at least one lesion marked in a case with no lesions)

Results summary

  • The per lesion normalized score (NS) = per lesion score / total number of lesions x 100%
  • The per case normalized score (NS) = per case score / 20 X 100%
  • In both cases, a perfect score = 100%
  • In both cases, false positive and false negative markings could result in a negative score
  • The overall performance score for each participant will be calculated as:
    [ [ per lesion NS ] + [ per case NS ] ] ÷ 2
  • A perfect score (all lesions and cases marked correctly) = 100%

Tie breaker

  • In the event 2 or more submissions receive the same overall performance score, the per lesion normalized score will be used as a tie breaker. If the per lesion score normalized scores are equal (which implies the per case normalized scores are also equal), JAFROC figure of merits taking into account reader confidence will be used.

Timeline

  • January 15, 2016 - Midnight: Registration closes
  • January 18, 2016: Training patient cases available
  • February 1, 2016: Test patient cases available
  • April 30, 2016 – Midnight: Submission of noise-reduced data sets closes
  • May 2016: Images read by radiologists at the host site
  • June 2016: Data analysis at the host site
  • July 2016: Winners contacted to plan symposium participation
  • August 2016: Winners announced at AAPM Annual Meeting