| Question 1: The Medical Imaging and Data Resource Center (MIDRC) is a multi-institutional initiative driven by the medical imaging community and aimed at accelerating the transfer of knowledge and innovation in the current COVID-19 pandemic. The aim of MIDRC is to foster machine learning innovation through data sharing for rapid and flexible collection, analysis, and dissemination of imaging and associated clinical data by providing researchers with unparalleled resources in the fight against COVID-19. Machine intelligence techniques in medical imaging include multiple aspects such as: |
| Reference: | Sahiner B, Pezeshk A, Hadjiiski LM, Wang X, Drukker K, Cha KH, Summers RM, Giger ML. Deep learning in medical imaging and radiation therapy. Medical Physics, 2019 Jan;46(1):e1-e36. doi: 10.1002/mp.13264. Epub 2018 Nov 20, 2019. |
| Choice A: | Object classification |
| Choice B: | Natural language processing |
| Choice C: | Human engineered features |
| Choice D: | Deep learning |
| Choice E: | All of above |
| Question 2: What attributes make a data commons desirable for developing robust, unbiased machine learning algorithms in medical imaging? |
| Reference: | Vyas DA, Eisenstein LG, Jones DS. Hidden in plain sight – Reconsidering the use of race correction in clinical algorithms. NEJM June 17 2020.
Langlotz CP, Allen B, Erickson BJ, et al. A Roadmap for Foundational Research on Artificial Intelligence in Medical Imaging: From the 2018 NIH/RSNA/ACR/The Academy Workshop. Radiology. 2019;291: 781–791 |
| Choice A: | Large number of imaging studies. |
| Choice B: | Imaging studies that span the various disease states and normal presentations. |
| Choice C: | Imaging studies from diverse populations. |
| Choice D: | Imaging studies obtained on various manufacturer systems. |
| Choice E: | All of the above. |
| Question 3: Which of the following tasks are supported using the DICOM image standard? |
| Reference: | A. Fedorov, R. Beichel, J. Kalpathy-Cramer, D. Clunie, M. Onken, J. Riesmeier, C. Herz, C. Bauer, A. Beers, J.-C. Fillion-Robin, A. Lasso, C. Pinter, S. Pieper, M. Nolden, K. Maier-Hein, M. D. Herrmann, J. Saltz, F. Prior, F. Fennessy, J. Buatti, and R. Kikinis, “Quantitative Imaging Informatics for Cancer Research.,” JCO Clinical Cancer Informatics, vol. 4, pp. 444–453, May 2020. |
| Choice A: | Display of medical images on a vendor-specific workstation. |
| Choice B: | Display of medical images from different manufacturers scanners on third-party workstations. |
| Choice C: | Recording region of interest (ROI) definitions and information for the ROI. |
| Choice D: | Recording structured information. |
| Choice E: | All of the above. |
| Question 4: The definition of image quality is: |
| Reference: | H. H. Barrett, J. Yao, J. P. Rolland, and K. J. Myers, “Model observers for assessment of image quality,” P Natl Acad Sci Usa, vol. 90, no. 21, pp. 9758–9765, Nov. 1993. |
| Choice A: | The Modulation Transfer Function (MTF). |
| Choice B: | Image noise from multiple realizations. |
| Choice C: | Image contrast to noise ratio. |
| Choice D: | How well the desired information can be extracted from the image. |
| Choice E: | All of the above. |
| Question 5: Which of the following would be considered a clinical task related to COVID-19 that would utilize image data: |
| Reference: | Rubin GD, Ryerson CJ, Haramati LB, et al. The Role of Chest Imaging in Patient Management During the COVID-19 Pandemic: A Multinational Consensus Statement From the Fleischner Society. Chest. 2020 Jul;158(1):106-116. doi: 10.1016/j.chest.2020.04.003. Epub 2020 Apr 7. |
| Choice A: | The Diagnosis of COVID-19 when a Rapid COVID-19 test is not available. |
| Choice B: | Monitoring a patient known to have COVID-19 for signs of worsening respiratory status. |
| Choice C: | Distinguishing COVID-19 from Community Acquired Pneumonia. |
| Choice D: | The diagnosis of a hospitalized patient with moderate to severe clinical features consistent with COVID-19, but with a negative COVID-19 test. |
| Choice E: | All of the above. |
| Question 6: When evaluating the performance of a machine learning system designed to detect the presence/absence of COVID-19 in a CXR (e.g. COVID-19: Yes or No) when the reference standard from PCR testing is available, an appropriate figure of merit would be: |
| Reference: | Metz, C. E., Herman, B. A. and Shen, J. H., “Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data,” Stat. Med. 17(9), 1033–1053 (1998) |
| Choice A: | Accuracy |
| Choice B: | Pearson’s correlation coefficient |
| Choice C: | Dice Coefficient |
| Choice D: | Area Under the Curve (AUC) from an ROC Analysis |
| Choice E: | Cost-benefit ratio. |
| Question 7: The FDA allows the use of performance data summarizing results of the testing of AI/ML algorithms on publicly available datasets in submissions for premarketing authorization. (T/F) |
| Reference: | Guidance for Industry and Food and Drug Administration Staff Computer-Assisted Detection Devices Applied to Radiology Images and Radiology Device Data - Premarket Notification [510(k)] Submissions https://www.fda.gov/media/77635/download |
| Choice A: | True |
| Choice B: | False |
| Question 8: In regulatory reviews of AI/ML algorithms, questions about the quality and representativeness of the test data sets occupy a central place. Data collected through a third party like MIDRC may streamline device review because: |
| Reference: | See above and Qualification of Medical Device Development Tools Guidance for Industry, Tool Developers, and Food and Drug Administration Staff https://www.fda.gov/media/87134/download |
| Choice A: | Test data sets will represent multiple collection sites, increasing the likelihood of generalizability of the results to real-world use. |
| Choice B: | Performance comparisons between different devices with similar indications for use will be less susceptible to variability of the test data. |
| Choice C: | If a data set goes through the FDA’s Medical Device Development Tool (MDDT) process, it can be used in a submission to the Agency without the need for extensive documentation during regulatory review. |
| Choice D: | All of the above. |
| Question 9: Grand challenges provide value to the medical imaging research community by: |
| Reference: | Armato SG III, Drukker K, Li F, Hadjiiski L, Tourassi GD, Engelmann RM, Giger ML, Redmond G, Farahani K, Kirby JS, Clarke LP: The LUNGx Challenge for computerized lung nodule classification. Journal of Medical Imaging 3: 044506-1–044506-9, 2016. |
| Choice A: | Making available a common set of images to all participating groups |
| Choice B: | Advertising a specific clinical need |
| Choice C: | Increasing the intellectual property value of participants’ methods |
| Choice D: | Connecting academic groups with industry partners |
| Question 10: Grand challenges require careful planning to ensure: |
| Reference: | Armato SG III, Hadjiiski L, Tourassi GD, Drukker K, Giger ML, Li F, Redmond G, Farahani K, Kirby JS, Clarke LP: The LUNGx Challenge for computerized lung nodule classification: Reflections and lessons learned. (invited) Journal of Medical Imaging 2: 020103-1–020103-5, 2015. |
| Choice A: | The collected data represent unbiased distributions of cases. |
| Choice B: | The rules of the challenge are clearly identified for participants |
| Choice C: | Training, validation, and test cases are provided in sufficient number |
| Choice D: | All of the above. |
