| Question 1: Which one of the following is the least reliable measure for clinical commissioning of AI auto-segmentation method? |
| Reference: | 1. Valentini V, Boldrini L, Damiani A, Muren LP: Recommendations on how to establish evidence from auto-segmentation software in radiotherapy. Radiother Oncol 112:317-20, 2014.
2. Vaassen F, Hazelaar C, Vaniqui A, Gooding M, van der Heyden B, Canters R, van Elmpt W: Evaluation of measures for assessing time-saving of automatic organ-at-risk segmentation in radiotherapy. Physics and Imaging in Radiation Oncology 13:1-6, 2020 |
| Choice A: | Dice similarity coefficient |
| Choice B: | Surface distance |
| Choice C: | User editing times |
| Choice D: | DVH derived metrics |
| Question 2: Which of the following are reasonable approaches to deal with few labeled datasets for CBCT? |
| Reference: | 1. Jia X, Wang S, Liang X, Balagopal A, Nguyen D, Yang M, Wang Z, Ji J.X, Qian X, Jiang S. Cone-Beam Computed Tomography (CBCT) Segmentation by Adversarial Learning Domain Adaptation. In: Shen D. et al. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. MICCAI 2019.
2. Zhou B, Augenfield J, Chapiro S, Zhou K, Liu C, Duncan J.S. Anatomy-guided multimodal registration by learning segmentation without ground truth: Application to intraprocedural CBCT/MR liver segmentation and registration”, Medical Image Analysis. 2021.
3. Jiang J and Veeraraghavan H. One shot PACS: Patient specific anatomic context and shape prior aware recurrent registration-segmentation of longitudinal cone beam CTs. IEEE TMI 2022. |
| Choice A: | Cross-domain adaptation combining CT with CBCT |
| Choice B: | Multi-tasked learning |
| Choice C: | Registration based segmentation |
| Choice D: | All of the above |
| Question 3: What are sources of bias that could affect AI technologies? |
| Reference: | McIntosh C, Conroy L, Tjong MC, et al. Clinical integration of machine learning for curative-intent radiation treatment of patients with prostate cancer. Nat Med 2021;27:999–1005. doi:10.1038/s41591-021-01359-w |
| Choice A: | Limited training dataset |
| Choice B: | Variation in testing settings |
| Choice C: | Lack of clinical oversight in algorithm development |
| Choice D: | Limited enduser involved in training and/or testing |
| Choice E: | All of the above |
| Question 4: What are the advantage(s) of data curation? |
| Reference: | Conroy L, Khalifa A, Berlin A, et al. Performance stability evaluation of atlas-based machine learning radiation therapy treatment planning in prostate cancer. Phys Med Biol 2021;66:134001. doi:10.1088/1361-6560/abfff0 |
| Choice A: | Reduce dataset size |
| Choice B: | Ensure dataset is highly population-specific |
| Choice C: | Reduce impact of outliers and imaging artifacts |
| Choice D: | None of the above |
| Choice E: | A + B + C |
| Question 5: Trustworthiness as applied to AI can be defined as: |
| Reference: | Cutillo CM, Sharma KR, Foschini L, et al. Machine intelligence in healthcare—perspectives on trustworthiness, explainability, usability, and transparency. npj Digit Med 2020;3:47. doi:10.1038/s41746-020-0254-2 |
| Choice A: | Assess validity and reliability of AI derived outputs |
| Choice B: | Understand and evaluate inner working of AI algorithms |
| Choice C: | Achieve specific goals in the defined setting |
| Choice D: | Understand inputs that could influence outputs |
| Question 6: What are barriers to NLP being used clinically? |
| Reference: | Danielle S Bitterman 1, Timothy A Miller 2, Raymond H Mak 3, Guergana K Savova 2. Clinical Natural Language Processing for Radiation Oncology: A Review and Practical Primer. Int J Radiat Oncol Biol Phys. 2021 Jul 1;110(3):641-655. doi: 10.1016/j.ijrobp.2021.01.044. Epub 2021 Feb 3. |
| Choice A: | Methods are not mature enough for clinic use |
| Choice B: | Insufficient training data available in hospital systems |
| Choice C: | Lack of collaboration/guidance on high priority tasks |
| Choice D: | Classification performance is subjective for language tasks |
