| Question 1: Which of the following is NOT true about calibration, discrimination and clinical utility? |
| Reference: | Ewout W. Steyerberg, et al. Assessing the performance of prediction models: a framework for some traditional and novel measures, Epidemiology (Cambridge, Mass). 2010;21(1):128-138. doi:10.1097/EDE.0b013e3181c30fb2. |
| Choice A: | A prediction model with a high discrimination will be clinically useful, unless it is miscalibrated. |
| Choice B: | Calibration and discrimination are important to help statisticians build better models, but they don’t tell us much about whether a prediction model is of value. |
| Choice C: | Calibration is what is most important to the patient. |
| Choice D: | Decision analytic methods are needed to help us determine whether a model is of clinical value. |
| Choice E: | Discrimination is a property of the predictors, not the model. |
| Question 2: You are analyzing a training set to create a model that you intend to test on external validation sets. Which of the following is true:
|
| Reference: | Ewout W. Steyerberg, et al. Assessing the performance of prediction models: a framework for some traditional and novel measures, Epidemiology (Cambridge, Mass). 2010;21(1):128-138. doi:10.1097/EDE.0b013e3181c30fb2. |
| Choice A: | Machine learning methods will be superior because they can take into account complex interactions and multiple variables. |
| Choice B: | You should feel free to change the resulting model if you feel like it. |
| Choice C: | You should feel free to modify the estimated discrimination if you feel like it. |
| Choice D: | It is critical to assess calibration. |
| Choice E: | It isn’t important dot every i and cross every t on the model, because you can always change it
so that it fits the validation set better. |
| Question 3: Which of the following is true about external validation |
| Reference: | Ewout W. Steyerberg, et al. Assessing the performance of prediction models: a framework for some traditional and novel measures, Epidemiology (Cambridge, Mass). 2010;21(1):128-138. doi:10.1097/EDE.0b013e3181c30fb2. |
| Choice A: | If a model is externally validated on a cohort from one hospital, the results only apply to patients from that hospital. |
| Choice B: | Once a model has been externally validated on at least one cohort, it can be widely implemented |
| Choice C: | “There is no sell-by date on truth”: once a model has been validated, it is applicable to similar
cohorts for the foreseeable future. |
| Choice D: | Application of the results of an external validation study to a new cohort depends on careful consideration of how the new cohort compares to the cohort in the study. |
| Choice E: | If the model does not work well on external validation, changes can be made to the coefficients or intercept: if the model then works better, the new model can be considered validated. |
| Question 4: The highest level of validation for a predictive model is from: |
| Reference: | Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMC medicine. 2015 Dec;13(1):1. |
| Choice A: | Testing against an adequate dataset from an outside group (external validation). |
| Choice B: | Testing against data that is set aside from the original dataset before building the model (internal validation.) |
| Choice C: | Testing the consistency of model building by repeating the model building process on bootstrap resamples. |
| Choice D: | Testing the predictive power of model fits by holding out data samples and testing the model on the fraction of data held out (cross-validation.) |
| Question 5: The problem of multiple comparisons is critical in predictive model building. Strategies to avoid over optimistic estimates of predictive power include: |
| Reference: | Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the royal statistical society. Series B (Methodological). 1995 Jan 1:289-300. |
| Choice A: | Establishing a statistical analysis plan prior to seeing the data. |
| Choice B: | Including variable selection inside of the cross validation model fitting loop. |
| Choice C: | In selecting variables for multi variable model fitting, adjusting p values using a false discovery rate approach. |
| Choice D: | All of the above. |
| Question 6: The p value is: |
| Reference: | Biau, D.J., et al. P value and the theory of hypothesis testing: an explanation for new researchers". Clin Orthop Relat Res. 463 (3): 885–892. doi:10.1007/s11999-009-1164-4. |
| Choice A: | Used in the context of null hypothesis testing. |
| Choice B: | Helps to determine the significance of your results. |
| Choice C: | The level of marginal significance within a statistical hypothesis test representing the probability of the occurrence of a given event. |
| Choice D: | All of the above. |
