Question
A company has deployed an XGBoost prediction model in production to predict if a customer is likely to cancel a subscription. The company uses Amazon SageMaker Model Monitor to detect deviations in the F1 score.
During a baseline analysis of model quality, the company recorded a threshold for the F1 score. After several months of no change, the model's F1 score decreases significantly.
What could be the reason for the reduced F1 score?
- A. Concept drift occurred in the underlying customer data that was used for predictions. Correct
- B. The model was not sufficiently complex to capture all the patterns in the original baseline data.
- C. The original baseline data had a data quality issue of missing values.
- D. Incorrect ground truth labels were provided to Model Monitor during the calculation of the baseline.
Explanation
The correct answer is A because concept drift refers to changes in the underlying patterns of the data over time, which can lead to a decrease in model performance metrics like the F1 score. This aligns with AWS documentation on monitoring and maintaining machine learning models where concept drift is recognized as a common issue affecting model accuracy over time.
