In search and advice techniques, predictive fashions typically endure from temporal instability when sure enter options introduce volatility in output scores. This instability can degrade mannequin reliability and person expertise particularly in multi-stage techniques the place constant predictions are crucial for downstream resolution making. We introduce Fortress, a common framework for enhancing mannequin stability and accuracy by figuring out and pruning options that contribute to inconsistent prediction scores over time. Fortress leverages historic snapshots temporally partitioned datasets capturing rating fluctuations for a similar entity throughout durations and follows a four-step course of: (1) accumulate historic snapshots, (2) determine samples with unstable predictions, (3) isolate and take away instability-inducing options, and (4) retrain fashions utilizing solely steady options. Whereas semantic options from LLMs and BERT-based fashions enhance generalization, they typically lack full question or entity protection. Engagement-based options supply sturdy predictive energy however are inclined to introduce temporal instability. Fortress mitigates this trade-off by suppressing the volatility of engagement alerts whereas retaining their predictive worth resulting in extra steady and correct fashions. We validate Fortress on a query-to-app relevance mannequin in a large-scale app market. Offline experiments reveal notable enhancements in prediction stability (measured by Coefficient of Variation) and classification efficiency (measured by PR-AUC).
- ** Work accomplished whereas at Apple
