Authors: Augusto Cerqua, Marco Letta, Gabriele Pinto
studying (ML) has gained a central position in economics, the social sciences, and enterprise decision-making. Within the public sector, ML is more and more used for so-called prediction coverage issues: settings the place policymakers purpose to establish items most susceptible to a adverse consequence and intervene proactively; for example, focusing on public subsidies, predicting native recessions, or anticipating migration patterns. Within the non-public sector, related predictive duties come up when companies search to forecast buyer churn, or optimize credit score danger evaluation. In each domains, higher predictions translate into extra environment friendly allocation of sources and simpler interventions.
To realize these targets, ML algorithms are more and more utilized to panel information, characterised by repeated observations of the identical items over a number of time durations. Nonetheless, ML fashions weren’t initially designed to be used with panel information, which function distinctive cross-sectional and longitudinal dimensions. When ML is utilized to panel information, there’s a excessive danger of a delicate however major problem: information leakage. This happens when info unavailable at prediction time by accident enters the mannequin coaching course of, inflating predictive efficiency. In our paper “On the Mis(Use) of Machine Studying With Panel Knowledge” (Cerqua, Letta, and Pinto, 2025), just lately revealed within the Oxford Bulletin of Economics and Statistics, we offer the primary systematic evaluation of knowledge leakage in ML with panel information, suggest clear pointers for practitioners, and illustrate the implications by way of an empirical utility with publicly accessible U.S. county information.
The Leakage Drawback
Panel information mix two constructions: a temporal dimension (items noticed throughout time) and a cross-sectional dimension (a number of items, equivalent to areas or companies). Customary ML observe, splitting the pattern randomly into coaching and testing units, implicitly assumes unbiased and identically distributed (i.i.d.) information. This assumption is violated when default ML procedures (equivalent to a random break up) are utilized to panel information, creating two primary sorts of leakage:
- Temporal leakage: future info leaks into the mannequin through the coaching section, making forecasts look unrealistically correct. Moreover, previous info can find yourself within the testing set, making ‘forecasts’ retrospective.
- Cross-sectional leakage: the identical or very related items seem in each coaching and testing units, that means the mannequin has already “seen” a lot of the cross-sectional dimension of the information.
Determine 1 reveals how totally different splitting methods have an effect on the chance of leakage. A random break up on the unit–time degree (Panel A) is probably the most problematic, because it introduces each temporal and cross-sectional leakage. Alternate options equivalent to splitting by items (Panel B), by teams (Panel C), or by time (Panel D), mitigate one kind of leakage however not the opposite. Because of this, no technique utterly eliminates the issue: the suitable selection depends upon the duty at hand (see beneath), since in some circumstances one type of leakage might not be an actual concern.
Determine 1 | Coaching and testing units underneath totally different splitting guidelines
Two Varieties of Prediction Coverage Issues
A key perception of the examine is that researchers should clearly outline their prediction objective ex-ante. We distinguish two broad courses of prediction coverage issues:
1. Cross-sectional prediction: The duty is to map outcomes throughout items in the identical interval. For instance, imputing lacking information on GDP per capita throughout areas when just some areas have dependable measurements. The very best break up right here is on the unit degree: totally different items are assigned to coaching and testing units, whereas all time durations are saved. This eliminates cross-sectional leakage, though temporal leakage stays. However since forecasting just isn’t the objective, this isn’t an actual challenge.
2. Sequential forecasting: The objective is to foretell future outcomes primarily based on historic information—for instance, predicting county-level earnings declines one yr forward to set off early interventions. Right here, the right break up is by time: earlier durations for coaching, later durations for testing. This avoids temporal leakage however not cross-sectional leakage, which isn’t an actual concern because the similar items are being forecasted throughout time.
The improper strategy in each circumstances is the random break up by unit-time (Panel A of Determine 1), which contaminates outcomes with each sorts of leakage and produces misleadingly excessive efficiency metrics.
Sensible Tips
To assist practitioners, we summarize a set of do’s and don’ts for making use of ML to panel information:
- Select the pattern break up primarily based on the analysis query: unit-based for cross-sectional issues, time-based for forecasting.
- Temporal leakage can happen not solely by way of observations, but in addition by way of predictors. For forecasting, solely use lagged or time-invariant predictors. Utilizing contemporaneous variables (e.g., utilizing unemployment in 2014 to foretell earnings in 2014) is conceptually improper and creates temporal information leakage.
- Adapt cross-validation to panel information. Random k-fold CV present in most ready-to-use software program packages is inappropriate, because it mixes future and previous info. As an alternative, use rolling or increasing home windows for forecasting, or stratified CV by items/teams for cross-sectional prediction.
- Be certain that out-of-sample efficiency is examined on actually unseen information, not on information already encountered throughout coaching.
Empirical Software
As an instance these points, we analyze a balanced panel of three,058 U.S. counties from 2000 to 2019, focusing solely on sequential forecasting. We contemplate two duties: a regression drawback—forecasting per capita earnings—and a classification drawback—forecasting whether or not earnings will decline within the subsequent yr.
We run a whole bunch of fashions, various break up methods, use of contemporaneous predictors, inclusion of lagged outcomes, and algorithms (Random Forest, XGBoost, Logit, and OLS). This complete design permits us to quantify how leakage inflates efficiency. Determine 2 beneath stories our primary findings.
Panel A of Determine 2 reveals forecasting efficiency for classification duties. Random splits yield very excessive accuracy, however that is illusory: the mannequin has already seen related information throughout coaching.
Panel B reveals forecasting efficiency for regression duties. As soon as once more, random splits make fashions look much better than they are surely, whereas appropriate time-based splits present a lot decrease, but sensible, accuracy.
Determine 2 | Temporal leakage within the forecasting drawback
Panel A – Classification activity
Panel B – Regression activity
Within the paper, we additionally present that the overestimation of mannequin accuracy turns into considerably extra pronounced throughout years marked by distribution shifts and structural breaks, such because the Nice Recession, making the outcomes notably deceptive for coverage functions.
Why It Issues
Knowledge leakage is greater than a technical pitfall; it has real-world penalties. In coverage functions, a mannequin that appears extremely correct throughout validation could collapse as soon as deployed, resulting in misallocated sources, missed crises, or misguided focusing on. In enterprise settings, the identical challenge can translate into poor funding selections, inefficient buyer focusing on, or false confidence in danger assessments. The hazard is very acute when machine studying fashions are supposed to function early-warning programs, the place misplaced belief in inflated efficiency may end up in pricey failures.
In contrast, correctly designed fashions, even when much less correct on paper, present trustworthy and dependable predictions that may meaningfully inform decision-making.
Takeaway
ML has the potential to rework decision-making in each coverage and enterprise, however provided that utilized accurately. Panel information supply wealthy alternatives, but are particularly susceptible to information leakage. To generate dependable insights, practitioners ought to align their ML workflow with the prediction goal, account for each temporal and cross-sectional constructions, and use validation methods that forestall overoptimistic assessments and an phantasm of excessive accuracy. When these ideas are adopted, fashions keep away from the lure of inflated efficiency and as an alternative present steering that genuinely helps policymakers allocate sources and companies make sound strategic decisions. Given the speedy adoption of ML with panel information in each private and non-private domains, addressing these pitfalls is now a urgent precedence for utilized analysis.
References
A. Cerqua, M. Letta, and G. Pinto, “On the (Mis)Use of Machine Studying With Panel Knowledge”, Oxford Bulletin of Economics and Statistics (2025): 1–13, https://doi.org/10.1111/obes.70019.
