Long-Term Risks in ML Systems

Machine learning systems don’t just make one-off decisions — they often operate in environments that change in response to those decisions. Over time, this back-and-forth can create feedback loops: the system’s outputs influence the world, and the resulting changes feed right back into the system.

Not all feedback loops are bad — in control systems, they’re essential for stability — but in socio-technical ML systems, certain self-reinforcing loops can spiral into harmful, hard-to-reverse states. As we wrote,

The decision of an ML-based system induces certain changes in the environment, which, in turn, influences the system’s future behaviors through its input.

Left unchecked, this cycle can amplify errors, entrench bias, degrade safety, and cause long-lasting harm to people and society.

Consider predictive policing. If a model predicts a particular neighborhood has high crime, more patrols are sent there, leading to more recorded arrests, which the model interprets as even higher crime. The same pattern shows up in other domains — loan approvals affecting credit scores, or medical risk scoring influencing treatment access — where each decision subtly shapes the environment, sometimes with devastating cumulative effects.

Our early work (Towards Safe ML-Based Systems in Presence of Feedback Loops, SE4SafeML 2023) made the case that these loops should be treated as first-class design concerns. We introduced a conceptual framework for modeling how ML systems, decision policies, and dynamic environments interact over time, allowing developers to reason about questions like:

• What feedback patterns could emerge?
• How might they affect safety, fairness, utility, or other critical properties?
• Which interventions could break a harmful cycle?

Building on that foundation, our ICSE 2025 paper (FairSense: Long-Term Fairness Analysis of ML-Enabled Systems) presented FAIRSENSE — a simulation-based framework to study these long-term dynamics before deployment. While FAIRSENSE can evaluate fairness, its real power is in exploring any evolving system property. It runs Monte Carlo simulations to generate possible futures, then uses sensitivity analysis to pinpoint which design or environmental factors most influence the trajectory. This means we can identify the small number of parameters that truly matter, monitor them closely, and design targeted interventions.