Researchers at the University of Sydney have identified a fundamental mathematical reason why artificial intelligence systems can learn different behaviours even when trained under almost identical conditions, a discovery that could help improve the reliability and trustworthiness of AI technologies.
The study by Professor Pulin Gong from the School of Physics and his PhD student Andrew Ly, has been published in Nature Machine Intelligence, one of the world’s leading journals in artificial intelligence research.
As AI systems become increasingly embedded in healthcare, business, research and everyday life, ensuring they behave consistently and predictably has become a major challenge. The new research suggests that some forms of AI unpredictability are not due to engineering flaws or inconsistencies computer hardware, but in the underlying mathematics of how neural networks learn.
“Researchers have often attributed differences in AI behaviour to external sources of randomness, such as the order in which training data is presented or tiny numerical variations introduced by hardware,” Professor Gong said.
“Our findings point to something deeper. Even when these factors are removed, extremely small differences in a model’s starting conditions can still lead to different outcomes.”
Neural networks, the technology behind today’s most advanced AI systems, learn by adjusting millions or even billions of internal parameters during training. The Sydney team investigated why networks sometimes arrive at different solutions despite using the same data, training procedure and deterministic computing environment.
Andrew Ly said: “Imagine two AI systems trained to detect cancer using the same medical images and the same training procedure. Tiny differences in their initial parameters could lead them to learn different behaviours. Our research helps explain how such differences arise and might inform methods for making these systems more consistent.”
Understanding the source of these differences could guide the design of more dependable systems and reduce the cost and uncertainty associated with developing large-scale models.
AI TRAINING ‘MAP’ IS A FRACTAL
To explain the phenomenon, the researchers use the analogy of a map. Imagine every possible starting point for training as a location on that map, coloured according to the outcome it eventually produces. Regions leading to the same outcome form what mathematicians call a “basin of attraction”.
The team found evidence that these basins are organised as highly intricate fractals known as “riddled basins”. In such a landscape, every point lies arbitrarily close to points leading somewhere else. No matter how far you zoom in, the different regions remain interwoven.
The implication is striking. Increasing the precision of a network’s starting conditions may do little to improve confidence about where training will end up. In one larger image-recognition network, making the starting settings 10 billion times more precise produced no measurable improvement in reproducibility. Altering that network’s starting settings by the smallest amount a computer can register, a single “bit”, sent training to a different internal configuration in close to half (about 48 percent) of cases.
To test the idea, the researchers trained approximately 500,000 neural networks, analysing systems used for tasks including image recognition and text processing. Evidence consistent with “riddled basins” fractal structure was found across a wide range of network architectures, datasets and training methods.
Importantly, the effect remained even in fully deterministic environments where repeated training runs would otherwise produce identical results.
MODEL HELPS EXPLAIN DIFFERENT AI OUTCOMES
The findings may help explain several longstanding observations in AI research, including why independently trained models often behave differently and why both large and tiny disturbances can sometimes have similar effects on outcomes.
The work also revealed a surprising trade-off between performance and reproducibility. The settings enabling networks to learn most effectively also made their outcomes harder to predict and reproduce. However, reducing the ‘learning rate’ – the size of the step a network takes each time it adjusts itself – made the results reproducible, but the networks learned less effectively.
Rather than viewing unpredictability as a problem that can be completely engineered away, the researchers argue it may represent a fundamental feature of how many neural networks operate. The study introduces new tools for measuring this sensitivity and points towards future approaches, such as redesigning network architectures to reduce the mathematical conditions that give rise to instability.
“AI is becoming increasingly important across society,” Professor Gong said. “To build trustworthy systems, we need not only accurate predictions but also confidence that those predictions can be reproduced. This research provides a new way of understanding the limits of predictability in AI, and how we might overcome them.”
