Reinforcement Learning (RL) is the branch of machine learning where an 'agent' (the actor, in an environment) learns by trying an action, observing the result, and adjusting its behaviour accordingly to achieve its goal. It is not unlike teaching a dog a new trick: the dog hears a command, tries something, and a treat either arrives or it does not, and over enough repetitions the dog works out which action gets the reward. This creates a hard engineering problem, because the agent has to decide how much to stick with actions that are already working versus how much to keep testing other options. This is a concept known as the 'exploration-exploitation trade-off'. Understandably, when you don't know what you don't know, it's tricky knowing how well you're doing and if you should change tactics. Too much exploitation and the agent settles on a decent but not great solution (or never finds a solution), while too much exploration means it never actually makes use of what it has learned.
Traditionally, most artificial intelligence (AI) training setups handle this trade-off with a fixed schedule, where the agent explores a lot early on and progressively less as training continues, following a curve that is decided before training even starts. The reasoning behind this is that early on, the agent knows almost nothing, so wide exploration is worth the cost, and later on it has presumably learned enough that it makes more sense to exploit that knowledge and make more careful, refined decisions. It is the same shape of reasoning behind learning rate decay elsewhere in machine learning, where a model takes large update steps early in training and progressively smaller ones as it converges. This approach is simple and remains the industry default for good reason, but it has one significant limitation: it does not take into account what the agent is actually experiencing during training and instead adjusts purely based on elapsed time.
A separate strand of AI research, explainable AI, focuses on making models interpretable to the humans overseeing them, and in the context of RL this means answering questions such as why an agent chose a particular action or how confident it was in that decision. These tools are useful for auditing agent behaviour, but they are built to answer those questions only after training has finished, in the same way that a flight recorder reveals what happened during a flight only once the flight is already over.
In a recent piece of research at Airbus, we examined what would happen if the interpretability findings were plugged straight back into the agent's training loop, instead of simply logging them for human review after the agent has attempted to reach its goal. Essentially, the idea was to let the agent act on the information it was already generating about itself (e.g., confidence, surprise, sense of risk). In practice, the agent computes a handful of simple, low-cost signals about its own performance as training proceeds, including how much its expectations are being violated by what actually happens and how large the gap is between its best-valued action and its worst-case one (essentially, how much it stands to lose if it picks wrong). These signals are then used to adjust how much the agent explores at that particular point in training, replacing the fixed schedule with an approach that responds to what is actually happening rather than to how much time has passed.
When we tested this on a couple of very basic training environments (at the level of basic, single objective games), the best-performing configurations we found trained faster and more consistently than the fixed-schedule baseline. However, not every combination of signals helped, and which one worked best changed depending on the environment and algorithm. The more interesting result was in the shape of the exploration itself.
It did not taper off smoothly from high to low in the way a fixed schedule assumes, but oscillated in waves instead. The agent would settle into exploiting what it already knew, encounter a state that surprised it or looked risky, increase its exploration sharply at that point, and then settle back down again once conditions stabilised, before the cycle repeated at the next point of genuine uncertainty. This behaviour was not programmed into the system or agent; it emerged from the agent reading its own uncertainty as training went along. The pattern was seen consistently across several different starting points (called 'seeds'), each showing the same up-and-down rhythm on its own, so it appears to be a real, repeatable pattern of the agent's learning rather than a fluke or noise.
The exploration coefficient, labelled 'entropy coefficient' on the chart, plotted over the course of training. Riskiness is the specific signal driving the adjustment here, the same "gap between the best-valued action and its worst-case one" described above. The test itself was run on CartPole, a simple task where the agent balances a pole by moving a cart side to side, using PPO, one of the two training algorithms used in the study. Labels like 'Function 1' and the different seed numbers refer to specific experiment settings and can be ignored here. What matters is that the coefficient rises and falls repeatedly rather than decaying smoothly, as the agent responds to risk in real time. Source: Quintana-Amate et al., "Explain to Gain: Introspective Reinforcement Learning for Enhanced Performance," Fig. 7.
It is a slightly odd thing to picture at first, a machine's willingness to search rising and falling repeatedly, and following a seemingly complex strategy rather than a simple rule like a fixed schedule. But it makes sense once you think about what a fixed schedule is actually assuming: that the value of exploring is highest at the start and lowest at the end, when in fact the value of exploring at any given moment depends on how well the agent currently understands the specific situation it is in, and that understanding does not improve at a constant rate. Some parts of a task stay genuinely uncertain much longer than others, and some previously well-understood parts can become uncertain again if the situation shifts. A schedule fixed in advance has no way to know any of that, whereas a system reading its own uncertainty in real time does (though the same research also found the agent can sometimes be too quick to react, adding some variance).
It doesn't take a wild imagination to then try and project this research from an agent in a simulated environment onto a human in the real world, and see what sticks and what doesn't. What this research suggests, at least for the simple systems it was tested on, is that steady progress does not actually work by picking a strategy and never changing your mind, but rather by adapting to your external and internal environments. It looks more like a series of local decisions to search harder exactly when something stops making sense, and to stop again once it does.
One thing that came to mind, personally, is how quotes, sayings, or proverbs like "actions speak louder than words" sometimes make us roll our eyes and glaze over, yet other times the same one may strike a chord and give us goosebumps. For example, you might hear "a bird in the hand is worth two in the bush", "if it ain't broke, don't fix it", and "a rolling stone gathers no moss" when you're being encouraged to stay put and stick with what you or others know works. Conversely, you may hear "nothing ventured, nothing gained" and "fortune favours the bold" when you're being encouraged to give an idea a shot, take a risk, or try something new.
These are all well-known and commonly repeated sayings, but they are contradictory: a decision to explore is a decision not to exploit, and vice versa. Moreover, our mood and state of life can affect which ones resonate. Sometimes the exploration-encouraging ones will stir more within us, and sometimes the exploitation ones will stir more within us. At the risk of making an apophenic observation, and in admission of the very basic environments and agentic behaviour tested in the research, this might seem like a loose, human-scale echo of the same oscillation observed in the agent. Sometimes, the unquantifiable conditions and parameters of our lives, like ripples propagating across a lake, may converge to produce a peak or a trough; an urge to explore or exploit.
Perhaps quite embarrassingly, one of the most powerful encapsulations I have read of this exploration-exploitation trade-off was a top comment on a YouTube video of someone who takes possession of, and begins the first of many years of repairing, a 1903 stone barn in the Italian Alps. The comment, from @jonathansteinhauser6760, says "This series has me again asking myself the question of whether it's more enriching for one's soul to see 1000 different places or to see one place, in a thousand different lights and a thousand different moods on a thousand different days and a thousand different nights."
References
- Capaldo, P., Ketheeswaran, V., Quintana-Amate, S., Stevens, D., Hall, M. (2026). Explain to Gain: Optimising Performance Through Explainable Reinforcement Learning Parameter Investigation. In: Marcelloni, F., Madani, K., van Stein, N., Filipe, J. (eds) Computational Intelligence. IJCCI 2025. Communications in Computer and Information Science, vol 2829. Springer, Cham. https://doi.org/10.1007/978-3-032-15638-9_4
- Quintana-Amate, S., Stevens, D., Ketheeswaran, V., Capaldo, P., Sheldon, D., Hall, M. (2026). Explain to Gain: Introspective Reinforcement Learning for Enhanced Performance. In: Guidotti, R., Schmid, U., Longo, L. (eds) Explainable Artificial Intelligence. xAI 2025. Communications in Computer and Information Science, vol 2577. Springer, Cham. https://doi.org/10.1007/978-3-032-08324-1_1
- Martijn Doolaard. (2021, October 19). #1 First days at my cabin in the Italian Alps [Video]. YouTube. youtube.com