Coordination between artificial delegates
We are entering an era in which autonomous agents - software systems capable of perceiving, deciding, and acting on our behalf - are quietly becoming part of our society’s invisible infrastructure. Agentic AI now handles everything from booking our dinner reservations and routing delivery drones to balancing energy loads and orchestrating warehouse fleets.
Unlike traditional software, these agentic AI systems operate with goals, autonomy, and initiative. They’re not just reactive tools ingesting data; they also decide on actions to take. That decision-making should reflect the interests of their user or owner. And while this may seem obvious, it brings with it a subtle but serious challenge: what happens when many individually rational agents, each optimising for their own user, must operate in conflict-of-interest situations where individual profit is at stake with a positive collective outcome for the group?
In a joint PhD between DECIDE and prof. Lenaerts of VUB, Inês Terrucha has investigated the question of delegation of conflict-of-interest decisions to autonomous agents.
When smart agents face a social dilemma
Balancing self-interest and collective welfare is the essence of a social dilemma. Many of the systems that underpin modern life rely on public goods. A public good is a shared resource that benefits everyone but requires collective responsibility to sustain. When agentic AI systems are deployed to represent individuals or organizations, and each of them acts solely in their own best interest, they risk undermining the public good they depend on.
These situations are not theoretical. Agentic AI systems face such public-goods dilemmas in several emerging domains:
- Smart meters. AI agents decide when to run the dishwasher, charge a home battery, or top up an EV. If every household agent chooses the same “cheap” time slot—say, the moment commuters plug in after work—the local grid can overload. Similar coordination pitfalls plague other parts of the electricity system.
- Autonomous cars. Self‑driving vehicles that minimise their own travel time may refuse to yield at merges, exacerbating congestion and even causing stop‑and‑go waves (PNAS 2019).
- Data pods (e.g. SOLID). Delegate agents can grant or deny access to personal data case‑by‑case. Hoarding data preserves privacy or competitive edge, yet collective openness is crucial for breakthroughs—think drug discovery or public‑health modelling.
In each of these applications, what is locally optimal for one agent depletes a shared public good: a stable grid, smooth traffic, or a rich data commons. Humans often cooperate instinctively; AI agents, trained to maximise a single owner’s utility, must be explicitly designed to do so.
The Collective‑Risk Experiment
To study delegation under risk, we abstracted away from those domains and used the collective‑risk dilemma.
In this scenario, a group of players must work together to reach a shared contribution threshold that protects a public good. Each participant receives an initial endowment and is asked to contribute a portion of it over a series of rounds to a common pool. If, by the end of the game, the total contributions meet or exceed the threshold, players keep whatever remains of their endowment. But if the group fails to reach the threshold, everyone risks losing their remaining tokens entirely. The loss isn’t guaranteed—but determined by a known probability, called the risk factor, which players are informed about upfront.
Our experiment set-up is shown on the follwing figure.
- Setup. Participants first configured an artificial agent that would play once on their behalf, choosing a single contribution rule rather than acting round by round.
- Surprise restart. After the first game, we launched an unannounced second round and let participants re‑program their agents—revealing how they adapt after seeing others’ behaviour.
- Control group. A separate cohort played both games themselves, deciding contributions manually each round.
Do We Program Our Agents to Free-Ride?
No. Agents programmed by participants contributed more on average than humans playing directly. Even when the first game ended in failure or displayed extreme inequality, participants re‑programmed their agents to remain generous.
This contrasts with human‑to‑human delegation, where humans sometimes appoint decision delegates precisely to shift blame for selfish actions (Kouchaki et al., Nat Comm 2016). In our setting with AI agents, selfish behaviour may look to the participants like a bug, and not a clever exploit of the human behind the defecting agent. Another interpretation is that delegation to agents requires participants to devise their strategy in advance. This might lead to more deliberative decision-making and less emotional reactions compared to the control condition where participants can immediately respond to defecting players.
But Do We Program Our Agents Accurately?
Not quite. Because delegate agents commit to a static strategy, they cannot tune their contributions mid‑game. Human players, in contrast, adjust on the fly, thereby lowering donations if the threshold is clearly in reach or upping them to cover a shortfall. Consequently, agent groups show larger variance in the total contribution at the end of the game.
Where Next?
Achieving reliable cooperation among autonomous agents that represent different principals demands:
- New algorithms for cooperative AI that balance individual and collective objectives.
- Better communicaiton interfaces so humans can convey nuanced intentions to their delegates.
At DECIDE we are excited to push this frontier. Designing agentic AI that not only thinks for you, but thinks of us.
Curious about the full study? Read the open‑access paper in PNAS here.