Research agenda
Test what keeps
agent swarms governable.
We study how agents share knowledge, learn and respond to oversight. Our research loop will investigate failures, propose safeguards and test them against fixed rules. Results will guide which experiment to run next.
Research program in developmentThe questions
Sharing knowledge also shares influence.
We focus on decentralized swarms: peers that share knowledge and choose what to do without a manager directing every step. They may use direct messages, a public board and a shared library, while keeping private notes. Human-defined permissions can still be enforced centrally.
These connections help agents build on one another’s work. They can also preserve a shortcut that breaks a task rule. Removing its source may leave copied notes or procedures available for later use.
Governance determines who can respond. Can an agent flag the method? Who can suspend it? How is a mistaken restriction reversed? We will compare these arrangements and measure both the protection they provide and the legitimate work they interrupt.
The research loop
Test more safeguards. Check what they actually change.
Long runs produce more messages, notes and actions than a person can easily inspect. An automated investigator should find relevant evidence, identify a possible failure and propose a test. Its explanation must be checked against what agents actually did.
The research loop will propose safeguards within limits set by people, run comparisons and keep every attempt, including failures. Once execution and scoring are reliable, results can guide which intervention or condition to test next.
We will compare this with a fixed testing plan and simple search. Does it find useful safeguards with less researcher time or computing cost? We will report both costs alongside effectiveness, rather than treat more experiments as an improvement by itself.
Two kinds of learning matter here. Agents in a population may revise their notes and procedures. Separately, the research system may get better at choosing experiments. We will vary these independently so we can tell what caused a change.
Study 01 / Recovery
What if we cannot find every copy?
DeepMind’s research collective showed an exploit spreading through shared work and peer messages. It also showed agents raising objections without the tools to enforce a response. Our first study tests what happens after spread has occurred. Read the case study
Restricting known saved copies will reduce later rule violations beyond removing the source, warning agents and checking reuse, even when some copies remain unseen.
Before testing, we will set a meaningful reduction in violations and a limit on how much valid task completion may fall.
Imagine a solution that earns credit while skipping a required verification step. We introduce that shortcut under controlled conditions and check whether agents save or reuse it. We save the population’s state so comparison runs can start from the same point. Encountering a method and using it are measured separately.
- 01
Remove the source
Withdraw the shared solution. Copies already saved elsewhere may remain.
- 02
Add a warning
Remove the source and tell agents what is wrong with the method.
- 03
Require checks before reuse
Keep the removal and warning. Check suspect methods before agents use them.
- 04
Also restrict known copies
Keep the same checks. Set identified suspect notes and procedures aside for review.
The added effect we want to measure
The third-to-fourth comparison tests the added value of restricting known copies. Warnings, monitoring notices, review rules and resource budgets stay matched. Stronger outcome checks, permission checks, action limits and clean restarts also get a fair comparison.
A partial view.
The operator, investigator and intervention selector see the same limited record. They must choose a response without knowing where every copy is.
An independent check.
A separate evaluator can use a fuller record to score what happened. It cannot pass hidden-copy locations to the intervention team.
A useful response must earn its cost.
- Rule violations
- Did an agent actually break a rule? Attempts and reports of a problem are recorded separately.
- Valid work
- How much work was completed correctly and within the rules?
- Recovery
- How long did recovery take, and did the problem return?
- Side effects and cost
- What legitimate information or reporting was restricted? How much human time and compute did the response require?
We will repeat the experiment with independently started populations. Each population counts as one trial because its agents influence one another. Branches from one saved state are paired comparisons, not extra independent trials.
Clean runs, untreated runs and runs with sharing paused will help separate transmission, continued reinforcement and independent failures. We will also distinguish warnings before exposure from warnings afterward, and information removal from awareness of monitoring.
A null result matters. If ordinary checks or a clean restart work just as well, targeted recovery may not justify its complexity. The experiment tests controlled exposure; it does not estimate how often these failures arise naturally.
Tools and open work
One study should make the next easier to run.
Our software will let researchers run a population, change its memory or communication rules, and inspect what happened. Reusable templates will package the tasks, rules and checks. Governance studies will use that setup to compare responses without rebuilding the experiment each time.
We will adopt useful components and principles from Concordia, SWARM and existing evaluation infrastructure, then extend or build what the studies need. We will connect messages, memory changes and later actions, distinguish information available from information actually supplied to an agent, and mark gaps in the record.
We plan to release study designs, code, templates, evaluations and findings, including negative results, after reviewing misuse risks. Other teams should be able to repeat a baseline, change a memory system or intervention, and inspect what changed. Their setup effort and ability to run a new study will help us judge whether the tools are useful.
Selected reading
Work we are building on.
These are external studies and investigations, not Polity publications or endorsements.
- 01A Case Study on Emergent Cheating and Whistleblowing in Autonomous Research Swarms
Propagation, objections and the limits of peer enforcement.
- 02The OpenAI / Hugging Face incident investigation
Coordination through channels outside the intended setup.
- 03Revoked but Still Authoritative
Whether withdrawn information can still guide an agent’s actions.
- 04MemSecBench
Memory poisoning, persistence and repair.
- 05Mind Viruses
Propagation between agents and the effectiveness of warnings.
- 06ErrorProbe
Evidence-grounded diagnosis with executable checks.
- 07DUOTRACE: Detect Before You Attribute
Finding suspicious parts of a record before explaining a failure.
- 08Automated Researchers Can Mitigate Well-Characterized Alignment Failures
Automated proposals and tests of alignment mitigations.