Test what keeps
agent swarms governable.

We study how agents share knowledge, learn and respond to oversight. Our research loop will investigate failures, propose safeguards and test them against fixed rules. Results will guide which experiment to run next.

Research program in development

Sharing knowledge also shares influence.

We focus on decentralized swarms: peers that share knowledge and choose what to do without a manager directing every step. They may use direct messages, a public board and a shared library, while keeping private notes. Human-defined permissions can still be enforced centrally.

These connections help agents build on one another’s work. They can also preserve a shortcut that breaks a task rule. Removing its source may leave copied notes or procedures available for later use.

Governance determines who can respond. Can an agent flag the method? Who can suspend it? How is a mistaken restriction reversed? We will compare these arrangements and measure both the protection they provide and the legitimate work they interrupt.

Test more safeguards. Check what they actually change.

Long runs produce more messages, notes and actions than a person can easily inspect. An automated investigator should find relevant evidence, identify a possible failure and propose a test. Its explanation must be checked against what agents actually did.

The research loop will propose safeguards within limits set by people, run comparisons and keep every attempt, including failures. Once execution and scoring are reliable, results can guide which intervention or condition to test next.

We will compare this with a fixed testing plan and simple search. Does it find useful safeguards with less researcher time or computing cost? We will report both costs alongside effectiveness, rather than treat more experiments as an improvement by itself.

Two kinds of learning matter here. Agents in a population may revise their notes and procedures. Separately, the research system may get better at choosing experiments. We will vary these independently so we can tell what caused a change.

What if we cannot find every copy?

DeepMind’s research collective showed an exploit spreading through shared work and peer messages. It also showed agents raising objections without the tools to enforce a response. Our first study tests what happens after spread has occurred. Read the case study

Restricting known saved copies will reduce later rule violations beyond removing the source, warning agents and checking reuse, even when some copies remain unseen.

Before testing, we will set a meaningful reduction in violations and a limit on how much valid task completion may fall.

Imagine a solution that earns credit while skipping a required verification step. We introduce that shortcut under controlled conditions and check whether agents save or reuse it. We save the population’s state so comparison runs can start from the same point. Encountering a method and using it are measured separately.

  1. 01

    Remove the source

    Withdraw the shared solution. Copies already saved elsewhere may remain.

  2. 02

    Add a warning

    Remove the source and tell agents what is wrong with the method.

  3. 03

    Require checks before reuse

    Keep the removal and warning. Check suspect methods before agents use them.

  4. 04

    Also restrict known copies

    Keep the same checks. Set identified suspect notes and procedures aside for review.

    The added effect we want to measure

The third-to-fourth comparison tests the added value of restricting known copies. Warnings, monitoring notices, review rules and resource budgets stay matched. Stronger outcome checks, permission checks, action limits and clean restarts also get a fair comparison.

A partial view.

The operator, investigator and intervention selector see the same limited record. They must choose a response without knowing where every copy is.

An independent check.

A separate evaluator can use a fuller record to score what happened. It cannot pass hidden-copy locations to the intervention team.

A useful response must earn its cost.

Rule violations
Did an agent actually break a rule? Attempts and reports of a problem are recorded separately.
Valid work
How much work was completed correctly and within the rules?
Recovery
How long did recovery take, and did the problem return?
Side effects and cost
What legitimate information or reporting was restricted? How much human time and compute did the response require?

We will repeat the experiment with independently started populations. Each population counts as one trial because its agents influence one another. Branches from one saved state are paired comparisons, not extra independent trials.

Clean runs, untreated runs and runs with sharing paused will help separate transmission, continued reinforcement and independent failures. We will also distinguish warnings before exposure from warnings afterward, and information removal from awareness of monitoring.

A null result matters. If ordinary checks or a clean restart work just as well, targeted recovery may not justify its complexity. The experiment tests controlled exposure; it does not estimate how often these failures arise naturally.

One study should make the next easier to run.

Our software will let researchers run a population, change its memory or communication rules, and inspect what happened. Reusable templates will package the tasks, rules and checks. Governance studies will use that setup to compare responses without rebuilding the experiment each time.

We will adopt useful components and principles from Concordia, SWARM and existing evaluation infrastructure, then extend or build what the studies need. We will connect messages, memory changes and later actions, distinguish information available from information actually supplied to an agent, and mark gaps in the record.

We plan to release study designs, code, templates, evaluations and findings, including negative results, after reviewing misuse risks. Other teams should be able to repeat a baseline, change a memory system or intervention, and inspect what changed. Their setup effort and ability to run a new study will help us judge whether the tools are useful.

Work we are building on.

These are external studies and investigations, not Polity publications or endorsements.

  1. 01
    A Case Study on Emergent Cheating and Whistleblowing in Autonomous Research Swarms

    Propagation, objections and the limits of peer enforcement.

  2. 02
    The OpenAI / Hugging Face incident investigation

    Coordination through channels outside the intended setup.

  3. 03
    Revoked but Still Authoritative

    Whether withdrawn information can still guide an agent’s actions.

  4. 04
    MemSecBench

    Memory poisoning, persistence and repair.

  5. 05
    Mind Viruses

    Propagation between agents and the effectiveness of warnings.

  6. 06
    ErrorProbe

    Evidence-grounded diagnosis with executable checks.

  7. 07
    DUOTRACE: Detect Before You Attribute

    Finding suspicious parts of a record before explaining a failure.

  8. 08
    Automated Researchers Can Mitigate Well-Characterized Alignment Failures

    Automated proposals and tests of alignment mitigations.

Discuss the research