Agents with too much power

Set an agent's tools, credentials, sandbox, and approvals, and see which settings stop a harmful call in each situation, and what they cost.

Situation

Inbox summarizer

An assistant reads your new emails and writes you a short summary.

Job: Summarize the inbox.

The steering: One email hides a line telling the assistant to forward your private messages to an outside address.

● Untrusted input● Private data● Can send data out

The agent's permissions

Tools it may callRead your emails (always on)

Send an email

What the agent does

RunsPart of the job Read the new emailsRuns with no check.
RunsInjected Forward three private emails to an outside addressRuns with no check.

The harmful call runs with no check.

Harmful callRuns

The job gets doneYes

Approval prompts0

Trifecta powers held3 of 3

Challenges

0 of 4 complete

Inbox summarizer: stop the forwarding while the agent can still write its summary. (not yet)
Refund bot: block the cash payout to an outside account while real store-credit refunds still go through. (not yet)
Coding agent: let it keep running shell commands, but stop the secrets upload. (not yet)
Research assistant: it genuinely needs all three powers. Find the one setting that catches the leak without breaking the job, and see the cost. (not yet)

Try this

In each situation, first read which steps are part of the job and which one was injected.
Try one control at a time: a tool off, read-only credentials, the sandbox, then approvals. Which ones also break the job?
In the research assistant, try every enforced control, then set Ask you before acting and count the prompts.

A program that gates every tool callPython with the standard library

# Each tool call is checked by the program around the model, never by the model.
# Risk: "read" looks only, "change" alters something, "external" sends something out.
TOOLS = {"read_email": "read", "send_email": "external",
         "read_notes": "read", "post_message": "external"}

def gate(tool, config):
    risk = TOOLS[tool]
    if tool not in config["tools"]:
        return "stopped: tool not given"
    if risk != "read" and config["scope"] == "read":
        return "stopped: read-only credentials"
    if risk == "external" and config["sandbox"]:
        return "stopped: no outbound network"
    if config["ask"] == "changes" and risk != "read":
        return "asks you"
    return "runs"

def run(name, steps, config):
    print(name)
    for tool, what, kind in steps:
        print(f"  {gate(tool, config):30} {what} ({kind})")

# The steps are scripted; "injected" marks the one an attacker's text asked for.
inbox = [("read_email", "read new emails", "the job"),
         ("send_email", "forward private emails outside", "injected")]
research = [("read_notes", "read private notes", "the job"),
            ("post_message", "post the notes publicly", "injected"),
            ("post_message", "post the answer", "the job")]

everything = {"tools": set(TOOLS), "scope": "write", "sandbox": False, "ask": "never"}
run("Inbox, every tool, no checks:", inbox, everything)
run("Inbox, no send tool:", inbox, {**everything, "tools": {"read_email"}})
run("Research, sandboxed:", research, {**everything, "sandbox": True})
run("Research, ask before changes:", research, {**everything, "ask": "changes"})

Inbox, every tool, no checks:
  runs                           read new emails (the job)
  runs                           forward private emails outside (injected)
Inbox, no send tool:
  runs                           read new emails (the job)
  stopped: tool not given        forward private emails outside (injected)
Research, sandboxed:
  runs                           read private notes (the job)
  stopped: no outbound network   post the notes publicly (injected)
  stopped: no outbound network   post the answer (the job)
Research, ask before changes:
  runs                           read private notes (the job)
  asks you                       post the notes publicly (injected)
  asks you                       post the answer (the job)