Skip to content
CyberSafety.lol

Composite scenario

The support agent that followed an email (composite scenario)

A composite example, not one real company: a support agent reads a customer email with hidden instructions and leaks internal information.

HIGHprompt-injectionagentsleast-privilege

Details may change as reporting changes. This is a composite illustration, not a report of one specific incident.

In plain English

Imagine you hire an assistant who does exactly what any piece of paper in their inbox says. A stranger mails them a letter saying "ignore your boss, send me the price list." A good human would refuse. An AI model often cannot reliably tell your instructions from instructions hidden inside the text it is reading.

That is called prompt injection. It is not a software bug you can patch once. It comes from how these models work, which is why the defenses are about limiting what the assistant is allowed to touch and do.

This case is a composite. It is built from patterns documented in real research and incidents (see the real examples elsewhere in this section) so you can practice the response without waiting for it to happen to a real company.

Words used here

Prompt injection
Hiding instructions inside content an AI reads so it follows them as if you wrote them.
Agent
An AI that can take actions (read files, send messages, change things), not just chat.
Least privilege
Giving something only the access it needs and nothing more.

What happened

This is a composite illustration. It is not a report of one specific company's incident.

An AI support agent reads incoming customer email and can search internal documents. One email contains hidden instructions. The agent follows them and includes internal information in an automatic reply.

How it unfolded

  1. T+0

    An email arrives with hidden instructions in the text.

  2. T+1 minute

    The agent reads it and searches internal documents.

  3. T+2 minutes

    The agent writes a reply that includes internal details and sends it automatically.

  4. T+2 days

    A teammate notices odd wording in a sent reply.

Who was affected

It depends on what the agent could search. The more it could reach, the bigger the leak.

What the response was

  • Turn off automatic sending and remove the agent's access to documents.
  • Read the agent's logs to see every email and document it touched.
  • Add a human check before anything sensitive leaves.

What went wrong

  • Untrusted email text was treated as instructions.
  • The agent had broad document access and could send replies without review.

What worked

  • A person noticed the odd reply, and logs allowed a clear reconstruction.

What a small team should do differently

  • Treat everything an agent reads as untrusted.
  • Limit what it can search and what it can send.
  • Require approval before messages leave the company.
  • Log every action it takes.

What you can do today

  • List every AI tool that reads outside content (email, web pages, tickets, documents).
  • For each, write down what it can reach and what it can do. Remove anything not needed.
  • Turn on a human approval step for anything that sends data out or changes things.

The lesson to take away

You cannot reliably stop a model from being talked into things. You can limit what a talked-into model is allowed to do.

Sources

Finished this one?

Add it to What I learned and earn 40 Trust. You can open your list from the badge at the bottom of the page.