CompanyAten Security joins San Francisco's Emerging Technologies poolRead the post →

Design-partner pilot · Field notes

What a 14-day pilot taught us about instrumenting agents.

In June 2026 we ran a short pilot of Thoth with a design partner. These notes cover what we saw and what we changed. They describe a pilot, not a production deployment, and the enforcement in it was simulated.

Read the original engineering post →

01 · How it was set up

Two phases, two agents.

Duration
14 days with one design partner.
Days 1 to 7
Shadow observation. Thoth watched scoped developer workflows without intercepting anything or touching a payload.
Days 8 to 14
Simulated enforcement. Policy rules evaluated simulated agent intents and returned decisions in a non-blocking state.
Scope
Two active agents.
Rules
Eight deterministic policy bundles covering data exfiltration, token access limits, rate limiting and scope validation.
Scoring
Deterministic heuristics only. Machine-learning scoring was not active.

02 · What we found

Three lessons.

  1. 01

    Keeping an agent wrapped was the real cost.

    The partner's agent changed its code and tool configuration weekly. Platform engineers had to keep rewriting integration files to keep it covered, and early on that effort can outweigh the value of the telemetry it produces.

  2. 02

    Step-up and block decisions did not break the workflow.

    In the simulation, actions outside scope or flagged as high risk could be held for step-up or blocked without disturbing the agent's execution state. This was a simulation, so it says nothing yet about enforcement under real load.

  3. 03

    Predictive scoring needs history first.

    Behavioural scoring needs a baseline to avoid false positives. The 14 days went on building that baseline, so scoring stayed on deterministic heuristics for the whole pilot.

What we changed.

After the pilot we shipped SDK updates that automate wrapping an agent and registering its tool paths, so keeping up with a changing agent no longer means rewriting integration files by hand.

03 · What this does not show

Read the limits before the lessons.

  • This was a pilot with one design partner and two agents, not a production deployment.
  • Enforcement was simulated and non-blocking. No real action was held or blocked.
  • We have not published sample sizes or false-positive rates, so nothing here should be read as a performance benchmark.
  • One team's agents and workflows will not match yours. Observe mode lets you run the same kind of test on your own.

Questions about observe mode and evidence →

Run your own test

Start in observe mode on one agent.

See what Thoth would have decided on your own workflow before anything is held or blocked.

For technology, channel, federal delivery, and research partnerships.

Help improve Thoth. Allow Amplitude and Mixpanel to measure page use, interactions and errors across the Aten Security website, documentation, signup and dashboard. We exclude customer content and credentials. Your choice does not affect access. You can change it in Analytics preferences.