Design-partner pilot · Field notes
What a 14-day pilot taught us about instrumenting agents.
In June 2026 we ran a short pilot of Thoth with a design partner. These notes cover what we saw and what we changed. They describe a pilot, not a production deployment, and the enforcement in it was simulated.
01 · How it was set up
Two phases, two agents.
- Duration
- 14 days with one design partner.
- Days 1 to 7
- Shadow observation. Thoth watched scoped developer workflows without intercepting anything or touching a payload.
- Days 8 to 14
- Simulated enforcement. Policy rules evaluated simulated agent intents and returned decisions in a non-blocking state.
- Scope
- Two active agents.
- Rules
- Eight deterministic policy bundles covering data exfiltration, token access limits, rate limiting and scope validation.
- Scoring
- Deterministic heuristics only. Machine-learning scoring was not active.
02 · What we found
Three lessons.
01
Keeping an agent wrapped was the real cost.
The partner's agent changed its code and tool configuration weekly. Platform engineers had to keep rewriting integration files to keep it covered, and early on that effort can outweigh the value of the telemetry it produces.
02
Step-up and block decisions did not break the workflow.
In the simulation, actions outside scope or flagged as high risk could be held for step-up or blocked without disturbing the agent's execution state. This was a simulation, so it says nothing yet about enforcement under real load.
03
Predictive scoring needs history first.
Behavioural scoring needs a baseline to avoid false positives. The 14 days went on building that baseline, so scoring stayed on deterministic heuristics for the whole pilot.
What we changed.
After the pilot we shipped SDK updates that automate wrapping an agent and registering its tool paths, so keeping up with a changing agent no longer means rewriting integration files by hand.
03 · What this does not show
Read the limits before the lessons.
- This was a pilot with one design partner and two agents, not a production deployment.
- Enforcement was simulated and non-blocking. No real action was held or blocked.
- We have not published sample sizes or false-positive rates, so nothing here should be read as a performance benchmark.
- One team's agents and workflows will not match yours. Observe mode lets you run the same kind of test on your own.
Run your own test
Start in observe mode on one agent.
See what Thoth would have decided on your own workflow before anything is held or blocked.
For technology, channel, federal delivery, and research partnerships.