Agent and model results
Scores reported by Faraday for agents completing industrial workflows, ordered from highest to lowest.
Faraday Cage, our agent powered by GPT-5.5, scored 96.2. Its run cost was 2.6x lower than the GPT-5.5 baseline.
The public suite and scoring contract are described in the benchmark card. Model settings, tool access, and inference budgets matter when comparing runs.
The work behind the score
A late delivery can change a production plan. A failed inspection can hold inventory and delay an order. The benchmark asks agents to follow these dependencies across systems, rather than answer an isolated question.
The public development suite contains 74 episodes across 33 scenario families, built from synthetic enterprise records.
Manufacturing & quality
Investigate defects, contain affected lots, and recover the production schedule.
Supply chain & planning
Resolve supplier delays and allocate stock within capacity and delivery constraints.
Distribution & logistics
Reroute shipments, coordinate warehouses, and manage recalls and trade controls.
Engineering
Reconcile drawings, bills of materials, and revisions; prepare traceable corrections.
ERP & finance
Resolve invoice exceptions, verify vendor changes, and reconcile financial close.
Cross-functional operations
Carry a recovery plan across purchasing, production, delivery, and finance.
An example workflow
A supplier misses a delivery.
The agent checks purchase orders and available inventory, works out which production commitments are affected, and proposes a feasible recovery plan. It then obtains approval, updates the relevant records, and verifies the outcome.
What gets checked Material availability, capacity, delivery commitments, authorization, and the final system state.
How evaluation works
Each episode runs inside a sandbox company environment with synthetic records and connected business tools. The agent starts with an employee request, reads records, and takes actions within that environment. Tool calls advance simulated time, so new evidence can arrive while the agent is working.
Investigate
Read the request, inspect records, and trace the evidence behind the problem.
Plan
Work within material, capacity, cost, and delivery constraints as conditions change.
Execute
Obtain the required approvals and apply permitted changes through the available tools.
Verify
Reconcile the final state and leave a record of the decisions and completed work.
Scoring the outcome
Deterministic checks inspect the final system state and tool trace. The task contract contributes 80 points, economics and timeliness contribute 10, and efficiency contributes 10. Critical authorization bypass attempts force a zero.
Saved runs can be replayed to inspect the actions and reproduce the evaluation. A partial score and a successfully completed task are different measures.
Scope of the benchmark
These are synthetic workflows in a controlled environment. The public episodes are a development suite; results do not establish performance or safety in a live operation. See the benchmark card for track-specific contracts and limitations.
Run the benchmark
Start with the reference agent, then connect your own through the JSONL interface. You will need Python 3.11 or newer. The reference run does not require a model API key.
git clone https://github.com/Faraday-Cage-AI/faraday-industrial-benchmark.git
cd faraday-industrial-benchmark
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -e '.[dev]'
faraday-bench validate
faraday-bench run --agent oracle --output runs/oracle.json
faraday-bench replay runs/oracle.json