AI and blockchain: a 2,000-case alert exercise

A base-rate exercise you can reproduce

DeFiSec constructed two cohorts of 1,000 fictional cases and ran the accompanying evaluator on September 5, 2026. Each row has an authored ground-truth label and an authored alert decision. The prevalence of positive cases is 10% in one cohort and 1% in the other. We deliberately assigned alerts to yield 80% recall and a 10% false-positive rate in both.

This is an educational arithmetic example, not a model benchmark or a measurement of any AI system's accuracy. There are no real transactions, inferred labels or learned predictions. The rates are inputs to the construction.

Precision is true positives divided by all alerts. Recall is true positives divided by all actual positives, including missed cases. False-positive rate divides false positives by all actual negatives. With fewer positive cases, the same false-positive rate creates a much larger share of the alert queue: 99 of 107 alerts are false in the second cohort.

Download the synthetic dataset, evaluation script and computed results. Save them together and run python3 evaluate_alerts.py. The method notes explain regeneration and limitations.

For a real pilot, report the sampled population and label-review process. Include unalerted cases when estimating recall; reviewing only the alert queue leaves missed incidents unknown. Keep confirmed findings separate from duplicates and unresolved candidates. These synthetic proportions cannot predict performance on your protocol.

Separate AI triage from testing and proof

A proposed explanation, an execution trace and a checked theorem support different conclusions. Record which one each tool actually produced.

The Ethereum testing documentation explains these testing distinctions. An AI system can help draft a harness or specification; the resulting test or proof still needs review.

Reproduce a candidate before accepting it

The Ethereum Foundation's July 2026 account of agent-assisted protocol research requires an independently runnable reproducer against real code. It describes misleading candidates whose inputs cannot reach the vulnerable path and proofs whose statements fail to capture the intended property.

For each candidate, preserve the target revision and build configuration. State the entry point an attacker controls, the property allegedly violated and the observable failure. Run the reproducer in an isolated environment that matches the relevant production behavior. Have another reviewer check reachability and impact, then compare the result with known or fixed issues. Record who accepted it and why.

A successful run of generated test code is insufficient if the test bypasses the real entry point or asserts the wrong property. Keep the candidate open until those questions are resolved.

Treat repository context as untrusted input

Comments, README files and retrieved documentation can contain instructions aimed at the assistant reviewing them. OWASP's prompt-injection guidance covers external content that changes model behavior and recommends separating untrusted material from instructions.

Apply that boundary to audit inputs: a repository cannot grant permission to disclose secrets, install arbitrary software or suppress a finding. Label external content, validate structured outputs and keep credentials outside model context. Run generated code with restricted filesystem and network access. Test whether hostile documents can alter tool calls or report content; a prompt telling the model to ignore attacks is insufficient protection.

Keep response authority outside the model

Start monitoring integrations with read access and proposed actions. OWASP's excessive-agency guidance places authorization in downstream systems and calls for approval of consequential actions.

For blockchain operations, enforce permitted contracts and functions in the execution layer. Keep signing keys separate from the assistant. Require a human to review any proposal to pause a protocol, change an approval or move funds. Check the transaction target and simulated effects before approval, then log the authorized action and its outcome. Rate limits constrain damage but do not establish that an action is appropriate.

When selecting a workflow, use the security tools directory to identify the analysis methods you need. Consult security reports for supporting evidence and the DeFiSec methodology for how we assess it. Request reproducible artifacts and a defined response policy before relying on an AI security claim.