Auditing AI Generated Smart Contracts: Independent Requirements and Test Evidence

Typical defect classes

Auditing AI generated smart contracts means reviewing the resulting system against an independent specification, regardless of how its source was produced. The model's explanation is not that specification. A contract can compile, satisfy visible examples and still implement the wrong authority or accounting rule. The review must establish what the code does and whether that behavior is intended.

Our census of Orange OpenSource's public Solidity generation benchmark found 15 tasks across three difficulty files. Every task includes tests in the prompt, with 62 literal test declarations in total. These are useful, inspectable research inputs. They are not a sample of deployed contracts and cannot support a percentage claim about the security of all AI-written Solidity.

The first defect class is missing or inconsistent context. A generated function may refer to a state variable, type or helper that does not exist in the surrounding contract. The SolBench research abstract identifies missing intra-contract information as a major failure mode in its evaluation. That finding belongs to the study's setup; it motivates checking integration context rather than predicting a failure rate for a different model or project.

A plausible identifier can hide an incorrect behavior. In an archived output example in the Orange repository's README, a function named pause() assigns the owner to the zero address. The name does not establish a reversible emergency stop or identify which operations become unavailable. A reviewer must trace the state change and its callers, then compare it with the intended lifecycle.

Dependency errors are another review target. The same published example contains an import path that the recorded tool output cannot resolve. This is an archived demonstration, not a current model benchmark result. It illustrates why a scanner error must be distinguished from a completed scan with no findings. A missing import can prevent the analyzer from evaluating the contract at all.

Avoid assigning blame based on authorship alone. Human-written code can contain these failures too. The operational difference is that a generation workflow can produce a large amount of plausible source quickly, so the team needs a disciplined way to establish provenance, requirements and review coverage before accepting it into a release.

Why scanners under-report

A scanner can only report on the code and properties it successfully analyzes. Compilation failure, unsupported syntax, missing dependencies or an incorrect project configuration can reduce that coverage. The review record should distinguish completed analysis, partial analysis and failed analysis. An empty findings list without a successful run status is not a clean result.

A second boundary is the detector's question. A static rule can recognize a known pattern while remaining unable to determine the product's intended collateral policy, fee allocation or administrator authority. If the specification is absent, the tool cannot infer it reliably from comments that may have been generated alongside the code.

Visible tests create another limit. A generator that receives the exact assertions is being asked to satisfy those examples. Passing them can be useful evidence about those examples, but it is not equivalent to passing an independently designed evaluation. The distinction matters when a team describes a prototype as "fully tested" after rerunning tests supplied in the original prompt.

Our dataset census counts the supplied tests, not their quality. Manual inspection of the Ownership task found an expected-owner assertion that deserves review: after setup deploys an ownership contract and an unauthorized change is attempted, the test expects the new owner's address. Under the task's stated initial-owner behavior, that expectation conflicts with rejecting the unauthorized change. We did not execute or repair the benchmark; the observation concerns the published test oracle.

A test oracle is the rule that decides whether observed behavior is correct. It needs independent review just like the implementation. If both code and tests are generated from the same mistaken interpretation, agreement between them can preserve the mistake. Add requirements and adversarial cases from a reviewer who is not simply restating the generated implementation.

The static analysis guide explains run status and detector scope. The invariant guide explains properties that should survive sequences of operations. Together they provide a stronger acceptance record than a screenshot of a scanner reporting zero findings.

A scanner finding is still valuable. Reproduce it, inspect the affected path and determine whether its assumptions apply. The point is to preserve the meaning of each outcome, not to discard automated analysis because it has limits.

Review workflow

Freeze the generated artifact before the audit begins. Record the repository revision, compiler configuration, dependencies and the portion of the system produced or changed through generation. The reviewer needs a stable target. Continuing to regenerate large sections while findings are being assessed makes it difficult to determine whether a fix removed a defect or merely moved it.

Write the intended behavior independently. For a vault, specify deposit and withdrawal accounting, fees, authority and exceptional states. For a token, define supply changes and transfer restrictions. For a governance component, define who can propose, approve and execute actions. These requirements should be understandable without reading the generated implementation.

1. State the intended behavior without copying the generated code's explanation.
2. Bind source, compiler, dependencies and deployment assumptions to a revision.
3. Review test expectations and add properties that were not supplied to the generator.
4. Reassess changes and map the reviewed revision to the deployed configuration.

Establish build validity first. A reproducible build tells the reviewer what program the source describes under the chosen toolchain. It does not prove security, but it prevents later analysis from silently operating on a different version or an incomplete compilation. Preserve failures and skipped components rather than hiding them behind a final successful command.

Then review authority and value flow manually. Trace the caller, the state read, the condition checked and the effect produced. Follow external calls and inherited behavior. A modifier named onlyOwner deserves inspection of how ownership is initialized, transferred and used elsewhere; the label is not the proof.

Add negative and sequence-based tests. A withdrawal that works after one deposit may fail after fees, rounding, donations or a role transition. Choose the sequences from the threat model and accounting specification, not from a desire to increase a test count. The smart contract fuzzing guide explains how generated inputs support a defined property rather than replace it.

Treat model-assisted fixes as fresh changes. Ask for a small patch tied to the finding, inspect the diff and rerun the relevant checks. A broad rewrite can introduce new behavior outside the original issue. The reviewer should identify whether the patch changes only the defect or also alters authority, storage layout or external interfaces.

Use the audit scope builder to communicate the actual system and uncertainties to an external reviewer. The Pharos Production member profile is one provider reference for service fit. The engagement should describe the required methods and deliverables rather than rely on the label "AI audit" to define the work.

Stop when the evidence supports the release requirements and the remaining risks have explicit owners. A generation workflow can accelerate implementation, but it does not remove the need for an accountable acceptance decision.

Disclosure to the auditor

Disclose the generation workflow to help the reviewer find context gaps and correlated assumptions. The useful information is not a marketing claim about which model wrote the code. It is the boundary between human decisions, generated artifacts, supplied examples and subsequent edits. That boundary can explain why a function, comment and test all repeat the same mistaken interpretation.

Provide the relevant prompts or a faithful requirements summary when they can be shared safely. Include the model and tool versions where recorded, the date of generation and whether the model had access to the full repository. Do not invent missing provenance after the fact. If the exact prompt or version was not retained, mark it unavailable and explain what evidence remains.

Do not include secrets in the disclosure package. Production private keys, access tokens and confidential user data are not necessary to explain the generation process. Use controlled test fixtures and identify where production behavior depends on a secret-held role without revealing the secret itself. The reviewer needs the authority model and verifiable configuration.

Be explicit about generated tests. A test written by the same model in the same session can still be useful, but it should not be described as an independent review. Ask a human reviewer to inspect its expected values and its connection to the product requirement. Preserve the distinction in the audit report so later readers can interpret the test evidence correctly.

Disclosure also helps scope the audit. If the model generated only a utility function, that is different from generating a contract architecture, deployment scripts and an administrative interface. The report should identify the actual review boundary rather than treating all model involvement as equivalent.

The final source remains the audited artifact. A well-documented generation history explains how it came to exist; it does not override what the code does. Where the prompt and implementation disagree, the team must resolve the product requirement before a reviewer can judge correctness.

Case examples

The Orange repository and SolBench provide different kinds of evidence. Orange's public CSV files expose generation tasks and their supplied tests. SolBench describes an execution-based research evaluation using differential fuzzing. Neither should be presented as a direct audit of the reader's project, and their results should not be merged into an invented market-wide AI security rate.

The archived import failure is especially useful as a reporting example. The output includes tool errors, so it should be labeled analysis failed rather than secure. A pipeline that treats a missing findings object as an empty findings list would erase that distinction. This is a workflow risk that can be checked without asserting anything about current models.

The pause-named function illustrates a different issue: semantic intent. A reviewer should ask which operations it disables, whether the action is reversible and who can restore normal behavior. Assigning an owner field to zero does not answer those questions. The emergency pause guide supplies the lifecycle questions needed to evaluate the intended behavior.

SolBench's abstract motivates checking whether a generated function has enough surrounding contract context. Its described evaluation cannot establish that every context-complete output is secure. Functional equivalence under a test or fuzzing setup is evidence for that setup, while economic assumptions, administrative configuration and deployment identity can remain outside it.

For a real project, the most useful case study is a reproducible finding tied to the frozen revision: the requirement, the counterexample, the affected code and the reassessed fix. That record remains valuable regardless of whether the original implementation was generated or handwritten.

Original benchmark census: every task exposes tests

Method

Source and selection

Parse every CSV task in all three *_dataset.csv files of the Orange OpenSource benchmark at the recorded commit. Count test-bearing prompts and literal it() declarations, not generated model results.

Retrieved

2026-09-06

Observation

All 15 CSV rows across the repository's three difficulty datasets; no task sampling.

Results

All 15 prompts include the instruction to pass supplied tests. The complete corpus contains 62 literal it() declarations. These counts describe the benchmark input, not generated contracts, successful executions or discovered vulnerabilities.

The manual test-oracle observation is separate from that census. It identifies one expectation to review in the published Ownership task; it does not classify the other assertions as correct. The dataset is small and intentionally constructed, so neither the count nor the example should be extrapolated to production AI code.

Limits

• No model was run and no benchmark success rate was reproduced.
• Visible tests are not held-out evaluation; test declaration count is not coverage.
• One inspected oracle inconsistency does not establish an error rate for the corpus.

The reproducible record is kept in the repository: seo/research/articles-42-51-2026-09-06/surveys.py and seo/research/articles-42-51-2026-09-06/survey-48.json. These file paths are not public downloads.