# Synthetic alert base-rate exercise

Published by DeFiSec on September 5, 2026 for the article [AI and blockchain: a 2,000-case alert exercise](https://defisec.info/ai_and_blockchain).

This is an educational arithmetic example. No AI model, security detector or vendor product was evaluated. No real transaction data was used. The labels and alert decisions are authored, with fixed rates chosen to illustrate the effect of prevalence on precision. The cases have no associated code, vulnerability or transaction.

## Files

- `synthetic-alerts.csv`: 2,000 rows, divided into two cohorts of 1,000 cases.
- `evaluate_alerts.py`: a Python standard-library script that can regenerate the fixed dataset and count its outcomes.
- `results.json`: the output of the executed evaluator, including the dataset's SHA-256 digest.

## Reproduce the calculation

Save the files together, then run:

```sh
python3 evaluate_alerts.py
```

To regenerate the same CSV and evaluate it:

```sh
python3 evaluate_alerts.py --generate
```

The script prints JSON to standard output. It does not contact a network, invoke a model or require an external Python package. `--generate` overwrites the selected dataset path; without that option the script only reads it.

## Construction

Each row contains `scenario`, `case_id`, `ground_truth` and `alerted`. Labels use `1` for positive and `0` for negative. Case IDs are unique across the file.

For `prevalence_10_percent`, the first 100 cases have positive ground truth. The first 80 of those receive an alert. Of the remaining 900 negative cases, the first 90 receive an alert.

For `prevalence_1_percent`, the first 10 cases have positive ground truth. The first 8 of those receive an alert. Of the remaining 990 negative cases, the first 99 receive an alert.

This deliberately gives both cohorts 80% recall and a 10% false-positive rate. These are construction assumptions, not estimated detector capabilities. Row order is fixed and no randomness is involved.

## Executed results

The generator and evaluator ran successfully on September 5, 2026. Evaluation from the saved CSV reproduced these counts:

| Metric | 10% prevalence | 1% prevalence |
|---|---:|---:|
| Cases | 1,000 | 1,000 |
| True positives | 80 | 8 |
| False positives | 90 | 99 |
| False negatives | 20 | 2 |
| True negatives | 810 | 891 |
| Alerts | 170 | 107 |
| Precision, rounded | 47.1% | 7.5% |
| Recall | 80% | 80% |
| False-positive rate | 10% | 10% |

Using TP, FP, FN and TN for the four outcome counts:

- Precision = TP / (TP + FP).
- Recall = TP / (TP + FN).
- False-positive rate = FP / (FP + TN).
- Prevalence = (TP + FN) / all cases.

The evaluator returns `null` for any ratio with a zero denominator. It rejects empty datasets, duplicate case IDs and nonbinary labels.

The CSV SHA-256 digest is `c1a52ec31d101016f73f675186790e52428b890db05baf64ef288a02867e7c5a`.

## Interpretation limits

The exercise demonstrates a mathematical dependency: holding recall and false-positive rate fixed while reducing prevalence lowers precision. The 80% and 10% assumptions were selected for simple counts, without evidence that they describe a security tool.

These deterministic fictional rows do not estimate real attack prevalence, operational detection quality, review time or losses prevented. There is no sampled population and no empirical confidence interval. A real evaluation needs independently reviewed labels, an explicit sampling method and coverage of unalerted cases to assess missed incidents. Do not report these percentages as AI accuracy.