PRISM · Release v1
How the coding was checked, and its limits
Validation
Agreement with an independent labeller
Labeller: glm-5.3-flash on Ollama Cloud. Confirmer: nemotron-3-ultra on Ollama Cloud. Pass precision and the changes at confirmation per check are counts of launches and stay under disclosure control, so a small or suppressed value reads as words.
| Check | AC1 (95% CI) | Pass precision | Changed at confirmation |
|---|---|---|---|
| The artefact or action can be reproduced or inspected. | not estimable | suppressed (disclosure control) | 30% (59 of 195) |
| Important supplied context, assumptions, and inferences are visible. | not estimable | 87% (83 of 95) | 35% (33 of 95) |
| Check | AC1 (95% CI) | Pass precision | Changed at confirmation |
|---|---|---|---|
| The intended use and boundaries are explicit. | 0.89 (0.84–0.94) | 93% (229 of 246) | 31% (78 of 252) |
| Acceptance criteria are explicit and testable enough for the use. | not estimable | 75% (60 of 80) | 55% (47 of 85) |
| Evidence shows the acceptance criteria were met. | not estimable | suppressed (disclosure control) | 49% (42 of 86) |
| Relevant failure modes and fallback behaviour were evaluated. | not estimable | suppressed (disclosure control) | <10 |
| Known limitations and out-of-scope conditions are stated. | 0.86 (0.80–0.92) | 95% (172 of 182) | 11% (21 of 183) |
| Check | AC1 (95% CI) | Pass precision | Changed at confirmation |
|---|---|---|---|
| The Deliverable gate for the intended use has passed. | agreement not estimable: fewer than ten gold passes | not estimable: agreement not estimable: fewer than ten gold passes | <10 |
| Operational ownership is explicit. | not estimable | suppressed (disclosure control) | suppressed (disclosure control) |
| The assurance mechanism is defined for the important failure modes. | not estimable | suppressed (disclosure control) | suppressed (disclosure control) |
| Fallback or escalation is defined. | 0.72 (0.64–0.80) | suppressed (disclosure control) | 33% (22 of 66) |
| Recovery or rollback has been tested where failure consequence warrants it. | agreement not estimable: fewer than ten gold passes | <10 | <10 |
| Support, maintenance, debugging, and change responsibility are defined. | not estimable | 83% (160 of 192) | 46% (88 of 193) |
| The relevant operating-cost boundary is known well enough for the decision. | 0.82 (0.75–0.88) | 88% (130 of 148) | 30% (45 of 150) |
| Check | AC1 (95% CI) | Pass precision | Changed at confirmation |
|---|---|---|---|
| The outcome measure is defined. | not estimable | suppressed (disclosure control) | suppressed (disclosure control) |
| A comparable baseline or counterfactual is defined. | not estimable | <10 | <10 |
| The outcome was measured after the intervention. | not estimable | <10 | suppressed (disclosure control) |
| Before/after measurements use the same relevant definition. | agreement not estimable: fewer than ten gold passes | <10 | <10 |
| Material confounds, uncertainty, or attribution limits are named. | not estimable | <10 | <10 |
| Check | AC1 (95% CI) | Pass precision | Changed at confirmation |
|---|---|---|---|
| Outcome evidence relevant to the decision is available. | not estimable | <10 | suppressed (disclosure control) |
| The full relevant cost boundary is accounted for. | not estimable | <10 | suppressed (disclosure control) |
| Material risks and trade-offs are assessed. | not estimable | suppressed (disclosure control) | 38% (14 of 37) |
| Reasonable alternatives or the status quo are compared. | not estimable | 60% (31 of 52) | 75% (39 of 52) |
| The rule for scale, renew, expand, or stop is explicit. | not estimable | withheld: too few coder passes to publish under the disclosure rule | <10 |
Process figures (addendum A14)
- screening error of the audited exclusions
- Screening error 12.2% (5 of 41, 95% CI 5.3–25.5%), above the 10% target; see Limitations. Published under addendum A14, outside the k = 10 rule: a property of the screener, not of any launch.
- supported and raw collection recall
- Supported recall 0.894 (59 of 66); raw recall 0.887 (141 of 159). Published under addendum A14, outside the k = 10 rule: a property of the collector, not of any launch.
- the count of labels edited by a person
- 0 of 7200 labels edited by a person. Agent roles, no human edits. Published under addendum A14, outside the k = 10 rule: a property of the process, not of any launch.
- the total share of proposed positives changed at confirmation
- 634 of 1610 proposed positives (39.4%) changed at confirmation: Proposed positives rejected at confirmation: 411 of 1610 (26%, 95% CI 23–28%); Proposed passes downgraded to claimed at confirmation: 223 of 1610 (14%, 95% CI 12–16%). Published under addendum A14, outside the k = 10 rule: a property of the coder and confirmer, not of any launch.
Limitations
- Screening missed its pre-registered target: 12.2% of audited exclusions were in error (5 of 41, 95% CI 5.3–25.5%) against a 10% bar. This is a selection limitation. A re-measure at 2.4% is exploratory: the rule was tightened on the same 41 cases and was not re-applied to the 300.
- Supported recall 0.894 (59 of 66; 10 of 10 projects contributed); raw recall 0.887 (141 of 159).
- Of 159 gathered items, 66 (42%) were supported and 93 (58%) unsupported, 27 (17%) of them boilerplate such as calls to action.
- The paid search collector was replaced by a free own-site collector for all bulk coding (addendum A8), so every launch was collected the same way. Exploratory: on 142 projects the free and the paid collectors agree at a positive-level agreement of about 0.78. The all-states agreement of 0.915 is raised by states both found nothing for and is not the headline.
- A collector defect produced whole-page evidence spans. Every project was re-segmented into spans of at most 1,500 characters, and the decisions made on the defective spans are archived as exploratory and feed no published figure (addendum A9).
- Every role is an agent: screener (nvidia/Nemotron-3_5-Lightning on Nebius Token Factory); coder (deepseek-ai/DeepSeek-V4-Flash-0731 on Nebius Token Factory); second coder (nemotron-3-super on Ollama Cloud); independent labeller (glm-5.3-flash on Ollama Cloud); confirmer (nemotron-3-ultra on Ollama Cloud); screening auditor (kimi-k3 on Ollama Cloud); recall gatherer (kimi-k3 on Ollama Cloud). No human edited a label. Agreement figures are agreement with an independent labeller.
- Evidence is read from text extracted from each launch's own pages. What the extractor could not see, such as pages behind a login or text inside images, does not count as public evidence. Launches whose evidence came only from provider-extracted text: 218 of 300 (73%, 95% CI 67–77%).
- Cohorts are described as they are evidenced today. Older launches have had longer to publish, and launches whose sites disappeared are not in the sample.
- Not publicly evidenced does not mean not done.
- The sampling frame is led by Show HN launches, so the figures describe launches posted there and not AI launches in general.
- Every decision requires at least one check that was withheld at validation. A withheld check enters the deterministic gate as unknown, so no launch can reach PASS for any decision on published states alone. The decision figures show what the released checks can support, not how far launches meet each decision.
Exploratory analyses
Exploratory: collector cross-check (addendum A8)
Exploratory: on 142 projects the free and the paid collectors agree at a positive-level agreement of about 0.78. The all-states agreement of 0.915 is raised by states both found nothing for and is not the headline.
Exploratory: whole-page-span archive (addendum A9)
A collector defect produced whole-page evidence spans. Every project was re-segmented into spans of at most 1,500 characters, and the decisions made on the defective spans are archived as exploratory and feed no published figure (addendum A9).
Exploratory: screening re-measure
A re-measure at 2.4% is exploratory: the rule was tightened on the same 41 cases and was not re-applied to the 300.
Exploratory: community layer (addendum A4)
The community layer (addendum A4) is not part of this release.