PRISM · Release v1

Evidence by discovery source

Each bar is the share of one discovery source's included launches. A source with no included launches is shown as not in this release, and small groups as <10 or suppressed.

Reported descriptively: at least one source has fewer than 30 included launches, so the pre-registered test was not run. The Holm correction covers the remaining hypotheses.

Figure 8. Discovery sources: Keep exploringshare of the source's included launches
Figure 8. Discovery sources: Keep exploringShow HN: <10. Reddit: not in this release: no Reddit launches met the inclusion criteria. GitHub-only: not in this release: no GitHub-only launches met the inclusion criteria0%25%50%75%100%Show HNShow HN: Fewer than 10 launches; suppressed by disclosure control<10RedditReddit: not in this release: no Reddit launches met the inclusion criterianot in this release: no Reddit launches met the inclusion criteriaGitHub-onlyGitHub-only: not in this release: no GitHub-only launches met the inclusion criterianot in this release: no GitHub-only launches met the inclusion criteria
Table view
Figure 8. Discovery sources: Keep exploring
RowLaunchesShare95% CI
Show HN<10
Redditnot in this release
GitHub-onlynot in this release
Figure 9. Discovery sources: Permit reliance for the named useshare of the source's included launches
Figure 9. Discovery sources: Permit reliance for the named useShow HN: <10. Reddit: not in this release: no Reddit launches met the inclusion criteria. GitHub-only: not in this release: no GitHub-only launches met the inclusion criteria0%25%50%75%100%Show HNShow HN: Fewer than 10 launches; suppressed by disclosure control<10RedditReddit: not in this release: no Reddit launches met the inclusion criterianot in this release: no Reddit launches met the inclusion criteriaGitHub-onlyGitHub-only: not in this release: no GitHub-only launches met the inclusion criterianot in this release: no GitHub-only launches met the inclusion criteria
Table view
Figure 9. Discovery sources: Permit reliance for the named use
RowLaunchesShare95% CI
Show HN<10
Redditnot in this release
GitHub-onlynot in this release
Figure 10. Discovery sources: Sell, operate, support, or staff repeatedlyshare of the source's included launches
Figure 10. Discovery sources: Sell, operate, support, or staff repeatedlyShow HN: <10. Reddit: not in this release: no Reddit launches met the inclusion criteria. GitHub-only: not in this release: no GitHub-only launches met the inclusion criteria0%25%50%75%100%Show HNShow HN: Fewer than 10 launches; suppressed by disclosure control<10RedditReddit: not in this release: no Reddit launches met the inclusion criterianot in this release: no Reddit launches met the inclusion criteriaGitHub-onlyGitHub-only: not in this release: no GitHub-only launches met the inclusion criterianot in this release: no GitHub-only launches met the inclusion criteria
Table view
Figure 10. Discovery sources: Sell, operate, support, or staff repeatedly
RowLaunchesShare95% CI
Show HN<10
Redditnot in this release
GitHub-onlynot in this release
Figure 11. Discovery sources: Determine whether the initiative changed the resultshare of the source's included launches
Figure 11. Discovery sources: Determine whether the initiative changed the resultShow HN: <10. Reddit: not in this release: no Reddit launches met the inclusion criteria. GitHub-only: not in this release: no GitHub-only launches met the inclusion criteria0%25%50%75%100%Show HNShow HN: Fewer than 10 launches; suppressed by disclosure control<10RedditReddit: not in this release: no Reddit launches met the inclusion criterianot in this release: no Reddit launches met the inclusion criteriaGitHub-onlyGitHub-only: not in this release: no GitHub-only launches met the inclusion criterianot in this release: no GitHub-only launches met the inclusion criteria
Table view
Figure 11. Discovery sources: Determine whether the initiative changed the result
RowLaunchesShare95% CI
Show HN<10
Redditnot in this release
GitHub-onlynot in this release
Figure 12. Discovery sources: Scale, renew, expand, or stopshare of the source's included launches
Figure 12. Discovery sources: Scale, renew, expand, or stopShow HN: <10. Reddit: not in this release: no Reddit launches met the inclusion criteria. GitHub-only: not in this release: no GitHub-only launches met the inclusion criteria0%25%50%75%100%Show HNShow HN: Fewer than 10 launches; suppressed by disclosure control<10RedditReddit: not in this release: no Reddit launches met the inclusion criterianot in this release: no Reddit launches met the inclusion criteriaGitHub-onlyGitHub-only: not in this release: no GitHub-only launches met the inclusion criterianot in this release: no GitHub-only launches met the inclusion criteria
Table view
Figure 12. Discovery sources: Scale, renew, expand, or stop
RowLaunchesShare95% CI
Show HN<10
Redditnot in this release
GitHub-onlynot in this release

Checks by discovery source

Table view: checks by discovery source
Checks by discovery source: share of the group's included launches, per check and state.
CheckShow HNRedditGitHub-only
The artefact or action can be reproduced or inspected. (publicly evidenced)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
The artefact or action can be reproduced or inspected. (claimed, no artefact linked)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
Important supplied context, assumptions, and inferences are visible. (publicly evidenced)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
Important supplied context, assumptions, and inferences are visible. (claimed, no artefact linked)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
The intended use and boundaries are explicit. (publicly evidenced)169 of 300 (56%)not in this releasenot in this release
The intended use and boundaries are explicit. (claimed, no artefact linked)62 of 300 (21%)not in this releasenot in this release
Acceptance criteria are explicit and testable enough for the use. (publicly evidenced)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
Acceptance criteria are explicit and testable enough for the use. (claimed, no artefact linked)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
Evidence shows the acceptance criteria were met. (publicly evidenced)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
Evidence shows the acceptance criteria were met. (claimed, no artefact linked)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
Relevant failure modes and fallback behaviour were evaluated. (publicly evidenced)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
Relevant failure modes and fallback behaviour were evaluated. (claimed, no artefact linked)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
Known limitations and out-of-scope conditions are stated. (publicly evidenced)162 of 300 (54%)not in this releasenot in this release
Known limitations and out-of-scope conditions are stated. (claimed, no artefact linked)<10not in this releasenot in this release
The Deliverable gate for the intended use has passed. (publicly evidenced)<10not in this releasenot in this release
The Deliverable gate for the intended use has passed. (claimed, no artefact linked)<10not in this releasenot in this release
Operational ownership is explicit. (publicly evidenced)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
Operational ownership is explicit. (claimed, no artefact linked)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
The assurance mechanism is defined for the important failure modes. (publicly evidenced)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
The assurance mechanism is defined for the important failure modes. (claimed, no artefact linked)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
Fallback or escalation is defined. (publicly evidenced)44 of 300 (15%)not in this releasenot in this release
Fallback or escalation is defined. (claimed, no artefact linked)<10not in this releasenot in this release
Recovery or rollback has been tested where failure consequence warrants it. (publicly evidenced)<10not in this releasenot in this release
Recovery or rollback has been tested where failure consequence warrants it. (claimed, no artefact linked)<10not in this releasenot in this release
Support, maintenance, debugging, and change responsibility are defined. (publicly evidenced)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
Support, maintenance, debugging, and change responsibility are defined. (claimed, no artefact linked)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
The relevant operating-cost boundary is known well enough for the decision. (publicly evidenced)104 of 300 (35%)not in this releasenot in this release
The relevant operating-cost boundary is known well enough for the decision. (claimed, no artefact linked)27 of 300 (9%)not in this releasenot in this release
The outcome measure is defined. (publicly evidenced)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
The outcome measure is defined. (claimed, no artefact linked)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
A comparable baseline or counterfactual is defined. (publicly evidenced)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
A comparable baseline or counterfactual is defined. (claimed, no artefact linked)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
The outcome was measured after the intervention. (publicly evidenced)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
The outcome was measured after the intervention. (claimed, no artefact linked)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
Before/after measurements use the same relevant definition. (publicly evidenced)<10not in this releasenot in this release
Before/after measurements use the same relevant definition. (claimed, no artefact linked)<10not in this releasenot in this release
Material confounds, uncertainty, or attribution limits are named. (publicly evidenced)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
Material confounds, uncertainty, or attribution limits are named. (claimed, no artefact linked)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
Outcome evidence relevant to the decision is available. (publicly evidenced)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
Outcome evidence relevant to the decision is available. (claimed, no artefact linked)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
The full relevant cost boundary is accounted for. (publicly evidenced)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
The full relevant cost boundary is accounted for. (claimed, no artefact linked)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
Material risks and trade-offs are assessed. (publicly evidenced)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
Material risks and trade-offs are assessed. (claimed, no artefact linked)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
Reasonable alternatives or the status quo are compared. (publicly evidenced)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
Reasonable alternatives or the status quo are compared. (claimed, no artefact linked)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
The rule for scale, renew, expand, or stop is explicit. (publicly evidenced)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule
The rule for scale, renew, expand, or stop is explicit. (claimed, no artefact linked)withheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rulewithheld: withheld under the pre-registered shortfall rule