Concepts
Experiments
Every change Loop ships can go out as an experiment: half the audience gets it, half is held back, and the verdict says what happened, including when the honest answer is that nothing did.
What an experiment is made of
This area has no published API yet.
Segments, analytics, experiments, messages, imports and retention are reached through the console's own API, which exists to serve one first-party client and changes whenever that client does. Rather than print a shape we would have to break, this page describes what is structurally true. The public reference lands when the interface is stable enough to promise.
| Part | What it does |
|---|---|
| variants | The arms. One is the control; the rest are what you are trying. |
| holdout | The people deliberately shown nothing, so there is something to compare against. |
| primary metric | The one number that decides it, with a direction. |
| guardrail metrics | Numbers that must not get worse, even if the primary one improves. |
The direction has no default, deliberately.
Whether up is good is a fact about your business, not about the metric. Support tickets going up is bad; invites going up is good. Loop refuses to guess, so the form will not submit without it.
The holdout, and why null is not it
Someone in the holdout is in the experiment and was shown the untreated path on purpose. Someone with no arm at all is not in the experiment: it is not running, or they are not eligible, or the decision has not landed.
Render the holdout deliberately.
In your own UI, loop.variant() returns null for 'not in this test' and the arm key for everyone else, including 'holdout'. Treating null and holdout the same way is how a holdout stops being measurable.
The four verdicts
| Verdict | What it means |
|---|---|
| measuring | Running. Not enough has happened to say anything yet. |
| not_significant | Enough has happened, and the difference is not distinguishable from noise. |
| improvement | The primary metric moved the way you said was good, beyond noise. |
| decline | It moved the other way. |
not_significant is a result, not a failure.
It is the answer to 'did this work?', and the answer is no, or not enough to tell. Loop never dresses it as a win, never shows a lift it cannot support, and gives it the same weight as the other three. An interval is always shown with the headline, because a number without one is an opinion.
Deciding, and who shipped it
An experiment ends in a decision, roll out or revert, and the record says whether a human or the agent made it. That is the audit trail the approval gate exists to produce.
There are no email arms.
An experiment's arms ship in-app guides. A lifecycle email cannot currently be one side of a test, so an email you send alongside an experiment reaches both arms and will muddy the read.