What you can honestly conclude without the traffic to test
When a controlled test cannot reach a trustworthy answer in a useful time, use evidence matched to the decision and record what it can and cannot establish.
Skip this if head-of-ecommerce at scaling: with testing infrastructure and volume, the relevant lesson is how to read results honestly, not how to proceed without them
If the controlled test you want cannot reach a trustworthy answer in a useful time, do not force a winner. First check whether the test is feasible. Define the outcome metric and baseline, the smallest effect worth acting on, desired confidence and power, eligible traffic and allocation, and acceptable run time. Sample needs change with those inputs, especially effect size and outcome variability. Order count by itself cannot settle the question.[2]
The decision: when a trustworthy controlled test is not feasible, choose a claim-appropriate non-causal evidence route and label the decision untested. Use the route that matches the question. Combine routes when one source cannot carry the decision.
The rest of this lesson is available once the Atlas is unlocked.