Revenue Experiments: Test Offers on Customer Segments Without Guessing

Why run experiments instead of campaigns

A campaign tells you that a message went out. An experiment tells you that a message caused revenue. The difference is the holdout group: a randomly assigned slice of the audience that receives nothing, which becomes your baseline for what would have happened without the offer.

• Test one hypothesis at a time so the result is attributable.
• Keep assignment persistent — a customer must never switch variants mid-experiment.
• Use a holdout whenever possible; without one you get raw metrics, not incremental revenue.
• Decide in advance what number will tell you the experiment worked.

Start with a hypothesis, not a metric

A useful hypothesis names the audience, the offer, and the expected outcome. “Customers who completed a book but have not published will pay €149 for done-for-you publishing” is testable. “Let’s send something to everyone” is not.

1. 1

   Name the audience

   Use a segment you can defend: completed a book, exported it, has not bought publishing support yet.

2. 2

   Name the offer

   State the exact offer, price, and call to action the treatment will receive.

3. 3

   Name the outcome

   Pick one primary conversion event — a purchase — and attach a value (the price, or tracked revenue).

4. 4

   Name the decision

   Write what you will do if the treatment wins, ties, or loses against the holdout.

Choose the audience with a segment

Build the audience with the same segment engine you already use for campaigns, and preview the matching customers before starting. Check that the count is meaningful and that the rules describe the people the hypothesis is about. Most experiments fail before they start because the audience was wrong, not because the offer was bad.

• Preview the matching customers and spot-check a few.
• Exclude people who already bought the offer or opted out of marketing.
• Do not start an experiment with an empty or tiny audience.
• Record the segment and its count with the experiment so the definition survives.

Design variants and the holdout

Allocations must total exactly 100% and the holdout should never receive campaign messaging. A simple two-arm design — one treatment and a holdout — is enough for most tests; add a second treatment only when the message really differs. Assignment should be deterministic: the same customer and experiment always resolve to the same variant, and re-running assignment must never change existing buckets.

Sync audiences to Brevo

SegFuse creates one Brevo list per treatment variant in a dedicated folder, so several experiments can run at once without touching your existing lists. The holdout is deliberately not synced to any campaign audience.

• Sync is idempotent — re-running it does not duplicate contacts or re-bucket anyone.
• Holdout customers are never added to treatment lists.
• Unsubscribed contacts are excluded automatically.
• Keep the sync status and last sync time visible so a stale audience is obvious.
• Send the campaign from Brevo; SegFuse does not send email.

Track the funnel to conversion

The experiment records the journey from assignment to purchase: exposure, click, offer viewed, checkout started, converted. Where the customer is known (logged in), attribute by customer; where they arrive from a link, use signed attribution parameters on the offer URL so the visit stays linked to the right variant.

• Append experiment and variant parameters to every offer link.
• Report conversion events with customer id, amount, and currency through the events API.
• Deduplicate conversions so one purchase is never counted twice.
• Match revenue to the variant when the customer is identifiable.

Calculate incremental revenue honestly

Incremental revenue compares each treatment against the holdout baseline. Baseline conversion rate is holdout conversions divided by holdout assignments. Expected conversions among a treatment are its assignments times that baseline rate; incremental conversions are actual minus expected; incremental revenue is incremental conversions times the tracked average revenue (or the configured conversion value when revenue is not tracked).

• Label the result as an estimate — holdout comparison, not statistical significance.
• Require a minimum holdout size and a minimum number of conversions before claiming anything.
• A treatment must beat the holdout by a meaningful margin to be called promising.
• With no holdout, report raw conversion and revenue metrics and say nothing about lift.

Read the results and decide

The recommendation is deterministic and explainable — it states which variant is leading and why. Do not treat a small-sample lead as a guarantee; the point of the experiment is a clearer decision, not a fake p-value.

• 🟢 Promising — a treatment beats the holdout by a meaningful margin; keep collecting data.
• 🟡 Needs more data — the holdout or conversion count is still too small to conclude anything.
• 🔴 No evidence of lift — treatments have not outperformed the holdout; consider pausing or changing the offer.
• Apply the decision you wrote in the hypothesis: continue, adjust, or roll the winner into a campaign.

A launch checklist

• Hypothesis names audience, offer, outcome, and decision.
• Segment rules reviewed and matching customers previewed.
• Allocations total 100%, one holdout, no holdout messaging.
• Assignment run and counts verified per variant.
• Brevo lists synced; holdout absent from every campaign list.
• Offer links carry attribution parameters; events API wired to the checkout.
• Conversion dedupe and unsubscribe exclusion confirmed.
• Minimum holdout size and conversion thresholds chosen before launch.