Outbound A/B test Claude skill

Use this to run outbound tests that produce decisions. Most outbound "tests" compare 40 sends to 40 sends and crown a winner by noise.

Inputs

• The current campaign and its baseline numbers.
• The metric to improve (usually positive reply rate or meetings per 100 contacts).
• How many contacts per week the team can send to the test.

The play

1. Test the biggest lever first

Impact order, roughly: list and segment > angle and offer > first line and personalization > CTA > subject line > send time. Testing send time while the list is wrong wastes a month.

2. Write a hypothesis

"Leading with the hiring trigger instead of the pain will raise positive replies for SDR leaders, because the trigger shows why now." One variable changes; everything else stays the same.

3. Size it

Required contacts per variant depend on baseline and the lift you want to detect:

(Approximate, 80% power, 5% significance.) Rare outcomes need big samples. If volume cannot reach the size, test bigger changes (detect 2x), use total reply rate as a leading metric (it has a higher baseline), or run the test across several weeks of sends.

4. Split fairly

Random assignment within the same segment, same senders spread across both variants, same days and times. Never run variant A this week and B next week.

5. Decide in advance

Write the stop rule: the sample size, the metric, and what counts as a win. Check for guardrails too: bounce and unsubscribe rates must not rise.

6. Read and log

Report both variants with counts, the difference and whether it clears the rule. "No difference" is a result: keep the cheaper or simpler variant. Log every test so the team does not re-run old ones.

Output format

Test: <name>   Hypothesis: <...>
Variable: <one>   Held constant: <list, senders, timing, steps>
Metric: <primary> | Guardrails: bounces, opt-outs
Size: N per variant | Expected duration: N weeks
Decision rule: <...>
Readout: A x/N (x%) vs B x/N (x%) → winner / no difference → action

What good looks like

• One variable per test.
• Sample size decided before launch.
• The log shows what was learned, including tests that changed nothing.
• The common mistake is judging on open rates, which are inflated by privacy features and say nothing about replies.

Rules

• MUST change only one variable per test.
• MUST set sample size and decision rule before launch.
• NEVER declare a winner below the planned sample, and never on fewer than 200 contacts per variant.
• NEVER use open rate as the deciding metric.

Example prompts

• "Design a test: trigger-led vs pain-led first email for VP Sales. We send 400 a week."
• "Here are results for two variants. Is B really better?"
• "Plan the next 3 outbound experiments for our team, in order of impact."