Optimize testing design leads to better business outcomes
- Estimate P&L Impact of proposed Loyalty Program features
- ROI maximization and risk mitigation
- Improved decision making via higher confidence in testing outcomes balanced rigor vs testing speed and size
IIdentify measurable KPI Metrics
- Identify hypothesis and number of testing/control groups
- Identify KPIs where client can measure true lift in the shortest amount of time (e.g., increased onboard spend, 60-day rebooking rate)
- Identify key population segments and determine whether they can feasibly support an A/B/n test
Identify Test and Control Groups
- Determine sample sizes and test duration using Power Analysis to ensure feasibility
- Select testing/control sailings to maximize similarity between testing groups using clustering and optimization techniques
- Optimize guest population to measure KPIs against using a matched-pairing design (not random) and indexing
Evaluate Test's Business Benefit
- Quantify difference in guest behavior through statistical tests to determine the signal from the noise
- Evaluate immediate business impacts of the pilot by monitoring short-term KPIs
- Monitor long-term guest behavior with more robust KPIs to augment findings