State Farm presentation at the Chief Analytics Officer, Fall 2016

1. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization Do You Know or Do You Think You Know? Incorporating Business Experiments Into Strategy Development Andy Pulkstenis Director, Advanced Analytics Andy.Pulkstenis.re6i@StateFarm.com
2. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 1 • Setting the context • State Farm examples, from simple to sophisticated • Blueprint for getting testing started (or growing testing) at your firm • Responding to resistance • Setting the context • State Farm examples, from simple to sophisticated • Blueprint for getting testing started (or growing testing) at your firm • Responding to resistance What are we doing today?
3. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 2 What is a “designed experiment?” Designed testingis a way of treatingthe business environment like a controlled experimental setting, randomlyassigning strategiesto customersas in a clinical drug study or lab Benefits • Clear, accurateinsightsabout causation • Ultra-cleandatafor model development = more accurate prediction • Enables strategyoptimizationatthe customer level
4. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 3 Innovators, on Testing “By thorough experimentation and trial-and- error in the market, [successful companies] iterate towards a strategy and a business model that really is viable.” - Harvard Business School Professor Clayton Christensen, a recognized authority on “disruptiveinnovation”
5. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 4 - Rich Fairbank, Chairman & CEO Capital One “What we’ve done is to create an innovation machine. This is not just a credit card strategy, testing is a revolution that can be applied to many businesses.” Innovators, on Testing
6. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 5 - Jeff Bezos, Amazon Founder “If you double the number of experiments you do per year, you’re going to double your inventiveness.” Innovators, on Testing Fun fact: Amazon once fired a team for implementing strategic change without proper statistical testing first!
7. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 6 How do businesses use testing? Strategy Optimization Targeted Messaging Pricing Real-time Website Design/Optimization Customer-level Decisioning Retention & Save Strategies Product & Strategy Development Operations Improvement
8. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 7 Specific business examples Sources: “How to Design SmartBusiness Experiments” by Tom Davenport,HBR 2/2009 “The Discipline of BusinessExperimentation” by Stefan Thomkeand Jim Manzi, HBR 12/2014 Do visible lobster tanks increase lobster sales? Are higher bids driven by CC payment option? Will opening stores an hour later lead to a drop in sales? Nearly all aspects of business.
9. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 8 Typical Testing Evolution None A/B “one factor at a time” (OFAT) Multivariate (MVT) MVT w/ covariate analysis Pick “one winner forall” Increased businessinsight while progressingleft to right Enables customer or segment- level strategy/message optimization Companies typically flow through an evolution when embarking on testing agendas Continuoustesting A pervading culture of evolving,continuous,and cyclical testing provides strategic foundation Many stop at A/B or OFAT through lack of knowledge or lack of skillset, missing out on benefits that increase with each successive milestone.
10. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 9 • Setting the context • State Farm examples, from simple to sophisticated • Blueprint for getting testing started (or growing testing) at your firm • Responding to resistance What are we doing today?
11. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 10 Which banner led to the most visits to the “Agent Search” page? Version B: Version A: (Click leads to “Drive Safe & Save” page) (Click leads to “Agent Search” page)
12. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization Answer: Version B led more people to the Agent Search page
13. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 12 Which statefarm.com banner led to the most visits to the “Drive Safe & Save” page? Version B: Version A: (Click leads to “Drive Safe & Save” page) (Click leads to “Agent Search” page)
14. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization “Find an Agent” drove more people to the “Drive Safe & Save” page than the “Drive Safe & Save” experience did!
15. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 14 Version A: Version B: Which mailer generated the highest lift in Life insurance sales?
16. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 15 Claims Contact Center Claims Contact Center – “Out of Scope” Calls Can proactive sharing of contact info reduce misdirected or “out of scope” Claims calls?
17. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 16 Multivariate Discount Double Check Test • Explored 7 different page features as tested ‘factors’ (text, graphics, layout, images, colors, etc.) • Results in 384 distinct web page variations (with a subset of 200+ valid combinations) • Only needed to physically test 58 combinations • Improved quote starts by over 50% and quote completes by 10%
18. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 17 • Setting the context • State Farm examples, from simple to sophisticated • Blueprint for getting testing started (or growing testing) at your firm • Responding to resistance What are we doing today?
19. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 18 My roadmap for launching testing Be Persistent It will take legwork and determination to find the partner in step two Be Ambitious Aim for moderate or greater business impact Be Careful Carefully execute the approach with business partners Be Ready Build the “success story” deck before you need it Be A Teacher There is a significant initial education component Be Opportunistic Follow the path of least resistance
20. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 19 Best Practices: Set Up for Success Assemble the right structure Identify key roles at the outset (business, stats/DOE, execution, structural or creative design) Assemble the right players Don’t just fill seats with “good people.” If you need an SME in a certain area, then staff that role with an SME Assemble the right questions A surprising amount of time in early DOE engagements is spent helping the business partner define the problem as a quantifiable hypothesis Assemble the right scope Too ambitious and you fail spectacularly Not ambitious enough and you really don’t move the needle Assemble the right controls Make sure to monitor execution and data collection from Day 1 and throughout the process
21. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 20 • Setting the context • State Farm examples, from simple to sophisticated • Blueprint for getting testing started (or growing testing) at your firm • Responding to resistance What are we doing today?
22. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 21 Common Arguments Regulation Initial resistance to statistical testing typically falls into three categories: Complication Organizational Tradition
23. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 22 Regulation Our industry is too heavily regulated for this Didn’t stop Finance, Pharma, and others from successful transition to DOE Even if allowed, the regulators would never understand it It’s not “fair” to randomly assign strategies to customers in a test Successful testing industries proactively educate regulators on key concepts • Current strategy isn’t “fair” either • Jonas Salk & polio • “Beautiful experiment over which the epidemiologist could become quite ecstatic but would make the humanitarian shudder” Perceived Barrier Response
24. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 23 Complication Too complicated Nonsense – no more complicated than predictive modeling, just new to your firm Ignoring complex relationships doesn’t mean they aren’t there Start small, manageable • Design accordingto cost constraints • Much cheaper than hoping we guessed correctly (think of it as “buying data”) This can be handled by applying exclusions to the design Complexity of analysis Too difficult to execute Some combinations don’t make business sense Too expensive Perceived Barrier Response
25. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 24 “We tried it once and we saw no improvement” (i.e. learned nothing) “We intuitively know our business” Perceived Barrier “Our external consulting firms tell us all we need to know about course correcting” Organizational Tradition • You don’t hit a home run every time • You learned that those strategies were equivalent • Often gain fresh, real insights that defy historical logic or perceivedwisdom • “The greatest obstacle to discoveryis not ignorance - it is the illusion of knowledge.” - Daniel Boorstin Response Designed testing consistently outperforms consultant-provided solutions “I have not failed 10,000times.I have succeededin provingthatthose 10,000 wayswill not work. When I have eliminatedthe ways thatwill notwork, I will find the way thatwill work.” - ThomasEdison Today WD-40 can be found in 4/5 UShomes WD-1 WD-39 “didn’twork!”
26. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 25 Questions/Comments Andy.Pulkstenis.re6i@StateFarm.com
27. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 26 Appendix
28. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 27 Engagement Model Business Partner Statistical Design Implementation/Execution Structural and Creative Design TestingTeam Ensure alignment with broader strategic goals Strategic Approval “Testing Council” of SME’s ensures alignment with statistical best practices and production constraints Structural Approval • StatisticalDOE • ProductionEnvironmentDocument key hypotheses, findings, participants, and actions taken Test Results Repository Execute/Monitor/Analyze/Take Action Ensure alignment with broader strategic goals Strategic Approval Execute/Monitor/Analyze/Take Action “Testing Council” of SME’s ensures alignment with statistical best practices and production constraints Structural Approval
29. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 28 What do I mean by “test-and-learn culture?” • Designed experiments inform (and determine) strategy development • Understanding of when to settle for simple “A/B” testing and when to use more sophisticatedmultifactor tests • Internal standards/bestpractices for test design • Formal internal governance A true testing culture demonstrates the following elements:   
30. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 29 Four “Test Factors”: • Engagement (3) • Primary Emphasis (2) • Tabs (2) • George (2) “Quote” “Yes” “Phone” A Multivariate Test Example “Simple” “Active” “Yes” “No” “Passive” “No” This results in 24 combinations or “test cells” Metric of interest: quote starts
31. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 30 So we tested a statistically chosen subset of 12 that allows us: • To draw conclusions about all 24 combinations • Make inferences about the impact of individual factors and associated interactions Physically testing all 24 combinations is undesirable: • Implementation complexity I applied a “reduced” design
32. Contains CONFIDENTIAL information which maynot be disclosed without express written authorization 31 • We learned “Why?” not just “Which one?” • Discovered interactions that had been downplayed as unlikely • Better business result: - 12% > than current page - 10% > than vendor-recommended combo • Set the stage for a 7-factor 200+ combination comprehensive test • “George” Benefits of using multifactor testing