Hide Notes 

Slide 1: Deriving Key Insights from Blue Martini Business Intelligence Ronny Kohavi Vice President, Business Intelligence Blue Martini Software Anne Ford E-Commerce Project Manager MEC Bill Maginn Internet Controller Debenhams © Copyright 2003, Blue Martini Software. San Mateo California, USA 1
Slide 2: Agenda        Debenhams and MEC Web site, DSSGen, and Bots Easy insights – out of the box reports Harder insights – what you could write Actions at Debenhams and MEC BIG ROI project Q&A 2 © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Slide 3: Debenhams  Debenhams plc, “Britain's favourite department store” – UK's fashion retail leader for more than 200 years – Runs approximately 100 department stores in the UK and Ireland – Offering brand-name women's, men's and children's apparel as well as high-end housewares and cosmetics – Debenhams also offers the award winning ‘Queen's Award for Enterprise - April 2001’ bridal registry service  Debenhams online profile – Over 500,000 customers – About £18 million in total sales in the last year – Wedding list purchases account for about half of sales  Clickstreams – About 2M page views / week – About 6,000 new customers per week © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 3
Slide 4: MEC    MEC – Mountain Equipment Co-op Canada’s leading supplier of quality outdoor gear and clothing MEC has – 1.8 million members – Sales over $160 million – Seven physical retail stores  As a co-op, MEC supports the community in several ways. – One of them is gearswap, an area of the site for selling used gear, where MEC makes no money – Products link to information promoting environmental responsibility © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 4
Slide 5: Survey We asked early participants what they wanted to hear in this webinar 12 Observation: Sample count 10 8 6 4 2 0 Bo t M ig ra to r As so Cr c os sCa sell m pa ig n Us ab ilit y Se ar ch Ad ef f Re po rti ng G e Re o fe rre r 5 • Everyone wants to learn how to start fires • Few are thinking about fire safety • What you don’t know can hurt you - bots Vi z Options © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Slide 6: Bot Detection (Fire Safety)  Bots are automated programs, sometimes called crawlers/robots Examples: search engines, shopping bots, performance monitors   Significant traffic may be generated by bots Can you guess what percentage of sessions are generated by bots at MEC and Debenhams? 23% at MEC 40% at Debenhams  Without bot removal, your metrics will be inaccurate Blue Martini has good heuristics, but look at the bot report and make sure your performance monitors are recognized  © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 6
Slide 7: Website Checklist  Collect clickstreams at 100% – You can’t improve what you don’t measure      Turn ReverseDNS on to get host names, not IPs (note, this is off by default) Setup home page redirect correctly so that you do not lose referrers (avoid client-side redirects) When customizing site, don’t lose business events (search, checkout, etc) Do not run reports against web site DB, unless they are small. That’s why there is the DSS DB More details in “Business Intelligence - Getting Started Guide” on http://developer.bluemartini.com 7 © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Slide 8: Real Time Reporting Against Store   Sometimes executing a small query against live DB makes sense. For example, on the www.bluemartini.com site, we provide our salespeople with a live query JSP that shows visitors from any domain to help sales activites. Example report below. For you, this could be useful for customer support, for example IE6 user Running Win 2000 Came from our press release on Yahoo  © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 8
Slide 9: Setting Session Timeout   Set your session timeout to an hour Note the impact session timeout has on the percent of people who will lose their shopping cart (or see annoying message) Watch for an upcoming article on developer.bluemartini.com around this recommendation Recommended timeout duration is 60 mins 2.5% of sessions with an item in cart will experience timeout © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 9
Slide 10: DSSGen    Reports should run against a data warehouse (DSS) DSSGen builds the data warehouse from the store/click/main DBs Normally, such an ETL (Extract/Transform/Load) process takes months to develop, but with Blue Martini you get 95% of it out of the box The other 5% are: – Adding your custom tables – Marking performance monitors/bots – Changing things due to web site customizations   Common mistakes – Inappropriate hardware – Build and Reporting need strong hardware – Bad database setup (Oracle must be tuned) – Running DSSGen without –parallel flag – Running all reports every DSSGen run instead of daily, weekly, monthly options © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 10
Slide 11: Reports  Blue Martini provides two ways to build reports – Crystal Reports    Industry standard Easy to layout reports Limited capability to transform data (e.g., can’t sort by percentages requiring multiple passes) Use transformation chains Can integrate multiple investigations into unified report Generic reports easy, custom reports require JSP coding Visualizations are interactive 11 – Blue Martini Reports     © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Slide 12: Dataviz Webstart   Every graph has a “dataviz” icon Example: the standard dashboard Demo   View->Scatterplot Right click Settings, X: Day of week Y: hour of day, size: heatmap, color -> web visits 12 © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Slide 13: Easy Insights – Out of the Box Reports  With the Analysis Center, you get an extensive set of reports for web analytics and sales analysis Here are some examples from Debenhams and MEC  © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 13
Slide 14: Search Effectiveness at MEC   Customers that search are worth two times as much as customers that do not search Failed searches hurt sales Visit 10% 90% Search (64% successful) Avg sale per visit: 2.2X No Search Avg sale per visit: $X 70% Last Search Succeeded Avg sale per visit: 2.8X 30% Last Search Failed Avg sale per visit: 0.9X 14 © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Slide 15: Top Searches  Top searched keywords (percent of searches) – Empty search string (3.9%) returns over 160 results – GPS (1.2%) – sunglasses (0.8%) – watches (0.6%) – nalgene (0.5%) – ecological footprint (0.5%) Recommendation: - Do not allow empty search - Create custom pages for often searched keywords  Top failed keywords in the product category (percent of failed searches) – gift certificate(s) (0.98%) (already implemented since study) – arc’teryx (0.44%) – bear spray (0.44%) – pedometer (0.37%) – stroller(s) (0.36%) – north face (0.33%) – (gift) registry (0.21%) Recommendation: - Parse search string to remove special characters - Build extensive thesaurus - Consider carrying products 15 © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Slide 16: Referrers at Debenhams  Top Referrers – Google    5.8% of all visits are referred by google Visit to purchase conversion 1.6% Average purchase per visit = 1.8X – MSN (including search and shopping)    11% of all visits are referred by MSN Visit to purchase conversion 0.7% Average purchase per visit = X Recommendation Define an ad strategy based on ROI Emphasize AOL – AOL search    0.62% of all visits are referred by AOL Search Visit to purchase conversion 2.6% Average purchase per visit = 4.8X 16 © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Slide 17: Micro-Conversion Rate at Debenhams      Understand abandonment and conversions Not just visitor to purchaser, but also the micro-conversions Shopping Cart Abandonment 62% =55% + 45% * 17% Abandonment varies from about 25%-80% across sites Excellent opportunity to identify problematic steps in processes and improve Also a good way to identify abandoned products, send targeted e-mails if those products are on sale © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 2.0% 7.7% 25% 2.3% 6% 45% 83% 17%  55% 17
Slide 18: Acxiom   BMS supports ADN – Acxiom Data Network Seamless integration: get username/password Note: Acxiom recently changed their interface, so you will need a patch Comprehensive collection of US consumer and telephone data available via the internet – – – – Multi-sourced database Demographic, socioeconomic, and lifestyle information. Information on most U.S. households Contributors’ files refreshed a minimum of 3-12 times per year.  – Data sources include: County Real Estate Property Records, U.S. Telephone Directories, Public Information, Motor Vehicle Registrations, Census Directories, Credit Grantors, Public Records and Consumer Data, Driver’s Licenses, Voter Registrations, Product Registration Questionnaires, Catalogers, Magazines, Specialty Retailers, Packaged Goods Manufacturers, Accounts Receivable Files, Warranty Cards 18 © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Slide 19: Example - Income  Graph showing incomes for a company that targets high-end customers based on POS purchases Income of their customers in blue The US population in red Note highest bracket (30% vs. 5% for US) Percent   © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 19
Slide 20: Product Affinities   Which products sell well together Together these form a model which can be used as a Product Recommender Note: this does not ship by default as an AC report, but as an example investigation © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 20
Slide 21: Product Affinities at MEC Product Orbit Sleeping Pad Association Orbit Stuff Sack Bambini Crewneck Sweater Children’s Lift 222 Confidence 37% Website Recommended Products Cygnet Sleeping Bag Aladdin 2 Backpack Primus Stove Bambini Tights Children’s 195 52% Yeti Crew Neck Pullover Children’s Beneficial T’s Organic Long Sleeve T-Shirt Kids’ Silk Crew Women’s Silk Long Johns Women’s 304 73% Micro Check Vee Sweater Volant Pants Composite Jacket Cascade Entrant Overmitts    Polartec 300 Double Mitts 51 48% Volant Pants Windstopper Alpine Hat Tremblant 575 Vest Women’s Minimum support for the associations is 80 customers Confidence: 37% of people who purchased Orbit Sleeping Pad also purchased Orbit Stuff Sack Lift: People who purchased Orbit Sleeping Pad were 222 times more likely to purchase the Orbit Stuff Sack compared to the general population 21 © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Slide 22: Product Affinities at Debenhams Product Association Lift Confidence Website Recommended Products Fully Reversible Mats Egyptian Cotton Towels 456 41% J Jasper Towels Confidence 1.4% White Cotton T-Shirt Bra Plunge T-Shirt Bra 246 25% Black embroidered underwired bra    Minimum support for the associations is 50 customers Confidence 1% Confidence: 41% of people who purchased Fully Reversible Mats also purchased Egyptian Cotton Towels Lift: People who purchased Fully Reversible Mats were 456 times more likely to purchase the Egyptian Cotton Towels compared to the general population 22 © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Slide 23: Identifying Top Products Debenhams It’s hard to update top products manually Website recommended products for Homeware - Bedroom Bombay Quilt Cover Border Quilt Cover Quilt Cover Oxford edge border Polycotton Mattress Cover Set Units: 1.7X Revenue: 0.8Y Units: 10.1X Revenue: 7.3Y Units: 5X Revenue: 6.4Y Units: X Revenue: Y Top Homeware – Bedroom products for the last year Bombay Quilt Cover Elephant Parade Quilt Cover Units: 10.1X Revenue: 7.3Y Opulent Check Quilt Cover Elephant Parade Throw listed not listed x Units: 10X Revenue: 8.2Y not listed x Units: 9.7X Revenue: 6.1Y not listed 23 x Units: 8.7X Revenue: 2.6Y © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Slide 24: Campaign Analysis - Debenhams  Analyze the effectiveness of campaigns Recommendation: Send targeted emails Campaign Emails Sent Opens Clickthroughs 9.3% (52p/email) Orders Campaign 1 100% (4.8p/email) 22% (22.3p/email) 0.07% Campaign 2 100% (0.5p/email) 11% (4.8p/email) 3% (17.9p/email) 0.01% Campaign 3 100% (0.8p/email) 22% (3.6p/email) 5.3% (15.3p/email) 0.01% Click-through rates are high - good 24 © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Slide 25: Campaign and Ad “Tricks”  To track clickthrough from external ads – Use jump pages (e.g., www.foo.com/ad2.jsp) or – Add unused extra parameter to URL   http://www.bluemartini.com/bi&biwebinar=3 The biwebinar=3 will be ignored, but you can then see how many sessions have biwebinar=3 in the first request  Examples – Use with Google adwords – Use with rented lists, which you can’t mail using Blue Martini’s campaign management / RM © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 25
Slide 26: Page Effectiveness Report 14% 3% 2% 8% Percentage of visits clicking on different links 2% 13% 9% 0.6% Top Menu 6% 3% 2% 2% 0.3% 2% 18% of visits exit at the welcome page © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA Any product link 7% 26
Slide 27: Top Links followed from the Welcome Page: Revenue per session associated with visits 1.4X 2.3X X 2.3X 1.3X 5X 1.4X Top Menu 0.2X 4.2X 10X 10.2X 1.2X 1.7X 3.3X Note how effective physical catalog item #s are Product Links 2.1X © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 27
Slide 28: “Harder” Insights  Blue Martini collects a lot of data for which there are no out-of-the-box reports yet Some will be in future releases Some need to be written depending on your attributes, hierarchy, etc Some answer specific questions you have BMAS has developed many such reports 28     © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Slide 29: Top Searches On Search Engines / Internet Portals  Which keywords do people type on popular search engines/internet portals to get to our site? The ‘Referrer’ recorded as part of the Blue Martini clickstream data contains these keywords For example – Google Search: http://www.google.com/search?hl=en&ie=ISO-8859-1&q=jasper+conran &btnG=Google+Search – AOL Search: http://aolsearch.aol.co.uk/web.adp?query=department%20stores%20uk    Extract the keywords, substitute HTML escape characters with their ASCII equivalents (such as ‘space’ for ‘%20’) Determine the top searched keywords  © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 29
Slide 30: Top Searched Keywords (Debenhams)  Variations of Debenhams are by far the most frequent – DEBENHAMS – debenhams.com – debenhams;direct – debenhams department store Recommendation Monitor top searched keywords to identify interesting trends  Other interesting search keywords Google Burberry Calvin Klein Crave Cravela / Cravela Shoes Jane Packer / Suede Boots MSN Search Faith shoes Nike Wedding invitations Luggage Swimwear AOL Search burberry carvela Mother of the bride Skiwear Jane Packer BRIDESMAIDS 30 © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Slide 31: Site Usability  Form errors logged on 5.0 and later – Any form that is filled and has an error caught by the Blue Martini architecture generates a Form Error event – MINE_EVENT table in the DSS database has these – BMAS has written initial reports to analyze typical form errors and help website designers improve the form design – For example, on MEC there were thousands of errors on the member application page and address change page   Many errors for fields that cannot be empty Many mismatches between postal code and region © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 31
Slide 32: Usability – Form Errors This was the Bluefly home page Looking at form errors, we saw thousands of errors every day on this page Any guesses? Approved by Bluefly © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 32
Slide 33: Improved Home Page This is the new Bluefly home page • Search box added • E-mail box clearly marked as email Approved by Bluefly © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 33
Slide 34: Building The Customer Signature  Building a customer signature is a significant effort, but well worth the effort A signature summarizes customer or visitor behavior across hundreds of attributes, many which are specific to the site Once a signature is built, it can be used to answer many questions. The mining algorithms will pick the most important attributes for each question Example attributes computed: – Total Visits and Sales – Revenue by Product Family – Revenue by Month – Customer State and Country – Recency, Frequency, Monetary – Latitude/Longitude from the Customer’s Postal Code     © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 34
Slide 35: Migration Study - MEC  Customers who migrated from low spenders in one 6 month period to high spenders in the following 6 month period Oct 2001 – Mar 2002 Apr 2002 – Sep 2002 s (5.5%) or t ra ig Spent Spent $1 to M under $200 $200 © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA Spent over $200 Spent over $200 (94.5%) 35
Slide 36: Key Characteristics of Migrators at MEC  During October 2001 – March 2002 (Initial 6 months) – Purchased at least $70 of merchandise – Purchased at least twice – Largest single order was at least $40 – Used free shipping, not express shipping – Live over 60 aerial kilometers from an MEC retail store – Bought from these product families, such as socks, t-shirts, and accessories – Customers who purchased shoulder bags and child carries were LESS LIKELY to migrate Recommendation: Score light spending customers based on their likelihood of migrating and market to high scorers. © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 36
Slide 37: RFM Analysis  RFM – Recency, Frequency, Monetary (described in appendix). Insights from Debenhams – Anonymous purchasers have lower average order amount – Customers who have opted out [of e-mail] tend to have higher average order amount – People in the age range 30-40 and 40-50 spend more on average © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 37
Slide 38: Customer Locations Relative to Retail Stores Heavy purchasing areas away from retail stores can suggest new retail store locations No stores in several hot areas: MEC is building a store in Montreal right now. Map of Canada with store locations. Black dots show store locations. © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 38
Slide 39: Distance From Nearest Store (MEC)  People farther away from retail stores – spend more on average – Account for most of the revenues © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 39
Slide 40: World Wide Revenue Detail Although Debenhams online site only ships in the UK, we see some revenue from the rest of the world. UK – 98.8% US – 0.6% Australia – 0.1% Low Medium High NOTE: About 50% of the non-UK orders are wedding list purchases © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 40
Slide 41: Other Results at MEC (See Appendix)  Free shipping changed to flat-fee (C$6 flat charge) – Orders - down 9.5% – Total sales - up 6.5%  Gear Swap (buy/sell used gear) – Visit-to-Purchase very low: 0.34% vs. 2.1% for non gear-swap – However, these visitors converted to purchasing customers (over multiple visits) at a rate 62% higher than visitors who never visited gear swap!  Visits where an FYI page (For-Your-Information) page was viewed had a Visit-to-Purchase conversion of 7.1% © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 41
Slide 42: Other Results at Debenhams (See Appendix)  People who got the timeout page for a high percentage of their sessions are less likely to migrate (to heavy spenders) Revenue due to wedding list item purchases is clearly affected by summer weather – Weddings are more common in the summer in the UK – In June/July, 65% of revenues were generated through the wedding list   A page-tagging-based service provider was used, but was about 30% inaccurate due to people hitting links before page download was complete © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 42
Slide 43: MEC Actions Resulting From Analysis  Done  Implemented links to on-line shop from Gear Swap (appendix has before and after pictures) Implemented gift certificates and increased their visibility on the site (#1 failed search) Used the content page view information to inform our IA redesign    Planned  Refine internal measures by removing bot, internal and production host visits Design and implement controlled experiments to help guide our content planning process in particular Product Recommender (as budget permits) 43   © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Slide 44: Debenhams Actions  Note: Debenhams analysis delivered recently, so there was less time to take action Done – Increased session timeout   Planned – Some failed searches were for products available at stores but not online. Will import all brands and offer a store locator for brands not available online – Looking to enable ‘save basket’ functionality © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 44
Slide 45: BIG ROI Project   The Business Intelligence Group (BIG) Guarantees Return On Investment in 6 months or You Don’t Pay Process – Client provides us with a backup of databases – BI group analyzes data and makes recommendations – BI group provides the JSP changes using a test/control group methodology   Half the people will see the “old” site (control group) Half the people will see our recommendations (test groups) – Client approves the changes, QAs, deploys – Client provides us with a second backup for assessment – Client pays 20% of the revenue delta between the test and control groups, extrapolated to the next 6 months, assuming it would be implemented for everyone 45 © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Slide 46: The ROI  Assuming your profit margin is about 20%, this project has a 6month ROI With a test/control group methodology, the delta revenue is clear: – Seasons, ads, campaigns will affect both the test group and the control group in the same way – Once the experiment is done, you can end the experiment and stop the control group (old site)   Example: – Measurement time: 2 weeks – Revenue for control group is $200,000 – Revenue for test group is $205,000 – Expected revenue if implemented for everyone: $410,000 – You pay: 26 weeks / 2 weeks * ($410,000-$400,000) * 20% = $26,000 © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 46
Slide 47: Qualifications    Must run 4.1.4 or later, preferably 5.0 or 5.5 Must have clickstream collection on at 100% Must commit to timeline – Must be able to generate backup of DBs and send to Blue Martini – Commit to implement changes in 3 weeks from the time code is provided by BMAS   We will not recommend significant site changes We will provide the code to implement changes © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 47
Slide 48: Examples and Commitments  Potential recommendations: – Improvements to usability – Improvements to search – Product recommender for cross sells, top products – Identification and reduction of abandonment  Commitments by Blue Martini – We will fix code that the BI group provided for this agreement, free of charge, to help deployment – This SOW (Statement Of Work) will be governed by your existing contract with Blue Martini. (Easy contractual agreement.) © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 48
Slide 49: Additional Resources  Business Intelligence - Getting Started Guide on http://developer.bluemartini.com Data Mining Tutorial on http://www.bluemartini.com/bi MEC case study at http://www.bluemartini.com/bi Debenhams case study at http://www.bluemartini.com/bi Appendix has more examples     For questions and a copy of these slides, send e-mail to bi-sales@bluemartini.com © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 49
Slide 50: Q&A   Questions and Answers Type your questions into the Q&A (upper-left) © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 50
Slide 51: Appendix  Here are additional slides with some interesting insights © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 51
Slide 52: RFM Analysis (Debenhams)  Recency, Frequency, and Monetary calculations are used extensively in retail for customer segmentation Implemented the Arthur-Hughes RFM Cube – R, F, and M scores are binned into 5 equal sized bins – Each dimension is labeled 1 (best) – 5 (worst)    Interactive visualization using Filter Charts Look at charts instead of cell-tables 52 © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Slide 53: Complete RFM Majority of customers have purchased once Low Medium High Low Medium High Recommendation More frequent customers have higher average order amount © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA Targeted marketing campaigns to convert people to repeat purchasers, assuming they did not opt-out of e-mails 53
Slide 54: Interacting with the RFM visualization  Explore sub-segments with filter charts Average Order Amount mapped to color  People in the age range 30-40 and 40-50 spend more on average Low Medium High  Anonymous purchasers have lower average order amount © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 54
Slide 55: RFM for Debenhams Card Owners Debenhams card owners Large group (> 1000) High average order amount Purchased once (F = 5) Not purchased recently (R=5) Recommendation Send targeted email campaign since these are Debenham’s customers. Try to “awaken” them! Low Medium High Low Medium High © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 55
Slide 56: Customers who have Opted Out  Customers who have opted out tend to have higher average order amount Low Medium High Recommendation Send targeted emails to prevent email fatigue Recommendation Log changes to opt out settings and track unsubscribes to identify email fatigue © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 56
Slide 57: Free Shipping Offer (MEC)     Free shipping stopped on Aug 14, 2002 A flat $6 Canadian Dollars shipping charge introduced Express shipping at higher charge continues Observations – Total sales – Orders up 6.5% up 2.8% – Revenue (excluding shipping and tax) down 9.5% up 18% – Average Sales per Order – © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 57
Slide 58: Free Shipping Offer (Cont.)  The distribution shows fewer orders from low spenders (probably a good thing) No impact on rest of buyers  Fewer low spenders (<= $50) © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 58
Slide 59: Free Shipping Offer (Cont.)   Breakdown of orders by shipping method More people used express shipping, probably because the delta to ship express wasn’t as large (from C$6 instead of from C$0) Free/Standard Shipping Express Shipping © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 59
Slide 60: Gear Swap Pages (Cont.) Recommendation: Link back to MEC Shopping from Gear Swap Shop MEC Cycling © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 60
Slide 61: Gear Swap Pages (Cont.) Done © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 61
Slide 62: Definitions for Gear Swap Analysis  A visitor is defined as someone who is registered (MEC member) or is identified by a cookie – Note that in the Blue Martini system a registered user will have all of his/her cookies combined into a single visitor ID  Comparing visitors who viewed gear swap with those who did not – Several non-bot sessions have 1 request that just visited the MEC homepage (Main/home.jsp) – To get to the Gear Swap section you have to click at least twice – To make a fair comparison we have excluded all 1 request sessions that just visited the MEC homepage (Main/home.jsp) from the following analysis © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 62
Slide 63: Distribution of Gear Swap Visitors  Visitors who viewed Gear Swap pages had a 62% higher visitor to purchaser conversion ratio as compared to those who did not view Gear Swap Visitors: Overall X MEC members: Y Purchasing Customers: Z Visitors who ever viewed Gear Swap Visitors who never viewed Gear Swap Visitors: 14.3% of X Visitors: 85.7% of X MEC members: 20.8% of Y Purchasing Customers: 21.1% of Z MEC members: 79.2% of Y Purchasing Customers: 78.9% of Z 63 © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Slide 64: Distribution of Orders (the real ROI) Orders: Overall X Average Basket Value: $Y Visitors who ever viewed Gear Swap Visitors who never viewed Gear Swap Orders: X Orders: 3,875 (78.3%) Average Basket Value: 1.05 * Y Average Basket Value: 0.98*Y © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 64
Slide 65: Distribution of Visits  Although, Gear Swap visitors have lower visit-to-purchase conversion than non Gear Swap visitors, they visit more often and their overall visitor-to-purchase conversion is higher Overall Visits: Visitors who ever viewed Gear Swap X Visitors who never viewed Gear Swap Visits: 24.8% of X Visits: 75.2% of X Visit to Purchase Conversion: 1.94% Visit to Purchase Conversion: 2.3% © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 65
Slide 66: Effectiveness of FYI Pages  People viewing FYIs are more likely to purchase Viewed FYI Visits: 6.2% of all Purchases: 23% of all Visit-to-Purchase: 7.1% Avg. Sales per Visit: 6.1X Did Not View FYI Visits: 93.8% of all Purchases: 77% of all Visit-to-Purchase: 1.2% Avg. Sales per Visit: $X Recommendation: Controlled experiment to study the effect of FYIs © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 66
Slide 67: FYIs (Cont.)  Setting up controlled experiments to study the causeeffect relationship of FYI – Select a handful of products (say 6) for introducing FYIs – Randomly show the new FYIs to 50% of the visitors viewing these products and don’t show the FYIs to the other 50% of the visitors – At the end of the trial period (say 2-3 weeks) measure the visit-to-purchase conversion of the two groups – Determine if there is a significant difference in the visit-topurchase conversion of the two groups © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 67
Slide 68: Debenhams Migrators: Timeout Some attributes are more useful when combined with other attributes For each visitor we computed the number of sessions which went to the page timeout.jsp This was binned as shown on the X axis of the chart The height shows the number of visitors in each bin and color shows the percentage of those visitors who migrated Just looking at this variable alone it is difficult to tell what the pattern is 68 © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA
Slide 69: Migrators: Timeout By combining the number of timeout sessions with the total number of sessions for each visitor a pattern emerges In this heatmap the X axis shows the total number of sessions, the Y axis shows the number of timeout sessions, and color shows the percentage of migrators at each pair of values The green along the diagonal shows that people who got the timeout page for a high percentage of their sessions are less likely to migrate © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 69
Slide 70: Migrators: Timeout The number of sessions a visitor has is a good indicator of whether or not they will migrate However there are some inconsistencies that are apparent. For example, why does the percent of visitors who migrate drop at 19 sessions? We can construct new attributes based on the relationship we saw between the number of timeouts and the number of sessions Two more attributes can be created: • Number of sessions that did not time out • Percentage of sessions that did not time out © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 70
Slide 71: Migrators: Timeout Number of sessions without timeout is a good predictor of migration © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA Percentage of sessions without timeout is also a good indicator of migration * 68,000 visitors with no timeout sessions have been filtered out 71
Slide 72: Distribution of Wedding Purchases over Time Revenue due to wedding list item purchases clearly affected by summer weather, when weddings are more common in the UK © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 72
Slide 73: Hidden Page Requests  Page Tagging – Debenhams used a third-party ASP that uses page tagging to track users – Tag generation implemented using a separate JSP – This causes two requests to be executed   Generates higher server load Hurts user experience  Many users are clicking on links before the complete page downloads – Page Tag jsp is never executed – Statistics are inaccurate: 34% of non-bot sessions did not execute the page tagging jsp © Copyright 2003, Ronny Kohavi, Blue Martini Software. San Mateo California, USA 73