Case Study: Kia
Kia is one of the world’s leading automobile manufacturers, and one of the world’s most recognized brands. Part of the world’s fifth largest automaker, Kia prides itself its power to surprise consumers with dynamic and exciting experiences which go beyond their expectations. Kia has demonstrated this power to surprise through innovations in every aspect of its business, from the cars it makes to the way they sell, the way they provide service, and how they communicate with their customers and prospects. Kia’s web sites are one example of this. The company strives for dynamic and exciting experiences in how it presents its vision, its products and its programs.
Kia’s Korean website (kia.com/kr) provides a deep resource of information on the company’s products and services to help consumers make informed buying decisions and get the most out of their Kia purchases.
The automaker wanted to learn how to better get its customers past the initial home page to increase engagement with vehicle-specific information. They had many ideas on how to accomplish this, but time was limited—Kia was working on a new overall site design and needed to provide feedback to its designers within two months to inform them on tactics for deeper engagement.
Running A/B or A/B/n tests on their various ideas for improvement would take well beyond the allotted time, and would only demonstrate the strength of individual changes or sets of changes on one aspect of the design. And with thousands of potential design combinations of their ideas, standard multivariate testing was a non-starter.
The marketing team had a quandary—proceed with the redesign without this critical information, or quickly find a new way to accomplish the research goals.
Fortunately, the Kia team had recently read about Evolv in the marketing trade press.
“It sounded too good to be true,” says Monica Kim, manager of digital marketing strategy for KIA. ‘But we understand that artificial intelligence is able to offer new solutions for existing processes, like experience optimization. We decided to put Evolv to the test.”
The Kia homepage features the same interface for accessing product-specific information, across multiple vehicles, as shown in the image below. Each vehicle panel is shown for seven seconds, and rotated along with other vehicles and various promotions.
The five circular buttons in the upper left (highlighted in teal) are the pathway to additional vehicle-specific information. The buttons lead, from left to right, to vehicle information, the electronic brochure, pricing information, the online quotation service, and the driving center, where users can schedule a test drive.
The goal of the experiment was to determine the factors that would lead users to click on any of these buttons more frequently.
Monica and her Kia team had five areas they wanted to investigate whether changing any of the following would increase the clickthrough rate:
- Changing the color of the buttons
- Changing the shape of the buttons
- Changing the size of the buttons
- Changing the order of the buttons
- Changing the time duration each vehicle and promotion panel was shown
The team determined to test multiple colors, a square button vs. the current round shape, two new button sizes, 5 new orderings of the buttons, and 8 new durations for the time each panel was shown—some faster and some slower.
All in all, the team tested 23 new variations on the 5 elements of the page. The number of potential combinations of the various changes—the “search space” in AI lingo—was 2,592.
The team set the audience for the experiment using Evolv’s filtering and segmentation tools to desktop users only. They then entered the improvements into the Evolv editor, used the QA tools to review them for workability and brand compliance, and hit the Start Experiment button.
Evolv automates massively multivariate experiments like the Kia experiment using evolutionary artificial intelligence to quickly and efficiently move towards the optimal design within the search space.
Kia’s experiment ran for 5 weeks, and tested 49 possible combinations of the test ideas across multiple “generations” of designs it created using various configurations of those ideas.
Each generation provides information about which individual ideas, and which combinations of ideas, are the best performing—the most fit—and then applies survival of the fittest math to combine the attributes of best performers, removing those elements that don’t work well, into new designs to test.
Evolv uses Bayesian statistics to project the performance of each of the designs—the reward—and also evaluates its likelihood to beat the control design by any amount—the risk factor.
The winning design in the experiment incorporate changes to 4 of the 5 items being tested:
- Maroon button (control was gray)
- 66 pixel button size (control was 44 pixels)
- Square button shape (control was round)
- 10-second rotation time for the different homepage panels (control was 7 seconds)
- The order of the buttons did not change in the winning design.
The winning design showed a 113% increase in clickthroughs vs. the control design, and was projected to deliver a gain of between 75 and 139% at a 95% confidence level, once deployed.
During the experiment run, Evolv delivered a real-time gain of 26%—this is the performance of all of the tested designs vs. control, vs. the stats above which are those of the best performing single design.
“We found testing multiple changes in support of a common goal—in our case, increasing engagement with our vehicle-specific information and services—to be an ideal use case for Evolv,” said Monica. “We were able to compress half a year’s worth of testing into a single month, and improve one of our key KPIs that much more quickly.”
The Kia team is now integrating what it learned into the design of their upcoming new website. In addition to the overall design-level reporting that Evolv provides, the system also provided Kia with in-depth information on the contributions of the individual ideas into the most successful design.
The team is excited to investigate using Evolv to optimize more of their sales funnels, and plans to evaluate Evolv’s server-side mobile application optimization capabilities as well.
“It seems like there’s never enough time in the day to get everything you want done,” said Monica. “Fortunately, artificial intelligence solutions like Evolv can augment your team through automation and accelerate the path to results.”
By The Numbers
Kia more than doubled engagement with their vehicle-specific content and services through Evolv massively scaled testing.
week test duration
possible designs evaluated
conversion uplift from the best design
"We were able to compress half a year’s worth of testing into a single month, and improve one of our key KPIs that much more quickly."
– Monica Kim
Manager of Digital Marketing Strategy, Kia