Sandra Cheng
Sandra Cheng is a Product Manager at Google where she is responsible for the overall product strategy and roadmap for Google Website Optimizer. She also manages ads optimization related products for AdWords. Previously, Sandra worked at Amazon.com, where she led product management for Endless.com.
Interview Transcript
Eric Enge: So why don’t you start with an overview of the program that Google Website Optimizer did with these four companies?
Sandra Cheng: We launched Website Workout last year to raise awareness of website testing and improve the ease with which you can increase the value a website delivers to your prospective and existing customers.
So, during a two week contest entry period, we had asked thousands of advertisers why their sites needed makeovers. Out of all of the entrants, we then chose four of the most compelling cases.
We paired each of those four advertisers with one of our Website Optimizer authorized consultants, who then helped them set up experiments to optimize their website in order to generate more leads, sales or signups, whatever their conversion goal may be.
Eric Enge: So an authorized consultant is a third-party organization like ZAAZ, or someone like that?
Sandra Cheng: Exactly.
Eric Enge: When did you actually start working with the companies?
Sandra Cheng: We announced the contest last June and started working with each of the companies shortly thereafter.
Then we went through a couple of different series of tests before we spent some time to shoot the videos. And we just announced the results two weeks ago.
Eric Enge: Let’s dive into one of the scenarios. Can you talk about Extra Space Storage and maybe tell me a little bit about what they felt their situation was, the scenarios that they tried and the results that occurred.
Sandra Cheng: Extra Space Storage is the second largest operator of storage space in the US and is based in Salt Lake City, and get about 40% of its customers online. And as they stated themselves, marketers and developers guessing what customers wanted designed their website.
This is something that I feel is pretty common in website design today, where people tend to guess what’s best based on cool ideas or a gut feeling, but never really knowing what’s actually going to resonate with their visitors. So, Extra Space Storage decided to test their facility pages, which are the pages that allow customers to find the storage unit that’s best suited for their needs.
The original version of the page included pictures, driving directions, the description of different unit types and some big select buttons to choose which one you are interested in.
They then tested three different variations. In the first variation, they increased the size of the picture, added a map thumbnail to driving directions and decreased the size of the information that they felt was less critical to decision making on that page.
In the second variation, they made the picture smaller and tried to bring more things above the fold. In the third variation, they moved the picture, the directions, hours of operation and everything else over to the left-hand side, and had the unit information front and center with larger select buttons.
Really what they were doing was testing some pretty drastic layout changes where the content more or less stayed the same. The first version that they tested led to an increase of almost 10% in conversion rates.
Eric Enge: Right, so 10% is a pretty significant lift for those pages.
Sandra Cheng: Absolutely.
Eric Enge: So, you talked about three different types of strategies that were tried. They tried pretty significant changes, and, there is an argument that could have been made for any of those scenarios.
Sandra Cheng: Absolutely, I agree. We see that a lot in our own testing and the moral of the story here is that things that are generally regarded as best practices aren’t necessarily best for your particular website. They are the result of someone else’s success, but they may not apply to you.
Eric Enge: Right. And, I think the underlying theme that drives that is that best-practices may be best-practices when you are dealing with a completely homogenized, un-described audience; but that’s not your audience.
Sandra Cheng: Right.
Eric Enge: Your audience has come to the site by a specific path with specific things in mind, and you need to be very, very aware of that.
Sandra Cheng: Agreed. I think that goes back to the earlier point that website testing takes out the guess-work of website design. It not only develops better pages for users because they get a better web experience, but it’s also good for site owners because they get higher ROI.
At Google, we fashion ourselves as being pretty well-informed web designers, yet we are constantly surprised at what customers tell us they prefer. We use Website Optimizer on a lot of our own Google properties, such as Gmail, AdWords, Website Optimizer, Google Earth, Google Maps, etc.
We are constantly testing Website Optimizer’s homepages to try to get more people to try it out and help them launch their first experiment.
Eric Enge: Basically what Extra Space gave you was the site that was designed based on the best thoughts of groups of developers and marketing people, but there wasn’t a specific science to it. I think it’s important for people in the Web Marketing world to really understand that the chances that they will get it perfect that way are basically nonexistent.
Sandra Cheng: Absolutely, I agree. You are paying money to drive traffic to your webpage, and not everybody that visits the page is going to do what you want them to do. So ultimately, it comes down to conversions and money. Every visitor is precious, so why not convert as many of them as possible and let them tell you what is and what’s not working for them.
Eric Enge: Can you tell us about the Colonial Candle example?
Sandra Cheng: Colonial Candle is a 100-year-old candle company based in Minnesota. They sell a large variety of candles and seasonal items, and they tested their product page by changing font size, headers and product photos. Their original site didn’t have a clear call-to-action and they had inconsistent imagery.
They had some product shots, some were stylized, some had colored background, and some didn’t. They ended up increasing the font size and changing the image size, the headers, and the colors, and they had two drastically different pages to test. One was shorter and had more stylized photos, and the other was a longer page that had blue headings which helped organize the page.
Everyone thought that the shorter design would win because conventional wisdom is that less is more. It turns out that the longer page won. They saw a 20% lift from that page, which is more than $20,000 a month in revenue. So, now they are a big fan of Website Optimizer and they run Website Optimizer tests on a regular basis.
Eric Enge: So if you want to have a quarter of a million extra dollars a year, use Website Optimizer.
Sandra Cheng: That’s what I am saying. If you are not testing, you are leaving money on the table.
Eric Enge: Absolutely. The idea that less is more is certainly conventional wisdom. People think that things that are over-cluttered are going to not perform very well, and simple, even sparse, designs are best. But that didn’t prove to be the case here.
Sandra Cheng: Right. You just never know. In some cases, less is in fact more. Website testing really is just like having insurance against making the wrong decision.
Eric Enge: Sure, what you are trying to do is figure out what the best practices are for your audiences.
Sandra Cheng: Exactly.
Eric Enge: So, when you have websites with more than one page, you should basically cycle through pages in some sequence?
Sandra Cheng: Yes. We find it really helpful to start with your primary landing page because that’s where people are first. You want your goal of that test to get them to move towards your conversion. Maybe it’s to learn more about your product, or to signup for a newsletter or for an account.
It could be to dive deeper into product categories and spend more time on the sites. You can also run tests to test how long people are staying on the site, and optimize for that. But, what we generally recommend is that you start with your landing page and make some pretty big changes there, and then start to refine your tests.
Maybe you start with A/B Testing to test a couple different variations of the page and then move on to multivariate testing, where you are testing different sections of the page. You might try different headings, different images different content, different text or different copy and then move down the conversion funnel until you test and optimize the entire process.
Eric Enge: One thing that you said that was important there was to not be afraid to be radical.
Sandra Cheng: Absolutely. Don’t be afraid. I think if you are going to test, you want to test something bigger than a semicolon versus a period; you want to be able to stand 6 feet away from the monitor and still be able to see the difference. Once you get the general layout correct, you can start tweaking the insides.
Eric Enge: Right, so you get the big gains from the big change, and once you are there you can do some follow-up things. In the case of Colonial Candle, they got 20% and maybe now they can refine it and pick up another 5%.
Sandra Cheng: Right. Sometimes small changes can result in big increases in conversion as well. Something as simple as changing a button color could make a difference. We have one customer, NerdyShirts.com, that tested a white button versus an orange button, and they saw a 7% increase in sales just due to that change in button color. You just never know. Small changes can make a big impact, so it’s worth testing any idea.
Eric Enge: Let’s talk about the mechanics of implementation. Can you outline what’s involved from a technical perspective?
Sandra Cheng: Okay, great. Website Optimizer offers two different types of testing. The first is A/B Testing, where you are testing different variations of the single page. The way that works is at the top of your page you paste in some Javascript that we call the control script, and that piece of Javascript will take care of cookie-ing the user and choosing whether or not to redirect the user to a separate page.
Then, you put a tracking script at the bottom of each of your variation pages, and a conversion tracking script at the bottom of your conversion pages.
Eric Enge: So basically, the control JavaScript goes at the top of the current pages?
Sandra Cheng: Right, the control script is at the top of the current page and the tracking script goes at the bottom of each of the different variations.
Then at the bottom of your conversion/goal page, you put a third piece of Javascript that tracks the fact that the conversion occurred.
Eric Enge: Of course, you also need to brainstorm the different versions of things you want to test.
Sandra Cheng: Exactly. We give you all the Javascript. All you have to do is copy and paste them in the right spot and upload the pages.
Eric Enge: So there is no need for the Website Optimizer to actually get intertwined with the nature of the changes.
Sandra Cheng: That’s right. Everything is pretty much controlled by the control script at the top of the original page. If a browser doesn’t have Javascript turned on, it just behaves as if everything was normal, the customer will just continue seeing the original page.
Eric Enge: And that doesn’t bias the test because they don’t have Javascript turned on.
Sandra Cheng: Exactly, it just doesn’t count. If they transact or not, it doesn’t matter.
Eric Enge: Putting aside the need to design different scenarios, it sounds like this is something that could be done in a day.
Sandra Cheng: Oh sure, it can be done in 5 minutes if you have access to the pages. The other kind of testing is called Multivariate Testing, and it basically allows you to change different sections of your page at the same time. You could say, “I would like to test four different copies for my header at the top.” For your front and center image, you could decide to test two different images. For the text on the right-hand side, you may have three different examples to test. So instead of creating a brand new page for each of these sections, you would put a control script at the top of the page and then wrap each of the sections that you want to test with the section script.
And then, within Website Optimizer, you define what the different variations are for each section.
Eric Enge: Right. So that sounds all very simple. It’s slightly more complex than a simple A/B test, but that’s just because you are allowing Website Optimizer to decide how to mix and match the elements. Do you literally test every single possible combination?
Sandra Cheng: We do. We do what is called a full-factorial test.
Eric Enge: Say I own a new company and I am interested in using Website Optimizer. How do I start?
Sandra Cheng: I think the most important thing is to have a very clear conversion goal. What number are you trying to drive?
It’s important that you have a fairly high rate of conversions, and by fairly high I mean about five conversions per day. Fewer could work, it just might take longer to finish the test.
Eric Enge: Right, you have to wait longer for the data to be statistically significant.
Sandra Cheng: Exactly. And then test a small number of variations. Our rule of thumb is to test no more than one variation per approximately 100 conversions. This is really just to make sure that your test doesn’t take forever to run.
Test a small number of big changes. If you can’t visually see the difference between two combinations in 8 seconds, visitors probably won’t either. Then all you have to do is set up the test and wait; don’t jump to conclusions. Wait for about two weeks and focus on the absolute difference in conversions. Once you have a winner, move on to testing other pieces of the site or refining the test in general, maybe getting more granular in the changes that you are making.
Eric Enge: Do you have any advice or tips on how to decide what kinds of things to test for? What are the typical things that people play with?
Sandra Cheng: A couple of things we have talked about, like page layouts, for example, are great candidates for testing. Things like font size and color are important too, especially for your first test. Then, what images should you show, how big should they be, where you should place them, etcetera.
You can test your call to actions, button sizes, and messages, where it is placed, etc. You can test different types of images like testing traditional product images versus action images, for example.
Extra Storage Space did a good job of this. They tested glamour shots of their storage space, using pictures of a space with people and furniture instead of empty storage containers, and they found that the glamour shots worked a lot better. You should test and see which works better for you. So that’s just the start, but really you can test anything.
Eric Enge: Does it make sense to try to approach these things with a philosophy, for example, one philosophy would be the less is more philosophy right, another might be hit them between the eyes, so does it make sense to think this way when trying to think about different scenarios?
Sandra Cheng: That’s certainly one way to go about it. I think there is an endless number of combinations that you could come up with, and ultimately I would say testing is not a one-time, hit-or-miss experience.
It’s really a mentality, it is a way of life for website design. Pick a couple that you think might work and test those. If you are not incredibly happy with the results take the winner and test it against some others, and keep iterating.
Eric Enge: Particularly for paid search marketers, where the margin tends to drive towards zero as everybody squeezes everything they can out of it, it is a great thing to accomplish a 20% lift. You have got a pretty substantial competitive advantage in that situation.
Sandra Cheng: Right. And, the web is constantly evolving, so what works for you last year may not be the best option for you this year. Especially in 2009, with the economy being the way it is, I think it’s really going to be the year where testing goes mainstream. People are looking to get more for their advertising dollars and landing page optimizations is one of the few ways you can do that without spending more.
Eric Enge: So the lesson here is that market forces can change the behavior and mindset of your customer?
Sandra Cheng: Exactly.
Eric Enge: So, one of the things people sometimes talk about is the personas of visitors. Some people are very disciplined or competitive, some are logical, some are spontaneous, others are emotional. People come to your site with many different mindsets.
Is that something you recommend people think about when they are considering how to try these scenarios?
Sandra Cheng: Sure, I think it’s great to try to design your website in such a way that appeals to your target customers. If you pick designs that you think will speak to each of those different types of customers, you have a really quick way of learning who your primary audience is and what is the most effective tactic to use. You could also argue the flip argument though, which is somebody who normally isn’t very spontaneous could come to your site and feel compelled to take action right then and there.
Eric Enge: Right. People who talk about this theory point out that everybody falls in all four of those groups at one time or another, depending on a lot of circumstances.
Sandra Cheng: I think the key is just to figure out what’s going to work the best for your visitors.
Eric Enge: Is there anything else that you want to say?
Sandra Cheng: I think that Google Analytics and Website Optimizer are both about helping people succeed online and improving their user experience. As I mentioned earlier, we expected 2009 to be the year where website testing goes mainstream. So in 2008, we saw most of our adoption from SEMs, web development firms, eCommerce companies, and other online marketing experts.
This year we are already seeing adoption across the board, from political campaigns, bed-and-breakfasts, to stationery companies. So, testing is going mainstream, and if you don’t test, you are going to get left behind.
Eric Enge: Thanks Sandra!
Sandra Cheng: Thank You Eric!
