Taking optimization from strategy to execution

I spoke at SES NY (now called ClickZ Live NY) about a year ago. Here's a post I wrote which summed up the talk:

I’m often asked how did we get ramped up to doing 3–5 tests a week. I have to admit, I’m impressed that we’ve got­ten this far, but we still have room to grow.

In short, it comes down to 3 things to get you there:

  • Align­ing your opti­miza­tion strate­gies with your busi­ness goals
  • Estab­lish­ing the right frame­work and processes
  • Exe­cut­ion 

I’ll break it down.

Align your opti­miza­tion strate­gies with your busi­ness goals

It may seem obvi­ous, but there’s often the impulse to just start test­ing every­thing every­where. You’re enthu­si­as­tic and want to show ROI imme­di­ately, but you need a strat­egy. What are your company’s goals? Most often there are rev­enue goals you’re look­ing to meet and there are busi­ness strate­gies you need to tap into to help get you there. Is there mes­sag­ing you can exper­i­ment with to sup­port the busi­ness strate­gies? Cre­ative expres­sion? Lay­out? Func­tion­al­ity? Answer­ing these ques­tions can help guide you.

Estab­lish the right frame­work and processes 

It’s no ques­tion that if your com­pany isn’t doing test­ing yet, it’s going to take a lit­tle time to shift the mind­set; it’s a cul­ture change. Depend­ing on how open teams are to change and tak­ing your busi­ness to the next level, will deter­mine how quickly you can imple­ment a pro­gram. You’ll need:

  • Oper­at­ing model align­ment: You’ll have to get buy in across your orga­ni­za­tion. It might mean ask­ing other teams to sup­port your efforts or ask per­mis­sion to take on some other tasks like design­ing pages.
  • Process design: It’s going to depend on your orga­ni­za­tion how you want to work. I’ll elab­o­rate more in the fol­low­ing paragraph.
  • Change impact and readi­ness: How does adding test­ing affect other teams?
  • Orga­ni­za­tional design: Do you keep it all in one group or do you share respon­si­bil­i­ties with other teams?
  • End user train­ing: Does the team have the skills they need?

If you’re doing it all your­self, you don’t have to work about processes as much. How­ever, if you’re work­ing across divi­sions or depart­ments it’s best to have roles, respon­si­bil­i­ties, and processes estab­lished and agreed upon. You’ll need to secure:

  • Gov­er­nance
  • Web strate­gist
  • Opti­miza­tion manager
  • Cam­paign enablement
  • Ana­lyt­ics support
  • Web pro­duc­tion
  • Project man­age­ment
  • Web devel­op­ment (for more sophis­ti­cated testing)
  • Demand and media gen­er­a­tion (to drive traf­fic to your site)
  • Cre­ative development
  • User expe­ri­ence expertise
  • Plat­form architecture

Exe­cute your tests 

Now you have your strate­gies, a slew of test ideas which sup­port your strate­gies and busi­ness goals, you have your team in place, now you have to exe­cute. How do you know what to test first? The first rule of thumb is ana­lyze which pages on your site yield the high­est con­ver­sion, have the high­est vol­ume of traf­fic, or demon­strate the biggest area of oppor­tu­nity. We’ve set up scor­ing mod­els to help us with pri­or­i­ti­za­tion. As the team grows, you have to man­age lots of tests get­ting ready to launch. Poten­tially all at the same time.

Our scor­ing model is sim­ple. What type of test is it: Con­tent, lay­out, func­tion­al­ity, or cre­ative? What’s the level of effort: Low, medium, or high? What is the poten­tial rev­enue lift: Low, medium, or high? The test type score is based on pre­vi­ous quarter’s suc­cesses. Have an ana­lyst help you with that. The level of effort is judged by the per­son doing the enable­ment of the test which might include devel­op­ment. And the poten­tial rev­enue impact is also eval­u­ated by an analyst.

The opti­miza­tion man­ager should develop their strat­egy and then come up with indi­vid­ual tests to sup­port those strate­gies. Then method­i­cally develop test char­ters or plans which out­line the goals of the test, jus­ti­fi­ca­tion, what page(s) the test should be run on. It’s good to have a hypoth­e­sis, but be cau­tious on includ­ing that in your test char­ter because it might sway you to want cer­tain results.

Set up your test using a solu­tion like Adobe Tar­get, run it 3–5 weeks (depend­ing on traf­fic and con­ver­sion rates), and eval­u­ate the suc­cess through the reports in the tool or if you want to dig deeper, use some­thing like Adobe Anal­ty­ics. Use what you learn to feed your next test idea and keep iterating.

Watch your con­ver­sions increase 

The more com­fort­able you get with test­ing, the quicker you can iter­ate on tests and reap the rewards of increased con­ver­sions. Happy testing!

Provide feedback on how you operate your optimization program.