A/B Testing - Alpha soup or worth it?

By Arti Gupta (Intel) (6 posts) on April 22, 2008 at 8:02 pm

Intel Software Network is A/B testing its new layout and design. I should say we are multivariate testing since these are 2 substantially different pages. I am still waiting for the data from the first week of the test. Has anyone done this testing? What has been your experience? 

I am interested in hearing if you were able to get meaningful data to act upon and what results you saw. 

here are the 2 versions of the ISN home pages being tested. 

Previous version

home-a1.bmp

New Version

home-b2.bmp

Categories: Intel® Software Network 2.0

Comments (2) Comments RSS Feed

By Sharon Greenfield on April 22nd, 2008 at 10:06 pm
Unfortunately you're right, this isn't true A/B testing since so many variables changed, and hence much much harder to determine causality. Any particular variable changed might or might not be an effect on high abandonment or higher click-through rate.

Because there are so many variables changed, I might suggest some hallway usability testing.

Also in terms of a knowledge base, I would suggest a starting point of
here: http://www.uie.com/articles/components_for_redesign/
and
here: http://www.uie.com/articles/usability_testing_mistakes/

both from Jared Spool.

By Arti Gupta (Intel) on April 23rd, 2008 at 4:10 am
Thanks Sharon for your comment. Yes this is Multivariate testing but my goal is to start at the bigger level of what changed and then start to narrow testing with specific areas from time to time.


What do you think?

Name (required)

Email (required; will not be displayed on this page)

Your URL (optional)

Comments (required)