Photo by David Travis
Table of contents
A/B testing
A/B testing in simple terms
A/B testing allows you to compare two versions of the same page to see which is more effective in helping the visitor complete a given action or goal. A/B testing is not a complete solution—it simply points to fine-tuning elements on a page. It doesn’t identify usability issues or your visitors’ frustrations and expectations.
A practical example
For example, on your home page, you have three news items, but you notice that no one clicks on the links to read more about them. This is unfortunate because this is the first action you want people to take when they visit your website. Why aren’t they clicking on the “read more” links? Are they not visible enough?
To find out, you can use a WordPress A/B testing plugin, create a variant of the homepage, and change the appearance of the links by underlining them. You can then randomly show your visitors the control page (the original version) and its variant (with the underlining) to see which one leads more people to click the news links. After 1,000 visits to the control page and 1,000 visits to the variant page, you may find that 5% of your visitors clicked on a news link on the control page, while 8% did so on the variant page. Therefore, the variant is more effective in clicks on that particular link. Still not satisfied? Then, you could test a new variant where the “News” title is more visible and check the impact of this change.
What A/B testing allows you to test
- Buttons
- Images
- Text
- The presence of videos
- The colour of an element
- The size of an element
Usability testing
Usability testing in simple terms
Usability testing is more thorough than A/B testing. It takes from 30 minutes to an hour and aims to verify that a website works as expected. To do this, you will invite a person to use your website while you observe. Ask the person to think aloud during the test to understand the progression of their thinking and the logic behind their behaviour.
The two main parts of the usability test
The test usually starts with an evaluation of the homepage (homepage tour) to confirm that the website’s purpose is clear to the visitor: who is behind the site, what can be done on it, what are their general impressions, what draws their attention, what would they click on first, etc. You then give them tasks to accomplish on the website. The tasks are those you want your visitors to perform easily. Find your best-selling product, for example, or make a purchase, register for an activity, or sign up for your newsletter. In performing these tasks, the person will express what they understand and don’t understand, and what they find useful or not so useful, easy or not so easy, annoying or frustrating, etc.
Mistakes to avoid
- Avoid leading questions that assume the user’s thinking or reaction. It is important to remain as neutral as possible. “Did you find this task difficult?” is a leading question, while “What’s your impression of this task?” is a more open-ended, neutral question. In Things a therapist would say (.pdf), Steve Krug, author of Don’t Make Me Think and Rocket Surgery Made Easy, gives tips for remaining neutral during a test.
- Avoid conducting a test without being prepared. Take the time to write a script that explains who you are and the purpose and procedures of the test. The script should include the question that will lead participants to comment on the homepage, the list of tasks you want to evaluate, and a paragraph as a conclusion and thank you. In his “Downloads” section, Steve Krug offers a script, a checklist, and a consent form that you can download and use free of charge.
- Avoid creating a test that is too long. Thirty minutes to an hour is sufficient. It is better to develop more tests for other tasks than trying to do everything in a single test.
Identifying and correcting problems
Ideally, ask someone to observe and take notes during the test: how long it took to do each task, what confused the participant, etc. Of course, Steve Krug also has instructions for observers (.pdf). If you can’t have an observer, watch a recording of the test, and take notes yourself.
For each participant, you can then list the three most important usability problems that came up. You will then need to decide which problems are the most urgent to correct and the means to correct them.
How many tests should you conduct?
Conducting the same test with three people from your target audience is usually enough to identify most problems. But testing with one person is always better than not testing at all.
When and how often?
Ideally, usability testing is conducted at several stages of the design process, including conceptual and graphic mock-ups. It is also best to continue testing regularly once the site is launched to test other tasks and verify the impact of certain changes made over time. For example, you could try to run three usability tests per month.
A practical example
Going back to the example of news links on your homepage, in a usability test, a test subject might avoid the news section and share their experience like this:
- I saw that there were news items here, but I didn’t click on the headlines because they don’t interest me. What really caught my eye was…
- The image at the top of this news item makes me think it’s an advertisement, so I’m moving on.
- That red button grabbed my attention, so I didn’t see the news section
- I don’t understand the title of this news at all, so I’m not wasting time here
As you can see, usability testing allows you to find out what your audience really thinks. It’s the best way to make sure a website is useful and easy to use because it allows you to understand how people use it.
If usability testing is still a mystery to you, you can see an example, led by Steve Krug.
Conclusion and comparison
In short, usability testing verifies whether a website is useful and user-friendly. It helps identify and understand usability problems on a website, and sometimes even gives hints on how to correct them. On the other hand, A/B testing aims to refine the effectiveness of specific elements on a page.
To conclude, there are key differences between the two tests.
| A/B testing | Usability testing |
|---|---|
| Aims to refine the effectiveness of an element on a web page | Aims to check the site’s usefulness and ease of use by testing the whole site, a particular page, or a specific task |
| Is performed once the website is launched | Can be done on very simple mock-ups as well as on a published website |
| Is done anonymously over a period of several days and doesn’t require the tester and the user to be present at the same time | Is done over a period of about an hour in the presence of the user |
| Gathers test results from thousands of visitors | Is conducted with one person at a time |
| Doesn’t provide answers or explain precise reasons for the behaviour tested | Answers questions and helps understand the reasons for a behaviour; the information collected is richer |
