User Research Apr 25, 2018

Actual price your business pays by not user testing

User Research Apr 25, 2018

Rule book states, the purpose of any product, is in its service and relevance for the users. In practise though, it’s amazing how repeatedly, we lose track of this basic.


I chanced upon a review by a customer about the web application of Singapore Airlines. The customer complained clearly about anomalies in using the website. Turned out the website kept returning to the reservations page even after entering the destination, date of travel and selecting flights. Each time the customer tried again, the price kept changing and becoming higher than before. He finally called at the customer service center to check the actual pricing. Sure enough, the executive told him a price higher than the first time he had tried.


Here, it wasn’t enough to create a convenient online system of booking. The company should have invested time on building an efficient website that would truly help their customers make faster bookings.    


Enter user-testing. In order to build a loyal customer base, user testing is the key in user-centric product development. It does take time, and recruiting the right people from your potential user base can be a painstaking task.


The temptation of not testing with users 


Preparing the test plans, executing them and getting the test results, on an average take around a week for a website. After that, analyzing the results to assess the user-friendliness of the product and making further changes take some more time. So why not go with your intuition and the opinion of your expert design panel? They are the experts after-all, and bypassing the user-testing does save upfront time and money. And as time is money, you are actually saving double the money. Sounds about right, doesn’t it?

Yes, there are different perspectives and arguments against or in favour of user-testing. And I would like to enlist what happens when companies don’t user-test.


Can user-testing be ignored?


The most publicized study that comes to mind is the $300 Million Button on The giant e-commerce player realized about an unknown bottleneck in its check-out process. After the user made the purchases and clicked on ‘checkout’, a form popped up that asked the user to Log In or Register. Turned out, many users didn’t want to register in order to shop and this little anomaly cost Amazon a whopping $300 million worth of cart abandonment. The company fixed it by replacing ‘Register’ with ‘Continue’ along with a message that said -

You do not need to create an account to make purchases on our site.  Simply click Continue to proceed to checkout. To make your future purchases even faster, you can create an account during checkout.


This little change resulted in a 45%  increase in a completed purchases and for the first year, the site saw an additional $300M in revenue.


User-testing cannot be ignored at any stage of your product. You may be an e-commerce conglomerate or a small business. When you define the objectives of usability testing clearly, determine relevant and open-ended tasks and create specific and open-ended questions, you get valuable insights on what your users are actually thinking.


Exploring alternatives to user testing


Once a rival website or app has introduced a new feature that everyone (including you) finds amazing, you are desperate to introduce something better than your competitor’s website in order to retain our customers. You have to try and find ways to be quick to the market, which most likely means, user-testing starts to sound like an overhead. So you try some quicker (and maybe cheaper, depending on your budget) alternative to user testing. More often than not, one of the following solutions.


Heuristic Evaluation: This is a usability inspection method . It involves evaluators examining the interface and judging its compliance with ‘heuristics’ or standard usability principles. At the end of evaluation, you receive a report which highlights the usability issues and their severity, potential solutions or workarounds present, and recommendations to improve the overall UX. This is a safe alternative provided your product already exists for UX experts to perform a Heuristics evaluation.


The quick and dirty method: Testing the product ourselves first, trying to surf through it as if we are a naive new user on the platform. Asking a colleague or a friend to spend some minutes on the platform, taking their reaction to the product and how they may use it in the long run. Using this knowledge as an internal review to improve the user-experience of your website or app is perfectly fine, but diving in the market on the basis of this method alone, is not. The infamous Dipstick so to say!


Low budget for user testing? Some tips

1] Ask family and friends: You can hire consultants or ask your friends, family and colleagues for their opinions. They can also help you get some volunteers which will help you as you get to know more about your product’s usability by testing it on as many different user-groups as possible. Focus on what these users ‘do’ more than what they ‘say’ about your product.

2] Open hallway testing: This is a method of usability testing where people selected randomly are asked to try the product. Except for the project designers and engineers, anyone can be used to test the product in this method.

3] URUT: In Online User Testing (Remote) aka URUT, the testers are present in their natural environment (home/office, etc) and are thus more likely to give you a less biased feedback on user behavior. This tremendously saves logistics costs.

The Perils of not undertaking user-testing


When you bypass the user-testing phases, here is what you risk. You are actually losing out on valuable user insight which may help you build a scalable product. Your business should be aligned with the needs of your users.  You also lose the referral factor, the best way to market your product, as if a user did not like your product, he definitely is not going to recommend it to others. For a new-product, it might just not click with the users the way you thought, or may not even make a good first impression. Since you didn’t test it with users, you may not even come to know why exactly it failed.


Being clueless about user abandonment


User-testing needs the establishment of the correct context. This is specially important while introducing a brand new feature into your already established app. Messaging app Whatsapp recently introduced the status feature to rival the Stories feature on Instagram and be more attractive than snapchat for the younger audience. The problem was, there already was a text status feature which was removed. The company faced severe backlash for their decision and had to quickly bring-back the text-status feature, this time as “About”. A new feature on a successful platform too may disrupt the existing user-experience. This may go against your app in future when other options are available to users.  


Another issue that businesses face in the absence of user-testing is the lack of proper understanding of their own product. If the product, or some features of it, does not impress the end-user, and the users abandon the product, it becomes harder to nail the problem. Some businesses have found a work-around for this issue, they ask those users to provide feedback on why they are leaving the product. I believe spontaneous reactions don’t work so well . It’s wiser to integrate user-testing at an early stage.


Failure to establish patterns


If you are not putting in efforts to understand your users, you will not be able to establish patterns and preferences  in their behavior, your product would fail to be relevant to them. You risk reducing your overall sales and probably losing your business too.


Recently, Google has started a new feature in their Chrome mobile app (PlayStore, App Store), when you open a new tab, you get news items popular in your area, and news items you might find interesting, based on what you have been reading in the past. This way, they keep their content relevant to the user, and the users spend more time reading the news articles, resulting on more advertisement revenue for google.


Losing customer advocacy for your product/service


When you don’t test with your initial users, your product gets into the market without any advocates. Not all customers are equal. Some are destined to become super-fans of your product or service. Customer advocates are people who will shell out the most money for your product.  They will bring others into the herd. They will wear your brand tees or shout out your name from wherever they can.

- 83% of brand advocates are likely to share information

- 50% are more likely to share a purchase decision than a regular customer


How do these customers become your advocates? Most likely because you tested your product with them first, honoured their feedback, adapted your app, website or product as per their needs. So even if they didn’t know exactly which product features were a result of their exact feedback, they feel like their need was fulfilled. This is the one factor of user-testing that you can never replicate.


Do you know what are some bad alternatives when it comes to user testing?


Manik Rathee (a former UX engineer for US President Obama’s website in 2012 elections) said - “You are almost always wrong about your users.” That includes your design team, who are sure that users will love their design. They may have been doing this for years, but that still does not guarantee that they will be right this time. I am, in fact, against any of these methods -

- Asking top executives or proactive employees to put on the user’s hat and test the product

- Designers taking their designs so personally, and choosing interfaces on the aesthetics more than usability

- Any team members choosing or giving feedback on interfaces based on their personal preferences

You may end up being another one of the billions of websites (hopefully one of the 200 million active ones) or one of the 3.5 million apps. At the end, you and your team are working for users, so why not confirm that they will like it before going out and throwing it all away.

Know that if  you want to build a long-term term business, you want to move forward with each release,  and keep going back to fix issues with the older ones. You may lose some valuable users in the process of this fixing. It takes five times more efforts and money to bring them back. User-testing is a crucial process of building products that really work.


- By Ashish

Ashish has over two decades of spearheading effective integrated campaigns and strategies across traditional and emerging mediums. He has worked in leading Advertising agencies for global clients across categories. Reach him at




Don’t miss out, get notified!

At UXArmy we are busy giving final touches to our Online User testing ecosystem.
Notify me!

This email is safe, we dont spam

Start user testing

Test with your users for free to get feedback on prototypes, websites and mobile apps. No credit card needed.