A Quick Trick for Identifying Usability Problems Without User Testing

Maintaining a website or mobile experience with a high degree of usability is essential to maximize business outcomes, and people who are frustrated often take for granted how easy it is in the digital world to simply click over to a competitor. Even worse are cases where determined customers simply cannot figure out how to proceed to complete a transaction, or otherwise achieve their goals.

At FROM, we regularly conduct both in person and online usability tests for our clients to observe "real" users engaging with their digital experience. This gives us enormous insight into where users are encountering frustration, confusion, or other difficulties, and while we are huge believers in robust usability testing as a tool to identify and prioritize which aspects of a digital touchpoint should be optimized (and really, it’s not terribly time-consuming or expensive), there is a little-known trick that can start to identify many problems. While not as comprehensive as user testing, it’s generally much faster, and therefore, a great place to start!

What is that place? The server's error logs.

First, the server will typically log if a page doesn’t load properly, errors occur, or if transactions fail to complete. Naturally, usability is hampered if your customers are receiving errors because the system not functioning properly, and yet it’s amazing how often server logs don’t get looked at. And since error logs can generally be viewed by browser and device, it’s not uncommon to find that a new version of Chrome or Edge is causing errors that previously didn’t exist, so this is something that need regular attention. In addition, many systems rely on external cloud services, increasing the points of failure. By monitoring server errors, you can make sure you are aware if your site is "breaking," a simple but often overlooked part of managing an effective digital experience.

Second, we have errors of user validation, i.e., a user enters an invalid email or phone number, tries to complete a transaction without checking the "terms and conditions" acceptance box, etc. Now, on the one hand, you might say “That's not my fault, my site worked. It was the user made a mistake!” Bzzzzt. Wrong answer. Especially if there are a lot of these types of errors, or if the number suddenly spikes.

It’s our job to design a solution that makes it unlikely that users will make errors. If they’re frequently overlooking something, or misunderstanding what they are meant to do, it’s a sign we need to look at that screen or field and consider how to redesign it to reduce confusion. It might be as simple as rewriting the instructions or moving a button.

One nuance we like to look for is circular errors. What’s a circular error? It’s when, during a single session, a user sends the same input multiple times and receives the same error. For example, a user submits a page, and the email is determined to be invalid (a logged error.) Then the user submits again, with the same email (and maybe then a third time, again with the same email.) These types of circular errors usually mean the error messaging system in your application is flawed. Perhaps the error text appears at the top of the screen, and the field itself is below the fold, so the user may not even be seeing the error text.

The third type of error is failed search or out of stock messages. The user wants to rent a car with a pickup at 2 am but that location is closed, or the user wants the pants in a 42 waist, but you don’t have any in stock. Or, the user is searching your site for information on bedwetting, but no articles match that term. These types of errors indicate a missed opportunity to meet a customer need, and you should scour these types of messages to consider what steps can be taken to meet commonly requested unmet needs.

All of this is based on the assumption that your site’s back-end code is logging errors properly. This is a standard coding practice, but just because it’s standard doesn’t mean it can’t get omitted, or that certain errors might not have code that logs them. It’s important to check with your technical team; if your site is not logging most errors, or not logging them with sufficient detail, this code can generally be added.

Additionally, you may include logging at different levels of your system, and therefore have multiple log files. For example, the web server may have one log file, the commerce layer may have a separate log file, and your security/authentication layer may have its own log files, and that’s fine. There are great tools that can combine them together and make them easy to analyze, filter, sort, etc.

The logging I’ve been referring to is generally done on the server. However, with each new generation of digital experiences, we push more and more code (including more and more error checking) to the client. Whether it’s javascript (in the case of web pages), or Java code (in the case of mobile apps.) These types of error events can be logged as well, it just requires a separate effort or technology (but it’s well worth it!) You can use analytics packages like Google Analytics to record "events" when certain things (like error messages) happen in the interface.

A one or two-day analysis of error logs can help you focus in on specific, frequently occurring error states that were previously off your radar. Sometimes, it’s still necessary to do user testing to figure out what the deeper reason for the confusion is, but even still, it’s helpful to know where the errors are occurring, so you can focus your testing there. In other cases, it’s easy to guess what’s tripping your users up, once the errors are there to act as signposts.