You opened your Uber App and added your destination. The price was 5X above expectations. What? Why?
Needless to say, you are not very happy! The App is broken, you assume.
Another click away, Google Maps shows you that the regular road is blocked for repair and the diversion would add 5 kilometers.
Ah! Now the 5X makes more sense.
Both Products told you the same fact in different flavors. You were happy with one and disappointed with the other.
All literature remotely related to Product management has established the criticality of user validation.
They recommend that you:
Validate user problems
Validate your assumptions
Validate your proposed solution.
The recommended mechanism to achieve all things validation surveys, focus group discussions, interviews, and whatnot.
Validating different aspects, however, requires an intended well-designed methodology that differs per the intent. It’s easier to witness how the user is interacting with that button, and if she has issues with navigation. It is, however, not as easy to measure customer happiness or sadness when the recommended price of an Uber Ride slowly increases from Rupees 200 to 500. It is however more critical to know, witness, and validate two other key aspects of your Products with your users. The often neglected aspects can be the make and break your Products. The first aspect is — How does the user interact with your corner case handling mechanism. Trust me when I say this, your Product is as strong as its weakest link. The second applies more to Products which are powered by or use an additional hidden layer of Algorithms, a complicated set of rules, or Artificial Intelligence.
Why are these aspects so crucial to Validate?
What happens behind the scenes does not stay behind-the-scene!
As a Product Manager, it is our job to know all the nooks and crannies of our Product behavior.
We give our engineering teams carefully crafted stories that call out loud the correct expected behavior of a Product feature.
When it comes to algorithms (Artificial Intelligence or not) backing the Product, it becomes heavily conceptualized and brought to life in the Engineering world.
There might be assumptions and decisions that were taken during
implementation which totally make sense to us. But do they do too to the users?
The contribution of these algorithms, although extensive, often does not demand critical attention from users. That is until something goes wrong.
The magnitude of “Implications of going wrong” here is huge!
A button color gone wrong you can change. A copy not making sense is easy enough to fix.
An algorithm behavior or a logical block takes more time to create or correct.
Any changes to this piece can have a domino effect on basically everything else.
The criticality of getting this piece right is immense.
Our user validation is usually ineffective because…
The effectiveness and correctness of the algorithmic output are often not very easy to validate through our conventional methods due to the following reasons:
We fail to put users in the right shoes
Taking someone near a well lighted beautiful rack of shoes, pointing out one and asking “Do you like it?” is not the same as handing it over to them, allowing them to walk on a cobblestone road, and then asking “Do you like it?”
The trustability of the user answer is miles apart.
Our biases take over our judgment
Amazon has the sale of Office Chairs. Steal prices! One of them, very suitably priced and looks great. So what if the Brand is unknown, the ratings look good. Well, I wanted a Blue colored chair to match my interiors but a black would do too!
What just happened?
You used good ratings and prices to convince yourself that the chair was perfect. The factors like Brand and Colour, which might earlier have held some importance were ignored.
When you want something to be perfect, it’s likely that in your mind it is
This is the most dangerous bias you can carry to your Product, and we do.