I recently came across (again) the classic logic puzzle the liar and the truth teller. My first introduction to this logic puzzle was when I was a kid watching the 80s movie Labyrinth. The hero needs to find the right door to freedom, but one guard tells the truth and the other one lies. She can ask only one question to find the right door. What question do you ask?
Now that I’m older, I can see the relationship to A/B testing. What is fascinating to me is that you can’t solve the puzzle without the liar and the truth teller. You need both to get the answer. (Spoiler ahead, try to solve it on your own.)
Let’s say you are testing a new home page design on the website. Which one leads to better conversion? Just like the puzzle, which one leads to freedom?
Like the puzzle, version A and B are not going to give you the answer, you must ask the right question.
ASKING THE RIGHT QUESTION: In the puzzle, asking the correct question is crucial to determine the door to freedom. In A/B testing, asking the correct question is what you do when defining the right hypothesis. Your hypothesis will drive your test and determine what changes you make to Version B.
UNDERSTANDING RESPONSES: In the puzzle, we need to comprehend the implications of the answers given by the guards, regardless of who is speaking. The answer they give is data, but we need to know how to correctly interpret the data in relation to our initial question.
GETTING TO THE TRUTH: The puzzle’s goal is to find the door to freedom; in A/B testing, the goal is to increase conversions. With the right hypothesis, the right question, the right data, we are getting closer to the truth or at least the truth between the two versions. That’s the point of experimentation, we don’t know the truth, but we have a system to find it.
The fun part about this puzzle (spoiler ahead) is that both guards will give you the same answer if you ask the right question. And then your choice is to take the opposite path of what they said. This is crucial to testing, you must know at all times what you are optimizing towards as to not get distracted by other data, biases, opinions, etc.
I think that many marketing organizations are not asking the right questions and don’t have a system for finding answers. It would be like people coming to the guards and the doors, not asking questions and then choosing a door because they like the design better, or they liked the guard’s attitude better or they said, “screw it, we have a 50/50 chance” and just picked a door. We know that there are so many methods people use that can end in disaster.
Think of this puzzle as you approach your website, you need to find the path to freedom, the answers are there in the data and can be unlocked by the right questions and approach.
Also, if you grew up in the 80s and you watched movies like Labyrinth, The Dark Crystal, The Secret of NIMH, etc. Reach out to me, we may need a therapy support group.
