Based on taste tests, it turns out that experts are very bad at predicting the future price of fine Bordeaux wines – so realised Orley Ashenfelter, a Priceton economist. As told by Daniel Kahneman in Thinking Fast and Slow, when Ashenfelter built simple rules of thumb to predict prices he was able to achieve accuracy far beyond that of the experts – his rules of thumb being based just season average temperature, rainfall during harvest and rainfall during winter.
So, why were experts so bad at predicting each wine’s future quality (effectively) price? – the answer, at least accord
ing to Ashenfelter and Kahneman, is that wine tasting is an example of a “low validity environment” – an environment where there is a lack of frequent high quality feedback. When there is a lack of quality feedback, and an abundance of complex information to consider (lots of aromas/notes/tastes), humans tend to be very inconsistent in making judgements because every time around they tend to focus on different information. Such behaviour, in my opinion, is also often the case in entrepreneurship.
In the assessment of future wine prices from tasting, there is both infrequent and low quality feedback. Infrequent because you only find out if the wine is good years later and low quality because even if you knew which wines would eventually be good, the abundance of aromas experienced during tasting makes it hard to pick out patterns. I think something similar is true with startups. Feedback is infrequent because it takes years to know if you are successful, and, feedback is also very low in quality because there are too many input parameters to know which ones were truly decisive (founder quality, technology, market characteristics etc.).
I think that recognising how startups are a low validity environment is very important. Knowing that feedback is infrequent and of low quality implies two things, I think:
- When making individual decisions, we should be very careful not to give much weight to what happened in previous startup decisions.
- When receiving advice based on someone else’s startup experience, we should be very careful to to give much weight to that advice.
To summarise, the problem here is that, in startups, feedback is of infrequent and of low quality and that means we should be careful when making decisions. One suggested solution is therefore to give more frequent, higher quality feedback but I think that this approach is misguided. The fact that feedback is infrequent and low quality is not because we are bad at teaching, it is because startups are, by nature, low validity environments – just like wine price prediction through taste. The lesson, therefore, is not to give more feedback* but rather to get entrepreneurs to focus on simple rules of thumb that have been borne out by large bodies of evidence. The development of such rules of thumb by entrepreneurs like Peter Thiel and researchers like Richard Dawes shall be the subject of these next blogs.
Many thanks to my brother Fergal for interesting ideas for and discussions around this blog, in particular the idea to use wine as an example.
*in fact, I think that can be very dangerous [see Kahneman on wicked feedback]