Have you ever tried to make a decision by writing down, weighting and then rating a few different factors you felt were important? M
aybe it was while picking a job, maybe it was picking a university course or maybe it was deciding on an employee to hire. If so, I think there’s a better way to make decisions that works particularly well when there are many uncertain and qualitative factors to consider – which is often the case for a startup.
The approach I’m going to give is not new – it goes back to a paper in 1971 by Dawes. Dawes took a look at graduate admissions and investigated whether the admissions committee was any good at predicting how students would be ranked, by professors, at the end of their graduate studies. The answer is that the admissions committee was very poor at this and the correlation between their assessment and the final ratings of students was just 0.1 . So Dawes built a simple predictive model that weighted, equally, incoming students’ GRE scores, grade point averages and the quality of their undergraduate institutions. This simple model, it turns out, had a correlation coefficient with the final ratings of students (after graduation) of 0.51, much higher than the admissions committee’s assessment! In other words, a simple model was much more accurate at predicting success than the admitting committee*.
To apply this model (called an “improper linear model”) I suggest the following steps:
- Decide on what the important factors are in making your decision and include only the most important ones (you should probably aim for 3-5 factors, no more).
- Weight all of these factors equally.
- For each factor, decide on what a score of 0/1 should look like and decide on what a score of 1/1 should look like. Thinking of what an intermediate score (0.5/1) should look like can be helpful too. Ideally these scores are derived from data (like GRE data) but this isn’t always possible.
- Rate each of your options on a scale of zero to one on each of the factors.
- Add together and rank the scores for each option to arrive at a decision.
To see an application of this approach, you can take a look at the model I developed to predict the success of startups in Part 2 of this series (where I use pretty qualitative data), or, even better, take a look at the work by Dawes (which uses qualitative data for factors).
The reason this approach works well, compared to the usual weighted approach, is that it turns out humans are inconsistent in how we assign weightings. In uncertain environments, the risk of getting the ratings wrong very often outweighs the benefits of trying to optimise the weightings. With equal weights assigned to each factor, you are essentially minimising the risk of getting the weightings wrong; for uncertain environments, as pointed out by Dawes 1979 paper, this is a clever thing to do. The second thing this approach tries to do (in point 3 above) is to shift the focus onto factors that are more measurable. The logic here is that often, in uncertain environments, it’s better to use a measurable parameter that isn’t quite right (for example, using the weight of a bull when trying to order to rank bulls by age) rather than to revert to something less quantitative (for example, how old his eyes look). In other words, the risks of making a bad qualitative judgement are very often greater than the loss of accuracy in using a more quantitative factor, even if that quantitative factor seems somewhat tangential.
I think startups are environments with lots of incomplete and uncertain information. For me, this makes thinking about how to make decisions well worth the while. Improper linear models are one tool I think is worth considering.
*It turns out that humans are notoriously inaccurate at gauging quality in interviews. Nobel laureate Daniel Kahneman had the same problem when he worked to evaluate interviewing techniques in the Israeli army. He also found that a simple model was better at predicting success.