-A basic primer on forecasting using the base line technique and drugs in development as an example.
-References to Tetlock’s work
-How it applies to second order/third order thinking and stock impacts
I’ve been dwelling on Tetlock’s book Superforecasting. I blogged his 10 principles before (see end). One important technique described in the book but not in his 10 guidelines is the importance of knowing or in estimating the base line or base rate for any forecast (or indeed for any piece of stat or data you are given).
The idea of knowing the base rate about an area gives you a sense of how significant a finding or number might be.
Example:
If I receive a dose of radiation of 100 micro sieverts (Sv) how good / bad is that? If you know the yearly base rate in the UK is around 300 micro Sv then you know that’s about ⅓ of the dose in a year. That seems fine to have once a year for instance from a scan, but perhaps not 100x in a year.
When forecasting the probability of the success of a drug, I (and most of the forecasting industry) tend to start with the average probability depending on the phase of testing.
In basic terms, this is 10% for phase 1, 40% for phase 2 and 70% for phase 3 (although ideally you need to know if a drug has passed this phase successfully and lots of other details)
[Note, in modern drug development there are other ways of understanding where a drug is in development for instance, has it passed proof of concept which is around phase 2]
On average, knowing nothing more about a drug other than it has successfully passed a phase 1 trial you would pencil in a 10% chance for that drug to make it all way to market (be given approval by the pharmaceutical drug regulator).
As you learn more information about the drug, you can then adjust this probability up or down.
Is it a totally new type of drug or is it based on biology / chemistry / mechanism we understand well?
Is it in a therapy area where the understanding of the science (and therefore success rate) is high or higher than average or average or is it in an area where our understanding is lower than average (eg Alzheimer’s and most of neuroscience)?
Were there signs in the trial of successful efficacy ? (There is then a range of questions you can ask about efficacy)
Were there signs in the trial of problematic safety ? (Same as efficacy, there are then a range of questions you can ask) For instance, is there such an unmet medical need that the regulator (and doctors/patients) would tolerate a higher degree of side effects (eg a cancer drug over a head ache drug)
How big was the trial ? How well designed was the trial ? Are there many trials or a single trial ?
Depending on your judgement of this information (which is not always available), you can then push your probability assessment up or down vs the average or base rate chances.
Taking the judgement second order and third order
When it comes to investment decisions there are then 2 more (amongst others) major judgements to make. First, a judgment on how much the drug will sell (and when). This will give you an approximate sense of value as sales x probability of success (discounted back by time).
You then need to make a judgement as to how different your judgement is from what everyone else in the market is thinking or judging. If your judgement is the same as everyone else - you don’t have anything superior to add.
However, if you can judge that everyone is thinking one way eg “this safety side effect is too bad to be approvable” but you are thinking another thing eg “this side effect can be managed by an appropriate process” then you might have a genuine forecasting edge.
This has been described as second order thinking. It’s not mentioned in Tetlock’s book that much (although in the first section about poker and guessing numbers), but it is by other many thinkers (Taleb, Maboussin, game theory etc.)
Along with all the other pieces in play, this ability to look beyond the first order effect to likely consequences at a sufficient order that you can look past most people, but not so far as to render the likely consequence as too unlikely. It’s hard but not impossible and you improve with practice and feedback,
Let’s look at a historic example. Back in the middle to late 2016, Kite Pharmaceuticals had finished / was finishing phase 3 testing for its CAR-T pharmaceutical product – now called Yescarta.
Significant numbers of investors were skeptical about the success of the product.
(Two pieces of evidence for this would be its weak stock price movement over 2016 and the amount of outstanding short selling interest in the stock.)
We can start with our baseline forecast.
On average we might expect about 70% of products with a successful phase 3 to make it through the FDA to approval.
Raising the approval risks would be:
Novel therapy with unproved mechanism
Side effects including cytokine release syndrome (CRS) making it potentially difficult to approve
Limited number of patient experiences
Relatively small number of trials
There are also added commercialization risks due to the novel way of producing and developing therapy.
Mitigating these risks would be:
Efficacy signal of significant response rates in what is typically terminal cancer
High unmet medical need allowing regulators to approve therapies with high side effects
Plus towards the end of 2016, there were several doctors publically suggesting they thought the CRS was manageable with specific treatment and management regimes.
Much of this discussion and data was available at a 2016 cancer conference (ASH).
At this point in time, we can probably suggest that a “consensus of investors” or in other waords a large part of the buying/selling market were assigning a lower than average probability that this product would make it market.
The interesting point here, is that you could probably make the case that investors were assigning somewhere between a 10% to 40% chance of success in aggregate. Say, you thought that consensus was around 20%.
If your own estimate was 45% - so still lower and still suggesting the drug is more likely to fail than pass, then that’s a bet you actually want to make.
Particularly if you have high confidence in both your estimate of success and your estimate of the market.
This gives you the – often times counterintuitive to people – result that good bets / investments are made even when (in fact often when) you actually believe an event has a low chance of happening. Even more particularly when that event has a large impact.
Back to Kite. Going into 2017, as more people start to judge the data, and more data and discussion is released, the Kite stock price rises – and we can impute that the consensus of people’s judgments are changing.
Fast forward into later 2017 and another company Gilead, makes the decision that Kite’s product and technology is valuable and buys the company at a premium.
That end result is not a forecast many people would have made, but you still might have got somewhere close to there by making your own judgements.
Now due to the feedback from markets and prices and real world data, we can also go back and see where our forecasts might have differed from reality.
In reality, the side effects from CRS etc. are significant and bad, but it turns out that doctors can manage the them and the regulators thought it was acceptable given the unmet medical need.
That’s only one small example of how this forecasting game (with lots of smart people in it£ can work in practice in fundamental markets.
Want more? Read Ray Dalio on his view as to whether we are at a major turning point in history.
https://www.thendobetter.com/investing/2019/7/19/ray-Dalio-investing-paradigm-shift
The 10 guidelines that Tetlock has for forecasting:
https://www.thendobetter.com/investing/2019/7/13/superforecasting-tips