The latest bit of misinformation on COVID is about absolute risk reduction of COVID vaccines. For example, Peter Attia posted a video on this topic a couple of weeks ago, stating that ARR is about 1% as measured by the trials, which is completely misleading:

ARR and number needed to treat are a staple for doctors. So why is it not applicable to these COVID vaccine trials?

To understand why, lets first briefly review how the COVID vaccine trials work.

In the COVID vaccine trial, people are randomly chosen to be placed into two groups: a control group and a treatment group. The treatment group is given the actual vaccine, whereas the control group is given some placebo (saline water or an unrelated vaccine).

Importantly, in the next step participants are told to observe all precautions that non-vaccinated people are advised to observe: to mask, to socially distance and in general to avoid situations where they may be exposed to the virus.

During this time, the prevalence of the virus is controlled by other means: distancing, stay at home orders, masks, periodic rapid antigen tests and any other tools at our disposal.

Despite precautions, a percentage of participants will be unfortunate enough to be exposed to someone infectious. When that happens, a further sub-percentage of them in both groups will become infected, and symptomatic.

The trials for Pfizer and Moderna were designed to stop at a fixed number of infections. Moderna had 30,000 participants, with 15,000 in the control and intervention group each. Pfizer had around

We can measure the relative risk reduction by comparing the number of symptomatic infected in the control group and in the intervention group. For example, if we had 100 people infected in the control group but only 10 in the intervention, the relative risk reduction of that vaccine would be 90%

But what about absolute risk reduction?

We can't measure that.

"Why not?" you might wonder. If we have the size of the control group - e.g. 10 000 people, and the size of the intervention group (e.g. also 10 000 people), shouldn't the absolute risk be really easy to calculate? i.e.

$ARR = { 100 \over 10 000 } - { 1 \over 10000 } = 0.9$

The answer is no, and here is why.

In medication experiments where we measure ARR / NNT, the medications are given to a population with relatively stable characteristics. The percentage of people who are at risk to develop sickness does not change all that much.

In contrast, in the COVID vaccine experiments, only a small percentage of the population is exposed to infectious virus because we're using other (very expensive) means to control the rate of growth. The typical prevalence while we run these experiments is less than 2%. For example, see the ONS survey data in England for January 2021 which measured 2% prevalence in England at the worst possible period of the pandemic.

How many people in the control and intervention group were exposed to COVID? We don't know, but its not likely to be much higher than 1-3%, depending on the community prevalence of the disease at the time the experiment was done. To really measure absolute risk reduction, we would have to ensure that we develop conditions for our trial groups that are similar to those when the restrictions are completely lifted. What would that mean? If restrictions are lifted, within a few months, perhaps a year, at least 80-90% of people would eventually be exposed to the virus. That is 30 times as many as during a typical COVID vaccine trial! (Numbers are illustrative, but an order of magnitude difference is almost certain)

So how do we achieve that in the experiment? Everyone in both the control and the intervention group would need to be acting without any precaution to get "naturally" exposed to COVID. Furthermore, everyone they're in contact with should also act without precautions to ensure there is a realistic probability that they're an infectious contact. Alternatively, the groups would deliberately need to expose themselves to the virus. Since its completely unethical to run an experiment like that, its not possible to recreate realistic conditions for the disease. Therefore, its not possible to measure ARR in the same way that its measured for medications.

The best we can do is try to extrapolate what that number would be. We know around 50%-60% are susceptible to developing symptomatic infection. If our absolute risk reduction measured when there was 1.5% prevalence was 1%, then the absolute risk reduction of developing symptomatic infection would probably be something closer to 35%

We don't know the exact ARR of COVID vaccines. But we know that its a lot closer to RRR, and nowhere near to the "ARR" number we can extrapolate from vaccine trials - that number is completely meaningless and depends heavily on the prevalence of the virus at the time of the trial.