Critical Appraisal and Statistics

Critical Appraisal is the process of carefully and systematically assessing the outcome of scientific research (evidence) to judge its trustworthiness, value, and relevance in a particular clinical context. When critically appraising a research study, you want to think about and comment on:

1. How solid is the rationale for the research question?
2. How important is the research question?
3. What is the potential impact of answering it? (clinically, research, health policies, etc…)
4. Can I trust the results of this research?

Having a systematic approach to critically appraising a piece of research can be helpful. Below is a guide.

Systematic errors (i.e. - biases) can threaten the validity of a study. There are two types of validity:

1. External Validity: How generalizable are the findings of the study to the patient you see in front of you
2. Internal Validity: Is the study actually measuring and investigating what it set out to do in the study?

There are many types of biases, and some of them are listed out in the table below. Researchers also use more comprehensive tools to measure and assess for bias. The most commonly used tool is the Cochrane Risk of Bias Tool. The components of the Cochrane Risk of Bias Tool is not described here.

More Reading:

• O’Sullivan, J. W., Banerjee, A., Heneghan, C., & Pluddemann, A. (2018). Verification bias. BMJ evidence-based medicine.
• Residual Confounding, Confounding by Indication, & Reverse Causality

A research question that leads to a research study is often quite general, but the answer that we get from a published research study is actually extremely specific. As an example, as a researcher, I might want to know if a new SSRI is more effective than older SSRIs. This seems like an easy question to answer, but the reality is much more complicated. When looking at the results of a study, you want to think how specific or generalizable are the findings?

Ask yourself, what is the target population in the research question? Then ask yourself, does the study sample itself actually represent this target population? Sometimes, the study sample will be systematically different from the target population and this can reduce the external validity/generalizability of the findings (also known as sampling bias). After this, you should look at the inclusion and exclusion criteria for the participants:

Different study designs have different limitations. Having an understanding of each type of common study design is an important part of critically appraising a study. The most common study designs in psychiatric research are experimental designs, such as randomized controlled trials, and observational studies, including: cross-sectional, cohort, and case-control.

The beauty of a properly done randomization is you can eliminate (or come close to eliminating) all the unknown factors. This way, you can be sure that the outcomes of your study are not affected by these factors since randomization should equally distribute all known and unknown factors amongst all treatment groups. Randomization is most effective when sample sizes are large. When studies have small sample sizes, they are called underpowered studies, which makes it hard to ensure the sample has been adequately randomized.

• Good for obtaining a prevalence of a disease or condition
  • e.g. - Point prevalence: at one specific point in time
  • e.g. - Period prevalence: within 1 month period, within 12 month period
• You cannot infer causality from cross-sectional studies, only associations
• There is a risk for sampling Bias because you have no control over who is being measured at this specific point in time

Case-control studies (also known as retrospective studies) are studies where there are “cases” that already have the outcome data (e.g. - completed suicide, diagnosis of depression, diagnosis of dementia), and “controls” that do not have the outcome (hence the name case-control). Rather than watch for disease or outcome development, case-control studies look for and compare the prevalence of risk factors that lead to the outcome happening. For example, a research might look at antidepressant use, and whether that affects the outcome they already know (e.g. - cases of completed suicides)

• Case-control studies are always retrospective (looking into the past!)
• Rather than watch for outcome (i.e. - disease development), compare prevalence of risk factors
• “Cases” have the outcome (i.e. - completed suicide, dementia diagnosis, depression diagnosis); “controls” do not
• Since you already have selected a predetermined number of people with and without the disease, you cannot calculate a relative risk (RR). However, you can calculate an odds ratio (OR).
• Advantages:
  • This is the only way to study rare diseases, since you already know how many cases there are!
  • It is inexpensive because it invariably uses pre-existing data
• Weaknesses:
  • Highly susceptible to different types of bias:
    • Sampling bias
      • You want to make sure that cases are not “atypical” presentations of a disease
      • To mitigate this: try to include all cases that are a representative sample of the population
    • Selection bias
      • You need to make sure that the outcome cases are relatively similar to controls other than on the exposure. Otherwise, the controls might never even have been exposed to the risk factor or have very little chance of developing the outcome!
      • To mitigate this: Select controls from same population as cases, match cases and controls on key variables, and have multiple control groups
    • Recall bias
      • The outcome cases more likely to remember an exposure than the controls (e.g. - asbestos and lung cancer)
      • To mitigate this: use records kept from before the outcome occurred, and in some cases, the keep exact hypothesis concealed from the case (i.e. - person) being studied

A cohort study (also known as a prospective study) follows a group of subjects over time.

• It is the only way to describe incidence (incidence = number of new cases/time)
• It is a little better for inferring causality, but can take a long follow up time, and you need a large sample size
• Unlike a case-control study, you can also calculate a relative risk (RR) of a given outcome (i.e. - you can compares outcome between 2 groups who differ on a certain risk factor)
• Usually you have an exposure or a risk factor (e.g. - antidepressant) plus an outcome (e.g. - dementia)
• However, since you can't control who gets exposed top what, there is a high risk for selection bias!
  • e.g. - those who take antidepressants might have more severe depression than those who do not
  • e.g. - depression might be a confounding factor for dementia
• When looking at cohort studies, it is important to pay attention to the demographics table (“Table 1”), and look for selection bias and important confounding factors. Also try to think of other factors that might not have been measured in the study. This is an important issue with retrospective studies (relative to RCTs), since researchers do not have control over what was measured!

Systematic Reviews (SR) and Meta-Analyses (MA) synthesize all the available evidence in the scientific literature to answer a specific research question. Systematic reviews describes outcomes of each study individually (always look for a forest plot in the paper). The meta-analysis is an extension of a systematic review that uses complex statistics to combine outcomes (if outcomes of different studies are similar enough). In most meta-analyses, there are usually strict criteria for study inclusion that usually weeds out flawed research study. However, this is not always the case! Poor study selection can often result in flawed meta-analysis findings (garbage in = garbage out)! It is also important to watch for publication bias (i.e. - certain articles might be favoured over others).

Fig. 1

• This is what you will often see in an RCT to illustrate the magnitude of an effect
• Can just mean a difference between means, or it can be standardized (e.g. Cohen’s d = difference between means/SD)
• Often described as small, medium or large
• For Cohen’s d: 0.2 is small, 0.5 is med, > 0.8 is large

Depending on the design of the study, the results section will look very different. However, all studies should:

1. Describe the sample (prevalence, incidence, and a “Table 1” of demographics)
2. Show the relationships between factors (e.g. - treatment and outcome, risk factor and outcome)
3. Tell you how large was a treatment effect (if looking at this)

Results may first compare the mean (average) between a treatment or non-treatment group:

• T-test, F-test (ANOVA): if p<0.05, then there is a significant difference
• See this in your “Table 1s” and in primary analysis of RCT outcomes
• If making multiple comparisons (tests) then you increase the chance that there will, by chance, be a significant result that is a false positive (Type I error), you can adjust for this using the Bonferroni correction.

</WRAP>

Fig. 3

Relative Risk (RR) is a ratio of the probability of an outcome in an exposed group compared to the probability of an outcome in an unexposed group

• Dichotomous variables can offer relative risk calculations of an event happening (i.e. - risk in group 1 relative to risk in group 2)
• RR is used for cohort studies, and randomized control trials
• RR tells us how strongly related 2 factors are but says nothing about the magnitude of the risk.
• If RR = 1, then both groups have equal risk

Relative risk reduction (RRR) measures how much the risk is reduced in the experimental group compared to a control group. For example, if 50% of the control group died and 25% of the treated group died, the treatment would have a relative risk reduction of 0.5 or 50% (the rate of death in the treated group is half of that in the control group).

Absolute Risk Reduction (ARR) is the absolute difference in outcome rates between the control and treatment groups (i.e. - CER - EER). Since ARR does not involve an explicit comparison to the control group like RRR, it does not confound the effect size of the treatment/intervention with the baseline risk.

Number Needed to Treat (NNT) is another way to express the absolute risk reduction (ARR). NNT answers the question “How many people do you need to treat to get one person to remission, or to prevent one bad outcome?” To compare, a RR and RRR value might appear impressive, but it does not tell you how many patients would you actually need to treat before seeing a benefit. The NNT is one of the most intuitive statistics to help answer this question. In general a NNT between 2-4 means there is an excellent benefit (e.g. - antibiotics for infection), NNT between 5-7 is associated with a meaningful health benefit (e.g. - antidepressants), while NNT >10 is at most associated with a small net health benefit (e.g. - using statins to prevent heart attacks).^[4]

• Odds Ratio = odds that a case is exposed/odds that a case is not exposed
• Used for case-control studies because there is no way to compute the risk of the outcome (i.e. - because you started with the outcome already)
• Very frequently used because it is the output of a logistic regression
• The OR approximates the RR for rare events (but becomes very inaccurate when used for common events!)
  • Note that as a rule of thumb, if the outcome occurs in less than 10% of the unexposed population, the OR provides a reasonable approximation of the RR.
• If OR = 1, then equal odds for both exposed and non-exposed groups (lower bound is 0)

• Relative Risk (RR) = (incidence in exposed) ÷ (incidence in unexposed)
  • = a/(a + b) ÷ c/(c + d)
  • = 2/(2 + 248) ÷ 3/(3 + 747)
  • RR = 2.0
• Odds Ratio (OR) = (odds of exposure among cases) ÷ (odds of exposure among controls)
  • = (a ÷ c) ÷ (b ÷ d)
  • = (a × d) ÷ (b × c)
  • = (2 × 747) ÷ (3 × 248)
  • OR = 2.0080645161 = 2.01

Note that even though RR and OR both = 2, they do not mean the same thing!

• An OR = 1.2 = there is a 20% increase in the odds of an outcome (not risk) with a given exposure
• An OR = 2.0 = there is a 100% increase in the odds of an outcome (not risk) with a given exposure
  • This can also be stated that there is a doubling of the odds of the outcome (e.g. - high pesticides exposure are 2 times as likely to have lukemia)
  • Note, this is not the same as saying a doubling of the risk (i.e. - RR = 2.0)
• An OR = 0.2 means there is am 80% decrease in the odds of an outcome with a given exposure

In psychiatry, studies may typically look at the time to relapse for an event (e.g. - time until a depressive episode relapse). There are several “buzz words” that may be used:

• Univariate analysis means there are Kaplan-Meier Curves
• Multivariable analyses most commonly use Cox proportional hazard modelling
• Both types of analyses allow for censoring
  • Censoring allows for analysis of data when the outcome [dependent variable] has not yet occurred for some patients in the study
  • e.g. - a 5-year study looking a diagnosis of dementia, but not all participants have dementia by the end of the 5-year study
• These analyses generate a “hazard ratio”, which is the ratio of:
  • [chance of an event occurring in the treatment arm]/[chance of an event occurring in the control arm], or
  • [risk of outcome in one group]/[risk of outcome in another group]… occurring over a given interval of time

Linear regression generates a beta, which is the slope of a line

• Beta = 0 means no relationship (horizontal line)
• Beta > 0 means positive relationship
• Beta < 0 means negative relationship

• Range of values within which the true mean of the population is expected to fall, with a specified probability.
• CI for sample mean = x ± Z(SE)
• The 95% CI (α = 0.05) is often used
• As sample size increases, CI narrows
• Remember:
  • For the 95% CI, Z = 1.96
  • For the 99% CI, Z = 2.58

Most studies only demonstrate an association (e.g. - antidepressant use in pregnancy associated with an increased rate of preterm birth). How can we decide whether association is, in fact, causation? (e.g. -Does anti-depressant use in pregnancy actually cause preterm birth?). The Bradford Hill criteria, otherwise known as Hill's criteria for causation, are a group of nine principles that can be useful in establishing epidemiologic evidence of a causal relationship between a presumed cause and an observed effect

1. Experimental evidence: from a laboratory study and or RCT
   • Researcher controls assignment of subjects and exposure
2. Strength of Association: what is the size of relative risk or odds ratio (remember, RR or OR = 1 means no association)
3. Consistency: same estimates of risk between different studies
4. Gradient: increasing the exposure results in increasing rate of the occurrence
5. Biological and Clinical Plausibility: judgement as to plausibility based on clinical experience and known evidence
6. Specificity: when an exposure factor is known to associated only with the outcome of interest and when outcome not caused by or associated with other risk factors (e.g. - thalidomide)
7. Coherence: also called “triangulation”: when the evidence from disparate types of studies “hang together”
8. Temporality: realistic timing of outcome based on exposure
9. Analogy: similar associations or causal relationships found in other relevant areas of epidemiology

Evidence-Based Medicine (EBM) typically will depend on the use of statistical and critical appraisal approaches. However, there are also inherent limitations and potential issues with the way EBM is applied in current medical research.

• Oza, A. (2023). Reproducibility trial: 246 biologists get different results from same data sets. Nature, 622(7984), 677-678.
• Bishop, D. (2019). Rein in the four horsemen of irreproducibility. Nature, 568(7753)
• Nature: Replication failures in psychology not due to differences in study populations
• Eric J. Topol, John P. A. Ioannidis. Ioannidis: Most Research Is Flawed; Let's Fix It - Medscape - Jun 25, 2018

• Sox, H. C., & Rennie, D. (2006). Research misconduct, retraction, and cleansing the medical literature: lessons from the Poehlman case. Annals of Internal Medicine, 144(8), 609-613.

• Substack: Stop and Think - If only critical appraisal was this good for *all* studies
• Rational Psychiatry: How To Read a Paper (Thomas Reilly )