Approaches in Observational Research

Approaches in Observational Research

• Approaches in Observational Research

• Our Start in Constructing Prior Knowledge

• Current Research in Quantifying the Knowledge Gap

• Our Future Research on Systematically generating Prior Knowledge

Traditional Epi Approach: An expert defines the study design, including the covariates, and asserts that the results would be unbiased. Expert knowledge driven approach

• Traditional Epi Approach: An expert defines the study design, including the covariates, and asserts that the results would be unbiased. Expert knowledge driven approach

• OMOP/OHDSI Approach: Try all designs and covariates, let the data decide. Data driven approach

Could we combine prior knowledge with a data driven approach?

• Could we combine prior knowledge with a data driven approach?

• We do not want to rely on a single expert to provide us the prior knowledge
• The data driven approach may not always be interpretable given covariates or sample size
• There needs to be a systematic way to capture and represent prior knowledge

Drug A causes/does not cause Condition B

• Drug A causes/does not cause Condition B

• Covariate C causes/does not cause Condition B

• Both may be expressed as probabilities

We can construct systematic representation of prior knowledge about causal effects by:

• We can construct systematic representation of prior knowledge about causal effects by:

• Manual construction of reference sets
• (Semi-) Automatic construction by using a statistical model synthesizing available evidence

List of drugs and condition pairs that are either considered:

• List of drugs and condition pairs that are either considered:

• Positive Controls: in an adverse event relationship
• Negative Controls: not in an adverse event relationship

• OMOP Reference Set

• EU-ADR Reference Set

• AZCERT Dataset

4 conditions (acute kidney injury, acute liver injury, acute myocardial infarction, and gastrointestinal bleed)

• 4 conditions (acute kidney injury, acute liver injury, acute myocardial infarction, and gastrointestinal bleed)

• Across the 4 conditions:

• 165 positive controls (drugs that are known to cause a condition, ground truth is 1)
• 234 are negative controls (drugs that are known to not cause the condition, ground truth is 0).

Process for Positive Controls:

• Process for Positive Controls:

• Drug product labels that listed condition in the ‘‘Black Box Warning” section, the ‘‘Warnings and Precautions”, or ‘‘Adverse Reactions” sections.
• Independent literature review
• Systematic literature review provided by Tisdale and Miller book “Drug-Induced Diseases: Prevention, Detection and Management”

• Process for Negative Controls:

• Same process but looking for a lack of evidence across all sources

Exploring and Understanding Adverse Drug Reactions (EU-ADR)

• Exploring and Understanding Adverse Drug Reactions (EU-ADR)

• 10 conditions (liver disorder, acute myocardial infarction, renal failure, acute, anaphylactic shock, erythema multiform, mitral valve disease, neutropenia, aplastic anemia, rhabdomyolysis, and gastrointestinal hemorrhage)

• Across the 10 conditions:

• 43 are positive controls
• 50 are negative controls

Conditions considered important from a pharmacovigilance/public health perspective

• Conditions considered important from a pharmacovigilance/public health perspective

• Drugs needed enough exposure in the EU-ADR database network

• Literature review to find associations between drugs and conditions.

• Positive Control: 3 drug-ADR association citations found in MEDLINE
• Negative Control: No literature citations and no World Health Organization (WHO) Vigibase mentions

• Manual review

Arizona Center for Education and Research on Therapeutics (AZCERT)

• Arizona Center for Education and Research on Therapeutics (AZCERT)

• The CredibleMeds group, which manages the AZCERT list, focuses on programs to reduce preventable harm from medication.

• This DB focused on two condition:

• Torsade De Pointes (TDP)
• QT Prolongation.

• Only contains positive controls.

First built a model and evaluated performance on individual reference sets EU-ADR and OMOP

• First built a model and evaluated performance on individual reference sets EU-ADR and OMOP

• Second built a model on the combination of EU-ADR and OMOP and evaluated performance on a third reference set AZCERT

Now that we are extracting knowledge, we want to learn what evidence already exists and what evidence does not

• Now that we are extracting knowledge, we want to learn what evidence already exists and what evidence does not

• This will start to give us a better feel for how to extract evidence as prior knowledge

Prior knowledge can be used in (at least) two ways:

• Prior knowledge can be used in (at least) two ways:

• As a reference set in an empirical evaluation, to measure how unbiased a method really is (previous work group meeting presentation by Martijn)
• To formulate explicit prior distributions in our models (future research project)

Frequentists Statistics – there is no prior belief, for example about effect size, any result is deemed equally likely.

• Frequentists Statistics – there is no prior belief, for example about effect size, any result is deemed equally likely.

• Bayesian Statistics – here there is a prior belief and some results are deemed more likely than others. The evidence generated by a study can modified our prior belief into a posterior.

Experts picking what goes into the model (e.g. Expert says “smoking is a risk factor for lung cancer”)

• Experts picking what goes into the model (e.g. Expert says “smoking is a risk factor for lung cancer”)

• Throwing everything into the model (e.g. the model would need to learn that smoking is a risk factor for lung cancer)

• Still throw everything into the model but use an informed prior to help the model learn (e.g. make the prior for smoking and lung cancer would be given a high probability)

Where we have come from: Not having an automated approach to utilize prior knowledge

• Where we have come from: Not having an automated approach to utilize prior knowledge

• What we can do currently: Using an automated approach to identify known drug/outcome relationships (aka negative controls) to calibrate our methods

• What we are working on now: Improving our evidence extraction by exploring the evidence gap

• Where we want to go: Using systematically generated prior knowledge to inform prior distributions in our models.