obsidian/🥙 Nutrition Vault/Nutritional Epidemiology Checklist.md

Evaluating nutritional epidemiology can be a right pain in the ass, and tons of studies get chucked around to support all sorts of claims. Here I have compiled a quick and dirty checklist for evaluating nutritional epidemiology, specifically prospective cohort studies. I've divided the checklist into two categories: "mandatory" and "cherry on top". The "mandatory" checklist contains study characteristics that **must** be assessed. The "cherry on top" list contains study characteristics that would be nice to assess, but aren't required. Enjoy!

**MANDATORY:**

**☐ Is there a meta-analysis of the research question?**

If you can find a meta-analysis on the same research question which includes either cohort studies or randomized controlled trials, you might not even need to look at this particular cohort study, as it would likely just be superseded by the findings of the meta-analysis.

**☐ What does the background population look like?**

Each population will have its distinct characteristics that could influence the results and could make comparisons with other cohort studies in other populations challenging.

-   **☐ Is the age range appropriate for the endpoint?**

Be sure that the mean/median age of the population being investigated is appropriate for the endpoint of interest. For example, heart attacks generally occur between the ages of 55-75, but if the median age of the population in your cohort is 45, you're not going to be likely to see a statistically significant association with the exposure.

-   **☐ Are the incidence rates sufficiently high?**

You may need to investigate other bodies of evidence to get a handle of what sorts of incidence rates are to be expected given a certain population and participant number. Once you have a handle on that, you can evaluate whether there is doubt as to whether the incidence rates would be sufficiently high to see a statistically significant result.

-   **☐ Is the exposure contrast sufficiently wide?**

Here we're just referring to the contrast between low and high intakes. For example, 10g versus 30g of chicken or 5g versus 200g of red meat. You may need to investigate other bodies of evidence to get a handle of what sorts of exposure contrasts typically present statistically significant associations with risk. Once you have a handle on that, you can evaluate whether there is doubt as to whether the exposure contrast would be sufficiently high to see a statistically significant result.

-   **☐ Is there something circumstantial about this cohort study?**

Some studies are idiosyncratic and use bizarre, unorthodox methodology. In order to get a handle on how to assess this, you'll have to run many, many cohort studies through this checklist.

-   **☐ Do other cohort studies tend to find contradictory results?**

Maybe this cohort study is an oddball. If it appears to be done either worse or no better than many others that find contradictory results, then the results shouldn't really move our needle much. If this cohort is special in that the methods are quite superior to other cohort studies on similar research questions, then this cohort study should be given more weight.

**☐ Are there a sufficient number of participants?**

You may need to investigate other bodies of evidence to get a handle of what sorts of participant numbers are desirable, given the follow-up time and incidence rates. Once you have a handle on that, you can evaluate whether there is doubt as to whether the participant numbers would be sufficiently high to see a statistically significant result.

**☐ Is there sufficient follow-up time?**

Follow-up time goes hand-in-hand with other variables, such as population age. For example, stroke generally occur between the ages of 70-90. Let's say the median age of the population in your cohort study is 55, and there is 15 years of follow-up. It's not clear that there is sufficient follow-up time to reliably detect statistically significant associations with risk.

**☐ How is the exposure being analyzed?**

The farther the authors' analysis gets from using continuous variables, the poorer the resolution of the analysis. It's typical to see variables represented as quintiles or quartiles in cohort studies.

-   **☐ continuous (best, but somewhat uncommon)**

Analyzing continuous variables is generally superior to analyzing discrete variables, like quintiles.

-   **☐ discrete variables (standard/sub-optimal)**

Analyzing quintiles of variables is generally superior to analyzing tertiles of variables. The lower the number of quantiles, the lower the resolution.

-   **☐ dichotomized (poor)**

Almost anything will usually be better than analyzing dichotomized variables. However, some variables are truly dichotomous and cannot be represented any other way. For example, hyperlipidemia at baseline as either true or false, and is thus a definitionally dichotomous variable.

**☐ What was the measurement method for the exposure?**

There are only a few different ways to measure nutritional exposures that are considered robust. Pay close attention to which measurement method was used, because it can often be the Achille's heel of the input data itself.

-   **☐ validated biomarkers (best exposure measurement)**

As long as the biomarker has been validated, meaning that it can be used as a good proxy for nutritional intake, then biomarker data is the cat's ass of intake measurement methods. Biomarker data should be favoured in all cases. 

-   **☐ validated food frequency questionnaires (best self-report method)**

Validated FFQs are also excellent, but secondary to biomarkers. But they're gold standard and still provide very good intake estimates in most cases.

-   **☐ 28-day food diaries (sub-optimal)**

If less robust methods are used, such as 28-day food diaries, try to ascertain how many repeat measurements were taken during the follow-up time. The lower the number, the lower our credence should be in the data.

-   **☐ 24-hour records/recalls (poor)**

Measurement methods like this are bottom of the barrier, and their utility as an intake measurement method questionable if chronic, longer latency period diseases are being investigated. Sometimes the measurement method is as simple as a phone call and a short conversation about what the participant ate within the last 24 hours, lol.

**☐ What was the measurement for the endpoint?**

The endpoint measurement method will depend on whether the endpoint is soft (change in a risk factor) or hard (disease event or death). These methods won't change a whole lot, but sometimes some researchers will throw you a curveball.

-   **☐ medicals records**

Standard methodology for assessing both hard and soft endpoints.

-   **☐ death records**

Standard methodology for assessing hard endpoints.

-   **☐ biomarkers (assays, etc.)**

Standard methodology for assessing soft endpoints.

-   **☐ some other fuckery**

Here is where the curveballs will be thrown. Sometimes the endpoints are very questionably measured, such as deaths being confirmed via a phone call to the next-of-kin. Recently, there was also a cohort(ish) study published out of Harvard on the carnivore diet, wherein the participants self-reported all of their endpoint data. Methods like this should lower our credence in the results.

**☐ Is the adjustment model comprehensive?**

The adjustment model is at the heart of the application of the author's causal model. The variables that the authors posit as being potentially confounding or covarying will be included in the model as a means of achieving what's called conditional exchangeability. Without getting it's the potential for over-adjustment, a general heuristic is that the more comprehensive the model, the better. But with caveats.

-   **☐ Are there plausible confounders and covariates missing?**

You may need to investigate other bodies of evidence to ascertain whether or not the adjustment model is indeed comprehensive. If there are any variables not included for which a sound causal inference can be made, that should lower our credence in the results, so long as such variables would be sufficient to explain the effect size observed.

-   **☐ Did the authors adjust for mediators or moderators?**

Mediators are variables that lie within the causal pathway between an exposure and an endpoint. Moderators are variables that influence the causal pathway between an exposure and an endpoint. Adjusting for mediators and moderators typically has the effect of rendering associations non-significant. This should be avoided, unless the authors are specifically trying to test for mediator or moderator effects.

**☐ Are the research questions and assumptions transparent?**

This should be self-explanatory. It's sub-optimal if authors are not transparent about what their even attempting to investigate, and how.

-   **☐ Well-defined endpoints**

The endpoints should be clear. For example, the endpoint of cardiovascular disease should be defined in terms of the endpoints that comprise it, such as stroke, hypertension, ischemia, etc. Ideally these endpoints are identified using ICD identifier codes.

-   **☐ Well-defined exposures**

Just as endpoints should be clear, so should be the exposures. For example, a cohort study could be investigating red meat, but if no attempt is made to qualify what that means, interpretation can be challenging. Red meat could include processed meat, or even pork in some cases. Without clear definitions, it's uncertain what specific exposure the association could actually tracking.

**☐ Was there a dose-response analysis performed?**

This isn't mandatory, but it really does help when trying to ascertain the shape of the risk curve.

-   **☐ Did the analysis show a dose-response?**

If there was a dose-response, keep the shape of the risk curve in mind. Ascertain whether or not it is consistent with other results from other bodies of evidence. If this is the first dose-response analysis you've seen on this research question, use these results as the standard against which future research will be compared.

**☐ Did the authors perform any sensitivity analyses?**

If there is evidence of significant imprecision in the results, typically characterized by unusually wide confidence intervals, the authors may attempt a sensitivity analysis. Basically the authors will test the influence of different variables, such as stratifying the results by sex, age, or region.

**☐ Did the authors preregister?**

If yes, read the preregistration to see if their stated methods match the methods used in the paper. If they don't, suspect fuckery. If they do, great. If there is no preregistration, it should lower your credence in the results slightly. If there is both no preregistration and an unusual finding, that should moderately lower your credence in the results.

**CHERRY ON TOP:**

**☐ Did the authors provide a transparent causal model?**

Something as simple as a directed acyclic graph is extremely helpful in understanding the authors' causal model. It's not necessarily, but it makes it clear if there is anything obvious missing, or if there are relationships that you were unaware of before. The relationships aren't always obvious by just reading the adjustment model.

**☐ Is the raw data and analysis code available for public access?**

Very rarely does this happen, but it is nice when the authors are transparent about these things.

**☐ Did the authors avoid affirming the null?**

This is a bit tongue-in-cheek, but if the authors say things like "no association" or "X does not cause Y", that should raise some red flags in your mind about the authors.

**☐ Is there a plausible biological mechanism?**

This is certainly not required for causal inference, but it is nice to have some sort of biologically plausible mechanism with some supporting data behind it.

**☐ Was their a power analysis?**

Rarely clearly seen in nutritional epidemiology of any sort, but if available it could shed some light on non-significant results.

**☐ Did the authors correct for multiple comparisons?**

The more endpoints you measure, the higher the need for a multiple comparisons correction method. If the authors measure an unusually high number of endpoints but do not disclose such a correct, your credence in the results should probably be slightly lower. It's not a make-or-break proposition, though.

#patreon_articles
#epidemiology 
#cohort_studies