class: center, middle, inverse, title-slide .title[ # Threats to Validity ] .author[ ### Yue Jiang ] .date[ ### STA 440 / Spring 2025 ] --- ### Acculturation among international students <img src="img/jrie1.png" width="100%" style="display: block; margin: auto;" /> --- ### Acculturation among international students <img src="img/jrie2.png" width="100%" style="display: block; margin: auto;" /> --- ### Acculturation among international students <img src="img/jrie3.png" width="100%" style="display: block; margin: auto;" /> .question[ What is your reaction to the *design* of this study? Is there a big concern regarding statistical inference somehow? ] --- ### Pregnancy, HIV, and AIDS <img src="img/hiv.png" width="60%" style="display: block; margin: auto;" /> **Image Credit**: Liza Gross, used under CCA 4.0 License --- ### Pregnancy, HIV, and AIDS Researchers are interested in the special population of `\(HIV+\)` women on antiretroviral therapy in sub-Saharan Africa. They would like to know whether in this population, a new pregnancy is related to the probability of having an AIDS-defining event (that is, their HIV being classified as AIDS). To test for an association, they recruit women from a large network of health care clinics and find the following: | | AIDS | No AIDS | | ------------- | -----: | -----:| | Pregnant | 31 | 44 | | Not Pregnant | 124 | 99 | .question[ Calculate an odds ratio and associated 95% CI using these data for AIDS-defining events based on pregnancy status. What might you conclude? ] --- ### Pregnancy, HIV, and AIDS Consider the following data, which is for all HIV+ women in the area, not just those who visited a clinic: | | AIDS | No AIDS | | ------------- | -----: | -----:| | Pregnant | 44 | 175 | | Not Pregnant | 248 | 990 | .question[ Calculate the same odds ratio as before and associated 95% CI using these data. What might you conclude? How might you explain this? ] --- ### Pregnancy, HIV, and AIDS The original sample was collected due to easier data collection (the women are already in the clinic). However, this is problematic since not all HIV+ women are equally likely to visit a health clinic - we have a .vocab[sampling bias] here: | | Visited clinic | Did not visit clinic | | ------------- | -----: | -----:| | Pregnant + AIDS | 31 | 13 | | Pregnant only | 44 | 131 | | AIDS only | 124 | 124 | | Neither | 99 | 891 | --- ### Everyone's heard of Simpson's paradox... The previous example is an example of .vocab[Berkson's paradox]. In these cases, two *marginally* independent events become *dependent* conditionally on the occurrence of either one of the two events. That is, for two independent events, if we only consider the cases where either of the events occur, then they become dependent (usually negatively). Note that `\begin{align*} P(A) &= \frac{P(A | A \cup B)}{P(A \cup B)}\\ &\le P(A | A \cup B) \times 1\\ &= P(A | A \cup B) \end{align*}` --- ### Berkson's paradox HIV+ women can go to the clinic to check on their pregnancy or because they are experiencing a drastic lowering in health status. In our original case, we've *excluded* every woman who was neither pregnant nor particularly worried about their HIV. We've conditioned on AIDS-defining event OR pregnancy when sampling these women. A non-pregnant HIV+ woman who went to the clinic is more likely to have AIDS than HIV+ women in general, because she's specifically going to the clinic for a *non*-pregnancy-related reason (maybe AIDS). This is responsible for the spurious observed conclusion of pregnant HIV+ women having lower AIDS risk than non-pregnant HIV+ women. --- ### Focusing on type 1 errors... Suppose we test the null hypothesis `\(H_0: \mu = \mu_0\)`. We could potentially make two types of errors: | Truth | `\(\mu = \mu_0\)` | `\(\mu \neq \mu_0\)` | | ----- | ------------- | ---------------- | | Fail to reject `\(H_0\)` | .eno[Correct decision] | .copper[Type II Error] | | Reject `\(H_0\)` | .copper[Type I Error] | .eno[Correct decision] | - .vocab[Type I Error]: rejecting `\(H_0\)` when it is actually true (falsely rejecting the null hypothesis) - .vocab[Type II Error]: not rejecting `\(H_0\)` when it is false (falsely failing to reject the null hypothesis) --- ### A famous XKCD comic <img src="img/significant.png" width="70%" style="display: block; margin: auto;" /> --- ### Family-wise type I error rate control .question[ Suppose you conduct ten (independent) hypothesis tests. Under the null hypothesis for each of them, what is the probability that you will have *overall* made at least one type I error rate among the ten tests? ] -- .question[ How have you dealt with these types of .vocab[multiplicity] issues in the past? What are some ways you might have unknowingly increased your chances of making a family-wise type I error? ] --- ### The Bonferroni-Holm method 1. Sort p-values in ascending order `\(p_{(1)} < p_{(2)} < \cdots < p_{(m)}\)` 2. Compare `\(p_{(1)}\)` to `\(\alpha/m\)` (just like Bonferroni) 3. If significant, compare the next smallest p-value `\(p_{(2)}\)` to `\(\alpha/(m - 1)\)` 4. ...and so on; stop at the first non-significant p-value The Bonferroni-Holm method also strongly controls family-wise error without being as conservative as the Bonferroni method.