class: center, middle, inverse, title-slide .title[ # Confidence and p-values ] .author[ ### Yue Jiang ] .date[ ### STA 440 / Spring 2025 ] --- ### Example 1 (again) <img src="img/int8.png" width="100%" style="display: block; margin: auto;" /> --- ### Example 1 (again) <img src="img/int9.png" width="100%" style="display: block; margin: auto;" /> .question[ Critique the following conclusion: ] > *There is a 58% chance that the results we observe for the cadmium model arose due to chance alone. Since this is so high, we conclude that this is an insignificant result: there is no association between cadmium exposure and reaction time (adjusting for age and sex), with there being a 95% chance that the slope is between -11.47 and 20.30.* --- ### Statistical inference A process that converts data into useful information, whereby practitioners - form a question of interest, - collect, summarize, and analyze the data, - and interpret the results --- ### Statistical inference The .vocab[population] is the group we'd like to learn something about. If we had data from every unit in the population, we could just calculate what we wanted and be done! Unfortunately, we (usually) have to settle with a .vocab[sample] from the population. Ideally, the sample is .vocab[representative], allowing us to use .vocab[probability and statistical inference] to make conclusions that are .vocab[generalizable] to the broader population of interest. We want to make inferences regarding population .vocab[parameters], which we do with .vocab[sample statistics]. --- ### *Ei incumbit probatio qui dicit* <img src="img/pius.jpg" width="70%" style="display: block; margin: auto;" /> --- ### The hypothesis testing framework 1. Start with two hypotheses about the population: the .vocab[null hypothesis] and the .vocab[alternative hypothesis] 2. Choose a sample, collect data, and analyze the data 3. Figure out how likely it is to see data like what we got/observed, IF the null hypothesis were true 4. If our data would have been extremely unlikely if the null claim were true, then we reject it and deem the alternative claim worthy of further study. Otherwise, we cannot reject the null claim --- ### Interpreting p-values .vocab[p-values do **not** allow you to make affirmative statements about the null hypothesis] Absence of evidence is not evidence of absence; moreover, you don't even have the ability to measure evidence "in favor" of the null at all! .vocab[p-values **cannot** be interpreted as the probability that "the results arose from random chance alone"] p-values do not give any information by the mechanism by which observed results arose; they are simply conditional probabilities of *observing* such results under a specific assumption about the parameters. .vocab[p-values should **not** be used to gauge real-world, scientific, or clinical significance] (see board for a few demonstrations of this fact) --- ### Making a conclusion We reject the null hypothesis if the conditional probability of obtaining our test statistic, or more extreme, given it is true, is very small What is very small? We often consider a cutpoint (the .vocab[significance level] or `\(\alpha\)` level) defined prior to conducting the analysis Many analyses use `\(\alpha = 0.05\)`: if `\(H_0\)` were in fact true, we would expect to make the wrong decision only 5% of the time (why?) If the p-value is less than `\(\alpha\)`, we say the results are .copper[statistically significant] and we .copper[reject the null hypothesis]. On the other hand, if the .vocab[p-value] is `\(\alpha\)` or greater, we say the results are not statistically significant and .vocab[fail to reject] `\(H_0\)`. --- ### But wait... What if `\(p \ge \alpha\)`? We **never** "accept" the null hypothesis - we assumed that `\(H_0\)` was true to begin with and assessed the probability of obtaining our test statistic (or more extreme) under this assumption When we fail to reject the null hypothesis, we are stating that there is *insufficient evidence* to assert that it is false Importantly, we have assumed from the start that the null hypothesis is true, and the p-value calculates conditioned on that event p-values do **NOT** provide information on the probability that the null hypothesis is true given our observed data --- ### What could go wrong? Suppose we test the null hypothesis `\(H_0: \mu = \mu_0\)`. We could potentially make two types of errors: | Truth | `\(\mu = \mu_0\)` | `\(\mu \neq \mu_0\)` | | ----- | ------------- | ---------------- | | Fail to reject `\(H_0\)` | .eno[Correct decision] | .copper[Type II Error] | | Reject `\(H_0\)` | .copper[Type I Error] | .eno[Correct decision] | - .copper[Type I Error]: rejecting `\(H_0\)` when it is actually true (falsely rejecting the null hypothesis) - .copper[Type II Error]: not rejecting `\(H_0\)` when it is false (falsely failing to reject the null hypothesis) .question[ In the example from the cadmium model (p = 0.58), what can you say about the probability that you have made a type II error, if anything? Suppose instead the p-value was 0.01. At `\(\alpha\)` = 0.05, what can you say about the probability that you have made a type I error, if anything? Explain. ] --- ### What is a confidence interval anyway? <img src="img/confidence.png" width="80%" style="display: block; margin: auto;" /> .question[ What does it mean to be "95% confident" that the parameter lies between -11.47 and 20.30? ] --- ### What is a confidence interval anyway? <img src="img/confidence.png" width="80%" style="display: block; margin: auto;" /> .center[(check out the interactive demonstration [here](https://digitalfirst.bfwpub.com/stats_applet/stats_applet_4_ci.html))]