Hello all,
I am having a lot of trouble understanding p-values (how they are similar and different from the critical values). If anyone could explain this or have any videos or links to something that could help me out with this, it would be greatly appreciated. Thank you!
I wrote an article on p-values that, despite the fact that I haven’t made the edits suggested by AF’s own tickersu, may be of some help here: http://financialexamhelp123.com/p-vs-α/
As usual, magician has a pretty crisp explanation. Let me try it from another angle:
The p-value is the probability of seeing a value of the test statistic that far from the null or greater by chance, given that the null is actually true.
As an example - let’s say you chose an observation from a normally distributed distribution with variance 1, and observed a value of 1.96. You’ve hypothesized that the true mean of the distribution is zero. The p-value of this test statistic (assuming you’re using a 2-sided null) is the probabilitiy of observing 1.96 or greater given that the true value of the mean is zero. We know that 95% of a normal distribution lies between 1.96 standard deviations from the mean, so there’s there’s a 5% chance of seeing a value greater than 1.96 or less than -1.96 (it’s a 2-sided test) BY CHANCE IF THE TRUE VALUE OF THE MEAN WAS ZERO. So, in this case, the p-value would be 5%.
Let’s take the same example and assume you saw a value of 2.20. What’s the p-value? For a std normal, 97.22% of the distribution lies within +/- 2.2 standard deviations. So, there’s a 2.78% chance of observing this value by chance assuming that the true mean is zero. In other words, the p-value of a z-score of 2.20 in a 2-sided test given a null of sero would be 0.0278.
Rejecting a null hypothesis at the 10% level of significance would require a z-score of +/- 1.645 (in other words, 1.645 would be the critical value of the z-statistic).
In other words, you’d have to be “far enough away” from the null of zero that there would only be a 10% chance of seeing a value greater than that by chance if the null were actually zero. Likewise, you’d need to be 1.96 standard deviations away to reject at the 5% level (i.e. the critical value for 5% level of siginificance is 1.96), and 2.58 is the critical value to reject at the 1% level.
So, given the example above of a z-score of 2.2, you’re far enough away from the null that you can reject at the 10% level (i.e. you’re farther away than the 1.645 critical value) and you can reject at the 5% level (critical value of 1.96), but not at the 1% level (you’re not 2.58 standard deviations away, so you don’t exceed the 1% critical value.
For the TL:DR version - the p-value is the lowest level of significance at which you can reject the null.
NOTE: Edited the last line to correct an error Magician caught.
As usual, I loved your post.
Until, unfortunately, the last line.
The p-value is the _ lowest _ level of significance at which you can reject the null, not the highest.
By the way, I just incorporated tickersu’s suggestions in my article. It’s better.
Thank you CPB. 
As usual, I brain-farted at the end. I’ve edited it so as not to confuse anyone.
It’s the weekend: nobody’s brain is in gear fully.
I’m glad you thought so-- thanks.
I’d like to point out something in bufprof’s awesomely done post that can sometimes be confusing.
Notice the " by chance" phrase followed by some variant of “if the null is true”. Specifically, look at the example where he says the p-value is 0.0278. He isn’t saying that the probability the results are due to chance is 0.0278. Why? because this isn’t true. When we “assume the null is true” we assume that any result we get is 100% due to random chance (i.e. sampling variation). So, it’s important to remember the definition busprof gave (and on S2000’s website) for a p-value. It only tells you how likely you are to observe something at least as extreme (think “in the distribution’s tails”) as what you observered, if that null hypothesis is true. For this same reason, you should also note that the p-value is not the probability of a Type I error (that’s alpha).
Now, if we observe that p-value of 0.0278, we can say that either our results were really unsual if the null is true, or we can say that we have enough evidence to reject the null (start thinking the null isn’t true). It should be intuitive now-- smaller p-values more strongly disagree with the null being true. “Small enough” or “significance” is determined by a preselected threshold, alpha. But, that’s a little off topic.
Hopefully, everyone’s contribution in this thread helped explain p-values (and statistical significance).
Definately bookmarking that page you wrote for later use, thank you