1 00:00:05,726 --> 00:00:07,052 [MUSIC]. So in the last slide we said that 2 00:00:07,052 --> 00:00:10,469 comparing two groups, the control group and the experimental group, is the name 3 00:00:10,469 --> 00:00:13,940 of the game. And we said that we're trying to take 4 00:00:13,940 --> 00:00:18,390 some test statistic and to measure the difference between those two groups. 5 00:00:18,390 --> 00:00:21,928 But, how different is different enough to be significant? 6 00:00:21,928 --> 00:00:24,690 Okay? So how, in other words, how do we know 7 00:00:24,690 --> 00:00:28,240 that the difference that we saw in the experiment is not attributable to just 8 00:00:28,240 --> 00:00:31,020 chance? Well, the answer is we don't, but we can 9 00:00:31,020 --> 00:00:35,550 calculate the probability that that, that it's attributable to chance. 10 00:00:35,550 --> 00:00:40,650 And that's what the p-value is, okay? So, the p-value is the following, if 11 00:00:40,650 --> 00:00:46,150 you'd repeat the experiment over and over again at the same sample size What 12 00:00:46,150 --> 00:00:50,700 percentage of the time would you see results that were at least as extreme as 13 00:00:50,700 --> 00:00:54,131 the ones you got in this experiment? And this is all assuming the null 14 00:00:54,131 --> 00:00:56,686 hypothesis is true. So, let me say that again. 15 00:00:56,686 --> 00:01:00,720 Assuming that there is no difference between the groups, right, the control 16 00:01:00,720 --> 00:01:03,263 group really is the same population as the experimental group Group. 17 00:01:03,263 --> 00:01:07,370 Alright, the treatment has no effect. If I were to do the same experiment over 18 00:01:07,370 --> 00:01:13,050 and over again, what percentage of the time would I, would I see, see a 19 00:01:13,050 --> 00:01:15,494 difference in the treatment group anyway, just by chance? 20 00:01:15,494 --> 00:01:20,630 Okay, and that's what the p-value is. Fine. 21 00:01:20,630 --> 00:01:23,070 So, more terminology, you know, you me, you could think about two sided versus 22 00:01:23,070 --> 00:01:25,100 one sided. So two sided is if we're measuring 23 00:01:25,100 --> 00:01:28,200 something in terms of the absolute value. Right, so the p-value is two times the 24 00:01:28,200 --> 00:01:32,480 probability that x is greater than the absolute value of the measured value, and 25 00:01:32,480 --> 00:01:36,440 if the test is one sided It's either greater than or less than and here the 26 00:01:36,440 --> 00:01:42,693 notation that we're using is mu, which is a mean, and mu, mus sub 0 is the, mean of 27 00:01:42,693 --> 00:01:50,309 the population of the Null Hypothesis. So this screenshot is taken from a nice 28 00:01:50,309 --> 00:01:55,090 applet that you can find. Online and play with here. 29 00:01:55,090 --> 00:02:00,620 But here if, if the null hypotheses is that the mean is 325, and we're doing a 30 00:02:00,620 --> 00:02:06,280 two sided test, where mu is not equal to 325. 31 00:02:06,280 --> 00:02:11,750 We're saying it must be something different than that. 32 00:02:11,750 --> 00:02:16,870 Either higher or lower, right. and the sample size is 10 and the 33 00:02:16,870 --> 00:02:21,660 observed sample mean is 328. Then when you click the Show P button on 34 00:02:21,660 --> 00:02:28,570 the applet, what you get is Is it computes the p value for you and shows 35 00:02:28,570 --> 00:02:31,826 this colored region. And so, these colored regions. 36 00:02:31,826 --> 00:02:37,614 And so these colored regions the area to that curve is the p-value. 37 00:02:37,614 --> 00:02:40,540 Okay. So that's the probability. 38 00:02:40,540 --> 00:02:45,500 That it's at least as extreme as the measured value. 39 00:02:45,500 --> 00:02:47,670 Okay. And if you get, you know, here the only 40 00:02:47,670 --> 00:02:52,290 change I've made is that the sample mean was 329 instead of 328, which means it's 41 00:02:52,290 --> 00:02:54,200 even less likely that you would see this by chance. 42 00:02:54,200 --> 00:02:55,900 And so the area into those curves is even smaller. 43 00:02:55,900 --> 00:03:03,810 And you notice the p value change. The p value went from 0.0574 to 0.0114, 44 00:03:03,810 --> 00:03:04,875 okay? So in order to make some sort of a 45 00:03:04,875 --> 00:03:08,020 decision, you know, did this treatment work, right? 46 00:03:08,020 --> 00:03:11,210 Do we invest in this treatment? Do we move on to the next stage of 47 00:03:11,210 --> 00:03:14,130 trials? We need some sort of a threshold, some 48 00:03:14,130 --> 00:03:18,976 sort of a cut-off for the p value. So what is that cut off? 49 00:03:18,976 --> 00:03:22,040 Well it's 0.05. Why? 50 00:03:22,040 --> 00:03:23,910 No good reason, it makes the math work out, okay. 51 00:03:23,910 --> 00:03:29,546 So this is a 1 in 20 chance. If you can show it's more rare than a 1 52 00:03:29,546 --> 00:03:35,490 in 20 chance then that's deemed to be good enough, okay. 53 00:03:35,490 --> 00:03:38,940 This is the subject of a lot of controversy, depending on what circles 54 00:03:38,940 --> 00:03:41,690 you what, what sort of literature you're reading. 55 00:03:41,690 --> 00:03:48,680 And we'll talk a little more about this in, in a few segments, but that's all I'm 56 00:03:48,680 --> 00:03:54,153 going to say about it right now. So 0.05 is what people are looking for. 57 00:03:54,153 --> 00:03:58,470 Alright. So now that you are armed with a little 58 00:03:58,470 --> 00:04:05,700 bit of basic terminology, let's go back to. 59 00:04:05,700 --> 00:04:09,970 This first slide from this New York article. 60 00:04:09,970 --> 00:04:14,790 And so the question that we raised was what accounts for this truth wearing off 61 00:04:14,790 --> 00:04:18,802 effect, how can we explain what's going on. 62 00:04:18,802 --> 00:04:25,950 Okay. So one reason is publication bias. 63 00:04:25,950 --> 00:04:27,420 All right. So let me read you a couple quotes of, 64 00:04:27,420 --> 00:04:30,815 from this article about publication bias. So, in the last few years several 65 00:04:30,815 --> 00:04:34,195 meta-analysis, and we're talk about what a meta-analysis is in a little bit, have 66 00:04:34,195 --> 00:04:37,410 re-appraised the efficacy and safety of antidepressants included a therapeutic 67 00:04:37,410 --> 00:04:42,464 value of these drugs, may have been significantly over-estimated. 68 00:04:42,464 --> 00:04:44,310 Okay. Although public, and, and there's other 69 00:04:44,310 --> 00:04:47,840 examples in this article as well, okay. So go back review the literature and find 70 00:04:47,840 --> 00:04:49,852 out that things have been overstated. Why? 71 00:04:49,852 --> 00:04:53,604 So, although publication bias has been documented in literature for decades and 72 00:04:53,604 --> 00:04:57,244 origins and consequences debated extensively, there is evidence suggesting 73 00:04:57,244 --> 00:04:58,980 that this bias is increasing. Alright. 74 00:04:58,980 --> 00:05:02,676 So I haven't told you about publication bias is yet but you may be familiar with 75 00:05:02,676 --> 00:05:05,720 the concept and see some of the effects. So a case in point in the field of 76 00:05:05,720 --> 00:05:13,000 biomedical research and autism spectrum disorder which suggest that in some areas 77 00:05:13,000 --> 00:05:15,971 negative results are completely absent. Alright. 78 00:05:15,971 --> 00:05:19,910 So what does that mean? That means that you're only publishing 79 00:05:19,910 --> 00:05:24,360 papers that show significant positive gains, right? 80 00:05:24,360 --> 00:05:28,930 If we try several treatments and none of them work except for one, we try 20 81 00:05:28,930 --> 00:05:31,660 treatments and only one works. How many papers do we publish? 82 00:05:31,660 --> 00:05:37,210 One not 20. Okay? 83 00:05:37,210 --> 00:05:42,750 So, how is this a problem? Okay. 84 00:05:42,750 --> 00:05:47,960 So, do we have an explanation for this decline effect with publication bias? 85 00:05:47,960 --> 00:05:51,970 So, how does this actually work? Well, Let's make a plot where those study 86 00:05:51,970 --> 00:05:55,530 size is on the x axis, and notice this is log scale. 87 00:05:55,530 --> 00:05:59,895 Right. So this is ten and this a 100 and this is 88 00:05:59,895 --> 00:06:02,216 1000 and so on. Alright? 89 00:06:02,216 --> 00:06:08,780 Well, somebody decides we need the number of patients, saying that are involved in 90 00:06:08,780 --> 00:06:14,415 the study. So the bigger the study size, the more 91 00:06:14,415 --> 00:06:18,740 statistical power you have and we'll define statistical power means precisely 92 00:06:18,740 --> 00:06:22,630 in a bit. But the better you are able to determine 93 00:06:22,630 --> 00:06:29,360 actual effects, right. And the assumption here is perhaps that 94 00:06:29,360 --> 00:06:34,620 you know, as time goes on and you see some results, you are able to, you or 95 00:06:34,620 --> 00:06:37,160 other researchers are able to garner more money. 96 00:06:37,160 --> 00:06:39,240 More funding to do larger and larger studies. 97 00:06:39,240 --> 00:06:40,830 Right? So this is maybe phase one, phase two, 98 00:06:40,830 --> 00:06:43,810 phase three trials of some new drug. They get bigger, and bigger, and bigger 99 00:06:43,810 --> 00:06:46,550 sets of patients as you get more momentum behind it. 100 00:06:46,550 --> 00:06:50,430 And this data is not real. This data is simulated. 101 00:06:50,430 --> 00:06:55,300 But imagine you see this kind of decline effect where the results,okay well, 102 00:06:55,300 --> 00:06:59,068 sorry, the y-axis is the effect size. And we'll talk about what the effect size 103 00:06:59,068 --> 00:07:05,520 is precisely in a little while but this is the degree of positive outcome, let's 104 00:07:05,520 --> 00:07:07,214 say. Let's say negative is bad and positive is 105 00:07:07,214 --> 00:07:11,550 good. So this is, you know, the number of 106 00:07:11,550 --> 00:07:17,712 smokers you were able to convince to quit with some intervention counseling method. 107 00:07:17,712 --> 00:07:24,010 Or the, you know, number of white blood cells increased as result of some 108 00:07:24,010 --> 00:07:25,480 treatment or so on. Okay. 109 00:07:25,480 --> 00:07:31,869 So, positive is good. Well, this decline, let's imagine, shows 110 00:07:31,869 --> 00:07:32,930 this sort of a pattern. Right? 111 00:07:32,930 --> 00:07:40,000 Where early studies with just a ten participants is up here, and as the study 112 00:07:40,000 --> 00:07:41,785 went up it sort of got worse, and worse, and worse. 113 00:07:41,785 --> 00:07:45,360 This is where, this is the effect that we see. 114 00:07:45,360 --> 00:07:50,910 How do we explain this? Well, this is directly explainable, this 115 00:07:50,910 --> 00:07:53,585 kind of and effect would be directly explainable just by publication. 116 00:07:53,585 --> 00:07:55,290 Bias. [COUGH] Right? 117 00:07:55,290 --> 00:07:59,960 Imagine that every dot, now is a test that done by some group somewhere for 118 00:07:59,960 --> 00:08:04,100 this phenomenon. What you'd expect is this kind of funnel 119 00:08:04,100 --> 00:08:07,820 shape, where, where the studies get more and more accurate as you get larger and 120 00:08:07,820 --> 00:08:10,740 larger and larger. Right, and they well, this is, this is, 121 00:08:10,740 --> 00:08:13,740 you can't get around this. Right, as the study size goes up, you do 122 00:08:13,740 --> 00:08:16,730 have more statistical power, you're able to better discriminate real effects from 123 00:08:16,730 --> 00:08:20,110 false effects and so on. But you'll notice that the actual effect 124 00:08:20,110 --> 00:08:26,210 that it's regressing to here is 0.0. There is no effect and yet, of course, 125 00:08:26,210 --> 00:08:29,700 you're going to get some just due to variability out here. 126 00:08:29,700 --> 00:08:38,160 And so if you only report the positive ones you'll end up with this mysterious 127 00:08:38,160 --> 00:08:45,684 decline effect. Okay.