WEBVTT Kind: captions Language: en 1 00:00:00.580 --> 00:00:02.790 Institute of Museum and Library Services 2020 Grants to States All States Conference May 2 00:00:02.790 --> 00:00:03.790 12, 2020 Virtual Proceedings by: CASET Associates, Ltd. caset@caset.net 3 00:00:03.790 --> 00:00:07.250 Agenda Item: State of the Nation: Analysis of Grants to States’ SPR Projects 4 00:00:07.250 --> 00:00:08.250 MS. 5 00:00:08.250 --> 00:00:11.050 DEVOE: Hi, this is Teri from IMLS. 6 00:00:11.050 --> 00:00:12.550 Let me just introduce our next speaker, who some of you have met at prior Grants to States 7 00:00:12.550 --> 00:00:13.550 conferences. 8 00:00:13.550 --> 00:00:14.889 Lisa Frehill is not part of our Grants to States team, but she is a wonderful asset 9 00:00:14.889 --> 00:00:16.000 to our program. 10 00:00:16.000 --> 00:00:21.789 She is from our Office of Digital and Information Strategy and is our senior statistician there. 11 00:00:21.789 --> 00:00:28.240 So along with Matt Birnbaum, she is going to be presenting some data analysis from your 12 00:00:28.240 --> 00:00:30.260 wonderful State Program Report projects. 13 00:00:30.260 --> 00:00:32.910 So take it away, Lisa. 14 00:00:32.910 --> 00:00:33.910 MS. 15 00:00:33.910 --> 00:00:35.780 FREHILL: Thank you so much, Teri. 16 00:00:35.780 --> 00:00:43.880 I want to say it is a real privilege to be able to work with the awesome folks in the 17 00:00:43.880 --> 00:00:50.170 Grants to States program and to share this data with you guys that many of you spend 18 00:00:50.170 --> 00:00:52.910 many hours entering into the system. 19 00:00:52.910 --> 00:00:58.820 So I am always very interested in hearing anything that you have to say or any questions 20 00:00:58.820 --> 00:01:00.920 that you might have on it. 21 00:01:00.920 --> 00:01:04.730 This is the fourth turn of the crank. 22 00:01:04.730 --> 00:01:10.730 So basically the way we started, we give you a little bit of an overview of what the SPR 23 00:01:10.730 --> 00:01:11.730 framework. 24 00:01:11.730 --> 00:01:16.800 And thank goodness, the talk that Madison and Michele just gave, gave you a bit of an 25 00:01:16.800 --> 00:01:17.910 introduction to that already. 26 00:01:17.910 --> 00:01:20.940 So I can fly through those slides wicked fast. 27 00:01:20.940 --> 00:01:25.950 And then I am going to talk a little bit about the SLAA Groupings to a high-level summary. 28 00:01:25.950 --> 00:01:32.110 And then go into the description profile which is basically what your basic journalistic 29 00:01:32.110 --> 00:01:33.110 type questions are. 30 00:01:33.110 --> 00:01:36.000 Who, what, when, how, why, and how much? 31 00:01:36.000 --> 00:01:40.580 So there are some optional fields. 32 00:01:40.580 --> 00:01:46.360 I am going to try to make sure that we keep track of some of the caveats. 33 00:01:46.360 --> 00:01:53.200 So one of the other things that we wanted to do was to give you a number of questions 34 00:01:53.200 --> 00:01:58.970 to be keeping in mind as I go through the slide deck. 35 00:01:58.970 --> 00:02:02.800 If you looked at it already, there are a lot of slides. 36 00:02:02.800 --> 00:02:03.800 There is a lot of data. 37 00:02:03.800 --> 00:02:09.650 So the idea is that a whole bunch of trees in this forest. 38 00:02:09.650 --> 00:02:16.130 Please help me understand what the whole forest really needs to look at. 39 00:02:16.130 --> 00:02:21.220 So what are the stories that are in these data that seem to really capture things best 40 00:02:21.220 --> 00:02:23.840 for you in your state or territory. 41 00:02:23.840 --> 00:02:27.780 Are there tools that would help us to make these data more useful to help you inform 42 00:02:27.780 --> 00:02:28.780 your outcomes? 43 00:02:28.780 --> 00:02:33.340 What support do you need from us with data? 44 00:02:33.340 --> 00:02:37.680 And then something we haven't done, and I was hoping to do it this year, was to do a 45 00:02:37.680 --> 00:02:44.890 little bit more data linking in terms of the system has been set up to permit us to do 46 00:02:44.890 --> 00:02:51.930 some linking with the public library survey data, common core data and IPEDS which is 47 00:02:51.930 --> 00:02:54.070 the higher education survey data. 48 00:02:54.070 --> 00:02:55.960 So we haven't done that yet. 49 00:02:55.960 --> 00:02:58.670 We would be interested in hearing what your thoughts are on that. 50 00:02:58.670 --> 00:03:03.380 So those are some questions to keep in mind as I race through the data slides. 51 00:03:03.380 --> 00:03:10.700 So the SPR, it has a number of very interesting analytical capabilities which thrills me to 52 00:03:10.700 --> 00:03:12.090 no end quite obviously. 53 00:03:12.090 --> 00:03:16.370 There are 14 intents, and these can be rolled up to six focal areas. 54 00:03:16.370 --> 00:03:24.920 But at the same time, we can take those intents and drill down into 38 different subjects. 55 00:03:24.920 --> 00:03:30.709 There is also, for no extra charge, three different levels of analysis. 56 00:03:30.709 --> 00:03:38.930 We can analyze the data at the activity level, the project level, and then for states and 57 00:03:38.930 --> 00:03:39.930 territories. 58 00:03:39.930 --> 00:03:43.599 So the intents, I am not going to go through all of these. 59 00:03:43.599 --> 00:03:47.099 I think that this was covered a little bit in the previous talks. 60 00:03:47.099 --> 00:03:48.239 There are 14 intents. 61 00:03:48.239 --> 00:03:52.790 The user chooses an intent. 62 00:03:52.790 --> 00:03:57.239 And then those intents get rolled up to these six focal areas. 63 00:03:57.239 --> 00:04:01.730 And then the top three of these focal areas which later in this presentation you will 64 00:04:01.730 --> 00:04:08.750 see me referred to as the big three really then align very closely with IMLS strategic 65 00:04:08.750 --> 00:04:12.459 goals for 2018 to 2022. 66 00:04:12.459 --> 00:04:18.979 And then there is what I sometimes call topical focal areas relate to human services, employment 67 00:04:18.979 --> 00:04:22.840 and economic development and civic engagement. 68 00:04:22.840 --> 00:04:24.740 And then there are subjects. 69 00:04:24.740 --> 00:04:30.389 So as I said, each of the intent is also related to subject. 70 00:04:30.389 --> 00:04:36.550 There are 37 subjects plus another, and once the user enters an intent, they can also choose 71 00:04:36.550 --> 00:04:37.610 a couple of subjects. 72 00:04:37.610 --> 00:04:40.289 These come in extraordinarily handy. 73 00:04:40.289 --> 00:04:48.879 And my colleague, Matt Birnbaum's talk tomorrow will have used some analysis based on our 74 00:04:48.879 --> 00:04:54.009 analysis on these subject, namely digital literacy and broadband adoption. 75 00:04:54.009 --> 00:05:00.949 In the past, we have done some work looking at summer reading programs, databases, and 76 00:05:00.949 --> 00:05:04.270 a little bit of work looking at some other topics as well. 77 00:05:04.270 --> 00:05:11.150 When we look at the SLAA groupings, we look at the general allotment levels. 78 00:05:11.150 --> 00:05:20.130 Now, a few years back in 2017 and 2018, and those two conference years, we had started 79 00:05:20.130 --> 00:05:24.819 saying, oh, well, let's do large and small by whether they do subgrants, you know, predominantly 80 00:05:24.819 --> 00:05:26.849 subgrants or predominantly SLAA awards. 81 00:05:26.849 --> 00:05:35.770 And we what we heard from the states and territories was you know something, some of us do subgrants 82 00:05:35.770 --> 00:05:40.069 one year, and then the next year, we go to predominantly SLAA. 83 00:05:40.069 --> 00:05:42.849 So that is not a persistent quality. 84 00:05:42.849 --> 00:05:48.689 So we did have a sense that we wanted to kind of bend things and to start to understand 85 00:05:48.689 --> 00:05:53.309 that there were some interesting differences or some differences across these different 86 00:05:53.309 --> 00:05:54.309 groups. 87 00:05:54.309 --> 00:05:58.199 And so what we ended up with was, well, let's look at the allotment levels. 88 00:05:58.199 --> 00:06:04.250 And you can see that we have these three allotment levels, smaller, larger, and what we call 89 00:06:04.250 --> 00:06:13.430 mega, not to be confused with mega Godzilla, just because those are so much larger than 90 00:06:13.430 --> 00:06:14.430 the larger. 91 00:06:14.430 --> 00:06:16.229 They really do differentiate. 92 00:06:16.229 --> 00:06:20.740 And you can see down below, I have got the 2018 population there. 93 00:06:20.740 --> 00:06:29.279 And on the 2018 population, in the 28 states and the five territories outlying areas, they 94 00:06:29.279 --> 00:06:32.210 had a 2018 population of less than five million. 95 00:06:32.210 --> 00:06:35.379 Those that were in the larger, five to 13 million. 96 00:06:35.379 --> 00:06:42.249 And then these four megas, California, Florida, New York, and Texas, all had very large populations. 97 00:06:42.249 --> 00:06:45.809 The allotment sizes are quite a bit different. 98 00:06:45.809 --> 00:06:52.009 And then as you can see, the average number of project activities for each project is 99 00:06:52.009 --> 00:06:53.009 a little bit different. 100 00:06:53.009 --> 00:06:57.279 They are about the same for the larger and the smaller states, but when you get into 101 00:06:57.279 --> 00:07:03.529 the mega states, they have slightly more activities, another half of activity for project than 102 00:07:03.529 --> 00:07:04.529 the others. 103 00:07:04.529 --> 00:07:08.710 So now, as I mentioned, this is going to be our description profile. 104 00:07:08.710 --> 00:07:16.650 We are going to go into how much, why, who the partners are, what kinds of activities 105 00:07:16.650 --> 00:07:19.800 there are, the types of activities, where. 106 00:07:19.800 --> 00:07:23.249 And then a little bit about the subject code. 107 00:07:23.249 --> 00:07:26.919 So the optional fields are partners and locales. 108 00:07:26.919 --> 00:07:30.569 Not all projects report those. 109 00:07:30.569 --> 00:07:32.389 Not all activities report those. 110 00:07:32.389 --> 00:07:37.869 And then as I mentioned before, we are doing a focus this year on broadband and digital 111 00:07:37.869 --> 00:07:39.509 literacy. 112 00:07:39.509 --> 00:07:43.819 So this is what we tend to show as our sort of high-level overview. 113 00:07:43.819 --> 00:07:47.800 I am not going to spend a lot of time on this because I end up having graph after graph 114 00:07:47.800 --> 00:07:51.050 after graph showing a little bit more detail. 115 00:07:51.050 --> 00:07:55.330 But I am going to point out just a couple of things given I can remember how to point 116 00:07:55.330 --> 00:07:58.339 things out on here. 117 00:07:58.339 --> 00:08:08.789 So you can see that what we have seen is that over time, FY15 was a pilot year, and then 118 00:08:08.789 --> 00:08:10.329 14, everybody entered. 119 00:08:10.329 --> 00:08:14.409 But over time, the numbers of projects have gone down. 120 00:08:14.409 --> 00:08:16.689 And as well, the number of activities. 121 00:08:16.689 --> 00:08:23.580 And that is really attributable, to a large extent, sort of learning as people have learned 122 00:08:23.580 --> 00:08:26.580 how to really work with the system, how to enter it. 123 00:08:26.580 --> 00:08:31.669 The excellent advice that Michele and Madison gave you earlier, those kinds of things, people 124 00:08:31.669 --> 00:08:33.950 are gaining a knowledge base on that. 125 00:08:33.950 --> 00:08:40.220 Now, 70 percent of projects had about one or two activities. 126 00:08:40.220 --> 00:08:44.510 So this one, you can't read on the screen and don’t even try. 127 00:08:44.510 --> 00:08:46.430 It is in the deck. 128 00:08:46.430 --> 00:08:52.120 You can then have access to the data, and the next couple of visualizations will show 129 00:08:52.120 --> 00:08:54.390 you some of the trends here. 130 00:08:54.390 --> 00:09:00.000 One of the things, even though you can't see it, what we are seeing is that fewer projects 131 00:09:00.000 --> 00:09:03.610 are specifying more than one intent. 132 00:09:03.610 --> 00:09:10.500 And that has been a real big gain in terms of learning about how to use the system. 133 00:09:10.500 --> 00:09:15.210 So this is a chart that needs a little bit of explanation. 134 00:09:15.210 --> 00:09:22.350 And what you are going to see is that this dark bar is the LSTA funds, median project 135 00:09:22.350 --> 00:09:23.720 budget. 136 00:09:23.720 --> 00:09:31.330 And then the light bit on top is the other funds, but includes your state match, your 137 00:09:31.330 --> 00:09:32.670 in-kind match. 138 00:09:32.670 --> 00:09:41.720 This number up at the top represents the total median budget for the project for that fiscal 139 00:09:41.720 --> 00:09:42.720 year. 140 00:09:42.720 --> 00:09:48.700 And then this percentage down here, this shows you what percentage of that total budget is 141 00:09:48.700 --> 00:09:50.480 from LSTA funds. 142 00:09:50.480 --> 00:09:56.480 So there is a lot going on in this particular chart. 143 00:09:56.480 --> 00:10:03.420 And you can see that the big thing is that we get this kind of interesting finding. 144 00:10:03.420 --> 00:10:06.940 And I am obviously not quite sure why this is. 145 00:10:06.940 --> 00:10:12.871 But the states that were in the 19 large states that were kind of in the middle there, their 146 00:10:12.871 --> 00:10:17.520 projects tend to have lower budgets. 147 00:10:17.520 --> 00:10:24.360 And now, in 2018, they have a proportionally higher spend from LSTA funds. 148 00:10:24.360 --> 00:10:31.840 68 percent as compared to the 62 percent for the megas and the 58 percent for the small 149 00:10:31.840 --> 00:10:35.550 states, the states with the smaller allotment. 150 00:10:35.550 --> 00:10:40.270 The median project budget is higher in the small states. 151 00:10:40.270 --> 00:10:42.510 So it has been kind of interesting. 152 00:10:42.510 --> 00:10:49.490 The percent overall, just to give you the benchmark here, about 68 percent is from LSTA, 153 00:10:49.490 --> 00:10:54.330 but 66 percent in 2018. 154 00:10:54.330 --> 00:10:59.720 So we get that kind of an interesting finding there. 155 00:10:59.720 --> 00:11:03.980 And I would entertain what ideas people have about why this might be the case. 156 00:11:03.980 --> 00:11:10.120 The other way that we have looked at budgets, and our original thought on this back when 157 00:11:10.120 --> 00:11:15.690 we started doing this analysis was, well, maybe there are certain types of projects 158 00:11:15.690 --> 00:11:20.620 that we might be able to identify that states could do quote unquote on a shoestring. 159 00:11:20.620 --> 00:11:27.330 So hey, what are some projects that might be under say $25,000 or under $7,500, like 160 00:11:27.330 --> 00:11:30.860 some very inexpensive projects. 161 00:11:30.860 --> 00:11:37.220 Because the notion in the system, my understanding from the get-go, was that this would provide 162 00:11:37.220 --> 00:11:41.540 a platform by which people could share information about projects. 163 00:11:41.540 --> 00:11:45.970 So we have been binning them by budget category. 164 00:11:45.970 --> 00:11:52.320 And as you can see, what we have got here is that, again, looking at the allotment size 165 00:11:52.320 --> 00:12:01.600 group that consistent with the last chart, the large allotment size group tend to have 166 00:12:01.600 --> 00:12:03.720 smaller budgets on their projects. 167 00:12:03.720 --> 00:12:07.000 They undo the other two. 168 00:12:07.000 --> 00:12:12.880 And the 49 percent of their projects have a total budget of less than $25,0000. 169 00:12:12.880 --> 00:12:18.620 So now, we are going to switch gears. 170 00:12:18.620 --> 00:12:25.880 We are going to talk a little bit about the activities. 171 00:12:25.880 --> 00:12:27.370 So up to now, it was about projects. 172 00:12:27.370 --> 00:12:30.200 Now it is about the activities per project. 173 00:12:30.200 --> 00:12:36.610 And this is really where you see another big difference between the allotment size groups. 174 00:12:36.610 --> 00:12:40.730 And you sort of saw that in that last table where you really couldn't see the numbers 175 00:12:40.730 --> 00:12:41.730 anyway. 176 00:12:41.730 --> 00:12:43.620 But you can see them a little bit better here. 177 00:12:43.620 --> 00:12:51.520 And that is that for the mega group, you can see that 57 percent have one or two activities 178 00:12:51.520 --> 00:12:58.150 per project which is quite a bit less than the small and the large. 179 00:12:58.150 --> 00:13:01.440 And that one-third of them have three to five. 180 00:13:01.440 --> 00:13:04.440 And that 10 percent of them have six or more. 181 00:13:04.440 --> 00:13:10.230 And having six or more is really a characteristic of the very large state. 182 00:13:10.230 --> 00:13:15.900 You don’t see it too much in the small states and in the large states. 183 00:13:15.900 --> 00:13:27.680 So we are done talking about the different size allotment groups and how that affects 184 00:13:27.680 --> 00:13:28.680 things. 185 00:13:28.680 --> 00:13:34.290 Now, we are going to spend a lot more time digging into things like focal areas, activities, 186 00:13:34.290 --> 00:13:36.910 who, how things get done, things like that. 187 00:13:36.910 --> 00:13:45.630 So as I said before, what we have got is two sets of focal areas that -- there are six 188 00:13:45.630 --> 00:13:49.290 focal areas that roll up those 14 intents. 189 00:13:49.290 --> 00:13:53.430 And you can see we have got the big three, information access, institutional capacity, 190 00:13:53.430 --> 00:13:55.060 and lifelong learning. 191 00:13:55.060 --> 00:14:01.090 And those tend to really dominate the numbers of projects that are there. 192 00:14:01.090 --> 00:14:11.960 The big drop that we see is that in FY18 versus FY15, there is a decline by 19 percent in 193 00:14:11.960 --> 00:14:13.780 the number of information access projects. 194 00:14:13.780 --> 00:14:22.430 Kind of hard to see much in the trend down here in the sort of topical categories, human 195 00:14:22.430 --> 00:14:25.510 services, civic engagement and economic development. 196 00:14:25.510 --> 00:14:34.440 But what we see is that because they are so small, they really have not changed in terms 197 00:14:34.440 --> 00:14:39.200 of relative size very much over this time. 198 00:14:39.200 --> 00:14:46.990 So now, we say, okay, well, how many states or territories have projects in each of the 199 00:14:46.990 --> 00:14:48.350 focal areas. 200 00:14:48.350 --> 00:14:56.510 And as you can see, almost everybody has something in institutional capacity, nearly everybody 201 00:14:56.510 --> 00:15:00.500 has something in information access and lifelong learning in the big three. 202 00:15:00.500 --> 00:15:08.640 But then, we have quite a bit fewer states who have projects in the topical areas, in 203 00:15:08.640 --> 00:15:12.040 civic engagement, human services and economic development. 204 00:15:12.040 --> 00:15:18.120 And really, where we have seen the gain over the past couple of years is in civic engagement, 205 00:15:18.120 --> 00:15:23.730 and we have seen a decline in the numbers of projects in human services. 206 00:15:23.730 --> 00:15:31.560 That said, what is kind of interesting is that the median LSTA budget by those focal 207 00:15:31.560 --> 00:15:34.730 area is very different. 208 00:15:34.730 --> 00:15:39.750 So if you recall, I said, well, human services, we have seen a decline in the number of projects. 209 00:15:39.750 --> 00:15:47.180 But we have actually seen the LSTA budget for those, is higher than that in civic engagement 210 00:15:47.180 --> 00:15:48.750 and economic development. 211 00:15:48.750 --> 00:15:57.910 And the highest LSTA spent is in the area of information access, and that is where a 212 00:15:57.910 --> 00:16:03.930 lot of database projects are going to pop up. 213 00:16:03.930 --> 00:16:12.420 So in this chart, this shows you, very similar to the last chart, although I didn’t put 214 00:16:12.420 --> 00:16:14.529 the totals up at the top. 215 00:16:14.529 --> 00:16:23.360 Again, you can get a sense of the relative budget of each of the different focus areas 216 00:16:23.360 --> 00:16:25.520 projects. 217 00:16:25.520 --> 00:16:28.310 And these are the big three. 218 00:16:28.310 --> 00:16:36.690 These here are the big three, information access, institutional capacity and lifelong 219 00:16:36.690 --> 00:16:37.690 learning. 220 00:16:37.690 --> 00:16:43.150 But you can see that they vary greatly in terms of the magnitude of their budget, their 221 00:16:43.150 --> 00:16:44.150 total budget. 222 00:16:44.150 --> 00:16:49.410 Just like the other chart, the dark part shows you the LSTA funds, and the light part shows 223 00:16:49.410 --> 00:16:50.720 you the non-LSTA funds. 224 00:16:50.720 --> 00:16:57.140 And then this percent gives you what percentage of the total budget is from LSTA. 225 00:16:57.140 --> 00:17:03.270 And so even though these are more expensive, they use proportionately less LSTA funds than 226 00:17:03.270 --> 00:17:05.240 the lifelong learning projects. 227 00:17:05.240 --> 00:17:14.029 The same thing we see in human services where about half of the budget is from LSTA funds, 228 00:17:14.029 --> 00:17:20.350 but they are much more expensive projects than are the projects in civic engagement 229 00:17:20.350 --> 00:17:22.760 and economic development. 230 00:17:22.760 --> 00:17:28.559 Economic development are the projects that have the highest percentage of budget from 231 00:17:28.559 --> 00:17:34.870 LSTA funds. 232 00:17:34.870 --> 00:17:37.110 So we have been talking about projects. 233 00:17:37.110 --> 00:17:41.150 Now, we are drilling one step down into the activities. 234 00:17:41.150 --> 00:17:51.899 And if you recall, for many states, 70 percent of the projects have one or two activities. 235 00:17:51.899 --> 00:17:54.720 It is just a handful that have more than that. 236 00:17:54.720 --> 00:18:00.499 So when we talk about activities, we can say, well, what types of activities? 237 00:18:00.499 --> 00:18:04.149 And there are four types of activities in the SPR. 238 00:18:04.149 --> 00:18:08.970 There is content, instruction, planning and evaluation, and procurement. 239 00:18:08.970 --> 00:18:12.009 And this is quite literally how they stack up. 240 00:18:12.009 --> 00:18:18.899 And the most common thing that you see here, the most prevalent thing, is the fact that 241 00:18:18.899 --> 00:18:26.249 the relative mix has not really changed markedly over the past four years in terms of what 242 00:18:26.249 --> 00:18:32.299 kinds of projects or what kinds of activities are engaged in. 243 00:18:32.299 --> 00:18:40.669 The one big difference that we do see is that in FY16, a little bit more slightly higher 244 00:18:40.669 --> 00:18:46.260 percentage and larger number of activities were related to evaluation. 245 00:18:46.260 --> 00:18:57.499 And that of course is when the evaluations for the five-year FY17 year-end projects were 246 00:18:57.499 --> 00:19:01.530 coming due. 247 00:19:01.530 --> 00:19:02.530 So why? 248 00:19:02.530 --> 00:19:10.610 So now, we can do those same stacks, so the types of activities by focal area for the 249 00:19:10.610 --> 00:19:11.610 activity. 250 00:19:11.610 --> 00:19:14.749 And so again, we have got the big three. 251 00:19:14.749 --> 00:19:21.690 And you can see some real dramatic differences in terms of how they allocate across the different 252 00:19:21.690 --> 00:19:23.749 types of activities. 253 00:19:23.749 --> 00:19:31.870 That institutional capacity has a lot more to do with content or with instruction. 254 00:19:31.870 --> 00:19:35.090 And that information access has to do with content. 255 00:19:35.090 --> 00:19:38.009 Getting more information, that is where those databases are showing up. 256 00:19:38.009 --> 00:19:43.279 And that lifelong learning has quite a lot to do with instruction. 257 00:19:43.279 --> 00:19:50.950 And then here to, you can see that light blue for instruction is prevalent in the topical 258 00:19:50.950 --> 00:19:53.000 areas as well. 259 00:19:53.000 --> 00:20:03.830 So now, if we drill into, just look at the FY18 data, and these are another way to look 260 00:20:03.830 --> 00:20:05.850 at those stacks. 261 00:20:05.850 --> 00:20:10.010 And hopefully, your eyes are not glazing over yet. 262 00:20:10.010 --> 00:20:16.700 It is kind of interesting because this gives you a sense of just how different the focal 263 00:20:16.700 --> 00:20:22.139 areas are in terms of the types of activities that end up sort of rolling out. 264 00:20:22.139 --> 00:20:27.710 And here, I have included the procurement which procurement is only allowed under institutional 265 00:20:27.710 --> 00:20:29.480 capacity. 266 00:20:29.480 --> 00:20:37.760 But you can see that it accounts for about 10 percent of the institutional capacity project. 267 00:20:37.760 --> 00:20:38.760 And those are the big three. 268 00:20:38.760 --> 00:20:44.539 If we look at the smaller ones, civic engagement, human services, and economic development, 269 00:20:44.539 --> 00:20:50.169 we can see that again they are overwhelming instructions. 270 00:20:50.169 --> 00:20:59.720 But you can see quite a lot of content associated with the economic development. 271 00:20:59.720 --> 00:21:07.660 So activity locale, in the system it asks is this a statewide or not a statewide project 272 00:21:07.660 --> 00:21:08.820 or activity. 273 00:21:08.820 --> 00:21:17.039 And so what we have seen over time is that the relative percentage of statewide projects 274 00:21:17.039 --> 00:21:23.919 has increased, even though as we see, the numbers of activities that are in the system 275 00:21:23.919 --> 00:21:24.919 have actually decreased. 276 00:21:24.919 --> 00:21:36.700 So now, as you can see, 47 percent of the activities in the STR are statewide. 277 00:21:36.700 --> 00:21:40.890 So locales are one of the optional items. 278 00:21:40.890 --> 00:21:52.119 And so this shows you what types of locales, where the activities are implemented and gives 279 00:21:52.119 --> 00:22:00.789 you the public libraries lead the way in terms of the 42 percent of FY18 activities had as 280 00:22:00.789 --> 00:22:03.990 at least one of its locales, the public library. 281 00:22:03.990 --> 00:22:11.559 And I must note that this is 42 percent of those is set by locales. 282 00:22:11.559 --> 00:22:15.100 They could specify more than one public library. 283 00:22:15.100 --> 00:22:17.419 So this isn't the whole total number of public libraries. 284 00:22:17.419 --> 00:22:22.279 This is the number of activities that have specified that a public library was one of 285 00:22:22.279 --> 00:22:23.279 its locales. 286 00:22:23.279 --> 00:22:31.279 We are seeing a bit increase in terms of the percentage that have specified the SLAA as 287 00:22:31.279 --> 00:22:32.279 a locale. 288 00:22:32.279 --> 00:22:38.019 And that is not a surprise given this previous chart that was showing that there were more 289 00:22:38.019 --> 00:22:39.119 statewide activities. 290 00:22:39.119 --> 00:22:43.480 And the SLAA becomes one of the important locations for that. 291 00:22:43.480 --> 00:22:47.909 And then the proportionate activities as an academic, a school or a special library has 292 00:22:47.909 --> 00:22:54.850 increased by about 3 percent points between FY15 and FY18. 293 00:22:54.850 --> 00:23:04.940 So partner areas are an optional item. 294 00:23:04.940 --> 00:23:10.470 About 56 percent of activities in FY18 reported them. 295 00:23:10.470 --> 00:23:13.301 That is a little bit higher percentage than reported them in FY15. 296 00:23:13.301 --> 00:23:23.450 And FY15, just 52 percent of the activities had a partner area that was reported for it. 297 00:23:23.450 --> 00:23:31.630 And you can see that local states, governments tend to be predominant when a partner area 298 00:23:31.630 --> 00:23:37.360 is specified, non-profit has stayed fairly constant. 299 00:23:37.360 --> 00:23:43.590 I know it is hard to tell because the bars are showing you the numbers of activities 300 00:23:43.590 --> 00:23:49.480 and then the percentages are showing you what percent of the activities that specified it. 301 00:23:49.480 --> 00:23:55.419 And as another point here, like the other chart, an activity can specify more than one 302 00:23:55.419 --> 00:24:00.210 partner area. 303 00:24:00.210 --> 00:24:08.429 And then partner types, so not all of them who specified a partner area specified a partner 304 00:24:08.429 --> 00:24:09.499 type. 305 00:24:09.499 --> 00:24:13.149 Just 38 percent specified a partner type. 306 00:24:13.149 --> 00:24:17.940 But when they did, they overwhelmingly specified libraries as a partner type. 307 00:24:17.940 --> 00:24:23.519 And then a very small percentage did specify museums, cultural heritage organizations and 308 00:24:23.519 --> 00:24:26.750 historical societies as partners. 309 00:24:26.750 --> 00:24:28.299 MS. 310 00:24:28.299 --> 00:24:31.440 GONSALVES: A question popped up. 311 00:24:31.440 --> 00:24:37.909 The question is, would tribal libraries be considered other in most states? 312 00:24:37.909 --> 00:24:38.950 MS. 313 00:24:38.950 --> 00:24:43.110 FREHILL: Actually, that is a very good question. 314 00:24:43.110 --> 00:24:53.360 I would probably want to see what folks would have said in the states because in a sense, 315 00:24:53.360 --> 00:25:00.820 you would have to cross the partner area, which is on this chart, because you can have 316 00:25:00.820 --> 00:25:08.919 a partner area as Native American, American Indian, Native Hawaiian organization with 317 00:25:08.919 --> 00:25:10.850 the partner type. 318 00:25:10.850 --> 00:25:15.169 And I think that that could tell you whether it is a tribal library. 319 00:25:15.169 --> 00:25:20.009 So that is a really good question and that sounds like something that I could have about 320 00:25:20.009 --> 00:25:21.759 20 minutes of fun with sometime. 321 00:25:21.759 --> 00:25:26.779 So I hope somebody writes that one down so I don’t forget it. 322 00:25:26.779 --> 00:25:30.259 But thank you, that is a really good one. 323 00:25:30.259 --> 00:25:34.619 Because sometimes, you could have the same thing like helpful heritage organizations. 324 00:25:34.619 --> 00:25:42.009 Could be tribal organizations as well. 325 00:25:42.009 --> 00:25:47.720 So who are the beneficiaries of the activity? 326 00:25:47.720 --> 00:25:53.809 And you can have the public, and within public, there could be general or targeted. 327 00:25:53.809 --> 00:26:03.739 If there was specific like group that is the focus, and then the library, I say library 328 00:26:03.739 --> 00:26:06.190 workforce, but really this is library more generally. 329 00:26:06.190 --> 00:26:11.850 And you will see that the moment the moment when I get to the next slide. 330 00:26:11.850 --> 00:26:19.700 And so what we see is that over time, there has only been a slight decrease in the percentage 331 00:26:19.700 --> 00:26:24.720 of the activities that are for targeted public audiences. 332 00:26:24.720 --> 00:26:33.600 And a slight increase in the relative proportion of activities that are focused on the library 333 00:26:33.600 --> 00:26:35.779 of an audience. 334 00:26:35.779 --> 00:26:36.779 MS. 335 00:26:36.779 --> 00:26:38.440 GONSALVES: One quick question. 336 00:26:38.440 --> 00:26:42.240 Someone wants to know what actually defines other. 337 00:26:42.240 --> 00:26:43.240 MS. 338 00:26:43.240 --> 00:26:45.830 FREHILL: That is a very good question. 339 00:26:45.830 --> 00:26:54.110 I was actually in the SPR book earlier today, trying to track that down myself. 340 00:26:54.110 --> 00:27:01.710 So I will ask if Matt has some probe of wisdom on that one. 341 00:27:01.710 --> 00:27:02.710 MR. 342 00:27:02.710 --> 00:27:09.210 BIRNBAUM: I don’t have the SPR in front of me, but I am 90 percent certain it is a 343 00:27:09.210 --> 00:27:10.230 type of content activity. 344 00:27:10.230 --> 00:27:16.220 It is the mode under content. 345 00:27:16.220 --> 00:27:17.909 MS. 346 00:27:17.909 --> 00:27:21.289 FREHILL: Okay. 347 00:27:21.289 --> 00:27:29.210 But I think is the question you are asking about this chart, though, where it is like 348 00:27:29.210 --> 00:27:30.999 the partner type says other. 349 00:27:30.999 --> 00:27:35.049 So actually, I think people can fill in other -- 350 00:27:35.049 --> 00:27:36.049 MR. 351 00:27:36.049 --> 00:27:37.049 BIRNBAUM: Yes. 352 00:27:37.049 --> 00:27:42.861 This would be partner was not one of the other categories. 353 00:27:42.861 --> 00:27:43.861 MS. 354 00:27:43.861 --> 00:27:48.570 FREHILL: I saw some other things listed as I was looking at the verbal entries there. 355 00:27:48.570 --> 00:27:51.119 And I could make a list. 356 00:27:51.119 --> 00:27:56.769 But it is a variety of things that don’t fit any of these categories. 357 00:27:56.769 --> 00:28:11.130 I will see if I can circle back with you, Joy, on that. 358 00:28:11.130 --> 00:28:16.720 So a moment ago, I mentioned that like it is really not just the library workforce, 359 00:28:16.720 --> 00:28:19.210 but library in general. 360 00:28:19.210 --> 00:28:28.659 And in terms of you look at the activities, about 40 percent in FY18 focus on the library. 361 00:28:28.659 --> 00:28:34.190 And of these, we break this down by the focal areas. 362 00:28:34.190 --> 00:28:39.200 And then within the institutional capacity focal area, which is where most of them are, 363 00:28:39.200 --> 00:28:44.110 almost two-thirds, we further drill down into the intent. 364 00:28:44.110 --> 00:28:50.989 So this is one of the only places within my presentation where I drilled down to the intent 365 00:28:50.989 --> 00:28:51.989 at all. 366 00:28:51.989 --> 00:29:00.059 And you can see that when we look at the 1,182 activities that were focused on the library, 367 00:29:00.059 --> 00:29:05.610 about 69 percent of the 63 percent were specific to the library workforce. 368 00:29:05.610 --> 00:29:11.960 But that quite a number of them really relate to improving library operations, accessing 369 00:29:11.960 --> 00:29:15.530 information, lifelong learning are the topical areas. 370 00:29:15.530 --> 00:29:25.190 So it is important to bear in mind that some of these activities have a lot to do with 371 00:29:25.190 --> 00:29:33.279 providing the support to library staff to be able to implement programming related to 372 00:29:33.279 --> 00:29:42.889 topics or to provide access for patrons to lifelong learning or other information sources. 373 00:29:42.889 --> 00:29:54.549 So if we look at the types of library workforce activities, almost two-thirds of the ones 374 00:29:54.549 --> 00:29:58.960 that are instructions have to do with institutional capacity. 375 00:29:58.960 --> 00:30:04.700 And in this case, the “all others” includes the other focal areas, namely lifelong learning 376 00:30:04.700 --> 00:30:06.789 and the three topical areas. 377 00:30:06.789 --> 00:30:11.840 So you can kind of see the breakdown here of how the institutional capacity continues 378 00:30:11.840 --> 00:30:18.230 to be an important focal area. 379 00:30:18.230 --> 00:30:26.289 And instruction is one of the primary ways in which the activities are implemented. 380 00:30:26.289 --> 00:30:31.220 So when we look at public activities, so now, the public activities account for about 60 381 00:30:31.220 --> 00:30:36.419 percent of the activities that are reported in the SPR. 382 00:30:36.419 --> 00:30:44.639 And as I mentioned earlier, the relative percentage of those that were for targeted groups has 383 00:30:44.639 --> 00:30:49.510 declined slightly, and those for general groups has increased. 384 00:30:49.510 --> 00:30:56.090 But what we can see is that the way in which they are distributed across the different 385 00:30:56.090 --> 00:30:57.559 focal areas really varies. 386 00:30:57.559 --> 00:31:05.849 So that 65 percent of the lifelong learning activities for the general public are targeted 387 00:31:05.849 --> 00:31:12.229 at a specific group, and that is much higher than say the institutional capacity or the 388 00:31:12.229 --> 00:31:16.820 information access funds. 389 00:31:16.820 --> 00:31:25.340 And I see a question, the difference between content and procurement. 390 00:31:25.340 --> 00:31:35.369 So content is a number of ways in which content actually -- it happens. 391 00:31:35.369 --> 00:31:42.599 So procurement is actually purchasing things, whereas content could be like digitizing things. 392 00:31:42.599 --> 00:31:48.760 It could be acquiring like curriculum materials and things like that. 393 00:31:48.760 --> 00:31:53.989 So I think that there is a very subtle difference there. 394 00:31:53.989 --> 00:31:56.470 Procurement is only for institutional capacity. 395 00:31:56.470 --> 00:32:03.769 So if you obtain content that is used for a program for the public, and it is for something 396 00:32:03.769 --> 00:32:16.859 other than institutional capacity, it gets lumped into the content category as an acquisition. 397 00:32:16.859 --> 00:32:28.500 So if we go back, 60 percent are public activities, and what we have is, okay, so among those, 398 00:32:28.500 --> 00:32:30.950 817 of them are targeted. 399 00:32:30.950 --> 00:32:37.649 So 817 represent 28 percent of all activities, 47 percent of the activities for the public 400 00:32:37.649 --> 00:32:39.179 are targeted. 401 00:32:39.179 --> 00:32:44.269 What types of targets are the focus of these? 402 00:32:44.269 --> 00:32:48.559 And age groups is really the number one thing. 403 00:32:48.559 --> 00:33:01.369 And what this is telling you is that 37 percent of the 1,754 public activities are targeted 404 00:33:01.369 --> 00:33:05.779 at some particular age group. 405 00:33:05.779 --> 00:33:10.289 And that is more than one-third of them. 406 00:33:10.289 --> 00:33:17.970 Then if you look down, you can see that the activities that are targeted at say immigrant 407 00:33:17.970 --> 00:33:25.489 groups or specific ethnic groups, are relatively small in terms of the numbers. 408 00:33:25.489 --> 00:33:30.519 So if we drill into the age groups that are targeted by public activities, what we see 409 00:33:30.519 --> 00:33:44.049 is that the youngest age group, 0 to 5, preschool age 410 00:33:44.049 --> 00:33:49.029 group, has actually had the biggest change since FY15. 411 00:33:49.029 --> 00:33:58.570 We have seen an expansion there in terms of the percentage of activities that target early 412 00:33:58.570 --> 00:34:00.230 learners. 413 00:34:00.230 --> 00:34:13.870 As you can see, the general distribution is about the same in FY18 versus FY15. 414 00:34:13.870 --> 00:34:18.409 And then if we look at ethnic groups that are targeted by public activities, what we 415 00:34:18.409 --> 00:34:25.110 see is that Hispanics are the largest group that is targeted. 416 00:34:25.110 --> 00:34:31.300 But that over time, what we have seen is proportionally fewer activities are targeted at various ethnic 417 00:34:31.300 --> 00:34:32.300 groups. 418 00:34:32.300 --> 00:34:37.100 But again, the distribution is about the same in both FY15 and FY18. 419 00:34:37.100 --> 00:34:46.420 I am going to pause there real quick because I have been seeing some of the questions come 420 00:34:46.420 --> 00:34:58.010 up and I am not sure if I have got them all answered. 421 00:34:58.010 --> 00:34:59.730 MS. 422 00:34:59.730 --> 00:35:05.120 GONSALVES: Terry wants to know, didn’t the SPR change in 2016? 423 00:35:05.120 --> 00:35:12.100 Was this included in the assessment of the number of projects, et cetera? 424 00:35:12.100 --> 00:35:13.350 MS. 425 00:35:13.350 --> 00:35:20.450 FREHILL: That is a good question because I am not certain. 426 00:35:20.450 --> 00:35:28.740 Matt, what material change for FY16 would there have been that Terry is referencing 427 00:35:28.740 --> 00:35:29.740 here. 428 00:35:29.740 --> 00:35:30.740 MR. 429 00:35:30.740 --> 00:35:31.740 BIRNBAUM: Yes. 430 00:35:31.740 --> 00:35:32.740 Sure. 431 00:35:32.740 --> 00:35:33.740 That is easy. 432 00:35:33.740 --> 00:35:39.210 The 2014, we started piloting the new SPR with about 15 states. 433 00:35:39.210 --> 00:35:43.230 In 2016, all the states joined the new system. 434 00:35:43.230 --> 00:35:45.270 MS. 435 00:35:45.270 --> 00:35:54.980 FREHILL: So what you were talking about, Terry, was the year 2016 rather than the reporting 436 00:35:54.980 --> 00:35:57.130 of the SPR, reporting year. 437 00:35:57.130 --> 00:35:58.130 Okay. 438 00:35:58.130 --> 00:36:01.690 PARTICIPANT: I believe all the states began to report in FY16. 439 00:36:01.690 --> 00:36:02.690 MS. 440 00:36:02.690 --> 00:36:07.660 FREHILL: Well, they all reported their FY15 data. 441 00:36:07.660 --> 00:36:13.710 It is just that they would have done that in 2016, right? 442 00:36:13.710 --> 00:36:19.410 Keeping our like FYs and calendar years, reporting years squared away. 443 00:36:19.410 --> 00:36:26.630 Any other questions before I like jump into output because this is the fun thing that 444 00:36:26.630 --> 00:36:30.750 I messed with this year that is a little different. 445 00:36:30.750 --> 00:36:32.270 MS. 446 00:36:32.270 --> 00:36:37.710 GONSALVES: I do not see any additional questions. 447 00:36:37.710 --> 00:36:38.710 MS. 448 00:36:38.710 --> 00:36:42.300 FREHILL: I am looking at the Q&A right now and trying to keep trac of that. 449 00:36:42.300 --> 00:36:43.810 The thing I can't see is the clock. 450 00:36:43.810 --> 00:36:44.810 I am sorry. 451 00:36:44.810 --> 00:36:46.410 I just can't see the green thing right now. 452 00:36:46.410 --> 00:36:52.620 I am just going to leap on forward into activity output. 453 00:36:52.620 --> 00:37:01.090 So there is about 40 some odd different output that are available in the SPR. 454 00:37:01.090 --> 00:37:09.580 And these outputs, those of you who have reported them, you see how this rolls out in terms 455 00:37:09.580 --> 00:37:16.730 of, okay, so once you have indicated the activity and said what type it is, what the mode was, 456 00:37:16.730 --> 00:37:22.670 what the format was, all of that kind of stuff, then you have a block where you enter the 457 00:37:22.670 --> 00:37:23.670 activities. 458 00:37:23.670 --> 00:37:29.440 And in fact, in Madison and Michele's presentations, they had a screenshot that showed you that 459 00:37:29.440 --> 00:37:35.260 which I think would be really helpful right about now as a visual. 460 00:37:35.260 --> 00:37:45.370 But what ends up happening then is that you have 40-some odd activities or activity outputs. 461 00:37:45.370 --> 00:37:52.520 So to start doing a little bit of analysis on those outputs that goes beyond just saying 462 00:37:52.520 --> 00:37:58.970 how many of different things came out of this which is what we kind of did for two years 463 00:37:58.970 --> 00:38:01.750 in 2017 and 2018. 464 00:38:01.750 --> 00:38:11.050 This year, I drilled into two types of outputs that seemed like you could sort of well define 465 00:38:11.050 --> 00:38:12.210 them. 466 00:38:12.210 --> 00:38:17.020 Program evaluations, and part of the reason there is that is something that is really 467 00:38:17.020 --> 00:38:21.680 important to do in all of the projects and all of the activities. 468 00:38:21.680 --> 00:38:23.411 And then databases. 469 00:38:23.411 --> 00:38:26.440 The reason is that databases come up all the time. 470 00:38:26.440 --> 00:38:31.730 And in fact, Jaime Bell, I know you are on, has put a bug in my ear that it would be really 471 00:38:31.730 --> 00:38:38.000 great to do what we can to get a handle on, on how much all the databases cost. 472 00:38:38.000 --> 00:38:43.180 So what I did was program evaluations, like I said, it is fairly well defined. 473 00:38:43.180 --> 00:38:48.290 Those outputs are all very well connected to a program evaluation activity. 474 00:38:48.290 --> 00:38:52.020 Databases, on the other hand, I had to do some limiting. 475 00:38:52.020 --> 00:38:56.560 So I am not looking at all the databases that got reported as outputs. 476 00:38:56.560 --> 00:39:01.740 I am only looking at the databases where it was an activity where only databases were 477 00:39:01.740 --> 00:39:05.120 required, and that this was only one activity. 478 00:39:05.120 --> 00:39:09.450 So that way, we were just saying, okay, we are going to really limit ourselves. 479 00:39:09.450 --> 00:39:16.800 And then the metrics that we assembled were how many states reported this, how many activities, 480 00:39:16.800 --> 00:39:21.660 and then how many of them did we get, how many evaluations, how many databases. 481 00:39:21.660 --> 00:39:24.600 What were the costs, the total and the LSTA budgets? 482 00:39:24.600 --> 00:39:33.020 And then we have got sort of what did each thing cost that we got? 483 00:39:33.020 --> 00:39:38.410 So this is a first time of doing this kind of analysis. 484 00:39:38.410 --> 00:39:44.680 And so no visualizations, just two tables of numbers. 485 00:39:44.680 --> 00:39:51.960 And the data are not adjusted for inflation as any of the other financial data. 486 00:39:51.960 --> 00:39:55.670 And basically, I just wanted to kind of point out a couple of things. 487 00:39:55.670 --> 00:40:05.020 And that is that evaluations were reported by 30 to 38 states over the period 2015 to 488 00:40:05.020 --> 00:40:06.020 2018. 489 00:40:06.020 --> 00:40:13.460 A number more in 2016 obviously is the five-year plans were being evaluated. 490 00:40:13.460 --> 00:40:15.300 And there were more activities obviously in 2016. 491 00:40:15.300 --> 00:40:23.790 So 2016, we are going to see kind of a hot bed evaluation activity going on. 492 00:40:23.790 --> 00:40:29.060 And so then we can look at the total number of evaluations. 493 00:40:29.060 --> 00:40:36.650 So this is the number of evaluations that were reported as having been completed in 494 00:40:36.650 --> 00:40:38.790 that year. 495 00:40:38.790 --> 00:40:43.480 And then this is the total cost, total budget, the LSTA budget. 496 00:40:43.480 --> 00:40:49.130 And then this cost per completed evaluation, just pick these numbers in their raw form, 497 00:40:49.130 --> 00:40:55.980 not in these rounded numbers that I have presented here, and by the number of evaluations and 498 00:40:55.980 --> 00:41:00.100 comes up with how much each one costs. 499 00:41:00.100 --> 00:41:06.690 So you can see this kind of trend where the total cost per completed evaluation went from 500 00:41:06.690 --> 00:41:09.300 $7,200 up to $21,000. 501 00:41:09.300 --> 00:41:13.880 But then the median budget per evaluation has actually gone up quite a bit. 502 00:41:13.880 --> 00:41:15.180 It has doubled. 503 00:41:15.180 --> 00:41:22.070 And that is not a surprise as institutions start out with doing evaluations. 504 00:41:22.070 --> 00:41:26.960 It takes a little time for that business practice to take hold. 505 00:41:26.960 --> 00:41:32.700 And it takes a little time for evaluations to start to realize just how much a good evaluation 506 00:41:32.700 --> 00:41:33.790 is going to cost them. 507 00:41:33.790 --> 00:41:38.930 And by good, I mean once that they will be able to actually use and be able to make some 508 00:41:38.930 --> 00:41:39.930 hay with. 509 00:41:39.930 --> 00:41:46.050 So what we are seeing, though, is that kind of interesting trend, too, where the overall 510 00:41:46.050 --> 00:41:53.670 percentage of the evaluation that was covered by the LSTA budget has gone from nearly half 511 00:41:53.670 --> 00:41:57.460 down to less than one-fourth of the evaluation. 512 00:41:57.460 --> 00:42:04.230 So that means that the states are investing in the evaluations themselves. 513 00:42:04.230 --> 00:42:09.670 So that is the evaluation, little outputs analysis on that. 514 00:42:09.670 --> 00:42:18.930 And then the databases which again this had to be limited to just only the outputs, all 515 00:42:18.930 --> 00:42:23.850 of the databases that were acquired based on the number. 516 00:42:23.850 --> 00:42:32.350 These are only activities for which it was a one activity project, the only activity 517 00:42:32.350 --> 00:42:36.140 was related to the databases. 518 00:42:36.140 --> 00:42:39.820 And so we can sort of done a one-to-one matching here. 519 00:42:39.820 --> 00:42:48.510 And so here, you can see that databases are not cheap as you guys all well know. 520 00:42:48.510 --> 00:42:55.210 They are quite expensive in that they have really fluctuated. 521 00:42:55.210 --> 00:43:00.420 The per unit cost has really changed a bit over time. 522 00:43:00.420 --> 00:43:12.780 But whereas the percentage of cost worn by LSTA funds on the evaluations is declined. 523 00:43:12.780 --> 00:43:19.290 It has kind of fluctuated a bit here for the databases. 524 00:43:19.290 --> 00:43:28.200 So that is a first crack at some outputs analysis just looking at databases and evaluations. 525 00:43:28.200 --> 00:43:33.160 Are there any questions about that or any comments? 526 00:43:33.160 --> 00:43:39.580 This is the first time we have presented these analyses to you folks. 527 00:43:39.580 --> 00:43:42.610 MS. 528 00:43:42.610 --> 00:43:49.300 GONSALVES: We don’t have any immediate questions right now, Lisa. 529 00:43:49.300 --> 00:43:50.300 MS. 530 00:43:50.300 --> 00:43:51.300 FREHILL: Okay. 531 00:43:51.300 --> 00:43:59.470 I thought I would give people a chance. 532 00:43:59.470 --> 00:44:02.691 I am glad to see that folks are appreciating that. 533 00:44:02.691 --> 00:44:07.430 I see like a couple of folks saying, interesting, and thank you. 534 00:44:07.430 --> 00:44:09.990 It is good to know my efforts are appreciated. 535 00:44:09.990 --> 00:44:11.160 Thank you, guys. 536 00:44:11.160 --> 00:44:16.000 So this is actually our second look at activity outcomes. 537 00:44:16.000 --> 00:44:20.370 And we did the first one last year. 538 00:44:20.370 --> 00:44:25.700 This is a very -- that is cool. 539 00:44:25.700 --> 00:44:31.780 I am going to mention this before I get into the -- someone said, it would be interesting 540 00:44:31.780 --> 00:44:34.330 to see database cost per use. 541 00:44:34.330 --> 00:44:37.090 Yes, that would be fun. 542 00:44:37.090 --> 00:44:46.630 So yes, if you have ideas about how I might be able to measure that, please let me know. 543 00:44:46.630 --> 00:44:51.080 I think that inquiring minds would like to know that answer. 544 00:44:51.080 --> 00:44:55.400 So I am going to jump into the outcomes now. 545 00:44:55.400 --> 00:45:02.920 This is about questionnaires and Michele and Madison's presentation talked about it. 546 00:45:02.920 --> 00:45:07.430 These outcome questionnaires are aligned with Project Outcome. 547 00:45:07.430 --> 00:45:14.250 And the idea was that this way, it wouldn't, at least in theory, present a huge burden 548 00:45:14.250 --> 00:45:18.820 on libraries to collect it because if they were already participating in Project Outcomes, 549 00:45:18.820 --> 00:45:21.430 they would be collecting these data. 550 00:45:21.430 --> 00:45:26.840 But there are only four instances where we have instructional activities for either the 551 00:45:26.840 --> 00:45:32.450 general public or the library workforce, or we have content or planning and evaluation 552 00:45:32.450 --> 00:45:33.540 for the library workforce. 553 00:45:33.540 --> 00:45:38.550 So there are only four types of things where we collect this. 554 00:45:38.550 --> 00:45:39.930 And you saw this in the previous talks. 555 00:45:39.930 --> 00:45:42.440 I won't spend a lot time on this. 556 00:45:42.440 --> 00:45:53.060 But you can see is the set you enter in the survey findings. 557 00:45:53.060 --> 00:45:57.240 And so that was what the survey looks like. 558 00:45:57.240 --> 00:46:01.470 And then this is what you enter in for the questionnaire. 559 00:46:01.470 --> 00:46:06.060 When you pull these data, there are 108 variables across the four survey types. 560 00:46:06.060 --> 00:46:12.050 The analysis that I end up doing means that there are a few caveats. 561 00:46:12.050 --> 00:46:20.670 So number one, sometimes the survey didn’t really align with the type of activity. 562 00:46:20.670 --> 00:46:26.490 So in those cases, I drop that data because I thought, there is something, I am going 563 00:46:26.490 --> 00:46:29.940 to use a technical term here, funky going on. 564 00:46:29.940 --> 00:46:32.150 And we don’t want to include that data. 565 00:46:32.150 --> 00:46:43.310 The second thing is that there were some activities for which fewer than 10 responses were reported. 566 00:46:43.310 --> 00:46:48.640 And so dropped those from the analysis as well because we know that those are going 567 00:46:48.640 --> 00:46:51.190 to potentially be problematic. 568 00:46:51.190 --> 00:46:55.180 And this is actually kind of a second pass we did. 569 00:46:55.180 --> 00:47:01.930 This year, we did a little drilling into some of the intents with the library workforce. 570 00:47:01.930 --> 00:47:06.650 And the other thing that we could be spending some time doing is looking at formats. 571 00:47:06.650 --> 00:47:15.660 For example, the programs can be administered as either a virtual, an in-person or a combination 572 00:47:15.660 --> 00:47:17.610 of virtual and in-person. 573 00:47:17.610 --> 00:47:23.900 And so, we have an opportunity here to be able to look at whether there might be some 574 00:47:23.900 --> 00:47:31.400 interesting differences and how those different formats are assessed by different audiences. 575 00:47:31.400 --> 00:47:38.030 But that is sitting on the back burner at the present moment. 576 00:47:38.030 --> 00:47:47.080 So the way that we do this is all of these scales that you saw here are what are called 577 00:47:47.080 --> 00:47:48.080 Likert scales. 578 00:47:48.080 --> 00:47:51.000 They are five-point Likert scales. 579 00:47:51.000 --> 00:47:56.850 Strongly disagree, disagree, people sitting right on the fence in the middle, neither 580 00:47:56.850 --> 00:48:00.550 agree or disagree, agree and strongly agree. 581 00:48:00.550 --> 00:48:07.910 And so those five-point scales, you can think of it as having there is a strength of agreement, 582 00:48:07.910 --> 00:48:15.200 and that the scale and the strength of disagreement. 583 00:48:15.200 --> 00:48:19.250 I am going to go ahead and click through and get some more stuff up here. 584 00:48:19.250 --> 00:48:23.130 So you can kind of see the scale here. 585 00:48:23.130 --> 00:48:24.490 And you have got the neutral ones. 586 00:48:24.490 --> 00:48:26.840 These are the people who said neither agree or disagree. 587 00:48:26.840 --> 00:48:31.140 We put half of them on one side of the fence and half on the other side of the fence. 588 00:48:31.140 --> 00:48:35.800 And then we have agree and strongly agree, and we have disagree and strongly disagree. 589 00:48:35.800 --> 00:48:38.140 And we leave the non-respondents in. 590 00:48:38.140 --> 00:48:45.700 I am going to show you why it is we leave the non-respondents in in just a moment. 591 00:48:45.700 --> 00:48:49.010 So mathematically, this is how we represent the Likert scale. 592 00:48:49.010 --> 00:48:52.470 These are the five questions that they are asked. 593 00:48:52.470 --> 00:48:57.470 And these are for the general public activity outcomes. 594 00:48:57.470 --> 00:49:02.570 There were 53,661 responses from 240 activities. 595 00:49:02.570 --> 00:49:09.230 So what we tend to see happen in such instances is what is referred to as regression to the 596 00:49:09.230 --> 00:49:17.880 mean, which means that there is a very high level of agreement, strong agreement on the 597 00:49:17.880 --> 00:49:18.880 item. 598 00:49:18.880 --> 00:49:27.620 Now, why is it that we put the non-respondents over here? 599 00:49:27.620 --> 00:49:34.960 What tends to happen in surveys of this type is a social psychological thing called satis-ficing. 600 00:49:34.960 --> 00:49:41.230 Namely, that people tend to not want to say something negative. 601 00:49:41.230 --> 00:49:47.360 And so it sort of follows that old thing, if you don’t have something nice to say, 602 00:49:47.360 --> 00:49:48.680 don’t say anything at all. 603 00:49:48.680 --> 00:49:52.980 And that is what non-response is, that saying nothing at all. 604 00:49:52.980 --> 00:49:59.980 However, in the particular case of this question, I am more aware of resources and services 605 00:49:59.980 --> 00:50:02.250 provided by the library. 606 00:50:02.250 --> 00:50:09.850 Non-response is actually also kind of a signal that within the context of a Likert scale, 607 00:50:09.850 --> 00:50:11.560 it can be kind of confusing for respondents. 608 00:50:11.560 --> 00:50:22.440 So around 30 percent of people did not respond to this item at all in terms of, like, well, 609 00:50:22.440 --> 00:50:25.500 I sort of knew about this ahead of time. 610 00:50:25.500 --> 00:50:30.170 But now, I can't say I didn’t learn more because, well, that is saying something negative, 611 00:50:30.170 --> 00:50:32.230 and I really liked the project. 612 00:50:32.230 --> 00:50:38.680 So they have this sort of difficult time really processing that, that they don’t want to 613 00:50:38.680 --> 00:50:40.450 say something negative. 614 00:50:40.450 --> 00:50:47.720 But to say something negative isn't necessarily negative, if that makes any sense at all. 615 00:50:47.720 --> 00:50:54.150 So if we look at the library workforce, we had so many responses here on different types 616 00:50:54.150 --> 00:51:01.060 of activities that I was able to say, okay, well, let me look at just in-person programs. 617 00:51:01.060 --> 00:51:04.960 And we have got 8,200 responses for 73 activities. 618 00:51:04.960 --> 00:51:10.161 And this is specifically for instructional programs that were focused on improving the 619 00:51:10.161 --> 00:51:12.050 library workforce. 620 00:51:12.050 --> 00:51:16.850 And you can see that again there is a very high level of agreement here. 621 00:51:16.850 --> 00:51:23.920 But it varies in terms of, yes, they learned something, but they are less likely to have 622 00:51:23.920 --> 00:51:28.610 the same strength of agreement that it will improve services for the general public. 623 00:51:28.610 --> 00:51:31.230 But they still overwhelmingly say this. 624 00:51:31.230 --> 00:51:37.690 So more than 80 percent of people said something positive on all of these four questions. 625 00:51:37.690 --> 00:51:38.690 MS. 626 00:51:38.690 --> 00:51:39.690 GONSALVES: Hi, Lisa. 627 00:51:39.690 --> 00:51:44.631 David wants to know when you get a low response rate for a specific question over time, would 628 00:51:44.631 --> 00:51:48.630 it be the time to re-evaluate that particular question? 629 00:51:48.630 --> 00:51:50.060 MS. 630 00:51:50.060 --> 00:51:52.530 FREHILL: You bet you. 631 00:51:52.530 --> 00:51:56.590 I think last year was our first year, David. 632 00:51:56.590 --> 00:52:01.040 We talked through these at the LSTA coordinators meeting. 633 00:52:01.040 --> 00:52:08.590 And that last one that I was talking about, about the non-response, there were a lot of 634 00:52:08.590 --> 00:52:13.480 options that people said, hey, well, they could not respond because of this or whatever. 635 00:52:13.480 --> 00:52:19.360 So yes, it does signal that you need to kind of go back into the question. 636 00:52:19.360 --> 00:52:24.401 I think what is interesting about these questions, too, is that there is a Likert scale item. 637 00:52:24.401 --> 00:52:28.710 And those are the only ones that get reported in the SPR. 638 00:52:28.710 --> 00:52:32.010 But then there are some open-ended questions. 639 00:52:32.010 --> 00:52:39.330 And I think that those are probably potentially more useful on the ground at the point where 640 00:52:39.330 --> 00:52:45.190 a program or activity is being actually implemented. 641 00:52:45.190 --> 00:52:52.190 And also the open-ended questions can often suggest ideas for other sort of closed-ended 642 00:52:52.190 --> 00:52:55.840 questions that one might wish to ask at a future time. 643 00:52:55.840 --> 00:52:58.870 So that is a really great question. 644 00:52:58.870 --> 00:53:02.290 And I don’t know what the timeline is for this. 645 00:53:02.290 --> 00:53:10.720 I do know that because this is aligned with Project Outcomes, there is a larger context 646 00:53:10.720 --> 00:53:13.690 in which this exists. 647 00:53:13.690 --> 00:53:17.670 And then do we have a process in place to make changes? 648 00:53:17.670 --> 00:53:21.270 I am going to go back to my colleague, Matt, on this. 649 00:53:21.270 --> 00:53:27.260 Do we have a process of that sort, Matt? 650 00:53:27.260 --> 00:53:28.450 MR. 651 00:53:28.450 --> 00:53:40.340 BIRNBAUM: Lisa, I am just trying to digest the question. 652 00:53:40.340 --> 00:53:54.300 Let me give a more complete answer as to why it is not so simple. 653 00:53:54.300 --> 00:54:01.900 So the questions that we have around the surveys, these are aligned to those in Project Outcome 654 00:54:01.900 --> 00:54:04.760 that are run by the Public Library Association. 655 00:54:04.760 --> 00:54:11.480 And that was systematic, so that libraries who are either participating in Project Outcome 656 00:54:11.480 --> 00:54:16.760 or are receiving a Grants to States grant, they wouldn't have to ask two sets of questions 657 00:54:16.760 --> 00:54:18.540 to the same grantees. 658 00:54:18.540 --> 00:54:24.220 So there were lots of thought and care to making sure that the questions that are in 659 00:54:24.220 --> 00:54:29.650 the SPR match those, the Project Outcome. 660 00:54:29.650 --> 00:54:36.030 We have not talked with PLA about what would that process be like for revisiting the questions. 661 00:54:36.030 --> 00:54:42.140 But it is certainly a fair one to be considering in the next couple of years. 662 00:54:42.140 --> 00:54:43.140 MS. 663 00:54:43.140 --> 00:54:47.590 FREHILL: Thank you, Matt. 664 00:54:47.590 --> 00:54:51.400 A lot of moving parts to this particular one, I think. 665 00:54:51.400 --> 00:55:00.720 So now, I also set this up for instructional programs for library operations. 666 00:55:00.720 --> 00:55:04.060 I am just going to kind of click through these. 667 00:55:04.060 --> 00:55:12.340 These are the kinds of programs where it is the library workforce, but the intent was 668 00:55:12.340 --> 00:55:14.840 to improve users, fill in the blank. 669 00:55:14.840 --> 00:55:21.360 So these are those activities where they are doing an instructional program with the library 670 00:55:21.360 --> 00:55:22.530 workforce. 671 00:55:22.530 --> 00:55:30.400 And the intent is so that the library workforce can then do something that will help their 672 00:55:30.400 --> 00:55:32.670 patrons. 673 00:55:32.670 --> 00:55:38.280 And what is interesting is that this is a very, very strong level of agreement that 674 00:55:38.280 --> 00:55:43.300 the library workforce folks did learn something. 675 00:55:43.300 --> 00:55:50.760 And in fact, that they really felt strongly that this was going to hep them improve services 676 00:55:50.760 --> 00:55:51.890 for the public. 677 00:55:51.890 --> 00:55:56.950 And notice how the levels of non-response on these are really quite a bit smaller than 678 00:55:56.950 --> 00:56:00.730 on some of the other questionnaires that we saw. 679 00:56:00.730 --> 00:56:07.260 So this is an example of where we have taken the data, and we have sliced it a couple of 680 00:56:07.260 --> 00:56:15.040 different ways to really get at more specific instances, more specific types of programming 681 00:56:15.040 --> 00:56:17.710 that have been implemented. 682 00:56:17.710 --> 00:56:24.670 This is that content acquisition and creation. 683 00:56:24.670 --> 00:56:28.850 So only the library workforce gets asked these. 684 00:56:28.850 --> 00:56:36.130 There are a small number of responses compared to the others and a number of activities. 685 00:56:36.130 --> 00:56:43.200 And you can again see that the non-response is pretty small, relatively speaking and that 686 00:56:43.200 --> 00:56:46.000 a fairly high level of agreement on it. 687 00:56:46.000 --> 00:56:50.700 But there are only two questions on it. 688 00:56:50.700 --> 00:56:57.590 And then this is the last type of questionnaire there is. 689 00:56:57.590 --> 00:57:03.540 And that is ask the library workforce about the planning and evaluation activities. 690 00:57:03.540 --> 00:57:08.070 And as you can see, it is only 12 activities that had 303 responses. 691 00:57:08.070 --> 00:57:10.390 So it is a very, very small number. 692 00:57:10.390 --> 00:57:17.920 But you see, it is kind of interesting because here, this is the one item on which there 693 00:57:17.920 --> 00:57:23.570 is actually a bit more disagreement than on the others. 694 00:57:23.570 --> 00:57:29.490 On all the others, you saw these tiny little orange and red yellow bars on this side of 695 00:57:29.490 --> 00:57:30.670 the zero. 696 00:57:30.670 --> 00:57:42.260 And here, about planning and evaluation, you see a higher level of disagreement with the 697 00:57:42.260 --> 00:57:43.260 statements. 698 00:57:43.260 --> 00:57:49.950 So on the one hand, you can see, you still have a very high level of agreement that, 699 00:57:49.950 --> 00:57:52.660 at some level, addresses the library needs. 700 00:57:52.660 --> 00:57:54.960 But they are not satisfied. 701 00:57:54.960 --> 00:58:02.011 They are not as highly satisfied with the extent to which the planning and evaluation 702 00:58:02.011 --> 00:58:06.980 activity addresses the library needs. 703 00:58:06.980 --> 00:58:12.620 And then this last item is whether I believe the information from the planning and evaluation 704 00:58:12.620 --> 00:58:14.780 will be applied addressed library needs. 705 00:58:14.780 --> 00:58:22.500 Then you can see there is a little bit going on with all three of these questions together. 706 00:58:22.500 --> 00:58:26.110 So it really is the case. 707 00:58:26.110 --> 00:58:33.650 Based on my sort of gut feel for what we are seeing with these questions, it is really 708 00:58:33.650 --> 00:58:41.950 important to slice them, to try to control as much as possible for the intent. 709 00:58:41.950 --> 00:58:50.100 This one is a much tighter control set of activities, planning and evaluation versus 710 00:58:50.100 --> 00:58:57.830 these others in these other charts where we are looking at instructional programs. 711 00:58:57.830 --> 00:59:03.020 This is a real mixed bag of activities. 712 00:59:03.020 --> 00:59:11.660 This is everything from a citizenship thing to like a healthy diet thing to a yoga, all 713 00:59:11.660 --> 00:59:15.100 kinds of things. 714 00:59:15.100 --> 00:59:19.402 I do see the yellow thing, so I am going to go ahead. 715 00:59:19.402 --> 00:59:22.670 And back to the questions that I started with. 716 00:59:22.670 --> 00:59:29.200 I apologize for not really keeping track of your questions as well as I could have. 717 00:59:29.200 --> 00:59:35.230 But I guess I would throw this back at you and say, are there things in here that you 718 00:59:35.230 --> 00:59:38.370 would like us to be spending more time calling attention to? 719 00:59:38.370 --> 00:59:46.290 Are there things that we could do to best provide these data for you to support your 720 00:59:46.290 --> 00:59:47.290 project? 721 00:59:47.290 --> 00:59:48.930 And I will leave it at that. 722 00:59:48.930 --> 00:59:52.050 And thank you all very much for your attention. 723 00:59:52.050 --> 00:59:53.050 MS. 724 00:59:53.050 --> 00:59:59.900 DEVOE: Thank you, Lisa. this is Teri. 725 00:59:59.900 --> 01:00:03.880 We are getting right to the break time now. 726 01:00:03.880 --> 01:00:10.840 So if you have comments or questions, Lisa put her information on the last slide here. 727 01:00:10.840 --> 01:00:14.060 I will just advance it there, Ifrehill@imls.gov.