WEBVTT Kind: captions Language: en 1 00:00:11.140 --> 00:00:14.976 I'm Crosby Kemper, the Director of the Institute of Museum and 2 00:00:14.976 --> 00:00:18.751 Library Services, the nation's largest cultural agency. Which 3 00:00:18.751 --> 00:00:22.100 is why we've been working on a National Museum survey. 4 00:00:22.420 --> 00:00:24.860 We've been working on this for a long time, actually. 5 00:00:24.860 --> 00:00:29.688 We've tried, made, made one or two attempts in the past and and 6 00:00:29.688 --> 00:00:33.384 had some difficulties to overcome because of the 7 00:00:33.384 --> 00:00:37.910 heterogeneity of the museum universe, the breadth and depth 8 00:00:37.910 --> 00:00:42.738 of the museum universe, which is quite extraordinary as many of 9 00:00:42.738 --> 00:00:46.360 you know, but it's necessary for us to do this. 10 00:00:46.360 --> 00:00:49.818 It's important for us to do this because of the huge cultural 11 00:00:49.818 --> 00:00:52.440 impact of the museum world, broadly conceived. 12 00:00:52.720 --> 00:00:58.810 Art museums, science museums, science and technology centers, 13 00:00:58.810 --> 00:01:04.606 history museums, history sites, aquariums, zoos, botanical 14 00:01:04.606 --> 00:01:10.992 gardens, The cultural center of the United States is centered in 15 00:01:10.992 --> 00:01:17.279 what we refer to as museums and there's been no good, credible, 16 00:01:17.279 --> 00:01:23.075 professional level national survey of all of this cultural 17 00:01:23.075 --> 00:01:23.959 activity. 18 00:01:24.080 --> 00:01:29.038 And we hope our National Museum survey will be that. We hope 19 00:01:29.038 --> 00:01:34.240 that it will represent, as our Public Library survey has of the 20 00:01:34.240 --> 00:01:39.199 library world, the extent, the cultural impact, the economic 21 00:01:39.199 --> 00:01:44.320 impact, the impact on our social lives that museums represent. 22 00:01:44.760 --> 00:01:50.528 Museums and libraries together have over 2 billion visitations 23 00:01:50.528 --> 00:01:55.840 a year and that makes us bigger than professional sports. 24 00:01:56.080 --> 00:02:01.463 It makes us the most important public spaces in America and so 25 00:02:01.463 --> 00:02:06.248 we need the incredible professional review and database 26 00:02:06.248 --> 00:02:11.631 of this cultural infrastructure that that you represent and in 27 00:02:11.631 --> 00:02:17.186 all its heterogeneous extensive glory, and we've been working on 28 00:02:17.186 --> 00:02:22.569 this for a while. We we think we're we're finally about there, 29 00:02:22.569 --> 00:02:27.269 but we need your ongoing participation to make this an 30 00:02:27.269 --> 00:02:29.319 instrument of knowledge. 31 00:02:29.760 --> 00:02:33.695 And as an instrument of knowledge, an instrument of 32 00:02:33.695 --> 00:02:38.312 change, instrument of change based on a huge impact that you 33 00:02:38.312 --> 00:02:42.550 have in our world, an impact on education, an impact on 34 00:02:42.550 --> 00:02:47.166 professions and professional development for the public, for 35 00:02:47.166 --> 00:02:51.329 our civic leadership, for the media, for our political 36 00:02:51.329 --> 00:02:53.600 leaders, for effect on policy. 37 00:02:54.240 --> 00:02:58.500 So you are a pillar of our culture, pillars plural of 38 00:02:58.500 --> 00:03:02.760 course, and we want to demonstrate that to the world. 39 00:03:03.000 --> 00:03:04.040 So welcome. 40 00:03:04.080 --> 00:03:14.240 And I'm now going to turn it over to Laura. 41 00:03:14.240 --> 00:03:17.600 Great. 42 00:03:22.880 --> 00:03:23.880 Well, thank you all. 43 00:03:24.360 --> 00:03:27.320 Thank you Crosby for that warm welcome. 44 00:03:27.320 --> 00:03:34.090 And we are so excited to have such a broad and rich attendance 45 00:03:34.090 --> 00:03:36.240 for today's webinar. 46 00:03:36.480 --> 00:03:39.640 As Crosby mentioned, I'm Laura Huerta Migus. 47 00:03:39.640 --> 00:03:45.650 I am the Deputy Director for the Office of Museum Services here 48 00:03:45.650 --> 00:03:51.473 at IMLS and I am so happy to be emceeing this presentation of 49 00:03:51.473 --> 00:03:56.920 the summary findings of our National Museum Survey pilot. 50 00:03:57.960 --> 00:04:02.396 I will be joined on this call by a number of colleagues from 51 00:04:02.396 --> 00:04:06.833 IMLS, including Doctor Matthew Birnbaum, the Director of our 52 00:04:06.833 --> 00:04:10.907 Office of Research and Evaluation, Jake Soffronoff, who 53 00:04:10.907 --> 00:04:15.344 is our Survey Statistician and project lead for the National 54 00:04:15.344 --> 00:04:19.490 Museum Survey effort, as well as Helen Wechsler who is a 55 00:04:19.490 --> 00:04:23.636 Supervisory Senior Program Officer here in the Office of 56 00:04:23.636 --> 00:04:24.800 Museum Services. 57 00:04:25.520 --> 00:04:30.873 But before we jump into the findings, I want to reiterate a 58 00:04:30.873 --> 00:04:35.960 really important messagefrom Crosby's opening comments. 59 00:04:36.320 --> 00:04:42.314 And that is how integral relationship with so many of you 60 00:04:42.314 --> 00:04:48.722 on this call has been over the last two years to plan, launch 61 00:04:48.722 --> 00:04:52.960 and implement this pilot survey project. 62 00:04:53.240 --> 00:04:59.616 We were informed throughout our project by a group of 10 subject 63 00:04:59.616 --> 00:05:05.796 matter experts that represent a variety of demographics of the 64 00:05:05.796 --> 00:05:12.075 museum community from respected researchers and data scientists 65 00:05:12.075 --> 00:05:14.920 to field based practitioners. 66 00:05:15.360 --> 00:05:20.454 We also highly engaged our association partners who helped 67 00:05:20.454 --> 00:05:25.721 put us in touch with many museum professionals that provided 68 00:05:25.721 --> 00:05:31.075 field testing and input that informed the design of the pilot 69 00:05:31.075 --> 00:05:31.680 survey. 70 00:05:31.880 --> 00:05:37.513 And so many, many thanks to all of you who provided great public 71 00:05:37.513 --> 00:05:41.760 service to help us launch this successful pilot. 72 00:05:42.720 --> 00:05:47.446 And with that, I'm actually going to hand the screen and 73 00:05:47.446 --> 00:05:50.680 presentation over to my colleague, Dr. 74 00:05:50.680 --> 00:05:56.815 Matt Birnbaum to start taking us through the findings from our 75 00:05:56.815 --> 00:05:57.400 pilot. 76 00:05:57.640 --> 00:05:58.040 Matt. 77 00:05:58.520 --> 00:06:00.730 Yeah, Thank you, Laura and Dorothy, can you move to the 78 00:06:00.730 --> 00:06:01.480 next slide, please? 79 00:06:01.480 --> 00:06:01.960 Thank you. 80 00:06:02.520 --> 00:06:03.960 So welcome everybody. 81 00:06:04.720 --> 00:06:08.588 On behalf of my colleagues, I am humbled by the opportunity to 82 00:06:08.588 --> 00:06:11.720 represent the agency to present the findings here. 83 00:06:12.840 --> 00:06:16.373 What you're looking at are all the pieces that went into the 84 00:06:16.373 --> 00:06:19.616 survey and that we're going to summarize in the next 10 85 00:06:19.616 --> 00:06:20.080 minutes. 86 00:06:21.080 --> 00:06:24.040 For simplicity, perhaps oversimplifying. 87 00:06:24.760 --> 00:06:29.424 In designing the survey, we were looking at three particular 88 00:06:29.424 --> 00:06:29.960 pieces. 89 00:06:30.560 --> 00:06:32.680 We had to develop a questionnaire. 90 00:06:33.800 --> 00:06:38.819 We had to figure out who is the population of people to 91 00:06:38.819 --> 00:06:41.240 participate in this survey. 92 00:06:42.040 --> 00:06:46.517 And three, we needed an approach to actually administer the 93 00:06:46.517 --> 00:06:47.040 survey. 94 00:06:47.720 --> 00:06:50.246 So a lot of what we're going to be talking about, it's about all 95 00:06:50.246 --> 00:06:50.480 of it. 96 00:06:50.480 --> 00:06:55.328 It's going to be looking at these three parts to this survey 97 00:06:55.328 --> 00:07:00.256 design, the questionnaire, the approach for administering the 98 00:07:00.256 --> 00:07:02.800 survey and the population frame. 99 00:07:03.280 --> 00:07:07.621 And we're setting it up because to be successful it's going to 100 00:07:07.621 --> 00:07:11.963 be based on that design we we set together and how well we can 101 00:07:11.963 --> 00:07:13.480 implement that design. 102 00:07:14.000 --> 00:07:15.000 Next slide please. 103 00:07:17.880 --> 00:07:20.596 Matt, my apologies, can I interrupt one piece of 104 00:07:20.596 --> 00:07:23.480 housekeeping that I forgot to share with our group. 105 00:07:24.160 --> 00:07:29.228 We are in webinar mode on Zoom and so the chat function is not 106 00:07:29.228 --> 00:07:31.240 enabled for participants. 107 00:07:31.400 --> 00:07:35.537 But you can feel free to submit questions throughout the webinar 108 00:07:35.537 --> 00:07:39.356 via the Q&A function that you will see at the bottom of 109 00:07:39.356 --> 00:07:40.120 your screen. 110 00:07:40.120 --> 00:07:43.960 We will be monitoring that and saving time for a Q and A at the 111 00:07:43.960 --> 00:07:45.400 end of the presentation. 112 00:07:45.920 --> 00:07:46.520 Thank you. 113 00:07:46.720 --> 00:07:47.320 Thanks Matt. 114 00:07:47.840 --> 00:07:48.400 Yeah, you bet. 115 00:07:48.400 --> 00:07:49.040 Thanks Laura. 116 00:07:49.640 --> 00:07:53.586 You're looking here at some of the punch lines, some of the 117 00:07:53.586 --> 00:07:57.729 highlights and I want to just jump quickly to that last bullet 118 00:07:57.729 --> 00:07:59.440 which is to announce here. 119 00:07:59.440 --> 00:08:03.986 We're about 12 months away from launching a new annual survey 120 00:08:03.986 --> 00:08:08.240 that will add to the nation's federal statistical system. 121 00:08:08.920 --> 00:08:12.262 And we're going to be talking about all the things that we've 122 00:08:12.262 --> 00:08:13.880 achieved to get to this point. 123 00:08:13.880 --> 00:08:15.240 We're 12 months away. 124 00:08:15.800 --> 00:08:19.134 And what I just mentioned earlier about us putting the 125 00:08:19.134 --> 00:08:22.833 design together based on the questionnaire, the approach for 126 00:08:22.833 --> 00:08:26.774 administering the survey in the population of frame, how well it 127 00:08:26.774 --> 00:08:27.319 succeeds. 128 00:08:27.320 --> 00:08:30.600 is going to be based on a whole array of factors. 129 00:08:30.600 --> 00:08:35.261 And in this case, we've tried to be as scientifically robust as 130 00:08:35.261 --> 00:08:39.922 possible, but at the same time being responsive to the audience 131 00:08:39.922 --> 00:08:44.438 of the wide diversity of the museum sector and thinking about 132 00:08:44.438 --> 00:08:49.173 those who want to contribute an influence to a healthy sector in 133 00:08:49.173 --> 00:08:50.120 this country. 134 00:08:51.120 --> 00:08:52.120 Next slide please. 135 00:08:57.500 --> 00:09:02.977 So in the pilot overview, if you look at the left column, when we 136 00:09:02.977 --> 00:09:04.720 talk about a museum, 137 00:09:05.160 --> 00:09:10.136 we want to recognize the breadth and variety of what it means to 138 00:09:10.136 --> 00:09:12.280 be a museum in this country. 139 00:09:12.680 --> 00:09:16.766 There are multiple disciplines, from botanical gardens to 140 00:09:16.766 --> 00:09:20.642 planetariums to children's museums to art museums, all 141 00:09:20.642 --> 00:09:21.840 types of museums. 142 00:09:22.920 --> 00:09:27.808 All of these museums were considered and tried to be 143 00:09:27.808 --> 00:09:33.619 counted on in thinking about who would be participating in the 144 00:09:33.619 --> 00:09:35.280 survey altogether. 145 00:09:35.520 --> 00:09:39.457 When we ran the survey in the depth of August, we had 146 00:09:39.457 --> 00:09:43.978 approximately sent invitations out to 7050 and that deep part 147 00:09:43.978 --> 00:09:48.572 of August when a lot of us are on vacation, are thinking about 148 00:09:48.572 --> 00:09:49.520 other things, 149 00:09:49.880 --> 00:09:52.800 we had a 17% response rate 150 00:09:53.360 --> 00:09:57.685 who responded to the survey and answering the need and 151 00:09:57.685 --> 00:10:02.560 demonstration that we can get enough to participate just from 152 00:10:02.560 --> 00:10:06.885 the get go to get us to a reliable statistically valid 153 00:10:06.885 --> 00:10:11.840 count of measuring what is going on with our nation's museums. 154 00:10:12.680 --> 00:10:13.680 Next slide please. 155 00:10:16.880 --> 00:10:20.093 The next two slides are going to be sharing you basically the 156 00:10:20.093 --> 00:10:21.960 same information in different ways. 157 00:10:22.360 --> 00:10:26.124 This is a visual and if we can go back several years ago when 158 00:10:26.124 --> 00:10:29.949 we first started to launch this current effort of the National 159 00:10:29.949 --> 00:10:30.799 Museum Survey, 160 00:10:31.400 --> 00:10:35.013 we started with background research, lit reviews, 161 00:10:35.013 --> 00:10:39.061 landscaping, benchmarking of who already was collecting 162 00:10:39.061 --> 00:10:41.880 information about our nation's museum. 163 00:10:41.880 --> 00:10:45.640 If we were to begin to do one, how can we do one? 164 00:10:45.640 --> 00:10:48.680 How can we add the most value? 165 00:10:49.080 --> 00:10:53.414 And so like a funnel, we went from these large issues and 166 00:10:53.414 --> 00:10:57.972 trying to conceptualize what would this survey look like all 167 00:10:57.972 --> 00:11:02.605 the way down to finer and finer details and homing in for the 168 00:11:02.605 --> 00:11:07.088 point when we got to doing a pilot to then looking at those 169 00:11:07.088 --> 00:11:09.480 who didn't respond to the pilot. 170 00:11:10.000 --> 00:11:14.784 So it was a very deliberate, scientifically driven approach 171 00:11:14.784 --> 00:11:19.089 using the engagement of hundreds, thousands of people 172 00:11:19.089 --> 00:11:21.960 across the country's museum sector. 173 00:11:22.360 --> 00:11:23.360 Next slide please. 174 00:11:26.880 --> 00:11:31.142 And again, this is basically the same information you saw on the 175 00:11:31.142 --> 00:11:34.880 prior slide, presented it in a slightly different, form. 176 00:11:35.520 --> 00:11:39.804 I want to just highlight here that one that when we started 177 00:11:39.804 --> 00:11:43.803 that lit review it started to drive what the intent and 178 00:11:43.803 --> 00:11:48.230 purpose would be to make sure that any data that we had would 179 00:11:48.230 --> 00:11:52.657 be meaningful, useful and in ways that could help people make 180 00:11:52.657 --> 00:11:54.800 more informed decision making. 181 00:11:56.320 --> 00:11:57.240 Next slide please. 182 00:12:00.960 --> 00:12:03.760 Indeed engagement was the key to success. 183 00:12:03.760 --> 00:12:07.843 From the very, very get go, we've been thinking of how we 184 00:12:07.843 --> 00:12:12.491 can best collaborate and partner and make this statistical survey 185 00:12:12.491 --> 00:12:16.645 a useful product, a tool, a process to enable those in the 186 00:12:16.645 --> 00:12:21.151 museum sector to use that data to make more informed decisions, 187 00:12:21.151 --> 00:12:22.559 more informed plans. 188 00:12:23.200 --> 00:12:27.339 As well as investors, public and private, who also have an 189 00:12:27.339 --> 00:12:31.758 interest in strengthening the capacity of our country's museum 190 00:12:31.758 --> 00:12:32.320 sectors. 191 00:12:32.680 --> 00:12:36.080 And to do so, engagement was key from the get go. 192 00:12:36.480 --> 00:12:41.087 It started with interviews and focus groups, moved all the way 193 00:12:41.087 --> 00:12:45.403 to 7 experiments that we ran with a pilot, surveys all the 194 00:12:45.403 --> 00:12:49.571 different ways and then having additional communications 195 00:12:49.571 --> 00:12:54.252 education, outgoing in the field as well as actively using some 196 00:12:54.252 --> 00:12:58.421 very valued subject matter experts who were part of this 197 00:12:58.421 --> 00:13:01.640 process and with us through the whole time. 198 00:13:02.080 --> 00:13:05.017 And we also were very thankful for having the American 199 00:13:05.017 --> 00:13:08.223 Institutes of a Research as one of our partners for helping 200 00:13:08.223 --> 00:13:10.520 administer and drive this whole endeavour. 201 00:13:11.880 --> 00:13:15.585 We particularly want to focus on that last bullet that makes it 202 00:13:15.585 --> 00:13:19.233 clear of trying to reach some of the small museums and hard to 203 00:13:19.233 --> 00:13:22.765 reach institutions to make sure that those their voices were 204 00:13:22.765 --> 00:13:24.559 heard just as loudly as others. 205 00:13:25.280 --> 00:13:26.240 Next slide please. 206 00:13:30.280 --> 00:13:34.252 So in developing the questionnaire survey topics, 207 00:13:34.252 --> 00:13:38.939 this was nothing that if not wasn't done by a few of us in 208 00:13:38.939 --> 00:13:41.800 and around Washington DC developed. 209 00:13:41.800 --> 00:13:45.876 It was instead informed by the extensive engagement with the 210 00:13:45.876 --> 00:13:50.154 hundreds and thousands from the museum sector and it led to us, 211 00:13:50.154 --> 00:13:54.498 developing a questionnaire that addressed what we heard were the 212 00:13:54.498 --> 00:13:58.574 key essential topics that we didn't have hard evidence about 213 00:13:58.574 --> 00:14:02.584 and that if they had, that evidence could make a meaningful 214 00:14:02.584 --> 00:14:03.320 difference. 215 00:14:03.640 --> 00:14:05.400 So you see it here on the slide. 216 00:14:05.760 --> 00:14:10.090 Institutional characteristics, facilities, finances, human 217 00:14:10.090 --> 00:14:14.788 resources, admissions, visitors and outreach, digital presence, 218 00:14:14.788 --> 00:14:19.559 diversity, equity, inclusion, as well as for those participating 219 00:14:19.559 --> 00:14:24.256 in the pilot, the experience of actually trying to complete the 220 00:14:24.256 --> 00:14:27.560 questionnaire and being part of that survey. 221 00:14:29.840 --> 00:14:31.720 Next slide please. 222 00:14:35.280 --> 00:14:39.212 So in developing the questionnaire, we used the 223 00:14:39.212 --> 00:14:43.391 preliminary research and cognitive interviews with 224 00:14:43.391 --> 00:14:47.324 hundreds of museum administrators, supplemented 225 00:14:47.324 --> 00:14:52.404 with the subject matter experts who were part of us, to shape 226 00:14:52.404 --> 00:14:55.600 the development of that questionnaire. 227 00:14:56.240 --> 00:15:00.390 In running the pilot, we didn't try to use the pilot to give 228 00:15:00.390 --> 00:15:04.200 national estimates on any of these questionnaire items. 229 00:15:04.200 --> 00:15:05.360 Still too early. 230 00:15:05.920 --> 00:15:10.170 It instead was to really figure out how to make this survey as 231 00:15:10.170 --> 00:15:14.352 most responsive, least burden, getting the most credible data 232 00:15:14.352 --> 00:15:14.960 possible. 233 00:15:16.120 --> 00:15:20.337 And then the things that we've learned was that a majority of 234 00:15:20.337 --> 00:15:24.623 people, about 60%, had a hard time answering questions related 235 00:15:24.623 --> 00:15:26.800 to the finances of their museum. 236 00:15:27.160 --> 00:15:30.541 And that will be a piece for follow up research over the next 237 00:15:30.541 --> 00:15:33.814 year to try to get a better handle on some of the issues of 238 00:15:33.814 --> 00:15:34.960 this cultural change. 239 00:15:35.400 --> 00:15:38.883 So suddenly trying to throw financial information out there 240 00:15:38.883 --> 00:15:42.541 that can be used for things such as benchmarking for help, and 241 00:15:42.541 --> 00:15:46.374 trying to think about what peers might be doing and ways in which 242 00:15:46.374 --> 00:15:49.800 certain types of museums are investing in their resources. 243 00:15:50.320 --> 00:15:54.021 For the large museums, it also turned out to be a challenge to 244 00:15:54.021 --> 00:15:57.487 provide report on facilities data and frequently, maybe it 245 00:15:57.487 --> 00:16:00.953 was just the complexity of their institution, maybe it was 246 00:16:00.953 --> 00:16:04.420 multiple locations, maybe it was different people covering 247 00:16:04.420 --> 00:16:05.360 different areas. 248 00:16:05.720 --> 00:16:07.720 And so that provided a little bit of a hiccup for them. 249 00:16:09.120 --> 00:16:10.120 Next slide please. 250 00:16:16.270 --> 00:16:20.704 In looking at the respondents survey taking experience, that 251 00:16:20.704 --> 00:16:25.211 first bullet of trying to find ways to reduce the burden, and 252 00:16:25.211 --> 00:16:29.718 we did so through multiple ways in how we designed the online 253 00:16:29.718 --> 00:16:33.716 survey questionnaire and that was the mode in which we 254 00:16:33.716 --> 00:16:34.880 administered it. 255 00:16:35.800 --> 00:16:39.847 We also found out that as much as we had expected it in terms 256 00:16:39.847 --> 00:16:43.960 of implementing the pilot, it seemed that we had large success 257 00:16:43.960 --> 00:16:48.073 with the majority of folks being able to finish it in under an 258 00:16:48.073 --> 00:16:48.400 hour. 259 00:16:48.720 --> 00:16:51.738 And that includes the time they had to gather the requested 260 00:16:51.738 --> 00:16:53.600 information to answer the questions. 261 00:16:54.000 --> 00:16:57.735 And that they reported back that the survey was relatively easy 262 00:16:57.735 --> 00:17:01.413 to complete, and the instrument was fairly easy to navigate on 263 00:17:01.413 --> 00:17:01.880 the web. 264 00:17:01.880 --> 00:17:04.480 Next slide please. 265 00:17:08.040 --> 00:17:13.800 But that response rate, 17% participated, therefore 83% 266 00:17:13.800 --> 00:17:14.520 didn't. 267 00:17:15.040 --> 00:17:18.742 And we know we did it the dead of summer, which is possibly the 268 00:17:18.742 --> 00:17:22.155 time when it would be hardest for many to participate in a 269 00:17:22.155 --> 00:17:22.560 survey. 270 00:17:23.280 --> 00:17:27.012 But we were nonetheless finding some other issues that we will 271 00:17:27.012 --> 00:17:27.960 need to work on. 272 00:17:28.520 --> 00:17:32.752 We ran 7 experiments to figure out what's the best way for a 273 00:17:32.752 --> 00:17:35.320 person to participate in the survey. 274 00:17:35.760 --> 00:17:39.703 And it turned out that if we administered it with giving 275 00:17:39.703 --> 00:17:43.855 contacting people by phone at some point to tell them to go 276 00:17:43.855 --> 00:17:46.000 online and complete the survey, 277 00:17:46.000 --> 00:17:50.343 that that was a very effective probe to induce people to 278 00:17:50.343 --> 00:17:55.296 complete the survey, and moving forward, we know we need to do a 279 00:17:55.296 --> 00:17:59.716 lot more outreach in terms of raising awareness about the 280 00:17:59.716 --> 00:18:04.670 survey and the benefits and uses for individual institutions and 281 00:18:04.670 --> 00:18:06.880 the museum sector as a whole. 282 00:18:07.920 --> 00:18:08.840 Next slide please. 283 00:18:13.480 --> 00:18:15.640 OK, uncovering the intended audience. 284 00:18:16.520 --> 00:18:20.904 So one of the challenges we had about 10 years ago when we last 285 00:18:20.904 --> 00:18:25.152 tried to really develop this survey was to get an estimate, a 286 00:18:25.152 --> 00:18:29.400 credible estimate of who are the actual museums in the United 287 00:18:29.400 --> 00:18:29.880 States. 288 00:18:29.880 --> 00:18:32.320 And that was a long standing challenge. 289 00:18:32.800 --> 00:18:38.043 And to break the hurdle, this time, using the expertise and 290 00:18:38.043 --> 00:18:43.637 wisdom and creativity and acumen of Jake Soffronoff and some of 291 00:18:43.637 --> 00:18:45.560 our colleagues at AIR. 292 00:18:45.920 --> 00:18:50.190 That we really moved to the pioneer threshold of data 293 00:18:50.190 --> 00:18:55.409 science, where we collected data using Crowdsource data from Yelp 294 00:18:55.409 --> 00:18:59.600 and supplemented from the official museum directory. 295 00:19:00.080 --> 00:19:04.247 And from that, we used a number of innovative approaches and 296 00:19:04.247 --> 00:19:08.687 data science, including ChatGPT, web scraping and Amazon Turk to 297 00:19:08.687 --> 00:19:12.854 take those lists into something that was credible and valid, 298 00:19:12.854 --> 00:19:16.816 coming up with a current estimate of about 20,000 museums 299 00:19:16.816 --> 00:19:20.573 across all the disciplines of the museum sector in the 300 00:19:20.573 --> 00:19:21.120 country. 301 00:19:21.640 --> 00:19:22.320 Moving ahead. 302 00:19:22.320 --> 00:19:25.812 Over the next year, we'll be refining that list and working 303 00:19:25.812 --> 00:19:29.596 to update it so that a year from now when we're ready to launch, 304 00:19:29.596 --> 00:19:33.147 the annual survey will have a very robust, credible estimate 305 00:19:33.147 --> 00:19:36.000 of the size of the museum sector in the country. 306 00:19:37.480 --> 00:19:38.440 Next slide please. 307 00:19:41.680 --> 00:19:46.230 And my last point is now looking ahead to the next 12 months, the 308 00:19:46.230 --> 00:19:50.160 good news is the reception has been relatively positive. 309 00:19:51.000 --> 00:19:53.440 The survey was quick and easy to take. 310 00:19:53.800 --> 00:19:57.080 The content is going to remain largely intact. 311 00:19:57.080 --> 00:20:00.731 Based on the feedback that we've gotten from the pilot, we'll be 312 00:20:00.731 --> 00:20:04.102 doing some tweaking, but we've largely scoped it just about 313 00:20:04.102 --> 00:20:04.440 right. 314 00:20:04.840 --> 00:20:08.682 But one of our challenges will now will be trying to improve 315 00:20:08.682 --> 00:20:12.588 our outreach in letting people know about the National Museum 316 00:20:12.588 --> 00:20:15.359 Survey, what it means to participate in it, 317 00:20:16.160 --> 00:20:20.553 and helping them ease that experience of being a partner in 318 00:20:20.553 --> 00:20:20.920 this. 319 00:20:21.320 --> 00:20:24.491 It'll include phone calls to museums, more conference 320 00:20:24.491 --> 00:20:27.838 presentations, and us continuing to do education through 321 00:20:27.838 --> 00:20:31.480 communications, both social and traditional media engagement. 322 00:20:33.960 --> 00:20:37.602 I'm going to thank you all for giving me this opportunity to 323 00:20:37.602 --> 00:20:38.080 present. 324 00:20:38.080 --> 00:20:39.840 Laura, I'm going to turn it back over to you. 325 00:20:43.240 --> 00:20:43.680 Great. 326 00:20:44.400 --> 00:20:45.200 Thank you, Matt. 327 00:20:45.840 --> 00:20:51.837 Thank you for that high level overview of some of the lessons 328 00:20:51.837 --> 00:20:56.480 learned from this endeavor as which, as Crosby, 329 00:20:56.680 --> 00:21:01.419 Matt, myself and I think you'll hear others from IMLS on the 330 00:21:01.419 --> 00:21:06.314 call, is really the culmination of more than two decades worth 331 00:21:06.314 --> 00:21:10.200 of efforts from IMLS to move this effort forward. 332 00:21:10.200 --> 00:21:14.262 So we are so excited to have gotten through this phase of the 333 00:21:14.262 --> 00:21:17.800 development and piloting of a National Museum survey. 334 00:21:18.160 --> 00:21:23.280 And so now we are going to move into our Q&A section. 335 00:21:24.680 --> 00:21:30.835 And so Dorothy, if I could have you maybe take down the slides 336 00:21:30.835 --> 00:21:36.990 and I'll invite my colleagues Helen and Jake to come on camera 337 00:21:36.990 --> 00:21:41.680 as we enter into this part of our presentation. 338 00:21:41.680 --> 00:21:46.312 I will remind folks online that this is your opportunity to 339 00:21:46.312 --> 00:21:49.400 start populating that Q&A function. 340 00:21:49.600 --> 00:21:54.001 And as we are settling in, we have multiple questions 341 00:21:54.001 --> 00:21:58.972 regarding the ability to access a recording or the slides of 342 00:21:58.972 --> 00:22:00.440 this presentation. 343 00:22:01.240 --> 00:22:06.123 While we will not be sharing this themselves, the recording 344 00:22:06.123 --> 00:22:11.170 of this session will be posted to imls.gov as will the actual 345 00:22:11.170 --> 00:22:16.216 summary report, which is going to be available on our website 346 00:22:16.216 --> 00:22:20.937 at imls.gov for you to dive into these findings in a more 347 00:22:20.937 --> 00:22:22.239 substantive way. 348 00:22:23.480 --> 00:22:29.081 And so with that, I am going to move into some of your great 349 00:22:29.081 --> 00:22:30.000 questions. 350 00:22:31.440 --> 00:22:36.280 And Jake, I think that this question is for you. 351 00:22:37.560 --> 00:22:42.431 We have a question from Janice Klein around creating the 352 00:22:42.431 --> 00:22:47.303 population frame and how did we involve different museum 353 00:22:47.303 --> 00:22:51.320 associations in developing some of that frame. 354 00:22:54.560 --> 00:23:00.126 So we started the we started the population frame by looking 355 00:23:00.126 --> 00:23:05.783 through every available sort of public resource that we could 356 00:23:05.783 --> 00:23:06.240 find. 357 00:23:07.040 --> 00:23:11.093 One of the concerns that we had was trying to make sure that it 358 00:23:11.093 --> 00:23:14.956 was not duplicative, which has been, you know, an issue that 359 00:23:14.956 --> 00:23:17.679 IMLS has been working through in the past. 360 00:23:18.760 --> 00:23:22.438 When you grab information from one list, it may be written up 361 00:23:22.438 --> 00:23:25.880 in one way and then in another list it's a different way. 362 00:23:26.600 --> 00:23:29.825 And the next thing you know, you've got five different lists 363 00:23:29.825 --> 00:23:32.840 with five different versions of the name for one museum. 364 00:23:33.880 --> 00:23:38.432 So what we wanted was effectively a phone book as much 365 00:23:38.432 --> 00:23:39.840 as we could have. 366 00:23:40.640 --> 00:23:46.623 And so we started, after talking to a large number of vendors in 367 00:23:46.623 --> 00:23:52.146 the private sector, talking to colleagues across the public 368 00:23:52.146 --> 00:23:58.037 sector from the, you know, the, the types of resources that you 369 00:23:58.037 --> 00:24:03.376 would expect, census, BLS, that kind of stuff, going back 370 00:24:03.376 --> 00:24:08.807 through IRS990 forms for non profits, talking to a good 10 371 00:24:08.807 --> 00:24:10.280 private vendors. 372 00:24:11.240 --> 00:24:18.704 The vendor that we finally chose was Yelp, which we selected 373 00:24:18.704 --> 00:24:23.600 because it serves more as a phone book. 374 00:24:23.600 --> 00:24:27.840 It's very location based and that's really useful for 375 00:24:27.840 --> 00:24:30.040 reaching out to the museums. 376 00:24:30.040 --> 00:24:33.336 If you go into some of those other data sources you end up 377 00:24:33.336 --> 00:24:36.688 with, for example, from 990 forms you might you might get a 378 00:24:36.688 --> 00:24:40.040 business address that's separate from the physical address. 379 00:24:41.040 --> 00:24:45.757 You also, if you go through a lot of those other resources, 380 00:24:45.757 --> 00:24:50.318 you are constrained to the people who are opting in to be 381 00:24:50.318 --> 00:24:52.519 in being in those resources. 382 00:24:53.240 --> 00:24:58.398 So you miss a lot of the ones that aren't members of the 383 00:24:58.398 --> 00:25:00.480 different associations. 384 00:25:01.240 --> 00:25:07.941 So we started with national Yelp data and then did our best to 385 00:25:07.941 --> 00:25:14.111 find the needles in that haystack of all of the different 386 00:25:14.111 --> 00:25:20.813 disciplines, which is another sort of complicating factor, and 387 00:25:20.813 --> 00:25:25.920 used all of the methods that were talked about. 388 00:25:25.920 --> 00:25:29.950 We used ChatGPT to try and assign them to different 389 00:25:29.950 --> 00:25:30.880 disciplines. 390 00:25:30.880 --> 00:25:36.804 We used some web scraping to try and get any missing business 391 00:25:36.804 --> 00:25:37.760 addresses. 392 00:25:37.760 --> 00:25:44.741 We used M Turk prominently for a lot of the work that would have 393 00:25:44.741 --> 00:25:49.360 been normally manual work for individuals. 394 00:25:49.360 --> 00:25:53.687 Because we had a list of, even using the categories available 395 00:25:53.687 --> 00:25:57.944 from Yelp, we had well over 100,000 that we started with and 396 00:25:57.944 --> 00:25:59.480 had to pair that down. 397 00:25:59.560 --> 00:26:05.133 We also married all of this data to data wherever we could find 398 00:26:05.133 --> 00:26:09.400 it, so that included our past museum data files. 399 00:26:10.160 --> 00:26:16.317 It also included the OMD, the Official Museum directory in 400 00:26:16.317 --> 00:26:21.640 order to try and get as many contacts as we could. 401 00:26:23.360 --> 00:26:26.788 Even through all that effort, we still had trouble reaching 402 00:26:26.788 --> 00:26:27.360 everybody. 403 00:26:27.360 --> 00:26:30.643 So one of the one of the main issues was finding the specific 404 00:26:30.643 --> 00:26:32.920 person inside of the specific institution. 405 00:26:33.840 --> 00:26:37.700 And so I'm hopeful that maybe when we reach out later this 406 00:26:37.700 --> 00:26:41.757 year, everyone will take our phone calls and make sure we get 407 00:26:41.757 --> 00:26:45.160 the right person as we try and reinforce that list. 408 00:26:46.680 --> 00:26:51.957 We did interact some with some state museum organizations, 409 00:26:51.957 --> 00:26:57.593 particularly New York, because they have a a centralized body, 410 00:26:57.593 --> 00:27:01.440 the different methods of defining museums. 411 00:27:01.440 --> 00:27:03.720 Ours is a very, very broad tent. 412 00:27:05.640 --> 00:27:09.335 Doesn't necessarily make that the approach that's going to 413 00:27:09.335 --> 00:27:13.155 ensure that everyone gets in, but it's one of the approaches 414 00:27:13.155 --> 00:27:17.101 that we're going to continue to triangulate as we move forward 415 00:27:17.101 --> 00:27:19.920 in solidifying the the frame that we've got. 416 00:27:20.880 --> 00:27:21.720 Thank you, Jake. 417 00:27:21.720 --> 00:27:26.559 And I think the other thing perhaps to mention and I'm going 418 00:27:26.559 --> 00:27:31.399 to move to Helen with some of these eligibility questions is 419 00:27:31.399 --> 00:27:36.318 that because this was a pilot, we were also experimenting and 420 00:27:36.318 --> 00:27:41.237 sort of the most rigorous way that we might have to reach out 421 00:27:41.237 --> 00:27:45.759 and sample the field and there were some pretty rigorous 422 00:27:45.759 --> 00:27:50.520 experiments in that that that shaped our approaches as well 423 00:27:50.520 --> 00:27:57.296 for creating the frame. Right, we included seven separate field 424 00:27:57.296 --> 00:28:01.320 experiments, each had its own sample. 425 00:28:01.440 --> 00:28:05.530 They were all based around different ways of contacting 426 00:28:05.530 --> 00:28:06.480 institutions. 427 00:28:07.320 --> 00:28:12.061 So they included pre mailers, reminder mailers, sending a pre 428 00:28:12.061 --> 00:28:16.878 Mailer that includes all of the information around the survey, 429 00:28:16.878 --> 00:28:21.696 sending a pre Mailer that's a a postcard and phone calls, both 430 00:28:21.696 --> 00:28:26.590 pre fields and midfield to, you know, reminder emails, reminder 431 00:28:26.590 --> 00:28:28.120 phone calls, rather. 432 00:28:29.200 --> 00:28:33.546 We did find that phone calls were the most effective method 433 00:28:33.546 --> 00:28:38.038 for driving people to take the survey and we feel, looking at 434 00:28:38.038 --> 00:28:42.240 the data now that a big reason for that is the difficulty 435 00:28:42.240 --> 00:28:46.586 finding the specific person inside the specific institution 436 00:28:46.586 --> 00:28:50.643 as opposed to saying, for example, sending an e-mail to 437 00:28:50.643 --> 00:28:52.600 info at Jake's Good Museum. 438 00:28:53.400 --> 00:28:56.685 It seems like you may need to send it to Jake at Jake's Good 439 00:28:56.685 --> 00:28:58.840 Museum in order to really hit the mark. 440 00:28:59.840 --> 00:29:00.080 Great. 441 00:29:00.200 --> 00:29:00.960 Thanks, Jake. 442 00:29:02.440 --> 00:29:05.773 And I think I want to remind folks that in the pilot survey 443 00:29:05.773 --> 00:29:09.274 what we were the big questions we were trying to answer Is, is 444 00:29:09.274 --> 00:29:12.553 it feasible to to launch a survey and and what would those 445 00:29:12.553 --> 00:29:13.720 mechanisms look like? 446 00:29:13.720 --> 00:29:19.691 And so Helen, we have a number of questions in the queue here 447 00:29:19.691 --> 00:29:23.640 that all revolve around what's a museum. 448 00:29:24.560 --> 00:29:29.133 And so I'd love if you could share the some of the framework 449 00:29:29.133 --> 00:29:33.482 that we used and again more detail will be in the report, 450 00:29:33.482 --> 00:29:38.205 but just for for folks asking that question, how we approached 451 00:29:38.205 --> 00:29:40.080 that concept, absolutely. 452 00:29:40.080 --> 00:29:41.000 Thanks Laura. 453 00:29:41.480 --> 00:29:45.920 So as Jake said, we wanted to build a great big tent. 454 00:29:45.920 --> 00:29:51.119 We know our field is extremely diverse and we do, at IMLS, have 455 00:29:51.119 --> 00:29:55.424 experience in having criteria that help us determine 456 00:29:55.424 --> 00:29:59.080 eligibility for applying to a grant program. 457 00:29:59.080 --> 00:30:03.610 So we started with those pieces which helped us put together a 458 00:30:03.610 --> 00:30:08.068 series of questions that allowed people or institutions to be 459 00:30:08.068 --> 00:30:11.520 screened and to move through the questionnaire. 460 00:30:11.520 --> 00:30:17.045 And those involved having or using collections, being 461 00:30:17.045 --> 00:30:23.696 educational, being in a facility that's owned or operated, being 462 00:30:23.696 --> 00:30:30.040 open to the public, you know for a period of time every year. 463 00:30:30.800 --> 00:30:35.096 So those types of questions that are probably very familiar to 464 00:30:35.096 --> 00:30:37.960 anybody who's applied for a grant at IMLS 465 00:30:40.680 --> 00:30:41.600 Great, thank you. 466 00:30:41.840 --> 00:30:49.672 And you know, we had some questions here also about 467 00:30:49.672 --> 00:30:58.258 looking at the response rate from small institutions and 468 00:30:58.258 --> 00:31:01.120 rural institutions. 469 00:31:01.120 --> 00:31:05.440 And Matt talked about this a bit in his summary of the findings, 470 00:31:05.440 --> 00:31:09.771 but I'm wondering, Jake, if you can talk about our sampling and 471 00:31:09.771 --> 00:31:13.425 the attention we were paying, you know, as we in this 472 00:31:13.425 --> 00:31:17.621 experiment and how we looked at different parts of the museum 473 00:31:17.621 --> 00:31:18.840 field for testing. 474 00:31:21.920 --> 00:31:22.360 Sure. 475 00:31:22.360 --> 00:31:25.727 So if you got four or five hours, I can talk about this, 476 00:31:25.727 --> 00:31:29.509 but I'll do my best to give the high level version knowing that 477 00:31:29.509 --> 00:31:31.400 I'm verbose on topics like this. 478 00:31:33.120 --> 00:31:42.145 So the sampling was based around #1 discipline was the largest 479 00:31:42.145 --> 00:31:45.440 sort of sorting factor. 480 00:31:45.920 --> 00:31:51.293 The way that we had to go about that because that this was our 481 00:31:51.293 --> 00:31:56.496 first run was to collapse a number of those different groups 482 00:31:56.496 --> 00:31:58.800 into sort of larger groups. 483 00:32:00.280 --> 00:32:05.032 For example, some of the types of museums that have animate 484 00:32:05.032 --> 00:32:09.548 collections where they kind of grouped together, the the 485 00:32:09.548 --> 00:32:13.746 history museums with the historical societies, those 486 00:32:13.746 --> 00:32:18.499 kinds of groups because before the field we had never asked 487 00:32:18.499 --> 00:32:23.648 everybody exactly how they would categorize themselves and their 488 00:32:23.648 --> 00:32:24.520 discipline. 489 00:32:24.960 --> 00:32:29.118 The survey itself gets that information from you, the 490 00:32:29.118 --> 00:32:30.120 practitioner. 491 00:32:30.840 --> 00:32:34.194 And so in the future we're going to have much better statistics 492 00:32:34.194 --> 00:32:35.400 and ability to do that. 493 00:32:35.840 --> 00:32:40.896 That'll probably also be a part of our outreach later this year 494 00:32:40.896 --> 00:32:45.794 to to learn more and make sure that we're able to really draw 495 00:32:45.794 --> 00:32:50.692 the kinds of statistics that everybody's going to want out of 496 00:32:50.692 --> 00:32:51.640 this effort. 497 00:32:55.720 --> 00:33:00.074 The segmenting was also done by geography to try and make sure 498 00:33:00.074 --> 00:33:03.600 that we had everybody in their own geographic mix. 499 00:33:03.640 --> 00:33:08.044 And there was, there were the seven experimental groups which 500 00:33:08.044 --> 00:33:11.880 effectively were trying to look like the main sample. 501 00:33:12.840 --> 00:33:17.446 And then we had the much larger main sample which is what we 502 00:33:17.446 --> 00:33:20.240 compared each experimental group to. 503 00:33:20.600 --> 00:33:24.182 So the response rate from the experiment Group One against the 504 00:33:24.182 --> 00:33:26.400 control group, that's the main sample. 505 00:33:26.400 --> 00:33:30.361 We don't want those differences to be based on, you know if 506 00:33:30.361 --> 00:33:34.720 experiment Group One has a whole lot of a certain type of museum. 507 00:33:35.920 --> 00:33:38.411 We don't want it to be disproportionate, which could 508 00:33:38.411 --> 00:33:39.680 end up skewing the results. 509 00:33:41.480 --> 00:33:45.907 And then the the big measure that we would really like to 510 00:33:45.907 --> 00:33:50.641 implement and hopefully will come out of the outreach that we 511 00:33:50.641 --> 00:33:54.839 do this year, we would love to have a measure of size. 512 00:33:55.480 --> 00:33:59.485 We didn't have that from any of the available data sources, but 513 00:33:59.485 --> 00:34:03.617 we really want to make sure that we include as many small museums 514 00:34:03.617 --> 00:34:07.685 as we possibly can get a hold of and they can be a little bit of 515 00:34:07.685 --> 00:34:08.999 a challenge to reach. 516 00:34:09.000 --> 00:34:12.861 So I'm hoping in the future we'll be more able to make sure 517 00:34:12.861 --> 00:34:16.658 that that outreach is effective after we've reached out to 518 00:34:16.658 --> 00:34:19.040 everybody in the interim, very good. 519 00:34:20.280 --> 00:34:26.592 You know, I think I want to move this next set of questions 520 00:34:26.592 --> 00:34:32.800 moving forward into just tying up this pilot a little bit. 521 00:34:32.800 --> 00:34:36.215 There are some some juicy questions in here and I'm 522 00:34:36.215 --> 00:34:40.221 actually going to invite Helen or Matt or Jake to respond to 523 00:34:40.221 --> 00:34:43.440 this, who'd ever whoever would like to go first. 524 00:34:43.560 --> 00:34:48.470 But you know, there are questions about response data 525 00:34:48.470 --> 00:34:53.290 from this pilot survey and whether that will be made 526 00:34:53.290 --> 00:34:54.200 available. 527 00:34:54.200 --> 00:35:01.080 And I'd love to let Matt or Jake respond to that and then talk. 528 00:35:01.080 --> 00:35:03.627 And there are questions about then the intention of 529 00:35:03.627 --> 00:35:05.440 availability of data for the future. 530 00:35:08.440 --> 00:35:09.920 Matt's muted. 531 00:35:09.920 --> 00:35:11.040 Do you want me to handle that Matt? 532 00:35:12.200 --> 00:35:12.360 Sure. 533 00:35:14.280 --> 00:35:18.822 So because this was a pilot we weren't able to draw the kinds 534 00:35:18.822 --> 00:35:23.291 of statistics that the federal government requires you to be 535 00:35:23.291 --> 00:35:27.467 able to draw to report out the data, which is one of the 536 00:35:27.467 --> 00:35:28.200 downsides. 537 00:35:28.200 --> 00:35:32.028 The upside is that we were able to get, I feel, all of the 538 00:35:32.028 --> 00:35:35.985 process based information that we needed in order to run the 539 00:35:35.985 --> 00:35:36.440 survey. 540 00:35:37.560 --> 00:35:41.217 One of those things that we got is about how we want to 541 00:35:41.217 --> 00:35:44.680 disseminate the data when we do run the full survey. 542 00:35:44.720 --> 00:35:49.470 So when we do run the full survey, we plan on having a data 543 00:35:49.470 --> 00:35:54.300 portal where respondents in particular will be able to go in 544 00:35:54.300 --> 00:35:59.289 and check their data, check the data of other similar types of 545 00:35:59.289 --> 00:36:04.277 institutions, go and find the particular pieces of information 546 00:36:04.277 --> 00:36:09.186 that may be relevant, whether they're a museum administrator, 547 00:36:09.186 --> 00:36:14.254 member of the media, member of the public, and get really solid 548 00:36:14.254 --> 00:36:18.925 national statistics, regional statistics, discipline based 549 00:36:18.925 --> 00:36:23.360 statistics, whichever ones you might need for your use. 550 00:36:24.560 --> 00:36:27.734 In our research, we found that access to data was the number 551 00:36:27.734 --> 00:36:30.440 one driver for people to participate in the survey. 552 00:36:31.360 --> 00:36:34.120 And so we're really putting a very strong emphasis on that. 553 00:36:34.120 --> 00:36:38.048 We're also putting an emphasis on trying to get that data out 554 00:36:38.048 --> 00:36:39.760 within six months of field. 555 00:36:40.680 --> 00:36:45.200 So we're trying to be timely, we're trying to be complete and 556 00:36:45.200 --> 00:36:49.501 we're most of all trying to be useful so that everyone who 557 00:36:49.501 --> 00:36:54.167 participates really feels buy in and that the survey that we're 558 00:36:54.167 --> 00:36:58.760 running is a survey that they need in terms of response rates. 559 00:37:00.560 --> 00:37:09.440 We have the number that's been published the 17% national. 560 00:37:09.600 --> 00:37:14.602 We have done analysis on how that breaks out by for example 561 00:37:14.602 --> 00:37:15.520 discipline. 562 00:37:16.640 --> 00:37:21.425 I don't know that we're going to be doing a lot more reporting 563 00:37:21.425 --> 00:37:26.210 from that on the pilot, but when we do get to national data, I 564 00:37:26.210 --> 00:37:30.160 think that IMLS has a strong commitment to being as 565 00:37:30.160 --> 00:37:34.870 transparent as we can and trying to provide as much as we can 566 00:37:34.870 --> 00:37:39.199 within the boundaries of the federal statistical system. 567 00:37:39.320 --> 00:37:43.071 So, you know, we probably won't be able to send you the 568 00:37:43.071 --> 00:37:46.689 individual response from a specific museum that's not 569 00:37:46.689 --> 00:37:50.642 yours, but we will be able to provide you data that's been 570 00:37:50.642 --> 00:37:54.662 aggregated up to a level that protects your institution and 571 00:37:54.662 --> 00:37:58.280 anyone else's institution from getting sort of outed. 572 00:37:58.480 --> 00:38:03.271 Just briefly to add to what Jake said, this survey, IMLS is doing 573 00:38:03.271 --> 00:38:06.320 this survey and putting the public first. 574 00:38:07.000 --> 00:38:10.240 The data is going to be free and accessible. 575 00:38:10.920 --> 00:38:13.440 There will be no special permissions. 576 00:38:13.440 --> 00:38:15.720 We're not going to be trying to make money on this. 577 00:38:15.720 --> 00:38:18.880 This is to improve the larger public well-being. 578 00:38:19.320 --> 00:38:23.253 Over the next 12 months, I anticipate that we'll be doing a 579 00:38:23.253 --> 00:38:27.187 lot of additional research and engaging people in different 580 00:38:27.187 --> 00:38:31.120 museums at different types of museums about how to make the 581 00:38:31.120 --> 00:38:35.250 data and the associated products of that data such as reports, 582 00:38:35.250 --> 00:38:39.118 analytical tools, etcetera, most meaningful and useful and 583 00:38:39.118 --> 00:38:39.839 actionable. 584 00:38:41.200 --> 00:38:43.945 That's a that's a great point and if we do reach out, I hope 585 00:38:43.945 --> 00:38:44.800 you'll participate. 586 00:38:45.240 --> 00:38:49.061 So if we reach out for people to help us with our user experience 587 00:38:49.061 --> 00:38:52.651 testing, we really hope that people will answer the call. And 588 00:38:52.651 --> 00:38:56.298 our approach is going to be able to be much more expansive and 589 00:38:56.298 --> 00:38:57.920 organic in this launch phase 590 00:38:57.920 --> 00:39:00.671 than it was in this pilot phase, which was a pretty tightly 591 00:39:00.671 --> 00:39:01.680 controlled experiment. 592 00:39:02.760 --> 00:39:05.736 Helen, I would like to move to you because there are a number 593 00:39:05.736 --> 00:39:06.600 of questions here. 594 00:39:06.600 --> 00:39:10.979 We know, you and I both know, being in the Office of Museum 595 00:39:10.979 --> 00:39:15.067 Services, you have been the program officer for many of 596 00:39:15.067 --> 00:39:19.154 these efforts, that there are a number of existing data 597 00:39:19.154 --> 00:39:23.680 collection activities in the field from the different service 598 00:39:23.680 --> 00:39:26.599 organizations, consultancies, etcetera. 599 00:39:26.920 --> 00:39:31.217 And we have a number of questions about where we see the 600 00:39:31.217 --> 00:39:33.480 NMS fitting in that landscape. 601 00:39:33.480 --> 00:39:37.277 And so I'd love for you to respond to that set of questions 602 00:39:37.277 --> 00:39:37.720 please. 603 00:39:38.800 --> 00:39:43.310 I think that's a great question and one that has future legs and 604 00:39:43.310 --> 00:39:47.266 a real future to talk about and to think about how these 605 00:39:47.266 --> 00:39:47.960 integrate. 606 00:39:49.120 --> 00:39:52.191 We we looked very carefully at all the other data collection 607 00:39:52.191 --> 00:39:54.960 that was going on as part of the preliminary research. 608 00:39:54.960 --> 00:39:59.036 And we tried hard when we did duplicate a question or or or 609 00:39:59.036 --> 00:40:03.316 ask this about the same topic to use language that people were 610 00:40:03.316 --> 00:40:07.121 familiar with to structure questions in similar ways to 611 00:40:07.121 --> 00:40:09.160 other data collection efforts. 612 00:40:10.440 --> 00:40:15.461 And we think as we continue this effort, we'd like very much to 613 00:40:15.461 --> 00:40:16.560 start, as this 614 00:40:17.000 --> 00:40:24.578 material becomes, and the data becomes, accessible to everybody 615 00:40:24.578 --> 00:40:28.960 to be able to meld efforts together. 616 00:40:28.960 --> 00:40:32.371 So that you can focus on the kinds of things that your 617 00:40:32.371 --> 00:40:36.340 discipline that you need to find out from your discipline while 618 00:40:36.340 --> 00:40:40.371 we're doing some core work that will be accessible and available 619 00:40:40.371 --> 00:40:41.239 for everybody. 620 00:40:41.640 --> 00:40:44.400 So you know that's the that's the hope for the future. 621 00:40:45.240 --> 00:40:48.168 It is the first time that all the disciplines are asked this 622 00:40:48.168 --> 00:40:48.600 together. 623 00:40:48.600 --> 00:40:50.960 So we can compare on the same questions. 624 00:40:51.440 --> 00:40:54.341 So I think it will be very valuable and we really look 625 00:40:54.341 --> 00:40:57.506 forward to collaborating with the other associations of the 626 00:40:57.506 --> 00:40:59.880 other data collection efforts in the future. 627 00:41:02.120 --> 00:41:02.680 Thank you. 628 00:41:03.520 --> 00:41:08.809 You know, there are a number of questions about access to 629 00:41:08.809 --> 00:41:14.008 different parts of both the survey pilot and then future 630 00:41:14.008 --> 00:41:14.920 data sets. 631 00:41:15.600 --> 00:41:19.462 I'll let everybody know that the current questionnaire is 632 00:41:19.462 --> 00:41:23.725 actually available right now on the IMLS website and so you can 633 00:41:23.725 --> 00:41:27.787 download, if you search on our website imls.gov for National 634 00:41:27.787 --> 00:41:28.720 Museum survey, 635 00:41:29.520 --> 00:41:34.098 there is a page for that survey for this effort and a link to 636 00:41:34.098 --> 00:41:37.200 the actual questionnaire and definitions. 637 00:41:37.200 --> 00:41:39.040 So that is already available. 638 00:41:40.600 --> 00:41:45.120 I do want to move forward. 639 00:41:45.120 --> 00:41:51.955 There are some future facing questions about more detailed 640 00:41:51.955 --> 00:41:58.790 and topical investigations beyond our current foundational 641 00:41:58.790 --> 00:42:00.760 set of questions. 642 00:42:00.760 --> 00:42:04.027 And so Matt or Jake, if you'd like to speak briefly to that 643 00:42:04.027 --> 00:42:07.240 'cause then I have a juicy question after they've all been 644 00:42:07.240 --> 00:42:09.800 juicy, but a really juicy question after this. 645 00:42:09.800 --> 00:42:13.640 Jake, I'll let you take the lead on additional questions. 646 00:42:15.840 --> 00:42:20.177 So one of the things that we want to do methodologically is 647 00:42:20.177 --> 00:42:24.659 make sure that the main body of the survey stays as stable as 648 00:42:24.659 --> 00:42:28.563 possible so that variations in response aren't due to 649 00:42:28.563 --> 00:42:31.600 variations in the instrument effectively. 650 00:42:32.440 --> 00:42:36.864 So one of our ideas moving forward is that we'll probably 651 00:42:36.864 --> 00:42:41.823 run what we're calling offshoot topical surveys, which will give 652 00:42:41.823 --> 00:42:46.324 us a special place to put surveys that cover material that 653 00:42:46.324 --> 00:42:51.359 either doesn't make sense to ask annually or material that may be 654 00:42:51.359 --> 00:42:55.326 responsive to a specific situation that occurs, for 655 00:42:55.326 --> 00:42:58.760 example, when the COVID outbreak took place. 656 00:42:58.760 --> 00:43:04.105 We might use an offshoot topical survey, or for issues that only 657 00:43:04.105 --> 00:43:09.121 affect a specific subset, so maybe a specific discipline, we 658 00:43:09.121 --> 00:43:12.000 may do an offshoot topical survey. 659 00:43:12.560 --> 00:43:18.849 Those are going to be, as we currently envision them, most 660 00:43:18.849 --> 00:43:25.671 likely to be put out right after the survey is done to a select 661 00:43:25.671 --> 00:43:31.214 sub sample of respondents so that they don't burden 662 00:43:31.214 --> 00:43:32.280 everybody. 663 00:43:32.280 --> 00:43:35.395 It's not a matter of, like, this year the survey is this much 664 00:43:35.395 --> 00:43:36.400 longer for everyone. 665 00:43:36.840 --> 00:43:42.312 It'll only be on a a smaller group and hopefully will not be 666 00:43:42.312 --> 00:43:47.515 overly burdensome, but will allow us to hit those topical 667 00:43:47.515 --> 00:43:52.898 issues as useful or needed kind of as we move forward while 668 00:43:52.898 --> 00:43:57.653 allowing us to maintain the fidelity of the original 669 00:43:57.653 --> 00:43:58.640 instrument. 670 00:44:00.040 --> 00:44:00.440 Great. 671 00:44:00.640 --> 00:44:05.268 And I there's some follow on questions and it it melds a 672 00:44:05.268 --> 00:44:10.221 little bit with Helen's response about you know, do we think 673 00:44:10.221 --> 00:44:14.200 about this effort replacing other field efforts. 674 00:44:15.040 --> 00:44:19.040 I might be so bold as to maybe answer that question if I could. 675 00:44:19.800 --> 00:44:21.640 I think our intention is no. 676 00:44:21.640 --> 00:44:26.553 And I would actually point folks if you're not familiar with the 677 00:44:26.553 --> 00:44:31.164 work that IMLS has done for more than 40 years on the Public 678 00:44:31.164 --> 00:44:35.321 Library survey, the PLS, which is a very rigorous data 679 00:44:35.321 --> 00:44:39.630 collection on our sibling community here at IMLS I think 680 00:44:39.630 --> 00:44:44.316 you can be assured that there is still plenty of field driven 681 00:44:44.316 --> 00:44:49.079 data collection in the Public Library sector, but it is driven 682 00:44:49.079 --> 00:44:51.120 and accelerated by the PLS. 683 00:44:51.120 --> 00:44:54.483 And Matt, you sat up straight if you wanted to share anything 684 00:44:54.483 --> 00:44:55.080 about that. 685 00:44:55.240 --> 00:44:58.960 Yeah, I would Now using Jake's example of COVID because we've 686 00:44:58.960 --> 00:45:02.320 been collecting the Public Library survey for 40 years, 687 00:45:03.000 --> 00:45:08.040 when we went to introduce two years of special topical surveys 688 00:45:08.040 --> 00:45:13.160 about COVID, it was supplemented by a wealth of data going back 689 00:45:13.160 --> 00:45:17.800 40 years on key core parts of public libraries, finances, 690 00:45:17.800 --> 00:45:20.680 staffing, other operational issues. 691 00:45:21.160 --> 00:45:24.928 That allowed us to get the real full picture of the extent of 692 00:45:24.928 --> 00:45:25.840 the disruption. 693 00:45:26.160 --> 00:45:29.588 We had something similar a decade earlier, 15 years earlier 694 00:45:29.588 --> 00:45:32.502 when we were looking at the aftermath of the Great 695 00:45:32.502 --> 00:45:36.102 Recession, particularly in how that shuffled federal and state 696 00:45:36.102 --> 00:45:38.960 spending patterns for those, for those libraries. 697 00:45:39.360 --> 00:45:42.942 So I would anticipate a very similar strategy here for the 698 00:45:42.942 --> 00:45:46.828 museum sector and keeping those core questions that Jake talked 699 00:45:46.828 --> 00:45:50.349 about so that we could be looking at things over time and 700 00:45:50.349 --> 00:45:51.200 for our space. 701 00:45:53.600 --> 00:45:54.040 Thank you. 702 00:45:55.000 --> 00:45:59.750 So I do want to move to the juicy question that I alluded to 703 00:45:59.750 --> 00:46:04.501 earlier that has come up in a couple of different ways and I 704 00:46:04.501 --> 00:46:09.019 think you can, I'll offer both adjectives and you all can 705 00:46:09.019 --> 00:46:11.199 respond as you are so moved. 706 00:46:11.200 --> 00:46:15.892 But there were a couple of questions around what were the 707 00:46:15.892 --> 00:46:20.827 most surprising or provocative findings from this experience 708 00:46:20.827 --> 00:46:21.960 that we found. 709 00:46:23.360 --> 00:46:25.800 I'll let you all cogitate and whoever would like to go first. 710 00:46:28.360 --> 00:46:34.467 I mean one of one of the things that I felt like we needed to be 711 00:46:34.467 --> 00:46:40.010 particularly responsive to and was... I don't know if it's 712 00:46:40.010 --> 00:46:45.460 not... I don't know if it's surprising is necessarily the 713 00:46:45.460 --> 00:46:45.930 word. 714 00:46:45.930 --> 00:46:51.208 But very early was driven home to me through our research and 715 00:46:51.208 --> 00:46:56.400 outreach to the field were the great differences between the 716 00:46:56.400 --> 00:47:01.253 different disciplines and between the different sizes of 717 00:47:01.253 --> 00:47:02.360 institutions. 718 00:47:03.680 --> 00:47:08.493 Understanding in as deep a way as we can, where those 719 00:47:08.493 --> 00:47:14.198 differences occur and how they affect people's responses to our 720 00:47:14.198 --> 00:47:16.160 outreach was a very... 721 00:47:16.880 --> 00:47:20.606 I mean it was provocative in that it really drove home how 722 00:47:20.606 --> 00:47:21.680 important it was. 723 00:47:21.680 --> 00:47:24.624 For example, a lot of institutions aren't going to 724 00:47:24.624 --> 00:47:26.760 call themselves necessarily museums. 725 00:47:26.760 --> 00:47:30.606 A Botanical Garden, if they get outreach for a National Museum 726 00:47:30.606 --> 00:47:33.720 survey, may not know that we want to talk to them. 727 00:47:34.680 --> 00:47:39.388 And so making sure that we do everything we can to reach those 728 00:47:39.388 --> 00:47:40.360 institutions. 729 00:47:40.360 --> 00:47:44.408 You know, if you look at any of our lists, they're always listed 730 00:47:44.408 --> 00:47:48.331 first because we want to make sure that if somebody's perusing 731 00:47:48.331 --> 00:47:50.200 the paragraph, they stick out. 732 00:47:50.720 --> 00:47:54.472 So there's a lot of intentionality around size and 733 00:47:54.472 --> 00:47:59.181 discipline that came out of our research that I'm sure those in 734 00:47:59.181 --> 00:48:03.227 the field might know, but that we really work to be as 735 00:48:03.227 --> 00:48:04.920 sensitive to as we can. 736 00:48:06.800 --> 00:48:09.720 Thanks, Jake, Matt, Helen. 737 00:48:10.080 --> 00:48:15.346 Yeah, it's not surprising, but I'll say challenging the finding 738 00:48:15.346 --> 00:48:20.118 that financial data was a real challenge for many museums 739 00:48:20.118 --> 00:48:24.480 participate that on its surface doesn't surprise me. 740 00:48:25.240 --> 00:48:29.687 What I'm stuck with is still the uncertainty of what it means for 741 00:48:29.687 --> 00:48:32.720 going forward to try to lower those hurdles. 742 00:48:33.160 --> 00:48:36.618 Because I think that that data will be so important and 743 00:48:36.618 --> 00:48:40.448 valuable for all museums to be able to use to benchmark their 744 00:48:40.448 --> 00:48:44.215 own growth and development and to be able to tell collective 745 00:48:44.215 --> 00:48:45.760 stories to policy makers. 746 00:48:49.240 --> 00:48:54.582 And I'll add on to that because we did have a question about how 747 00:48:54.582 --> 00:48:59.924 the financial data was collected and it was self-reported and we 748 00:48:59.924 --> 00:49:05.101 didn't collect it from any other source but we did try to link 749 00:49:05.101 --> 00:49:10.279 what we were asking for to 990 items so that if you filled out 750 00:49:10.279 --> 00:49:14.800 your 990 you know where to how to answer the question. 751 00:49:15.040 --> 00:49:18.427 We tried to structure it the way others might ask for financial 752 00:49:18.427 --> 00:49:21.868 information so that you weren't digging up different numbers for 753 00:49:21.868 --> 00:49:23.880 different surveys you're filling out. 754 00:49:24.360 --> 00:49:28.254 So that is a big question that how to make it even easier or 755 00:49:28.254 --> 00:49:31.000 what's how how do we get to those answers? 756 00:49:31.000 --> 00:49:33.480 Because that data is super, super important. 757 00:49:34.360 --> 00:49:38.463 And what, what's most surprising, I guess not 758 00:49:38.463 --> 00:49:44.083 surprising, but sobering, was familiarity with IMLS, which you 759 00:49:44.083 --> 00:49:46.760 know is going to be important. 760 00:49:46.760 --> 00:49:49.747 We have to be that the organization that's asking for 761 00:49:49.747 --> 00:49:51.960 all this information should be trusted. 762 00:49:51.960 --> 00:49:54.240 And if you don't know who we are, it's hard to trust us. 763 00:49:54.240 --> 00:49:58.080 So we've got some heavy lifting to do in that regard. 764 00:49:58.440 --> 00:50:02.329 So that the part of the field that isn't familiar with us will 765 00:50:02.329 --> 00:50:04.120 feel welcome and comfortable. 766 00:50:06.720 --> 00:50:07.440 Laura, how about you? 767 00:50:08.760 --> 00:50:14.106 Oh, I feel like the jaded person here as a former association 768 00:50:14.106 --> 00:50:18.160 person who has mounted these data collections. 769 00:50:18.320 --> 00:50:21.200 I felt like this two years was I told you so. 770 00:50:21.200 --> 00:50:25.584 I see it's not just us, but I think that the opportunity, all 771 00:50:25.584 --> 00:50:29.686 these challenges I think are lived experiences from folks 772 00:50:29.686 --> 00:50:34.000 who've been doing this kind of data collection in the field. 773 00:50:34.000 --> 00:50:39.279 And this opportunity gave us a chance to modify that as a 774 00:50:39.279 --> 00:50:42.920 public serving agency and organization. 775 00:50:42.920 --> 00:50:49.105 You know, we have a existential mandate to be stewards of this 776 00:50:49.105 --> 00:50:52.640 kind of data at the national level. 777 00:50:52.640 --> 00:50:57.796 And so I think bringing together a lot of lived experience from 778 00:50:57.796 --> 00:51:02.792 museums to this work and even documenting what the challenges 779 00:51:02.792 --> 00:51:07.465 are are going to be very cathartic for the field and help 780 00:51:07.465 --> 00:51:09.480 us move forward together. 781 00:51:11.160 --> 00:51:18.731 I also think that it's a call to some solidarity across our 782 00:51:18.731 --> 00:51:21.760 differences for museums. 783 00:51:21.840 --> 00:51:27.463 If we want to be able to tell, particularly a policy narrative 784 00:51:27.463 --> 00:51:32.640 in times of crisis, as well as times of bounty, about our 785 00:51:32.640 --> 00:51:37.906 presence and impact of this sector, there is going to need 786 00:51:37.906 --> 00:51:42.725 to be some uniformity and in some basic reporting and 787 00:51:42.725 --> 00:51:43.440 figures. 788 00:51:43.480 --> 00:51:49.252 And hopefully the future of this effort can help provide a clear 789 00:51:49.252 --> 00:51:54.581 path not to define it, but provide a common reference point 790 00:51:54.581 --> 00:51:58.399 to to keep us all iterating going forward. 791 00:51:59.760 --> 00:52:02.200 I will say that I am. 792 00:52:02.200 --> 00:52:07.241 I was happily surprised at the overall positive feedback that 793 00:52:07.241 --> 00:52:12.120 we received for the effort and the amount of engagement and 794 00:52:12.120 --> 00:52:17.323 interest that there's been in a time of such a busyness and and 795 00:52:17.323 --> 00:52:22.040 a crowded landscape for museum leaders and practitioners. 796 00:52:22.040 --> 00:52:26.715 So I think there's a real positive, positive story moving 797 00:52:26.715 --> 00:52:27.360 forward. 798 00:52:29.680 --> 00:52:36.680 I am looking at our time. 799 00:52:37.080 --> 00:52:43.935 I'm also checking in on our energy and I'd love to give my 800 00:52:43.935 --> 00:52:46.840 colleagues a last chance. 801 00:52:46.840 --> 00:52:54.250 I know that you can see Matt, I think that we answered that 802 00:52:54.250 --> 00:52:56.720 question if you can. 803 00:52:56.800 --> 00:53:00.962 But if you do see any other stragglers that we didn't 804 00:53:00.962 --> 00:53:05.664 capture or you want to share some final thoughts as we close 805 00:53:05.664 --> 00:53:07.360 out our time together. 806 00:53:12.560 --> 00:53:13.680 I'd just like to give another plug. 807 00:53:13.680 --> 00:53:19.784 If if you do see us looking for helpers between now and the full 808 00:53:19.784 --> 00:53:22.320 survey, please participate. 809 00:53:23.200 --> 00:53:27.160 And then I saw one or two questions about us presenting. 810 00:53:27.160 --> 00:53:32.072 We're going to be out there and presenting. We'll be happy to 811 00:53:32.072 --> 00:53:36.984 talk to people when we do that about what we've done here and 812 00:53:36.984 --> 00:53:39.520 what we're doing moving forward. 813 00:53:40.440 --> 00:53:46.374 The last thing I guess would be, if you haven't, I think I 814 00:53:46.374 --> 00:53:52.207 answered one question in the chat itself about having not 815 00:53:52.207 --> 00:53:57.839 been contacted or having not participated in the pilot. 816 00:53:58.280 --> 00:53:59.800 While the pilot is over. 817 00:53:59.800 --> 00:54:03.285 We do want everybody we can possibly get to participate when 818 00:54:03.285 --> 00:54:04.600 we run the full survey. 819 00:54:05.360 --> 00:54:10.207 And so if anybody is in a situation where you're not sure 820 00:54:10.207 --> 00:54:14.553 that we have the right contact information for your 821 00:54:14.553 --> 00:54:15.640 institution, 822 00:54:16.480 --> 00:54:20.339 I would be thrilled if you would send emails to nms@imls.gov to 823 00:54:20.339 --> 00:54:24.077 pass along that information and we'll make sure it, you know, 824 00:54:24.077 --> 00:54:27.756 gets in the right place so that you get the outreach for the 825 00:54:27.756 --> 00:54:28.480 full survey. 826 00:54:34.040 --> 00:54:35.400 Helen, any parting thoughts? 827 00:54:37.480 --> 00:54:41.709 Well, echoing Jake that our engagement with the field is not 828 00:54:41.709 --> 00:54:43.720 at all over after this pilot. 829 00:54:44.080 --> 00:54:48.960 We've got a lot of work to do between now and the 2025 launch. 830 00:54:48.960 --> 00:54:53.938 So we're going to be knocking on doors and associations are going 831 00:54:53.938 --> 00:54:57.560 to be with us, we hope and others in the field. 832 00:54:57.560 --> 00:55:00.784 We really are relying on you to make it something that works for 833 00:55:00.784 --> 00:55:01.280 everybody. 834 00:55:04.000 --> 00:55:07.586 And Matt, any final thoughts before final housekeeping, I'll 835 00:55:07.586 --> 00:55:09.880 just say that we we have an open door. 836 00:55:10.280 --> 00:55:12.040 We we are here to learn and share. 837 00:55:12.600 --> 00:55:15.920 So please don't hesitate to connect us. 838 00:55:15.920 --> 00:55:19.840 And you can already see what a force Jake and Helen are and we 839 00:55:19.840 --> 00:55:23.324 are fully confident they can answer everything and I'll 840 00:55:23.324 --> 00:55:26.560 conflate it and then Laura can come and correct it. 841 00:55:28.840 --> 00:55:29.920 Well, thank you for that. 842 00:55:29.920 --> 00:55:34.323 I think Dorothy, we can move to our last slide which is the most 843 00:55:34.323 --> 00:55:38.592 important one, which is really expressing our thanks to all of 844 00:55:38.592 --> 00:55:40.760 you for joining us on this call. 845 00:55:41.320 --> 00:55:47.599 As a reminder, if you want to be in touch with IMLS in general 846 00:55:47.599 --> 00:55:52.683 for this effort and future efforts, you can always 847 00:55:52.683 --> 00:55:57.567 subscribe to our mailing list via our website at 848 00:55:57.567 --> 00:55:59.960 imls.gov/news/subscribe. 849 00:56:00.400 --> 00:56:05.411 If you would like to follow up directly around the National 850 00:56:05.411 --> 00:56:10.590 Museum survey with some of the more fine grained details that 851 00:56:10.590 --> 00:56:15.768 Jake mentioned, you can e-mail us at N ms@imls.gov The report 852 00:56:15.768 --> 00:56:20.863 that we highlighted today is actually available right now on 853 00:56:20.863 --> 00:56:22.200 IML s s website. 854 00:56:22.200 --> 00:56:26.200 There is a press release at the very top of our home page and 855 00:56:26.200 --> 00:56:28.200 you can access that report now. 856 00:56:29.040 --> 00:56:35.640 We look forward to engaging with all of you in the future to help 857 00:56:35.640 --> 00:56:41.540 us ensure that our full NMS survey data collection in 2025 858 00:56:41.540 --> 00:56:42.840 is a success. 859 00:56:43.480 --> 00:56:48.221 And we are also going to be looking for field experts to 860 00:56:48.221 --> 00:56:50.800 join us in advisory capacities. 861 00:56:50.800 --> 00:56:54.467 And so again, if that is something that you are 862 00:56:54.467 --> 00:56:58.745 interested in or have expertise, please do e-mail us at 863 00:56:58.745 --> 00:57:03.253 nms@imls.gov In the meantime, please look for more updates 864 00:57:03.253 --> 00:57:05.240 from us and solicitations. 865 00:57:05.520 --> 00:57:08.549 Thank you again for your partnership and your interest 866 00:57:08.549 --> 00:57:11.853 and we look forward to talking to you about the full survey 867 00:57:11.853 --> 00:57:12.680 launch in 2025. 868 00:57:13.920 --> 00:57:14.520 Thank you. 869 00:57:15.640 --> 00:57:16.360 Thank you everybody.