1 00:00:00,940 --> 00:00:04,860 Module, in terms of the Big Data Architecture: Fundamental lecture here at 2 00:00:04,860 --> 00:00:08,510 the JPL-Caltech Virtual Summer School on Big Data Analytics. 3 00:00:08,510 --> 00:00:09,550 I'm Chris Mattmann. 4 00:00:09,550 --> 00:00:12,060 Thanks for sticking with me here to the end. 5 00:00:12,060 --> 00:00:17,340 So, we've done a sort of a hitchhiker's guide really fast paced 6 00:00:17,340 --> 00:00:21,280 history of the sort of research and software architecture, 7 00:00:21,280 --> 00:00:24,180 software engineering, how some of that might apply to big data. 8 00:00:24,180 --> 00:00:25,880 We've covered things like components and 9 00:00:25,880 --> 00:00:29,710 connectors, core architectural elements, styles, patterns, reference architecture. 10 00:00:29,710 --> 00:00:33,800 We've covered ways of capturing these sort of core elements in 11 00:00:33,800 --> 00:00:37,290 terms of architectural models, why you may want to visualize those. 12 00:00:37,290 --> 00:00:40,010 We've talked about the difference between code, as implemented and 13 00:00:40,010 --> 00:00:44,480 what can happen when code drifts or erodes from the actual architecture and 14 00:00:44,480 --> 00:00:48,270 design and what we can do to kind of deal with that, architecture recovery. 15 00:00:48,270 --> 00:00:50,390 And so here, in the final module, 16 00:00:50,390 --> 00:00:53,510 going to do an actual case study in the realm of grid, grid computing. 17 00:00:53,510 --> 00:00:56,090 To show you some of the value of software architecture and, and 18 00:00:56,090 --> 00:00:59,200 how you can actually apply it Within the context of big data. 19 00:00:59,200 --> 00:01:03,270 And your guy's applications and then maybe wrap up with some conclusions. 20 00:01:03,270 --> 00:01:06,540 So this, this lecture actually covers a specific example that's going to 21 00:01:06,540 --> 00:01:10,440 bring together soft architecture connection and Big data. 22 00:01:10,440 --> 00:01:13,560 So, you guys may be familiar or you might not be with the domain of Greek computing 23 00:01:13,560 --> 00:01:17,640 before clouds and big data and things like that, there was Greek computing. 24 00:01:17,640 --> 00:01:20,910 And the goal, as sort of defined by the fathers and, 25 00:01:20,910 --> 00:01:24,430 you know, forefathers and mothers of grid computing. 26 00:01:24,430 --> 00:01:27,750 Was to provide an infrastructure and an architecture for, for basically 27 00:01:27,750 --> 00:01:32,900 seamlessly bringing together resources for data and computer across organizations. 28 00:01:32,900 --> 00:01:40,090 To create virtual organizations that was the goal of grid computing systems circa, 29 00:01:40,090 --> 00:01:42,970 you know, around 2001, 2002 which was sort of the. 30 00:01:42,970 --> 00:01:47,470 Initial hot heyday that cloud computing era if you will of grid computing. 31 00:01:47,470 --> 00:01:48,880 Okay. So we were studying those in 32 00:01:48,880 --> 00:01:51,400 my research group around the time. 33 00:01:51,400 --> 00:01:55,430 We were studying as we found out there were two fundamental types of 34 00:01:55,430 --> 00:01:58,890 grid computing systems corresponding to the two fundamental types of 35 00:01:58,890 --> 00:02:00,320 resources that existed at the day. 36 00:02:00,320 --> 00:02:05,710 There were data grids which were really focused on federating data resources 37 00:02:05,710 --> 00:02:09,160 across institutions, searching them in sort of a nice and easy way. 38 00:02:09,160 --> 00:02:11,960 And then, there were computational grids, which were focused on how do 39 00:02:11,960 --> 00:02:16,530 we run jobs at your institution and my institution, and share sort of identities. 40 00:02:16,530 --> 00:02:18,990 And only let these science groups do these jobs, and 41 00:02:18,990 --> 00:02:22,170 let these other ones use these resources, and so on and so forth. 42 00:02:22,170 --> 00:02:25,090 So there are two predominant types of grid computing systems, data grids and 43 00:02:25,090 --> 00:02:26,330 computational grids from there. 44 00:02:27,630 --> 00:02:31,950 So, we were actually writing a paper on trying to understand the architecture of 45 00:02:31,950 --> 00:02:34,990 grid computing, trying to moosh them together, because we said, look, 46 00:02:34,990 --> 00:02:37,690 there were these sort of canonical papers on grid computing. 47 00:02:37,690 --> 00:02:41,110 There was the grid's anatomy, the anatomy of the grid, 48 00:02:41,110 --> 00:02:43,490 which was written by Foster and Kesselman and Alf. 49 00:02:43,490 --> 00:02:46,120 And then there's this paper called the Physiology of the Grid. 50 00:02:46,120 --> 00:02:46,880 The anatomy, there's more, 51 00:02:46,880 --> 00:02:50,070 the architecture, the physiology was more how do we implement this thing. 52 00:02:50,070 --> 00:02:54,480 Also its written by Foster, and Kesselman and Steven Tuck and a number of people. 53 00:02:54,480 --> 00:02:58,720 And so we were writing a paper on our grid technology at the time that we felt, 54 00:02:58,720 --> 00:03:03,640 kind of consistently met this sort of requirement of sharing data. 55 00:03:03,640 --> 00:03:06,100 We're sharing compute resources and things like that and 56 00:03:06,100 --> 00:03:08,230 we were writing in the context of mobile computing. 57 00:03:08,230 --> 00:03:12,180 We're building this project called Glide which was a technology for, 58 00:03:12,180 --> 00:03:14,180 it's like a mobile grid for sharing data and 59 00:03:14,180 --> 00:03:18,580 compute on wireless PDAs of the time before cell phones were super poplar. 60 00:03:18,580 --> 00:03:20,110 We submitted this paper to workshop. 61 00:03:20,110 --> 00:03:20,970 We got back a review. 62 00:03:20,970 --> 00:03:23,990 The workshop basically said the grid technology you're studying is nothing more 63 00:03:23,990 --> 00:03:26,190 than a simple object oriented framework. 64 00:03:26,190 --> 00:03:27,620 And so, we scratched our head and 65 00:03:27,620 --> 00:03:30,020 we said you know was the reviewer right about this? 66 00:03:30,020 --> 00:03:33,150 I mean, a simple object oriented framework it really seems to fit at 67 00:03:33,150 --> 00:03:38,500 least the definition of competing in terms of these anatomy and the physiology paper. 68 00:03:38,500 --> 00:03:41,530 You know, and the kind of current literature at the time. 69 00:03:41,530 --> 00:03:43,080 So we're wondering is, you're right. 70 00:03:43,080 --> 00:03:44,120 How do they know that? 71 00:03:44,120 --> 00:03:45,900 Okay. And we were also kind of annoyed, 72 00:03:45,900 --> 00:03:47,530 you know, for the review that we got. 73 00:03:47,530 --> 00:03:51,350 This Alleged Object Oriented Framework was the 2003 Runner-up NASA Software of 74 00:03:51,350 --> 00:03:52,810 the Year, okay? 75 00:03:52,810 --> 00:03:56,050 So, we basically started doing a little bit more research and 76 00:03:56,050 --> 00:03:58,510 looking around at the time which you guys might do here. 77 00:03:58,510 --> 00:04:00,100 You know, maybe as PhD students or 78 00:04:00,100 --> 00:04:03,490 post docs or as petitioners in the realm of big data. 79 00:04:03,490 --> 00:04:07,230 And we started to look at the research literature in terms of architectures and 80 00:04:07,230 --> 00:04:07,960 grid computing, and 81 00:04:07,960 --> 00:04:11,510 we basically found out that was, and still kind of persists to this day. 82 00:04:11,510 --> 00:04:14,110 That little was sort of known in terms of the architecture of 83 00:04:14,110 --> 00:04:16,690 the as implemented grid computing technologies of the day. 84 00:04:16,690 --> 00:04:18,590 There was a study by Finkelstein. 85 00:04:18,590 --> 00:04:21,040 And all of that was published in the Journal of Great Computing in which 86 00:04:21,040 --> 00:04:23,720 they were trying to look at things like requirements, and so forth. 87 00:04:23,720 --> 00:04:26,510 But, they didn't really know much about the architecture of it, the components, 88 00:04:26,510 --> 00:04:29,950 the connectors, the stuff that we're talking about here, in the summer school. 89 00:04:29,950 --> 00:04:30,840 Okay? 90 00:04:30,840 --> 00:04:32,070 Little was known about that. 91 00:04:32,070 --> 00:04:36,080 And so, you know, there's this big risk if you, you know, know a little bit 92 00:04:36,080 --> 00:04:39,870 about the requirements, but you're not exactly sure of how that maps. 93 00:04:39,870 --> 00:04:42,590 You know, and how that went through the design process, and 94 00:04:42,590 --> 00:04:46,140 how that went through the architectural process to the eventual implementation. 95 00:04:46,140 --> 00:04:48,570 It's a big risk of architectural drift or erosion, 96 00:04:48,570 --> 00:04:53,170 as we were just talking about, you know, in the second module of this course. 97 00:04:53,170 --> 00:04:53,670 Right? 98 00:04:53,670 --> 00:04:56,690 It seems like grid technologies all generally claim to 99 00:04:56,690 --> 00:04:58,650 have the same capabilities, but. 100 00:04:58,650 --> 00:05:01,030 They seem to be implemented in vastly different ways, 101 00:05:01,030 --> 00:05:04,380 have different sort of requirements that their satisfying and so on and so forth. 102 00:05:04,380 --> 00:05:07,890 Maybe evidenced by the reviewer that was reviewing our paper at the time. 103 00:05:09,160 --> 00:05:13,310 So we develop an approach that's base on what's your learning here in terms of 104 00:05:13,310 --> 00:05:14,200 software architecture. 105 00:05:14,200 --> 00:05:16,200 And related to big data and so forth, 106 00:05:16,200 --> 00:05:19,020 because grid computing errors, arguably within that realm. 107 00:05:19,020 --> 00:05:22,390 We developed an approach to studying, initially five, and 108 00:05:22,390 --> 00:05:27,290 eventually, almost 20, grid computing technologies and their implementations. 109 00:05:27,290 --> 00:05:30,900 And we developed an architectural approach to studying, these technologies, and I'll, 110 00:05:30,900 --> 00:05:31,540 and I'll talk about. 111 00:05:31,540 --> 00:05:33,800 So we had to select software codes. 112 00:05:33,800 --> 00:05:35,070 We decided to select, 113 00:05:35,070 --> 00:05:38,420 great computing technologies that were object oriented simply because some of 114 00:05:38,420 --> 00:05:43,320 the techniques we use to recover the the architecture were real amenable to that. 115 00:05:43,320 --> 00:05:46,230 We wanted them to be open source because we want to be able to look at the source 116 00:05:46,230 --> 00:05:50,670 code, the source code being the most, like I said, up to date version of the software 117 00:05:50,670 --> 00:05:55,660 and most sort of canonical representation of your system sort of as it evolves. 118 00:05:55,660 --> 00:05:59,330 And, we also wanted to use, kind of off the shelf, you know, systems, that seem to 119 00:05:59,330 --> 00:06:03,210 have a big user base, So, these were the initial five that we studied. 120 00:06:03,210 --> 00:06:06,700 We studied Globus, which was the de facto grid technology at the time, OODT, 121 00:06:06,700 --> 00:06:11,060 DSpace, GLIDE, our own technology because we wanted to put it to the test for 122 00:06:11,060 --> 00:06:12,790 building, and something called JCGrid. 123 00:06:12,790 --> 00:06:15,640 And that was where the initial five technologies that we studied. 124 00:06:15,640 --> 00:06:21,010 In 2005, 2006, having since expanded the study to include over, like I said, 125 00:06:21,010 --> 00:06:23,780 20 grid technologies circa 2009, 2010, 126 00:06:23,780 --> 00:06:28,730 including Hadoob, things like Wings, Pegasus and so on and so forth. 127 00:06:28,730 --> 00:06:30,970 So more modern technologies. 128 00:06:30,970 --> 00:06:35,050 So our approach was to initially do sort of a detailed literature review, and 129 00:06:35,050 --> 00:06:38,320 to try and look at the four seminal grid papers at the time. 130 00:06:38,320 --> 00:06:41,950 Just this data grid paper, by by Ann Chervenak, 131 00:06:41,950 --> 00:06:46,170 as well as three papers from Foster and Kesselman on the grid's anatomy, 132 00:06:46,170 --> 00:06:50,540 its physiology, and a checklist of what it means to be a grid, software technology. 133 00:06:50,540 --> 00:06:51,880 So looking at these allowed, 134 00:06:51,880 --> 00:06:54,820 these literatures allowed us to distill a set of reference requirements. 135 00:06:54,820 --> 00:06:56,360 Where have you guys heard that before, right. 136 00:06:56,360 --> 00:06:58,068 We talked about reference architectures and 137 00:06:58,068 --> 00:06:59,880 domain-specific software architectures. 138 00:06:59,880 --> 00:07:01,870 We came up with a set of reference architectures for 139 00:07:01,870 --> 00:07:06,280 grid, reference requirements for grid, the domain of grid computing. 140 00:07:06,280 --> 00:07:09,910 And so for each acquirement, requirement we derived from studying the literature we 141 00:07:09,910 --> 00:07:15,920 also defined a layer in the classic five-layer grid architecture, 142 00:07:15,920 --> 00:07:18,350 which I'll talk about here in the next slide. 143 00:07:18,350 --> 00:07:23,420 We mapped the requirement to which layer it actually had to do with. 144 00:07:23,420 --> 00:07:28,130 So if the requirement had to do with sharing resources across 145 00:07:28,130 --> 00:07:30,920 multiple organizations, it likely went in the collective layer, 146 00:07:30,920 --> 00:07:32,540 which is responsible for that. 147 00:07:32,540 --> 00:07:34,710 If it had to do with application or 148 00:07:34,710 --> 00:07:38,560 building sort of applications on top of the grid computing infrastructure. 149 00:07:38,560 --> 00:07:42,380 We mapped that requirement to the application layer, and so on and so forth. 150 00:07:42,380 --> 00:07:45,830 And then what we did is, for each of the technologies, initially five, 151 00:07:45,830 --> 00:07:51,610 eventually 20, like I said, we applied one of the sort of more automated 152 00:07:51,610 --> 00:07:55,630 clustering static analysis dynamic behavior architecture recovery techniques. 153 00:07:55,630 --> 00:07:57,254 We actually applied one called Focus. 154 00:07:57,254 --> 00:08:02,420 Just developed by Malevich and, and, and Yachabach, to these, these open 155 00:08:02,420 --> 00:08:06,610 source code bases to arrive at sort of a partial architecture recovery model. 156 00:08:06,610 --> 00:08:10,970 And we use these reference requirements as well as the layers that they map to, 157 00:08:10,970 --> 00:08:15,570 to help us guide and shoehorn which one, of the layers, 158 00:08:15,570 --> 00:08:18,190 the components that we recovered, then the connectors and 159 00:08:18,190 --> 00:08:21,100 this partial recovered model actually went into, okay. 160 00:08:21,100 --> 00:08:22,400 And then we studied that. 161 00:08:22,400 --> 00:08:24,970 So the approach is sort of depicted graphically here. 162 00:08:24,970 --> 00:08:26,590 There's a five layer grid architecture. 163 00:08:26,590 --> 00:08:29,020 Real quickly, the layers correspond to fabric. 164 00:08:29,020 --> 00:08:33,410 These are, like, the data disks, the storage devices, 165 00:08:33,410 --> 00:08:37,480 the actual physical processors and things like that across a grid. 166 00:08:37,480 --> 00:08:39,760 A connectivity is all the networking layer and 167 00:08:39,760 --> 00:08:43,530 things like that that make these available via network devices that make 168 00:08:43,530 --> 00:08:45,550 these resources available in a network way. 169 00:08:45,550 --> 00:08:49,370 Resources are specific individual resources services in 170 00:08:49,370 --> 00:08:52,160 a grid environment specific to a single institution. 171 00:08:52,160 --> 00:08:54,340 Collective is a layer on top. 172 00:08:54,340 --> 00:08:57,850 And by the way, you're seeing this in the upper left hand corner of this diagram. 173 00:08:57,850 --> 00:09:01,319 Which is sort of the five layered great architecture as defined in these papers. 174 00:09:02,610 --> 00:09:04,970 Collective is multiple resource, 175 00:09:04,970 --> 00:09:08,560 sort of services across multiple potential institutions. 176 00:09:08,560 --> 00:09:11,920 It's what allows sort of this virtual organization to deal with 177 00:09:11,920 --> 00:09:16,160 resources across multiple institutions, find them, find compute, find storage and 178 00:09:16,160 --> 00:09:16,840 so on and so forth. 179 00:09:16,840 --> 00:09:20,360 And then, that application, or applications, that are grid enabled that 180 00:09:20,360 --> 00:09:22,740 are built on top of this sort of underlying infrastructure and 181 00:09:22,740 --> 00:09:24,150 then interact with them. 182 00:09:24,150 --> 00:09:27,670 So, so that architecture was found in the physiology and the anatomy paper. 183 00:09:27,670 --> 00:09:28,590 We studied that. 184 00:09:28,590 --> 00:09:30,250 We pulled that right out of there. 185 00:09:30,250 --> 00:09:31,750 That's in the upper left. 186 00:09:31,750 --> 00:09:37,080 The, reference requirements were distilled by studying that, those papers and 187 00:09:37,080 --> 00:09:41,910 the data grid paper amongst others, these sort of four seminal, papers. 188 00:09:41,910 --> 00:09:44,430 And then for each grid technology source code, 189 00:09:44,430 --> 00:09:47,990 there on the right, we took the source code, any documentation, and 190 00:09:47,990 --> 00:09:51,790 anything else we found and we ran it through this sort of 191 00:09:51,790 --> 00:09:56,370 focus process to produce this initial sort of recovered architectual model. 192 00:09:56,370 --> 00:10:00,390 We sort of compared those reference requirements to see if 193 00:10:00,390 --> 00:10:03,390 it could give us any information about the components about the connectors. 194 00:10:03,390 --> 00:10:09,850 We shoehorned them into the upper left there, into the actual sort of perscribed. 195 00:10:09,850 --> 00:10:12,920 Great architecture to see how well the implementations of 196 00:10:12,920 --> 00:10:16,970 these grid technologies conform to what they said the grid architecture should be. 197 00:10:16,970 --> 00:10:19,090 And we use this as sort of a metric to determine, like, 198 00:10:19,090 --> 00:10:20,740 okay, how well does our technology do? 199 00:10:20,740 --> 00:10:22,360 Because, remember the original empadis for 200 00:10:22,360 --> 00:10:25,910 this was, someone told us our technology wasn't a grid technology. 201 00:10:25,910 --> 00:10:27,540 It wasn't a grid software system. 202 00:10:27,540 --> 00:10:29,600 So we wanted to develop sort of a tried and 203 00:10:29,600 --> 00:10:33,210 true research approach, a scientific approach for verifying this. 204 00:10:33,210 --> 00:10:37,010 Okay, so, we we basically, I basically talked, sort of, 205 00:10:37,010 --> 00:10:38,650 through this, visually, on the last diagram. 206 00:10:38,650 --> 00:10:41,690 Again, we clustered our components according to sort of the reference 207 00:10:41,690 --> 00:10:45,110 architecture and shoehorned them in there from that recovered architectural model, 208 00:10:45,110 --> 00:10:49,640 which we sort of gleaned according to that focused approach that we had. 209 00:10:49,640 --> 00:10:52,450 And then we used the reference requirements to sort of give us further 210 00:10:52,450 --> 00:10:55,760 information about what layer in the grid architecture or the components or 211 00:10:55,760 --> 00:10:57,130 the connectors that sort of go into. 212 00:10:57,130 --> 00:11:00,290 And so these are the reference requirements, the list of around 17, 213 00:11:00,290 --> 00:11:03,810 that we gleaned from that basic grid literature. 214 00:11:03,810 --> 00:11:08,440 And, the sort of impacted layer in that grid architecture, that sort of gave us 215 00:11:08,440 --> 00:11:11,965 some more insight as to well, you know, if it's something that's dealing with single 216 00:11:11,965 --> 00:11:14,940 sign-on it likely goes in the connectivity layer, and so on and so forth. 217 00:11:16,530 --> 00:11:21,540 So, this is a sort of sequence of steps that we took. 218 00:11:21,540 --> 00:11:24,080 We did these sequence of steps for 219 00:11:24,080 --> 00:11:26,130 all of the grid technologies that we were looking at. 220 00:11:26,130 --> 00:11:30,490 This happens to be, in the upper left, a static class diagram of 221 00:11:30,490 --> 00:11:35,420 the object Apache OODT software, object oriented data technology. 222 00:11:35,420 --> 00:11:37,300 And Starting from the upper left and 223 00:11:37,300 --> 00:11:41,530 following a, b, c, going down that column on the left. 224 00:11:41,530 --> 00:11:43,420 And up again to d on the right. 225 00:11:43,420 --> 00:11:45,780 Down all the way to f. 226 00:11:45,780 --> 00:11:47,640 There. Down at the bottom. 227 00:11:47,640 --> 00:11:49,680 That's sort of the steps of focus. 228 00:11:49,680 --> 00:11:51,390 Focus is a clustering technique. 229 00:11:51,390 --> 00:11:53,390 It's a graph sort of clustering technique. 230 00:11:53,390 --> 00:11:54,940 That sort of looks at coupling and 231 00:11:54,940 --> 00:11:58,700 cohesion, the relationships between these sort of classes, Remember they're 232 00:11:58,700 --> 00:12:03,420 object oriented codes that we're talking about classes and relationships between 233 00:12:03,420 --> 00:12:06,630 classes like generalization, inheritance, and things like that. 234 00:12:06,630 --> 00:12:09,860 So, focus is a graph clustering technique that allows us to sort of 235 00:12:09,860 --> 00:12:11,640 measure the coupling and cohesion. 236 00:12:11,640 --> 00:12:13,180 Between these to try and 237 00:12:13,180 --> 00:12:17,280 derive what classes are related and to suggest they might be a component. 238 00:12:17,280 --> 00:12:20,330 What classes are interacting with one another to suggest they might be 239 00:12:20,330 --> 00:12:22,180 a connecter, and so on and so forth. 240 00:12:22,180 --> 00:12:26,430 So we ran sort of ODT and all the other technologies up to 20, 241 00:12:26,430 --> 00:12:28,760 like I said, through this process. 242 00:12:28,760 --> 00:12:30,220 Ran them through Mr. Wizard here. 243 00:12:30,220 --> 00:12:33,370 And what we got out the other end is this sort of recovered architectural model, 244 00:12:33,370 --> 00:12:37,400 when none of the clustering techniques for focus, when it sort of synthesizes, 245 00:12:37,400 --> 00:12:40,980 when it sort of arrives at a point where they can't really be clustered anymore. 246 00:12:40,980 --> 00:12:43,220 That's basically this recovered architectural model. 247 00:12:43,220 --> 00:12:45,710 It's a partial recovered architectural model. 248 00:12:45,710 --> 00:12:48,900 It doesn't have all of the detail, the styles, the configurations,. 249 00:12:48,900 --> 00:12:51,820 And all of that, but it has some basic information about components and 250 00:12:51,820 --> 00:12:55,820 connectors and things that we can work with and which we did work with, 251 00:12:55,820 --> 00:12:57,660 according to the process that we'd talked about before. 252 00:12:57,660 --> 00:13:00,920 I took, we took this recovered architectural model, and 253 00:13:00,920 --> 00:13:05,260 then we shoehorned it into the grid reference architecture, those five layers. 254 00:13:05,260 --> 00:13:08,710 Okay, again, with those reference requirements that gave us some insight, 255 00:13:08,710 --> 00:13:12,590 which we distilled from the literature, to figure out which layer it went into. 256 00:13:12,590 --> 00:13:15,350 As well as, just with looking at things like documentation and 257 00:13:15,350 --> 00:13:16,750 any other information that we can find. 258 00:13:16,750 --> 00:13:20,300 And what you find is that, you know, does this sort of look perfect? 259 00:13:21,320 --> 00:13:23,690 well. Let's see, 260 00:13:23,690 --> 00:13:28,030 the canonical layered architectural style has the following constraints. 261 00:13:28,030 --> 00:13:31,990 First, any layer in the, layered architectural style should only be 262 00:13:31,990 --> 00:13:34,740 communicating with it's most, adjacent most layer. 263 00:13:34,740 --> 00:13:38,420 Communication should flow top to bottom, looking at this in a vertical way. 264 00:13:38,420 --> 00:13:43,810 The top-most components being sort of client consuming components and, or 265 00:13:43,810 --> 00:13:44,930 layers, if you will. 266 00:13:44,930 --> 00:13:49,210 And the bottom most layers, or components being service providing layers, 267 00:13:49,210 --> 00:13:53,480 to all of the above, sort of layers and components above it. 268 00:13:53,480 --> 00:13:57,730 Okay, so things on the bottom should be service providers to things on the top and 269 00:13:57,730 --> 00:13:58,470 layers on the top. 270 00:13:58,470 --> 00:14:00,470 Right? And again, any two layers, 271 00:14:00,470 --> 00:14:02,780 communication should only happen between any two layers, 272 00:14:02,780 --> 00:14:06,680 not from example the top layer all the way down to the bottom layer. 273 00:14:06,680 --> 00:14:08,630 Okay, so do you see anything wrong with this picture? 274 00:14:08,630 --> 00:14:12,580 It might be hard to see unless you sort of look at this, so let me help you here. 275 00:14:12,580 --> 00:14:14,360 There all sorts of things that are wrong. 276 00:14:14,360 --> 00:14:15,310 All sorts of violations. 277 00:14:15,310 --> 00:14:18,410 You see, components here in the application layer crossing a two 278 00:14:18,410 --> 00:14:22,550 layer boundary to get down to the connectivity layer, okay? 279 00:14:22,550 --> 00:14:26,670 You see upcalls, you see components in the actual fabric layer making an upcall to 280 00:14:26,670 --> 00:14:29,540 a component in the resource layer, okay? 281 00:14:29,540 --> 00:14:33,100 You see things like components that we couldn't determine the right layer that it 282 00:14:33,100 --> 00:14:34,710 went into, all right? 283 00:14:34,710 --> 00:14:37,550 We just didn't have enough information, or we couldn't tell. 284 00:14:37,550 --> 00:14:38,090 What about this one? 285 00:14:38,090 --> 00:14:40,550 This is Globus's recovered architectural model. 286 00:14:40,550 --> 00:14:41,090 Okay. 287 00:14:41,090 --> 00:14:43,130 Woah two layer boundary and up call. 288 00:14:43,130 --> 00:14:44,380 Three of them right there. 289 00:14:44,380 --> 00:14:45,510 What else? 290 00:14:45,510 --> 00:14:47,980 Five components that we couldn't determine which went in the right layer. 291 00:14:47,980 --> 00:14:51,803 And Globus is a comparitively larger system you know than but 292 00:14:51,803 --> 00:14:55,280 it still exhibits the same if not more sort of basic characteristics for that. 293 00:14:55,280 --> 00:14:58,710 And we found this in a number of the software systems that we're looking at. 294 00:14:58,710 --> 00:15:00,790 Up call, call, up call. 295 00:15:00,790 --> 00:15:01,580 Here. 296 00:15:01,580 --> 00:15:05,390 So, so basically, there are numerous violations of the reference architecture. 297 00:15:05,390 --> 00:15:08,400 There are things like component upcalls, which indicates sort of, 298 00:15:08,400 --> 00:15:10,510 definitely a violation of the layer architectural style, 299 00:15:10,510 --> 00:15:15,340 that potentially indicates architectural drift or erosion, components or code in 300 00:15:15,340 --> 00:15:18,850 the actual code that originally weren't designed to talk to one another but 301 00:15:18,850 --> 00:15:20,740 that are talking to one another there. 302 00:15:20,740 --> 00:15:22,410 Crossing two or more layer boundaries. 303 00:15:22,410 --> 00:15:25,260 This typically indicates developer sloppiness. 304 00:15:25,260 --> 00:15:27,510 You know, you had a library that you intended for 305 00:15:27,510 --> 00:15:30,450 other, sort of elements to sort of call. 306 00:15:30,450 --> 00:15:34,160 Or the most adjacent elements to that to have some sort of coupling to it. 307 00:15:34,160 --> 00:15:38,500 But you find later on down the road, shoehorning and bolting on components. 308 00:15:38,500 --> 00:15:41,270 That you just decided to just make call this library even though there no 309 00:15:41,270 --> 00:15:41,890 where related. 310 00:15:41,890 --> 00:15:43,450 In the software system at all or 311 00:15:43,450 --> 00:15:46,080 nowhere related in the actual architecture as well. 312 00:15:46,080 --> 00:15:47,120 All right? And then there was just, 313 00:15:47,120 --> 00:15:48,840 you know well it could be a refactorization problem. 314 00:15:48,840 --> 00:15:50,190 It could be a number of things. 315 00:15:50,190 --> 00:15:53,310 But And then there are some components that you just, 316 00:15:53,310 --> 00:15:54,720 at the end of the day you look at your coding. 317 00:15:54,720 --> 00:15:55,900 Why is this here? 318 00:15:55,900 --> 00:15:57,450 It's not being called by anything. 319 00:15:57,450 --> 00:15:58,370 You know, nothing. 320 00:15:58,370 --> 00:16:00,800 It has, it doesn't really have a purpose. 321 00:16:00,800 --> 00:16:04,250 So you know, look at all the types of information that we could glean from 322 00:16:04,250 --> 00:16:06,060 performing this type of process. 323 00:16:06,060 --> 00:16:08,480 Okay, what we'd glean from that is that you know, 324 00:16:08,480 --> 00:16:12,560 grid technologies tend to be sort of a domain specific software architecture for 325 00:16:12,560 --> 00:16:14,530 the realm of great computing. 326 00:16:14,530 --> 00:16:16,060 Right? They exhibit a core set of 327 00:16:16,060 --> 00:16:17,530 reference requirements. 328 00:16:17,530 --> 00:16:18,860 They have sort of, 329 00:16:18,860 --> 00:16:22,290 kind of similar components in style even though they're violations. 330 00:16:22,290 --> 00:16:25,720 They have a large variation in terms in their size and number of components. 331 00:16:25,720 --> 00:16:27,520 And cardinality, and things like that. 332 00:16:27,520 --> 00:16:29,990 They also have optional requirements, right. 333 00:16:29,990 --> 00:16:32,040 And these were identified through, you know, 334 00:16:32,040 --> 00:16:35,440 not all grids satisfy these sort of collective level requirements. 335 00:16:35,440 --> 00:16:40,050 Not all of them satisfy even these layer specific requirements for that, okay. 336 00:16:40,050 --> 00:16:42,500 So, you know, it seems like single sign on is optional. 337 00:16:42,500 --> 00:16:44,940 It seems like, you know, data grids are optional, okay. 338 00:16:45,980 --> 00:16:49,370 And also a distinction between data grids and computational grids, right? 339 00:16:49,370 --> 00:16:52,490 In, in terms of the identified requirements for that, okay? 340 00:16:52,490 --> 00:16:55,970 So, these were sort of the results of our study, just, just taking a sort of 341 00:16:55,970 --> 00:17:01,350 an analysis of you know, the actual code and its mapping to software architecture. 342 00:17:01,350 --> 00:17:04,410 And these actually led us to develop and publish. 343 00:17:04,410 --> 00:17:07,800 Basically we're working with the Journal of Good Computing to publish a new 344 00:17:07,800 --> 00:17:11,080 architecture for great computings that actually more carefully and 345 00:17:11,080 --> 00:17:14,490 accurately represents the architecture based on 346 00:17:14,490 --> 00:17:19,020 all of these purported grid technologies, these grid codes. 347 00:17:19,020 --> 00:17:19,940 And so on and so forth. 348 00:17:19,940 --> 00:17:24,420 And this was simply by looking at the architectural styles the components and 349 00:17:24,420 --> 00:17:26,220 things like that as derived from the code. 350 00:17:26,220 --> 00:17:28,710 And needing things like architectural recovery and 351 00:17:28,710 --> 00:17:30,300 processes like we've talked about here. 352 00:17:31,460 --> 00:17:34,880 So applying and figuring out how to think things through and how to 353 00:17:34,880 --> 00:17:38,850 sort of discern the differences between these large scale software systems and 354 00:17:38,850 --> 00:17:40,590 you have some big data too. 355 00:17:40,590 --> 00:17:43,050 Identifying which requirements are optional is really important. 356 00:17:43,050 --> 00:17:45,070 They may tell you when you build your next system, 357 00:17:45,070 --> 00:17:47,500 you don't need to include components or software that supports that. 358 00:17:47,500 --> 00:17:51,370 And save you money, time, resources and a number of other things. 359 00:17:51,370 --> 00:17:55,330 Okay, so just thinking about and having this sort of basic understanding of 360 00:17:55,330 --> 00:17:59,530 software, architecture, components, connectors, the basic elements, styles and 361 00:17:59,530 --> 00:18:02,340 patterns, how they apply, what they suggest, what. 362 00:18:02,340 --> 00:18:03,360 You know what should be true, 363 00:18:03,360 --> 00:18:07,225 what shouldn't be true about them has a real big impact on software systems. 364 00:18:07,225 --> 00:18:07,820 'Kay? 365 00:18:07,820 --> 00:18:08,880 And your big data systems. 366 00:18:10,200 --> 00:18:11,500 There's a lot of related work. 367 00:18:11,500 --> 00:18:12,690 I'm not going to go over this. 368 00:18:12,690 --> 00:18:16,260 I'll leave it here for the read, for you guys to sort of peruse. 369 00:18:16,260 --> 00:18:18,870 You know, the idea is grid systems, 370 00:18:18,870 --> 00:18:21,189 software systems typically have a hard time. 371 00:18:22,680 --> 00:18:25,780 Following the architectural style and that's just because there's a lot of 372 00:18:25,780 --> 00:18:29,600 overlap between grid layers and software systems in general are hard to understand. 373 00:18:29,600 --> 00:18:32,870 Unless from an architectural perspective especially when they get 374 00:18:32,870 --> 00:18:36,320 large you're dealing with data systems, frameworks, middle wares, libraries. 375 00:18:36,320 --> 00:18:39,360 The only way to truly properly understand them is to approach them from the realm of 376 00:18:39,360 --> 00:18:40,630 software architecture. 377 00:18:40,630 --> 00:18:43,920 Right, which is a summarization if you will, 378 00:18:43,920 --> 00:18:46,510 of the principle design decisions about your software system. 379 00:18:47,870 --> 00:18:51,980 There's lots of work kind of going on and, and, and areas for people to expand this. 380 00:18:51,980 --> 00:18:53,710 I encourage you guys to take a look. 381 00:18:53,710 --> 00:18:56,250 I encourage you to take a look at some of these pointers, 382 00:18:56,250 --> 00:19:00,010 again to my software architecture class at USC, to my homepage. 383 00:19:00,010 --> 00:19:03,690 And then all of data behind this particular study of grid middlewares is 384 00:19:03,690 --> 00:19:06,500 available on the following two links and websites, including. 385 00:19:06,500 --> 00:19:10,650 The data for the as submitted paper to Journal of Grid Computing. 386 00:19:10,650 --> 00:19:12,400 So I encourage you to take a look at that too. 387 00:19:12,400 --> 00:19:14,970 Encourage you to contact me if you have any questions, I'm Chris Mattmann. 388 00:19:14,970 --> 00:19:18,280 And thank you, this ends the first full module here on, 389 00:19:18,280 --> 00:19:19,950 on big data architecture fundamentals. 390 00:19:19,950 --> 00:19:20,450 Thanks.