1 00:00:00,860 --> 00:00:04,940 Hey everybody, welcome to the JPL-Caltec Virtual Summer School 2 00:00:04,940 --> 00:00:09,920 on Big Data Analytics, here in September, from the 2nd to the 12th in 2014. 3 00:00:09,920 --> 00:00:14,720 I am Chris Mattmann, I'm going to be, lecturing, about, 4 00:00:14,720 --> 00:00:18,560 a really interesting topic which is understanding soft architecture within 5 00:00:18,560 --> 00:00:22,330 the context of big data or, or big data architecture and it's fundamentals. 6 00:00:22,330 --> 00:00:27,280 So, the next set of slides is broken up into a series of, three talks, so 7 00:00:27,280 --> 00:00:31,140 I'll, stop sort of at each, at the end of each, set of the talks, 8 00:00:31,140 --> 00:00:33,730 giving you time to sort of review the materials and what's going on. 9 00:00:33,730 --> 00:00:36,840 So let's, let's go ahead and pop into it. 10 00:00:36,840 --> 00:00:41,260 We're going to cover a variety of topics in this lecture, we'll start out 11 00:00:41,260 --> 00:00:44,800 just giving you a basic introduction to the field of software architecture and 12 00:00:44,800 --> 00:00:46,812 software engineering research. 13 00:00:46,812 --> 00:00:48,722 As it pertains to big data. 14 00:00:48,722 --> 00:00:52,740 I'll talk to you guys about architectural styles, the difference between styles. 15 00:00:52,740 --> 00:00:54,770 I'll talk to you about patterns, and 16 00:00:54,770 --> 00:00:59,154 reference architectures are a little bit of an advanced topic. 17 00:00:59,154 --> 00:01:03,426 And then we'll get into some other areas within software architectural in terms of 18 00:01:03,426 --> 00:01:05,490 architectural modeling. 19 00:01:05,490 --> 00:01:08,900 visualization, how do we represent, things like components and 20 00:01:08,900 --> 00:01:11,200 connectors as models, how do we visualize them. 21 00:01:11,200 --> 00:01:15,390 We'll get into some other processes in terms of drift and recovery. 22 00:01:15,390 --> 00:01:18,820 What happens when your code actually drifts away from your, 23 00:01:18,820 --> 00:01:22,150 sort of intended architecture and how do you reconcile that. 24 00:01:22,150 --> 00:01:25,500 And then the latter part, the last sort of third of the lecture, 25 00:01:25,500 --> 00:01:29,110 is going to cover a case study on understanding software architecture within 26 00:01:29,110 --> 00:01:30,330 the context of great computing. 27 00:01:30,330 --> 00:01:32,010 So, how can you actually apply this. 28 00:01:32,010 --> 00:01:36,090 So this first lecture here is going to go ahead and cover an introduction to 29 00:01:36,090 --> 00:01:40,030 software architecture and then styles, patterns in reference architectures. 30 00:01:40,030 --> 00:01:41,590 So who's talking to you right now? 31 00:01:41,590 --> 00:01:44,620 Let me give you a little bit of background sort of on myself. 32 00:01:44,620 --> 00:01:48,060 My name is Chris Mattman, I sort of wear three hats, 33 00:01:48,060 --> 00:01:51,290 I'm the Chief Architect of the Instrument and Science Data Systems section 34 00:01:51,290 --> 00:01:55,190 at the jet propulsion laboratory at California Institute of Technology. 35 00:01:55,190 --> 00:01:57,960 I have a team of about 20 data scientist at JPL. 36 00:01:57,960 --> 00:02:01,690 And what they're doing is working across various projects from DARPA and 37 00:02:01,690 --> 00:02:05,450 defense investments, building out open source solutions for 38 00:02:05,450 --> 00:02:07,345 Earth science remote sensing missions. 39 00:02:07,345 --> 00:02:10,650 Uh,we work within the context of astronomy projects, 40 00:02:10,650 --> 00:02:13,000 specifically the square kilometer array and 41 00:02:13,000 --> 00:02:17,020 its 700 terabytes of data per second that its going to generate when it's online. 42 00:02:17,020 --> 00:02:18,670 So a wide variety of things, and 43 00:02:18,670 --> 00:02:22,610 those data scientists, the software that they're building, has sort of a dual role. 44 00:02:22,610 --> 00:02:27,370 Every piece of software that my team writes every line of code at JPL that my 45 00:02:27,370 --> 00:02:30,650 team develops is contributed to the Apache Software Foundation. 46 00:02:30,650 --> 00:02:33,740 Apache is a 501c3 non-profit organization. 47 00:02:33,740 --> 00:02:36,810 It's also home to the world's most deployed and 48 00:02:36,810 --> 00:02:39,240 used web server, the Apache Web Server. 49 00:02:39,240 --> 00:02:44,190 But also, relevant within the context of, of here in our big data summer school, 50 00:02:44,190 --> 00:02:45,050 to things like Hadoop. 51 00:02:45,050 --> 00:02:49,180 Which is the defacto big data technology, in terms of processing, right now. 52 00:02:49,180 --> 00:02:54,010 It's also home to emerging big data technologies, like Spark and Shark, 53 00:02:54,010 --> 00:02:59,219 and Which are, basically, these sort of in memory data analytics. 54 00:03:00,270 --> 00:03:02,450 And so, you know, you should be interested that, 55 00:03:02,450 --> 00:03:04,770 those types of things if you're interested in big data. 56 00:03:04,770 --> 00:03:07,510 So those software that we contribute at Apache where I'm on the board of 57 00:03:07,510 --> 00:03:12,350 directors, is also what I use to each to other students. 58 00:03:12,350 --> 00:03:15,050 I also professor at the University of Southern California in 59 00:03:15,050 --> 00:03:16,870 the Computer Science department there. 60 00:03:16,870 --> 00:03:18,810 I teach two classes, one in search engines and 61 00:03:18,810 --> 00:03:21,320 information retrieval, big surprise [LAUGH]. 62 00:03:21,320 --> 00:03:24,890 And then, which I'll be lecturing, to you guys later on, 63 00:03:24,890 --> 00:03:27,770 during, this, summer school here, at Cal Tech. 64 00:03:27,770 --> 00:03:30,160 And then, another class in, another big surprise, 65 00:03:30,160 --> 00:03:32,630 software architecture, which is what I'm going to cover today. 66 00:03:32,630 --> 00:03:33,540 Software Designs, so 67 00:03:33,540 --> 00:03:36,820 they're both graduate classes, some of them are available remotely, as well. 68 00:03:36,820 --> 00:03:39,580 I encourage you to reach out to me and I'll mention that sort of 69 00:03:39,580 --> 00:03:42,860 during the remainder of my lectures here for you guys today. 70 00:03:42,860 --> 00:03:47,490 So some quick notes on this lecture, its the talk is definitely optimized for 71 00:03:47,490 --> 00:03:49,710 breadth and not depth, and that's for a reason, 72 00:03:49,710 --> 00:03:54,080 I'm going to try and go through, I'm going to try and give you the history and 73 00:03:54,080 --> 00:03:56,800 the applicability of software architecture, the big data which. 74 00:03:56,800 --> 00:04:00,040 Software architecture in terms of the software engineering research community 75 00:04:00,040 --> 00:04:02,750 has a pretty storied history over the past 20 years. 76 00:04:02,750 --> 00:04:05,210 It has a golden age, a heyday, things like that. 77 00:04:05,210 --> 00:04:09,960 There's no way in a 45 minute lecture cut into three that I'm going to be able to 78 00:04:09,960 --> 00:04:13,890 cover the entire sort of depth of some of these topics. 79 00:04:13,890 --> 00:04:15,690 So I'm going to point you in areas. 80 00:04:15,690 --> 00:04:18,340 To things, give you sort of a breadth of that. 81 00:04:18,340 --> 00:04:20,110 And, you know where I don't cover something, 82 00:04:20,110 --> 00:04:23,920 I'll be sure, in the middle of the lecture and the topic, to kind of point and say, 83 00:04:23,920 --> 00:04:26,520 hey, you might want to look here for further research. 84 00:04:26,520 --> 00:04:28,360 And I have a slide, sort of at the end, 85 00:04:28,360 --> 00:04:32,529 that has some, pointers to future research at the end of the third lecture here. 86 00:04:33,600 --> 00:04:38,080 You might also want to take a look at the syllabus from my USC software 87 00:04:38,080 --> 00:04:42,680 architecture class CSCI 578, the link there is right there on the slide for 88 00:04:42,680 --> 00:04:45,220 you guys so have a look at that. 89 00:04:45,220 --> 00:04:48,340 And you can get slides and other things to download to get some of 90 00:04:48,340 --> 00:04:51,340 the depth in some of these topics that I'm going to cover. 91 00:04:51,340 --> 00:04:55,500 You know, so in other words, if you really are interested in architectural modeling. 92 00:04:55,500 --> 00:04:58,040 In terms of what I'm going to teach you guys about today and 93 00:04:58,040 --> 00:04:59,350 you know, you don't get enough, if 94 00:04:59,350 --> 00:05:01,970 you're really interested in architecture description language and things. 95 00:05:01,970 --> 00:05:03,180 You want to look that up. 96 00:05:03,180 --> 00:05:07,120 Take a look at the syllabus here that's like here my USC syllabus. 97 00:05:07,120 --> 00:05:07,760 It outta help you. 98 00:05:08,790 --> 00:05:10,790 And then I just encourage you I'm really open, send me and 99 00:05:10,790 --> 00:05:13,450 email, my email address is right here on the slides. 100 00:05:13,450 --> 00:05:14,940 Or you know, contact me on Twitter. 101 00:05:14,940 --> 00:05:16,646 You can probably find me on Facebook, 102 00:05:16,646 --> 00:05:20,620 Google+, LinkedIn [LAUGH], not a places that you can't find me. 103 00:05:20,620 --> 00:05:21,890 In an IRC chat room somewhere. 104 00:05:21,890 --> 00:05:24,660 So, you know, see if you can find me and I'll do my best to help, 105 00:05:24,660 --> 00:05:26,480 if you can kind of connect with me here after the class. 106 00:05:26,480 --> 00:05:28,330 And we really welcome you here to Cal Tech for 107 00:05:28,330 --> 00:05:31,940 the Cal Tech, JPL Cal Tech Big Data Virtual Summer School. 108 00:05:33,500 --> 00:05:36,360 Okay, so let's get started with some actual material. 109 00:05:36,360 --> 00:05:39,070 Software architecture and, and why do you care. 110 00:05:39,070 --> 00:05:42,260 So, you care about it because the way that you design your system, 111 00:05:42,260 --> 00:05:46,730 even before you make technology choices, has a lot of applicability to the way that 112 00:05:46,730 --> 00:05:50,280 your software and your big data system is going to sort of come out the other end. 113 00:05:50,280 --> 00:05:53,230 If you've ever heard of the analogy, garbage in, garbage out. 114 00:05:53,230 --> 00:05:56,260 And has a lot more to do with what stack you pick, 115 00:05:56,260 --> 00:05:59,700 what big data technology you're deploying. 116 00:05:59,700 --> 00:06:02,470 Then in terms of the way that you're actually thinking about the design of 117 00:06:02,470 --> 00:06:05,990 your system, there are a lot of analogies to building architecture, you know. 118 00:06:05,990 --> 00:06:09,950 When we construct, a building like the one that I'm presenting in to, 119 00:06:09,950 --> 00:06:15,210 here to you guys, and we don't start out by, you know simply laying, copper wire. 120 00:06:15,210 --> 00:06:16,710 And, you know, laying the plumbing and 121 00:06:16,710 --> 00:06:19,330 things like that, without having sort of a detailed plan. 122 00:06:19,330 --> 00:06:21,890 And so it's, just, the same is true in software. 123 00:06:21,890 --> 00:06:23,160 It's really important. 124 00:06:23,160 --> 00:06:27,490 All software architectures have some some software architecture whether or 125 00:06:27,490 --> 00:06:32,050 not it's implicit in the design, even if you start coding, you know, believe it or 126 00:06:32,050 --> 00:06:34,840 not there's a software architecture in your mind when you're doing that, 127 00:06:34,840 --> 00:06:36,620 or explicitly described. 128 00:06:36,620 --> 00:06:39,380 Okay, so there's a lot of analogies, you know, 129 00:06:39,380 --> 00:06:42,570 here, and there's some other sort of tangential analogies and 130 00:06:42,570 --> 00:06:45,680 differences, between building architecture and software architecture. 131 00:06:45,680 --> 00:06:49,430 One of them that's, that's really obvious, and really important, too, 132 00:06:49,430 --> 00:06:53,470 is that buildings and building materials are, are malleable. 133 00:06:53,470 --> 00:06:55,770 You can touch them, they're tangible, you can. 134 00:06:55,770 --> 00:06:56,470 Can change them. 135 00:06:56,470 --> 00:07:00,480 You know the impact, for example, if you decide, 136 00:07:00,480 --> 00:07:04,820 that a room should be of a certain, you know, height, an, and depth, and width. 137 00:07:04,820 --> 00:07:07,480 And you know the impact very quickly if your measurements, or 138 00:07:07,480 --> 00:07:10,510 if your thoughts about that, in your design were wrong. 139 00:07:10,510 --> 00:07:12,240 From an engineering and construction perspective. 140 00:07:12,240 --> 00:07:15,450 For example, a door won't fit, you know, where you intended it to go. 141 00:07:16,590 --> 00:07:20,140 In software, that's not exactly the same scenario. 142 00:07:20,140 --> 00:07:22,910 You don't really have that understanding. 143 00:07:22,910 --> 00:07:24,700 Software is, is not tangible. 144 00:07:24,700 --> 00:07:25,390 You can't touch it. 145 00:07:25,390 --> 00:07:28,690 It's not clear to us sometimes the impact of our decisions. 146 00:07:28,690 --> 00:07:32,050 Even at the design level but at the implementation level as, as well. 147 00:07:32,050 --> 00:07:33,640 But even more so, the design level. 148 00:07:33,640 --> 00:07:36,300 Before you start coding, if you're just simply thinking about. 149 00:07:36,300 --> 00:07:41,270 What's the impact of, if I pick a, request reply protocol here in my big data 150 00:07:41,270 --> 00:07:45,700 architecture, and what type of downstream impact will it have on data dissemination? 151 00:07:45,700 --> 00:07:49,070 Versus if I picked up peer-to-peer type of, you know, 152 00:07:49,070 --> 00:07:51,200 data communication protocol and things like that. 153 00:07:51,200 --> 00:07:55,620 So, so the, the impact to some of these decisions isn't as, sort of, obvious, so. 154 00:07:55,620 --> 00:07:58,360 Even though software architectures are related to building architectures, 155 00:07:58,360 --> 00:08:03,410 building architectures in tangential or I'm sorry, tangible materials give us 156 00:08:03,410 --> 00:08:07,280 a lot more sort of testability and coverage and thoughts than software. 157 00:08:07,280 --> 00:08:11,060 Which is why you know, this isn't a science exactly yet. 158 00:08:11,060 --> 00:08:15,210 And, and, a lot of it is based on the ability of software that we tested, 159 00:08:15,210 --> 00:08:16,810 have adequate test coverage. 160 00:08:16,810 --> 00:08:19,650 And to have an adequate architectural model, 161 00:08:19,650 --> 00:08:23,350 that effectively represents what you're actually implementing. 162 00:08:23,350 --> 00:08:23,990 Okay? And so we'll 163 00:08:23,990 --> 00:08:27,240 talk about that during the remainder of the lecture. 164 00:08:27,240 --> 00:08:31,280 Software architecture has sort of enjoyed at least 20 years of 165 00:08:31,280 --> 00:08:32,590 sort of fundamental research. 166 00:08:32,590 --> 00:08:36,380 Some of the early research on software architecture was done. 167 00:08:36,380 --> 00:08:38,670 By Dwayne Perry and Alexander Wolf in 1992. 168 00:08:38,670 --> 00:08:42,400 And they had a sort of seminal paper called the Foundations for 169 00:08:42,400 --> 00:08:44,030 the Study of Software Architecture. 170 00:08:44,030 --> 00:08:46,480 That they published in Software Engineering Notes. 171 00:08:46,480 --> 00:08:47,780 Which is an ACM publication. 172 00:08:48,790 --> 00:08:50,110 Or SEN. 173 00:08:50,110 --> 00:08:53,940 And basically what Perry and Wolf will tell you is that software architecture 174 00:08:53,940 --> 00:08:55,740 really boils down to thinking about the design. 175 00:08:55,740 --> 00:08:59,570 And thinking about the design of software and software as elements. 176 00:08:59,570 --> 00:09:00,670 Form, and rational. 177 00:09:00,670 --> 00:09:04,410 And what they're talking about there are the elements of your software system in 178 00:09:04,410 --> 00:09:09,520 the form of components, the units of computation, the form, the way that you 179 00:09:09,520 --> 00:09:13,950 put those components together, and the rational, for, the rational for doing so. 180 00:09:13,950 --> 00:09:17,950 Why did you pick this number of components to decompose and break. 181 00:09:17,950 --> 00:09:19,280 Your software system down into. 182 00:09:19,280 --> 00:09:23,240 Why did you pick a component representing, this particular protocol or 183 00:09:23,240 --> 00:09:24,440 data conversion? 184 00:09:24,440 --> 00:09:26,620 Or why did you, do this or why did you do that? 185 00:09:26,620 --> 00:09:28,530 So, the rationale for, describing. 186 00:09:30,220 --> 00:09:34,310 Another very, sort of kind of the next step in some of our architectural 187 00:09:34,310 --> 00:09:37,690 understanding, from a research perspective, came from David Shaw. 188 00:09:38,890 --> 00:09:41,630 I, I'm sorry, from Mary Shaw and David Garlan. 189 00:09:41,630 --> 00:09:45,130 In 1996 and they wrote a book called 190 00:09:45,130 --> 00:09:48,980 Software Architecture Perspectives On An Emerging Discipline and that book for 191 00:09:48,980 --> 00:09:54,240 many years up until recently was sort of the canonical software architecture 192 00:09:54,240 --> 00:09:57,760 text that was taught at the university and graduate level. 193 00:09:57,760 --> 00:10:00,090 And the contributions out of that book you know, 194 00:10:00,090 --> 00:10:04,660 all state basically you know, in my opinion are basically. 195 00:10:04,660 --> 00:10:07,590 The identification, the first identification; so they build upon Perry 196 00:10:07,590 --> 00:10:10,960 and Wolf's definition of Software Architecture, form elements and rationale. 197 00:10:10,960 --> 00:10:18,500 Except they sort of decompose the the elements, into more than just components. 198 00:10:18,500 --> 00:10:21,020 More than just the units of computation in your system. 199 00:10:21,020 --> 00:10:23,170 They basically argued that you know, 200 00:10:23,170 --> 00:10:26,180 basically the way components in a software system. 201 00:10:26,180 --> 00:10:30,530 Interact, or something called the connectors, which are models of 202 00:10:30,530 --> 00:10:33,950 the interactions between software components, they argued in their 203 00:10:33,950 --> 00:10:38,130 book that connectors should have a first class status in software architecture. 204 00:10:38,130 --> 00:10:42,650 That the way that the interactions occur is really important, for them. 205 00:10:42,650 --> 00:10:46,590 So we'll discuss some of that here, sort of in the reminder of the slides today. 206 00:10:47,620 --> 00:10:51,420 And then finally, if you're thinking about fundamental software architecture, 207 00:10:51,420 --> 00:10:56,750 research, Philippe Krutchen, who was one of the folks behind, the rational 208 00:10:56,750 --> 00:11:01,880 unified process, was heavily involved in rational rows and other things. 209 00:11:01,880 --> 00:11:07,280 He published a paper in 1995 called the 4+1 model view of software architecture. 210 00:11:07,280 --> 00:11:10,500 That's another paper that's in which he's basically trying to 211 00:11:10,500 --> 00:11:13,770 think how do we model software systems, effectively what, 212 00:11:13,770 --> 00:11:16,090 what types of things should it have in terms of their elements. 213 00:11:16,090 --> 00:11:18,680 And his definitions are very consistent with Perry and 214 00:11:18,680 --> 00:11:21,130 Wolf, and with Shaw and Garlan. 215 00:11:21,130 --> 00:11:25,350 Where Krutchen sort of added something is he stated, you know, there's a lot of. 216 00:11:26,460 --> 00:11:28,656 The, the, there's a lot of aesthetics to software architecture. 217 00:11:28,656 --> 00:11:32,420 Software architecture, what, what might be a good architecture to me, you know, 218 00:11:32,420 --> 00:11:37,250 where I maximizing some particular properties, extensibility, composibility, 219 00:11:37,250 --> 00:11:42,328 reliability, may not necessarily be a good architecture to you. 220 00:11:42,328 --> 00:11:42,980 right? 221 00:11:42,980 --> 00:11:45,070 You may care more about. 222 00:11:45,070 --> 00:11:48,160 Its capabilities in terms of data conversion or 223 00:11:48,160 --> 00:11:52,480 its flexibility in terms of the interaction protocol of things like that. 224 00:11:52,480 --> 00:11:55,740 So the same two software architectures though may they maybe similar in 225 00:11:55,740 --> 00:12:02,260 cardinality and size and eventually when we get down to code or implementation. 226 00:12:02,260 --> 00:12:07,750 Maybe different to other people based on their sort of user defined quality and 227 00:12:07,750 --> 00:12:09,170 their perspective on that. 228 00:12:09,170 --> 00:12:12,410 So for Krutchen, sort of introduced this notion of aesthetics, that, 229 00:12:12,410 --> 00:12:14,240 that's really, is sort of important there. 230 00:12:16,420 --> 00:12:19,830 Sort of another, emerging and, and building upon philosophy for 231 00:12:19,830 --> 00:12:22,910 software architecture came out of Medvidovic and Taylor. 232 00:12:22,910 --> 00:12:23,970 And where they, 233 00:12:23,970 --> 00:12:27,490 where, where they sort of built on is they're building on sort of Shaw and 234 00:12:27,490 --> 00:12:30,850 Garlan and the notion of, Perry and Wolf, the notion of components and connectors. 235 00:12:30,850 --> 00:12:34,900 They built on Krutchen having this sort of aesthetic capability. 236 00:12:34,900 --> 00:12:38,780 Medvidovic and Taylor basically suggested software architecture boils down to 237 00:12:38,780 --> 00:12:40,290 the following canonical things and 238 00:12:40,290 --> 00:12:42,440 that's going to be the definition we use here in the class. 239 00:12:42,440 --> 00:12:43,290 And that's sort of 240 00:12:43,290 --> 00:12:46,180 the thing I want to convey to you because it's consistent with the other. 241 00:12:46,180 --> 00:12:49,090 Definitions, and it's pretty much kind of what we're doing today. 242 00:12:49,090 --> 00:12:52,010 Software architecture consists of the components, 243 00:12:52,010 --> 00:12:54,710 which are the units of computation in the system. 244 00:12:54,710 --> 00:12:57,210 Right? The components may compute on things. 245 00:12:57,210 --> 00:12:58,870 They may maintain internal state. 246 00:12:58,870 --> 00:13:01,250 We may have components for data processing. 247 00:13:01,250 --> 00:13:05,910 We may have, components that basically are responsible for 248 00:13:05,910 --> 00:13:10,150 conversion of things, for sending, information. 249 00:13:10,150 --> 00:13:14,613 Connectors, which are effectively the way that components interact, 250 00:13:14,613 --> 00:13:20,248 request replied protocols, event-based, publish-subscribe interactions, 251 00:13:20,248 --> 00:13:23,190 implicit invocation, these types of things. 252 00:13:23,190 --> 00:13:27,250 How are the components interacting amongst one another, and how do other connectors. 253 00:13:27,250 --> 00:13:27,980 Interact too, 254 00:13:27,980 --> 00:13:31,330 sometimes you have connectors which interact with other connectors. 255 00:13:31,330 --> 00:13:36,650 Pub sub is a great example of that or request replier federated protocols. 256 00:13:36,650 --> 00:13:38,570 So we've got components and connectors and 257 00:13:38,570 --> 00:13:42,570 then we have configurations which are arrangements of components and connectors. 258 00:13:42,570 --> 00:13:44,790 In other words your architectural typology and 259 00:13:44,790 --> 00:13:47,440 any rules that sort of guide that composition. 260 00:13:47,440 --> 00:13:52,830 Okay, this component may be connected to, this connector, only two components may 261 00:13:52,830 --> 00:13:57,240 not be able to be connected to one another without a connector in between them. 262 00:13:57,240 --> 00:13:58,950 These two components can never communicate, 263 00:13:58,950 --> 00:14:01,920 because there's no path through the typology of the architecture for 264 00:14:01,920 --> 00:14:03,889 them to communicate, so on and so forth. 265 00:14:05,710 --> 00:14:10,420 So the core definition sort of building upon this of software architecture that 266 00:14:10,420 --> 00:14:12,980 we're, we're leveraging in this course and 267 00:14:12,980 --> 00:14:15,940 that's sort of defined in kind of this newer book on 268 00:14:15,940 --> 00:14:19,650 software architecture that's sort of rapidly merging as the standard textbook. 269 00:14:19,650 --> 00:14:25,400 Is this book from Taylor, Medvidovic and Dashofy on, on Software Architecture. 270 00:14:25,400 --> 00:14:29,800 And basically in their book, they're discussing architecture as 271 00:14:29,800 --> 00:14:32,810 the following in terms of the definition, and it sort of builds on this. 272 00:14:32,810 --> 00:14:36,750 They're the principal design decisions about a software system, right? 273 00:14:36,750 --> 00:14:39,960 So when you're thinking about your big data software system, what 274 00:14:39,960 --> 00:14:43,610 are the principal definitions that you're thinking about there for the design? 275 00:14:43,610 --> 00:14:45,500 And what makes a decision principal? 276 00:14:45,500 --> 00:14:48,720 Well, building upon sort of what we're learning about in, 277 00:14:48,720 --> 00:14:52,450 in the prior slides, a decision that makes it principle might be it affects one of 278 00:14:52,450 --> 00:14:53,860 the core architectural elements. 279 00:14:53,860 --> 00:14:58,740 It's related to some component some computation in the system. 280 00:14:58,740 --> 00:15:01,080 It's related to the interaction, maybe. 281 00:15:01,080 --> 00:15:03,440 Maybe it has to do with the implementation, but it's so 282 00:15:03,440 --> 00:15:07,000 important because it represents some sort of load bearing wall or 283 00:15:07,000 --> 00:15:09,120 a bottleneck in that implementation. 284 00:15:09,120 --> 00:15:11,040 Maybe it has to do with system evolution. 285 00:15:11,040 --> 00:15:13,300 Maybe it's a component we're going to want to switch out later, or 286 00:15:13,300 --> 00:15:18,030 it's a decision or principal decision that is going to have some impact on the way 287 00:15:18,030 --> 00:15:22,210 that we sort of further evolve the system after we initially develop its code. 288 00:15:22,210 --> 00:15:23,410 And then there are others. 289 00:15:23,410 --> 00:15:27,420 Okay, so these are various examples of sort of what makes a decision principal, 290 00:15:27,420 --> 00:15:29,180 and as you can tell from these various examples, 291 00:15:29,180 --> 00:15:32,340 it may be dependent on different stakeholders in the software system. 292 00:15:32,340 --> 00:15:32,890 So. 293 00:15:32,890 --> 00:15:36,840 Right, your manager may have a different, perspective, of what represents 294 00:15:36,840 --> 00:15:41,020 a principle architectural decision than, say, your buddy, tester, developer. 295 00:15:41,020 --> 00:15:41,520 Okay. 296 00:15:43,060 --> 00:15:46,500 What are some, sort of examples, you know, of software architectures? 297 00:15:46,500 --> 00:15:47,240 Here is an example. 298 00:15:47,240 --> 00:15:53,393 This is a, very popular, content detection analysis toolkit, called Apache Tika. 299 00:15:53,393 --> 00:15:56,030 Tika is been referred to as the digital babel fish, 300 00:15:56,030 --> 00:15:59,490 I'll be talking to you a little bit about Tika in one of my further lectures here in 301 00:15:59,490 --> 00:16:00,990 the summer school. 302 00:16:00,990 --> 00:16:04,390 Effectively what you can think of it is as a small library that 303 00:16:04,390 --> 00:16:07,910 handles identification of file types, extraction of text and 304 00:16:07,910 --> 00:16:09,600 metadata from those file types, and 305 00:16:09,600 --> 00:16:13,259 then identification of language, and potential translation of those language. 306 00:16:14,550 --> 00:16:18,000 Tika here if, is thinking about it in the context of software architecture. 307 00:16:18,000 --> 00:16:20,100 represents a number of components. 308 00:16:20,100 --> 00:16:24,280 I know that they are components because the nice person that made this diagram for 309 00:16:24,280 --> 00:16:28,370 you guys, me, [LAUGH] put in a legend for you that states when I see a box. 310 00:16:29,590 --> 00:16:32,850 Here in the diagram that represents a software component. 311 00:16:32,850 --> 00:16:35,460 And what are those arrows between the software components there. 312 00:16:35,460 --> 00:16:36,890 Well there's different types of variables. 313 00:16:36,890 --> 00:16:39,830 One has a dot or some type of dash lines on it. 314 00:16:39,830 --> 00:16:42,820 One is like a solid black line what do those mean. 315 00:16:42,820 --> 00:16:45,630 Control flow is the sort of dashed arrow. 316 00:16:45,630 --> 00:16:49,700 Control flow meaning the transfer of control from one component to the other. 317 00:16:49,700 --> 00:16:54,270 Data flow is a black line, that's how data flows between the components, okay. 318 00:16:54,270 --> 00:16:57,100 I see something here, called a, a, registry or 319 00:16:57,100 --> 00:17:01,760 a repository, seems to be represented by that sort of familiar data base E shape. 320 00:17:01,760 --> 00:17:03,450 And so on and so forth, right? 321 00:17:03,450 --> 00:17:08,280 So, so regularly software architecture is described in this way. 322 00:17:08,280 --> 00:17:11,450 You'll notice I haven't stated what programming language Tika 323 00:17:11,450 --> 00:17:13,900 is implemented in, I haven't stated. 324 00:17:13,900 --> 00:17:17,990 You know, what, what sort of physical host this is deployed on, is it put on your 325 00:17:17,990 --> 00:17:22,020 Mac Book, is it, you know, are you putting this on windows, is it a virtual machine? 326 00:17:22,020 --> 00:17:24,520 All that sort of discuss the sort of architecture at, 327 00:17:24,520 --> 00:17:27,820 is at the level of these components, the connectors, 328 00:17:27,820 --> 00:17:32,830 which represent in these diagrams those arrows, control and data flow interaction. 329 00:17:32,830 --> 00:17:34,190 Amongst the components. 330 00:17:34,190 --> 00:17:37,320 Right? And, and potentially some other 331 00:17:37,320 --> 00:17:41,150 meaningful or important annotations, here that are present on the diagram. 332 00:17:41,150 --> 00:17:41,970 Right? 333 00:17:41,970 --> 00:17:45,060 So, so this is a common way to represent software architecture. 334 00:17:45,060 --> 00:17:48,280 Now thinking about this, if we built a whole bunch of these. 335 00:17:48,280 --> 00:17:51,290 Sort of content detection and analysis tool kits we may find that our 336 00:17:51,290 --> 00:17:54,680 architectures after awhile may look the same. 337 00:17:54,680 --> 00:17:59,190 They may use familiar components you may use some type of 338 00:17:59,190 --> 00:18:02,270 language identification component you may reuse or 339 00:18:02,270 --> 00:18:07,250 use some similar text extraction parser or some type of component for that. 340 00:18:07,250 --> 00:18:10,860 So over time, what we find typically in domains, and you find this in the domain 341 00:18:10,860 --> 00:18:17,160 of big data too, is that an architectural style or sort of emerges. 342 00:18:17,160 --> 00:18:20,700 Common sets of things like components and connectors and we'll talk about that. 343 00:18:20,700 --> 00:18:23,910 Architectural patterns, which are kind of common arrangements of, 344 00:18:23,910 --> 00:18:27,580 of these things with a little bit of the information left unspecified. 345 00:18:27,580 --> 00:18:28,780 They start to emerge. 346 00:18:28,780 --> 00:18:31,270 A commonly used, maybe, vocabulary. 347 00:18:31,270 --> 00:18:34,490 For talking about architectures within this domain start to emerge 348 00:18:34,490 --> 00:18:35,030 sort of as well. 349 00:18:36,570 --> 00:18:42,740 So architectural styles boil down to commonly used sort of components and 350 00:18:42,740 --> 00:18:46,560 connectors and potential their types or their classes. 351 00:18:46,560 --> 00:18:51,210 That you see, in a family of software systems, built in a domain. 352 00:18:51,210 --> 00:18:54,520 Normally, across many years, okay? 353 00:18:54,520 --> 00:18:57,460 It takes a while for software architectural styles to emerge. 354 00:18:57,460 --> 00:19:00,018 Some examples are things like peer-to-peer, right? 355 00:19:00,018 --> 00:19:04,830 In a peer-to-peer software architectural style, and, and in software systems that 356 00:19:04,830 --> 00:19:08,630 implement the peer to peer style, we know that there's one component type. 357 00:19:08,630 --> 00:19:09,700 There is a peer. 358 00:19:09,700 --> 00:19:12,110 Okay, and we may instantiate many of those things but 359 00:19:12,110 --> 00:19:14,920 effectively we know that the type of component is a peer at 360 00:19:14,920 --> 00:19:17,760 maintains some internal state, potentially, 361 00:19:17,760 --> 00:19:21,300 we know that peers communicate directly with other peers, okay? 362 00:19:21,300 --> 00:19:24,980 So the connections and, and typically in an arbitrated way. 363 00:19:24,980 --> 00:19:29,200 Through some initial interaction in which they locate all the other peers on 364 00:19:29,200 --> 00:19:32,340 the network, and then those peers addresses or 365 00:19:32,340 --> 00:19:37,010 some root set of them, which forward along communication to other peers and so on and 366 00:19:37,010 --> 00:19:38,950 so forth in this sort of organically. 367 00:19:38,950 --> 00:19:42,790 So we know a lot about a system simply by stating it's a peer to peer system, right? 368 00:19:42,790 --> 00:19:46,470 We know about its components, its interactions, and things like that. 369 00:19:46,470 --> 00:19:50,880 Client server rest, layered, these are all other examples. 370 00:19:50,880 --> 00:19:52,920 Okay, of software architectural styles. 371 00:19:52,920 --> 00:19:54,080 Think about it, I have a question for 372 00:19:54,080 --> 00:19:58,040 you guys to ponder here; while you're learning here in our summer school. 373 00:19:58,040 --> 00:20:01,010 What types of architectural styles do you regularly see emerging in 374 00:20:01,010 --> 00:20:02,846 big data systems, Okay? 375 00:20:02,846 --> 00:20:05,570 Are they combinations of these styles, 376 00:20:05,570 --> 00:20:08,020 like do you ever see a system that's strictly peer to peer or 377 00:20:08,020 --> 00:20:11,360 strictly client server, or do you see ones that are more hybrid? 378 00:20:11,360 --> 00:20:15,730 Okay, and that's sort of an open question, there's no correct answer to that, 379 00:20:15,730 --> 00:20:17,860 just sort of think about that while we're talking. 380 00:20:17,860 --> 00:20:19,830 So, when I think about architectural styles, 381 00:20:19,830 --> 00:20:22,840 I like to think of styles as things like ingredients. 382 00:20:22,840 --> 00:20:26,400 If you're thinking about things like ingredients in recipes, and food, and 383 00:20:26,400 --> 00:20:27,260 stuff like that. 384 00:20:27,260 --> 00:20:30,790 And I do a lot, right, I'm a hungry guy, you can see by my stomach. 385 00:20:30,790 --> 00:20:33,570 I, I tend to, [LAUGH] tend to think about food a lot. 386 00:20:33,570 --> 00:20:36,630 So, so architectural styles think of them as sort of 387 00:20:36,630 --> 00:20:38,670 the ingredients in your recipe, right. 388 00:20:38,670 --> 00:20:40,943 Like, like what, how much, what, what salt or 389 00:20:40,943 --> 00:20:46,640 I need pepper or I need vegetables or I need, you know, carrots or this or that. 390 00:20:46,640 --> 00:20:50,030 Architectural styles are more, sort of ingredients, okay? 391 00:20:51,180 --> 00:20:54,230 So you might have heard, if you think about software architecture too, 392 00:20:54,230 --> 00:20:56,300 things like patterns, code patterns, 393 00:20:56,300 --> 00:21:00,440 if you've read the code patterns book by Gamma, and, and things like that. 394 00:21:00,440 --> 00:21:04,780 patterns, you might have heard in terms of things like model view controller or. 395 00:21:04,780 --> 00:21:07,800 Three tiered pattern or sense compute control. 396 00:21:07,800 --> 00:21:12,460 Architectural patterns are typically lower level than architectural styles and what I 397 00:21:12,460 --> 00:21:16,760 mean by lower levels is that styles are mostly like what I said think ingredients 398 00:21:16,760 --> 00:21:22,060 right so patterns take us closer to the actual implementation of software. 399 00:21:22,060 --> 00:21:26,510 Patterns may tell us, yes, there are these particular styles of components or 400 00:21:26,510 --> 00:21:28,020 connectors in our software system. 401 00:21:29,160 --> 00:21:35,090 And they may tell us basically that we arrange them in a particular way and 402 00:21:35,090 --> 00:21:36,180 they communicate in this way. 403 00:21:36,180 --> 00:21:42,752 Leaving sort of, outside of the box for that some information to be specified. 404 00:21:42,752 --> 00:21:45,310 We arrange in, for example, a three tiered pattern. 405 00:21:45,310 --> 00:21:46,780 We have three components. 406 00:21:46,780 --> 00:21:50,820 And three component types, we have a sort of UI or presentation tier. 407 00:21:50,820 --> 00:21:52,000 We have a logic tier. 408 00:21:52,000 --> 00:21:53,320 Sort of that the UI and 409 00:21:53,320 --> 00:21:57,120 presentation tier communicates with and then we have some back end data tier. 410 00:21:57,120 --> 00:22:00,040 Or, or, or component or you know, whatever you want to call it. 411 00:22:00,040 --> 00:22:01,980 So we know a lot about a system too, 412 00:22:01,980 --> 00:22:05,300 we know even more than a style by talking about patterns. 413 00:22:05,300 --> 00:22:09,710 Okay for that and so I like to think of patterns more like recipes right. 414 00:22:09,710 --> 00:22:14,000 We may take the ingredients like that we have from styles, salt and pepper and 415 00:22:14,000 --> 00:22:17,870 things like that and vegetables, and put them together in to a bolognese sauce. 416 00:22:17,870 --> 00:22:20,160 Okay leaving for you to specify in that, 417 00:22:20,160 --> 00:22:23,230 if you want to make a variation on the amount of salt or 418 00:22:23,230 --> 00:22:26,220 the amount of pepper or whether or not you like carrots in your bolognese or 419 00:22:26,220 --> 00:22:29,540 not or whether you strictly like things like celery or things like that. 420 00:22:29,540 --> 00:22:33,450 so it takes us closer to sort of the implementation of pattern does, but 421 00:22:33,450 --> 00:22:36,260 it doesn't prescribe or provide everything okay. 422 00:22:38,380 --> 00:22:40,080 And finally we're going to wrap up this first part of 423 00:22:40,080 --> 00:22:43,430 the module just talking about Reference Architectures which were a term that 424 00:22:43,430 --> 00:22:47,830 was originally coined in the architectural domain by Will Tracz in a paper in which 425 00:22:47,830 --> 00:22:51,540 he was talking about architectures and domain specific software architectures. 426 00:22:51,540 --> 00:22:54,100 And ACM Software Engineering Notes way back when in 1995. 427 00:22:54,100 --> 00:22:58,690 And Tracz basically said that reference architectures have sort of 428 00:22:58,690 --> 00:22:59,950 three common things to them. 429 00:22:59,950 --> 00:23:02,970 They've got kind of common architectural styles and patterns, 430 00:23:04,210 --> 00:23:08,600 found in a particular domain like grid computing or big data or avionics. 431 00:23:08,600 --> 00:23:12,040 They've got reference requirements that drove the sort of derivation of 432 00:23:12,040 --> 00:23:15,700 those components, those styles, those patterns, and things like that. 433 00:23:15,700 --> 00:23:18,520 Then they typically have a domain model or common vocabulary for 434 00:23:18,520 --> 00:23:21,980 talking about systems and data and components within that domain. 435 00:23:21,980 --> 00:23:27,190 Okay, so, so domain specific reference architectures are typically or 436 00:23:27,190 --> 00:23:30,540 domain specific software architectures are typically reference architectures for 437 00:23:30,540 --> 00:23:33,010 a particular domain, like for avionics. 438 00:23:33,010 --> 00:23:33,640 And so forth. So 439 00:23:33,640 --> 00:23:35,880 we use these sort of interchangeably for that. 440 00:23:35,880 --> 00:23:38,927 You know, examples within big data, you might think about [INAUDIBLE] being 441 00:23:38,927 --> 00:23:41,140 a reference architecture, okay, for a particular domain. 442 00:23:41,140 --> 00:23:45,710 Common [INAUDIBLE] of components and connectors and things like that. 443 00:23:45,710 --> 00:23:49,810 You might think about systems like Globus, which was very popular. 444 00:23:49,810 --> 00:23:52,030 You know, many years ago, but still has some popularity and 445 00:23:52,030 --> 00:23:53,530 remains that way today. 446 00:23:53,530 --> 00:23:56,360 Especially with things like Globus online and so on and so forth. 447 00:23:56,360 --> 00:23:59,980 So, that represents the end of the first, sort of ten minutes. 448 00:23:59,980 --> 00:24:01,180 or, sorry, not ten minutes. 449 00:24:01,180 --> 00:24:04,900 The first, part or module here, in our Big Data fundamentals. 450 00:24:04,900 --> 00:24:08,430 So we'll move on, in the next module, talking about some advanced. 451 00:24:08,430 --> 00:24:11,850 Architectural concepts in terms of architectural modeling. 452 00:24:11,850 --> 00:24:14,250 Architectural visualization and, and so on and so forth.