1 00:00:00,008 --> 00:00:06,740 [MUSIC] So let's talk a little bit about what distinguishes the term data science 2 00:00:06,740 --> 00:00:12,581 from other related fields. So one related field is business 3 00:00:12,581 --> 00:00:15,308 intelligence. Business intelligence systems are 4 00:00:15,308 --> 00:00:19,758 associated with a couple of concepts. One is a data warehouse, and the other is 5 00:00:19,758 --> 00:00:23,342 a set of dashboards or reports that consume data from the data warehouse, and 6 00:00:23,342 --> 00:00:29,230 are used to answer particular questions. So both of these components require a lot 7 00:00:29,230 --> 00:00:33,585 of upfront effort to design and build, and are therefore, not too adaptable when 8 00:00:33,585 --> 00:00:38,079 requirements change. Okay, and so therefore a software stack 9 00:00:38,079 --> 00:00:41,558 designed for business intelligence may or may not be appropriate for any particular 10 00:00:41,558 --> 00:00:46,330 data science problems, where changing requirements are considered the norm. 11 00:00:46,330 --> 00:00:48,996 And so it sort of warrants a new term, is that business intelligence became 12 00:00:48,996 --> 00:00:51,920 associated with a particular approach to a particular set of problems, and a data 13 00:00:51,920 --> 00:00:54,934 science in some sense broader. Okay. 14 00:00:54,934 --> 00:00:58,084 The other point I'd like to talk about business intelligence is that, the BI 15 00:00:58,084 --> 00:01:01,334 engineers are not typically expected to consume their own data products, and 16 00:01:01,334 --> 00:01:07,330 perform their own analysis, and make, and make the business decisions themselves. 17 00:01:07,330 --> 00:01:11,590 Usually they're building tools for others to make decisions with. 18 00:01:11,590 --> 00:01:13,906 Okay, as a data scientist, you'll be doing both. 19 00:01:13,906 --> 00:01:17,854 So, what about statistics? Well, statistical methods are at the 20 00:01:17,854 --> 00:01:20,310 heart of what a data scientist does, day to day. 21 00:01:20,310 --> 00:01:25,392 But a statistician will typically be comfortable with the, with assuming that 22 00:01:25,392 --> 00:01:32,440 any data set they encounter will fit in main memory on a single machine. 23 00:01:32,440 --> 00:01:35,542 And this makes sense, because the whole field was born out of the need to extract 24 00:01:35,542 --> 00:01:38,879 the most information possible from a very sparse, very expensive to collect and, 25 00:01:38,879 --> 00:01:42,735 and to typically therefpre, very small data set. 26 00:01:42,735 --> 00:01:46,761 Okay, so if you only have 20 patients in the world with a particular disease, you 27 00:01:46,761 --> 00:01:51,638 can't just go find 20 more cheaply. So therefore, you need to come up with 28 00:01:51,638 --> 00:01:54,566 new mathematics to squeeze as much information as you can out of the 20 you 29 00:01:54,566 --> 00:01:57,784 already have. But that's not always the problem 30 00:01:57,784 --> 00:02:00,730 anymore, right? So as we shift from a data-poor regime to 31 00:02:00,730 --> 00:02:04,525 a data-rich regime, the set of challenges move from the need for new mathematics to 32 00:02:04,525 --> 00:02:08,430 squeeze information out of a data set to new engineering to even handle or process 33 00:02:08,430 --> 00:02:12,524 very, very large data sets. Okay. 34 00:02:12,524 --> 00:02:16,276 However, some of the methods, some of the models you'll build are, are the same in 35 00:02:16,276 --> 00:02:21,883 both cases. So database experts, database programmers 36 00:02:21,883 --> 00:02:24,970 and, and administrators bring a lot of skills to the table that make them 37 00:02:24,970 --> 00:02:27,861 appropriate for data science tasks, but there's a, there's a focus on a 38 00:02:27,861 --> 00:02:32,198 particular data model. Which is usually the relational data 39 00:02:32,198 --> 00:02:35,600 model. So this is rows and columns. 40 00:02:35,600 --> 00:02:40,156 So we have data coming from sources that are video, or audio, or even text, or to 41 00:02:40,156 --> 00:02:46,360 some extent even graph graphs, nodes and edges which we'll talk about. 42 00:02:46,360 --> 00:02:49,948 a relational database may or may not be the right tool and even the concepts that 43 00:02:49,948 --> 00:02:54,330 transcend any particular database system may or not be appropriate. 44 00:02:54,330 --> 00:02:56,778 And we'll sort of explore when and where it isn't appropriate as we get into the 45 00:02:56,778 --> 00:02:59,805 course. Okay, so visualization experts also bring 46 00:02:59,805 --> 00:03:04,190 a lot of skills to the table. But like statisticians, are historically, 47 00:03:04,190 --> 00:03:09,612 less concerned with massive scale, data that spans many hundreds of machines. 48 00:03:09,612 --> 00:03:15,820 and then, finally, machine learning, is perhaps the closest to data science. 49 00:03:15,820 --> 00:03:20,649 But here and we'll try to make a, more of a point about this later. 50 00:03:20,649 --> 00:03:24,543 The, the, the, as a proportion of the time you'll spend on a data science 51 00:03:24,543 --> 00:03:29,031 problem, actually choosing the, the right model or algorithm, machine learning 52 00:03:29,031 --> 00:03:35,340 technique, and applying it and running it is a fairly small fraction. 53 00:03:35,340 --> 00:03:38,427 What you'll be spending much more time on is the preparation of the data, the 54 00:03:38,427 --> 00:03:41,808 manipulation of the data, the cleaning of the data the wrangling of the data, as 55 00:03:41,808 --> 00:03:46,181 some have been saying. And, for this, machine learning 56 00:03:46,181 --> 00:03:48,910 techniques are, are not particularly relevant. 57 00:03:48,910 --> 00:03:53,000 And so, this falls back more towards the the database managers. 58 00:03:54,290 --> 00:03:59,630 Managers, the database experts and database programmers, okay? 59 00:03:59,630 --> 00:04:04,240 So, there's a lot of courses that could be considered data science courses. 60 00:04:04,240 --> 00:04:06,650 Some of these use data science in the name, some of the newer ones. 61 00:04:06,650 --> 00:04:09,620 others have been around for a long time but are, but are obviously in this, in 62 00:04:09,620 --> 00:04:12,445 this same space. And so, I want to spend a little bit of 63 00:04:12,445 --> 00:04:15,415 time describing the dimensions by which you could describe these courses, and 64 00:04:15,415 --> 00:04:18,295 then choose a particular point in this design space that we'd use to motivate 65 00:04:18,295 --> 00:04:22,010 this course. Okay. 66 00:04:22,010 --> 00:04:24,614 So as a preface, let me show you this quote from Aaron Kimball, who's a CTO at 67 00:04:24,614 --> 00:04:28,478 the CTO at, at Wibidata. And so he said to me that he worries that 68 00:04:28,478 --> 00:04:34,030 the data scientist role is perhaps like the mythical webmaster of the 90's. 69 00:04:34,030 --> 00:04:37,112 That they were expected to do everything. Alright, the web manual, companies knew 70 00:04:37,112 --> 00:04:40,450 they needed to get on the Internet in the mid-90s but they didn't know how. 71 00:04:40,450 --> 00:04:44,295 And so they said, well, you know, we'll hire a webmaster, problem solved. 72 00:04:44,295 --> 00:04:48,114 Alright, the webmaster will write all of the content for the website, they'll do 73 00:04:48,114 --> 00:04:51,477 the design and manage the user experience, they'll write the code that 74 00:04:51,477 --> 00:04:57,124 will wire the website to the order fulfillment system in the back-end. 75 00:04:57,124 --> 00:05:01,417 they'll actually structure the pages and do the navigation they'll do the logging 76 00:05:01,417 --> 00:05:04,915 required to make sure that the site stays up all the time and, and has reasonably 77 00:05:04,915 --> 00:05:09,938 high availability. they'll design the schema to hold the 78 00:05:09,938 --> 00:05:13,490 data that will be served out through the website and so on. 79 00:05:13,490 --> 00:05:15,437 And so it wasn't really feasible that you were going to get this in a single 80 00:05:15,437 --> 00:05:18,835 person. And so instead the, the Internet strategy 81 00:05:18,835 --> 00:05:23,980 became a, a broader team. Similarly, that might be what we see 82 00:05:23,980 --> 00:05:26,980 happening with, with data science. But here's what it means to me. 83 00:05:26,980 --> 00:05:30,490 The term data science tells me, that if you're a data-based administrator and 84 00:05:30,490 --> 00:05:34,162 your skills are solely about relational databases, the current trend is you will 85 00:05:34,162 --> 00:05:39,730 need to learn more about unstructured data and statistical modelling. 86 00:05:39,730 --> 00:05:42,200 If you're a statistician, you will need to learn to deal with data that does not 87 00:05:42,200 --> 00:05:45,086 fit in memory. If you're a software engineer whose used 88 00:05:45,086 --> 00:05:48,467 to sort of building systems and working with files directly, you'll learn, you'll 89 00:05:48,467 --> 00:05:51,750 need to learn some statistical modelling and how to communicate your results to 90 00:05:51,750 --> 00:05:55,731 your managers. You'll need to work with these data sets 91 00:05:55,731 --> 00:05:58,480 and actually use them to, to make decisions. 92 00:05:58,480 --> 00:06:01,275 And if you're a business analyst who is trained to make your decisions based on 93 00:06:01,275 --> 00:06:03,941 data, you're going to need to start understanding a little bit more about the 94 00:06:03,941 --> 00:06:07,165 algorithms and tradeoffs, especially at scale. 95 00:06:07,165 --> 00:06:12,380 And for a couple of reasons. One is the cost changed dramatically 96 00:06:12,380 --> 00:06:17,229 based on the technology you're picking. What's happening with cloud computing 97 00:06:17,229 --> 00:06:20,153 that we'll talk a little bit about, and what's happening with these algorithms is 98 00:06:20,153 --> 00:06:22,733 that, you know, you can, you might be able to get an answer, but it may cost 99 00:06:22,733 --> 00:06:27,230 more or less than, than, than it did five years ago. 100 00:06:27,230 --> 00:06:30,848 The other reason is that, you know, as we do more fly-by-wire business, meaning, 101 00:06:30,848 --> 00:06:34,412 you know, we, we trust algorithms more and more to make some decisions for us 102 00:06:34,412 --> 00:06:39,570 they become these opaque black boxes. And, if you don't understand what's going 103 00:06:39,570 --> 00:06:43,450 inside, going on inside that black box, you're bound to misinterpret the results. 104 00:06:43,450 --> 00:06:46,700 And so, it's not, it, it's no longer safe to just, sort of, throw your trust over 105 00:06:46,700 --> 00:06:50,823 the wall to some algorithm. Or to, or to your staff that's running 106 00:06:50,823 --> 00:06:53,886 these algorithms. You may need to sort of understand, 107 00:06:53,886 --> 00:06:58,280 internalize the trade offs and choosing one model versus another yourself. 108 00:06:58,280 --> 00:07:01,380 Okay. So here are the dimensions by which I 109 00:07:01,380 --> 00:07:03,980 like to describe these different courses. The first one is breadth. 110 00:07:05,050 --> 00:07:08,140 and so I divide breadth into tools versus abstractions. 111 00:07:08,140 --> 00:07:13,835 And so every sophisticated course would prefer to cheat towards abstractions, 112 00:07:13,835 --> 00:07:16,606 right? You want, you want concepts that 113 00:07:16,606 --> 00:07:20,270 transcend any particular implementation. However, what students are interested in 114 00:07:20,270 --> 00:07:22,830 is, hands on experience using tools that they can use, you know, tomorrow at a 115 00:07:22,830 --> 00:07:24,759 job. And so you always have this tension 116 00:07:24,759 --> 00:07:27,100 between these two. Alright. 117 00:07:27,100 --> 00:07:30,172 And so some examples here are, you know, Hadoop, which we'll talk about, is an 118 00:07:30,172 --> 00:07:33,410 implementation of an abstraction called MapReduce. 119 00:07:33,410 --> 00:07:37,296 And then, the, the MapReduce abstraction certainly transcends its particular 120 00:07:37,296 --> 00:07:40,390 implementation in, in, in Hadoop. Okay. 121 00:07:40,390 --> 00:07:44,030 And so here, as we'll, as I'll mention in, in the next segment I want to cheat 122 00:07:44,030 --> 00:07:48,006 towards abstraction whenever possible to make sure that there are assignments that 123 00:07:48,006 --> 00:07:53,062 give you the hands-on skills that people are interested in. 124 00:07:53,062 --> 00:07:58,084 Alright, the next dimension here is depth, and so by depth, I intend the 125 00:07:58,084 --> 00:08:03,187 distinction between structural manipulation of data and statistical 126 00:08:03,187 --> 00:08:08,832 manipulation of data. Okay, and so here you can think about the 127 00:08:08,832 --> 00:08:12,077 elation to algebra as a structural formalism, a formalism for man 128 00:08:12,077 --> 00:08:16,207 manipulating data structurally while the linear algebra is perhaps a formalism of 129 00:08:16,207 --> 00:08:21,740 manipulating data statistically. Okay and so here, try to strike a 130 00:08:21,740 --> 00:08:25,460 balance, but I actually mean towards structure, and I'm going to defend that 131 00:08:25,460 --> 00:08:31,658 position in the next segment. The next dimension you can think about is 132 00:08:31,658 --> 00:08:34,290 scale. And so here is you know, on one end is, 133 00:08:34,290 --> 00:08:37,298 is yes, its a, a main, main member on a single machine versus what I'll call 134 00:08:37,298 --> 00:08:42,360 cloud, meaning the, or it might require hundreds of machines to, to work on it. 135 00:08:42,360 --> 00:08:45,460 And here I cheat towards cloud and the reasons that I've already sort of 136 00:08:45,460 --> 00:08:48,510 described are that you know, it's no longer safe to assume the data fits in 137 00:08:48,510 --> 00:08:51,710 main memory, and to train people to work only with data of that size, you know, 138 00:08:51,710 --> 00:08:54,760 the, the whole world changes when you start to move even two machines, let 139 00:08:54,760 --> 00:09:00,850 alone 100. and to not have, give you some exposure 140 00:09:00,850 --> 00:09:04,490 to that change, would no, would not equip you, to be, to be an effective data 141 00:09:04,490 --> 00:09:07,680 scientist. Okay. 142 00:09:08,870 --> 00:09:12,565 And then the final dimension I use is sort of the target audience, right? 143 00:09:12,565 --> 00:09:16,012 Is this for hackers or more for analysts? And by hacker I mean, you know, you 144 00:09:16,012 --> 00:09:19,246 already have significant programming experience and you're looking to sort of 145 00:09:19,246 --> 00:09:22,820 round out your skills in some of the mathematics. 146 00:09:22,820 --> 00:09:26,103 or are you more of a technology decision maker, who is trying to bring a little 147 00:09:26,103 --> 00:09:29,653 bit of technical depth? And here I like to actually strike a 148 00:09:29,653 --> 00:09:32,460 balance. I don't want this course to be solely 149 00:09:32,460 --> 00:09:36,890 assuming that you, that you are a a seasoned developer. 150 00:09:36,890 --> 00:09:41,140 but nor, nor can we sort of ignore all programming whatsoever. 151 00:09:41,140 --> 00:09:42,790 So we're going to try to strike a balance between these two. 152 00:09:45,790 --> 00:09:50,530 Alright, so here's the choices we, we sort of made in this course. 153 00:09:50,530 --> 00:09:53,610 So one is we cheat towards abstractions, we cheat towards structs. 154 00:09:53,610 --> 00:09:56,372 we definitely like large scale. And then we, I, I say we'll strike a 155 00:09:56,372 --> 00:09:59,192 balance. But we actually cheat towards the analyst 156 00:09:59,192 --> 00:10:00,861 side. We favor, we favor the fact that there 157 00:10:00,861 --> 00:10:02,752 are going to be analysts in the room who don't necessarily have significant 158 00:10:02,752 --> 00:10:05,190 programming experience. And I've already gotten a lot of 159 00:10:05,190 --> 00:10:07,830 questions from folks over email who say, hey look, you know I haven't done, been 160 00:10:07,830 --> 00:10:10,150 doing programming day to day, am I going to be able to take something away 161 00:10:10,150 --> 00:10:13,580 in this course? And I think the answer is, is yes. 162 00:10:13,580 --> 00:10:23,980 Although there will be some programming, so, so be ready.