My name is Matthew Graham.
And this is the last module in
a series of six on databases.
In previous modules, we've talked quite
extensively about relational databases,
the data model behind it, and
the supporting technologies.
I would maintain that
this is because a lot
of big data analytics can still
be done on relational databases.
However, there comes a point where a
relational database is not sufficient for
the type of problem that
you're trying to deal with.
Alex Szalay has said that
anything beyond 300 terabytes,
is generally difficult when you're trying
to do any form of data management.
And relational database systems
are tuned for small but
frequent read-write transactions, or
large batch transactions with
unfrequent write accesses.
This tends to mean that they're
scalable in terms of your dataset size.
Or read-write concurrency, you can always
spread the database onto multiple servers,
and clusters of services, distributed
clusters of services in some cases.
If the types of problems that you're
wanting to deal with are still, that you
have small but frequent read write
transactions, or large batch transactions.
Number of data, amount of data, greater
number of users doesn't really matter.
Problem comes, however,
when you move outside of that sweet spot.
When you've actually got more reads
happening than the database system
can actually cope with or, or
more writes are being put onto the data.
If you have too many joins going on,
because of the way that you've
structured your database.
Even if you have normalized it.
Well, particularly if you've normalized
it really because that encourages joins.
That can cause performance issues.
You find that the queries become slower.
Also, if you are doing a lot of
complex queries on the database making
use of a lot of stored procedures and
possibly even views, where
there's a lot of server site computation
required to support a particular query,
then you can find that you get performance
problems with a relational database.
Fortunately, as we've seen in Module 2
when we talked about different types of
data models, there are other ways of
representing your data and therefore,
different technological solutions
which may be more appropriate for
the type of problem that you're facing.
Different types of data store
that you may consider if you have
a scaling problem with your data.
This particular combination
of solutions called QServ,
this is an open source implementation
that's come out of the LSST Project.
This uses MySQL as the fundamental
data store at the backend.
So it's still relational.
Still makes use of SQL as the query
language, Maintains acidity,
and it uses an extra d layer on top.
And doesn't share anything between
individual instances of the service which
was supporting this, but this is a
scalable relational solution designed for
the petascale data volumes
that LSST is [INAUDIBLE].
So this may be a good solution for you,
if you're facing that sort of problem.
Moving away from the relational databases,
there's the SciDB project.
This is a column oriented database.
Column oriented databases
essentially are a,
a 90 degree rotation of
relational database.
It's the column which is
the important thing, not the row.
And instead of it using tables as
it's first order data type for
storing it's based around
the idea of numerical arrays.
So this is a very good database for
large amounts of numerical data,
large amounts of scientific data.
It has substantial industry backing.
So I would recommend going to SciDB.org to
look at this if you're interested in it.
It also maintains ACID, so it's
broadly the same types of transactions
as relational databases we're using.
Moving away from the relational
model even further,
in recent years there's been
a movement called the NoSQL movement.
And the idea is that they
want to reject many of the,
the precepts of relational databases and
move to something that is far more
performant essentially a glorified hash
table with that particular data structure.
So, these are largely
optimized key-value stores.
Where the type of data object that's
being stored is a key and the value.
So large lookup tables.
They're not ACID, but
they're very good for web scale solutions.
The Hadoop, project, has,
has produced a number of these.
And certainly they came out of
the Google projects with Mapreduce and
stuff like that as,
as supporting technology.
A more recent development is the so
called, NewSQL,
movement the idea here is to take the,
performance that you
get from NoSQL but you want to have
a relational model behind it with SQL.
Because that is what we're used
to from relational databases.
And a lot of people have technologies and,
and applications which are built
on that type of model.
NewSQL examples are H-Store or, or
Google Spanner technology examples.
Another possible example
is NuoDB which is a NewSQL
type graph based system but for
doing that sort of modeling.
And the NewSQL systems make use of
what's called a sharding middle layer,
for performance.
Sharding is a type of partitioning where
you have multiple horizontal partitions.
for, from the same schema.
So, the idea is to try and
optimize the particular data access.
But making use of horizontal partitioning.
But multiple schema versions.
So those are particular types of data
store you might want to look at,
if you've got scalability issues with
your particular big data problem.
In terms of alternatives to relational
data stores that are more based on
some of the other data
models we talked about,
it may be that you're actually working
with a large amount of XML data.
XML is the W3 standard,
the World Wide Consortium
standard for markup language for
structure data.
If you've ever seen a data file
which has got angle brackets in it.
That's probably an XML file.
It employs a hierarchical data model.
That's the, the type of tree structure.
Do you remember?
And there were a number of supporting
technologies for working with XML data.
Xpath is a standard for, for allowing you
to point at particular elements or, or
attributes or the values of those, within
an XML document, within an XML file.
Xslt so called style sheets.
Standard for
converting that XML angular format
to other formats, whether it's HTML,
or comma separated variable, or some other
alternate data structure that you want.
And xquery, is a standard for
querying XML documents in
the same way that SQL is a standard for
querying relational tables.
There are a number of databases
which are specifically designed for
working solely with XML data.
These are the so
called native XML databases.
eXist is the most popular open source one.
There are also a number of relational
databases which have XML support.
MySQL and SQL Server, for example,
both have added support for
XML-specific data types in
their most recent versions.
An example of an XML,
ML document is given here.
You can see that the, the indentation is
denoting particular hierarchical levels.
We have element names with
inside the angle brackets.
We also have attributes
inside the angle brackets.
This is a, a record describing
a catalog an astronomical
catalog that may exist in a,
in a directory service somewhere.
And you may have multiple examples of,
of such things, and
you actually want to work
with them in that format.
It may be because of the hierarchy that
there's not an actual translation into
the requisite relational data
structures in a relational database,
that we would require to represent
this type of information, so
using a native XML database
may be more performant.
This is an example of XQuery.
This is an equivalent to
the select statement.
This is in XQuery, this is called flower,
this type of construction.
And you can see that you're
defining some namespaces,
they're just defining areas of, concern.
And then you're defining particular
variables to be the result of
fx path statements, pointing to particular
elements inside an XML document.
And then there's a four loop
with a where statement in it.
The predicate argument of that where
statement, is exactly the same type of
idea as the predicate that
you have in a SQL statement.
There's an ordering and then you can
return the result and one of the powers of
XQuery is that you can actually
format the result that's returned,
in this case it would format the result
in, in a particular type of XML document,
a new XML structure that we would
be interested in working with.
This is an example of, of XQuery.
Another way of working with
data is in form of RDF,
resource description framework,
that I mentioned in the first module.
This is a W3C standard for
data interchange.
It makes use of
the associative data model.
And if you've heard of the Semantic Web,
it's the main underpinning
of the Semantic Web,
the so-called Web of Data, next generation
Internet, Web 2.0, that sort of thing.
The basic idea is that you
represent all data in the form of
subject-predicate-object triplets.
For example, Pluto is a dwarf planet.
Again there are a number
of supporting technologies.
SPARQL, is the,
query language for RDF data.
RDFS is a standard for
modeling RDFF, RDF data, in,
in the same way that you have database
schemas for, for modeling relational data.
SKOS is a standard for
representing controlled vocabularies.
this, this gets more into the,
the idea of knowledge management, the,
the upper echelons of the
Data/Information/Knowledge/Wisdom pyramid.
so, if you have a controlled
vocabulary that you're using in
your particular domain,
SKOS allows you to represent that in
a formal machine-processable format
that you can then program against.
And, ultimately, OWL is for representing
concept schemes, so-called ontologies.
Which represent full formal
encapsulizations of domain
knowledge in a particular area.
Triple stores are the names given
to databases for storing RDF data.
You also find that there are some
relational databases out there,
which have SPARQL interfaces,
so that they can participate in
distributed queries over RDF data.
This is the so called linked data,
or open data, open link data.
That a lot of, big data sources like
DBpedia, which is a version of Wikipedia,
have interfaces exposing
their data through SPARQL.
And so
you can issue queries against those.
This is an example of SPARQL.
The, the top part of the slide shows
again you're defining a prefix and
then you're selecting
using a select statement.
In SPARQL, you have the question
mark to notice the variables that
you're interested in.
And so you're just selecting your,
the capital and the country where
what we're asking for here is to find the
capitals of countries which are in Europe.
And we would be hitting a triple
store containing a set of triples.
Sample data in the,
the lower half of the slide shows
the sort of triples that we would have.
So you can see that the where statement in
the SPARQL example is a join of
a number of types of statement.
We're looking for
cities which have a name.
The city is designated as the capital
of a country which is in,
in the continent of Europe
in this particular case.
We have a schema which
defines a set of predicates.
The predicates, in this case,
would be cityname is capital
of countryname isInContinent.
So our sample data,
this example would return the first one,
that Berlin is the capital of Germany.
The answer we would get back is Berlin,
Germany.
But the other data that is in the database
would be required to check the types of
queries that we're interested in making.
Finally I'd like to talk about
ontology driven databases.
This is where data is modeled at both
the syntactic and the conceptual level.
So this is making use of RDF and owl.
You have your data represented in RDF and
it is structured but you also representing
the main knowledge behind the data [SOUND]
In terms of the ontology, capturing the,
the concepts and the relationships between
those concepts and their properties.
And making use of that in a formal way.
And, an ontology-driven database employs
this formally-expressed domain knowledge.
And it allows you to do consistency checks
not only against syntactic consistencies,
to make sure that data is in
the right data type association with
the right data structures but
also for conceptual checks.
That particular definitions of objects and
the source of properties that they may
want to have are not inconsistent.
So an example is you may have an ontology,
which is describing something like
a star and it may say that a star
has a certain set of properties.
It may be that data in your,
your database, in your triple store,
is defining a star to have may have,
there may be a particular style which has
got a measured property which
is inconsistent with that.
And by doing a consistency check
against the domain knowledge,
you would be able to identify that
inconsistency in your database.
This use of domain knowledge also allows
you to do logical inferencing, so
that you can do first order.
Logic, apply first order of
logic to your database and
identify new pieces of information which
are, are logically consistent or the,
the logical follow on from the,
the facts that you have in your database.
So, you may be able to identify new
properties based on a set number
of specified properties which are all
consistent within the domain knowledge.
Examples of this sort of thing are where
you have possibly a database of protein
information and you have an ontology which
is describing particular associations
between metabolic phenomena and
protein phenomena and
you can infer, particular metabolic
phenomena from the protein
phenomena that are specified in
the ontology and in your database.
You do not have that information
you've specified, but
you can infer it to generate these
new facts which are consistent with
the knowledge base that you are using.
This sort of ontology driven database
facilitates smart applications.
That means that I can write a,
a query against the database, which would
then infer information from the database,
consistent with the knowledge base that
I'm using, and give me answers
back that I'm interested in.
An example is that I may have
a database of zebra fish anatomical
images and their label atop with the
particular thing that they are displaying.
But by using an automony, and
anatomy infrastructure an anatomy ontolog,
I could infer what substructures of
those anatomical structures are,
or, or superstructures are and
so I might ask for
sample all images which display
the hindbrain, but then it
may also return to me in my search, things
related to the central nervous system.
Or to the brain in itself.
Or to individual parts of the hindbrain.
Or possibly to structures or
cell structures that develop into the hind
brain, making use of the ontology to
do the logical referencing
of that type of information.
Even though that's not necessarily
specified in the database itself.
Because it's in the the ontology, we can
infer it on the data in the database for
the sake of applications.
So with that I will end this series
of six modules on databases.
I hope you find it useful, thank you.