Chapter 4: Relational databases
All of the packages described later in this chapter provide clients to client/server databases.
The database can reside on the same machine or (more often) remotely. There is an
standard (in fact several:
/IEC 9075, also known as
is coming into use) for an interface language called
(Structured Query Language, sometimes
pronounced ‘sequel’: see Bowman et al. 1996 and Kline and Kline 2001) which these DBMSs
support to varying degrees.
The more comprehensive R interfaces generate
behind the scenes for common operations,
but direct use of
is needed for complex operations in all. Conventionally
in upper case, but many users will ﬁnd it more convenient to use lower case in the R interface
Arelational DBMS stores data as a database of tables (or relations) which are rather similar
to R data frames, in that they are made up of columns or ﬁelds of one type (numeric, character,
date, currency, . . .) and rows or records containing the observations for one entity.
‘queries’ are quite general operations on a relational database. The classical query is a
SELECT statement of the type
SELECT State, Murder FROM USArrests WHERE Rape > 30 ORDER BY Murder
SELECT t.sch, c.meanses, t.sex, t.achieve
FROM student as t, school as c WHERE t.sch = c.id
SELECT sex, COUNT(*) FROM student GROUP BY sex
SELECT sch, AVG(sestat) FROM student GROUP BY sch LIMIT 10
The ﬁrst of these selects two columns from the R data frame USArrests that has been copied
across to a database table, subsets on a third column and asks the results be sorted. The second
performs a database join on two tables student and school and returns four columns. The
third and fourth queries do some cross-tabulation and return counts or averages. (The ﬁve
aggregation functions are COUNT(*) and SUM, MAX, MIN and AVG, each applied to a single
SELECT queries use FROM to select the table, WHERE to specify a condition for inclusion
(or more than one condition separated by AND or OR), and ORDER BY to sort the result.
Unlike data frames, rows in RDBMS tables are best thought of as unordered, and without an
ORDER BY statement the ordering is indeterminate. You can sort (in lexicographical order)
on more than one column by separating them by commas. Placing DESC after an ORDER BY
puts the sort in descending order.
SELECT DISTINCT queries will only return one copy of each distinct row in the selected
The GROUP BY clause selects subgroups of the rows according to the criterion. If more
than one column is speciﬁed (separated by commas) then multi-way cross-classiﬁcations can be
summarized by one of the ﬁve aggregation functions. A HAVING clause allows the select to
include or exclude groups depending on the aggregated value.
If the SELECT statement contains an ORDER BY statement that produces a unique order-
ing, a LIMIT clause can be added to select (by number) a contiguous block of output rows. This
can be useful to retrieve rows a block at a time. (It may not be reliable unless the ordering is
unique, as the LIMIT clause can be used to optimize the query.)
There are queries to create a table (CREATE TABLE, but usually one copies a data frame to
the database in these interfaces), INSERT or DELETE or UPDATE data. A table is destroyed
by a DROP TABLE ‘query’.
Chapter 4: Relational databases
Kline and Kline (2001) discuss the details of the implementation of SQL in Microsoft SQL
Server 2000, Oracle, MySQL and PostgreSQL.
4.2.2 Data types
Data can be storedin adatabase in various datatypes. The rangeofdata types is DBMS-speciﬁc,
standard deﬁnes many types, including the following that are widely implemented
(often not by the
float(p) Real number, with optional precision. Often called real or double or double
32-bit integer. Often called int.
smallint 16-bit integer
ﬁxed-length character string. Often called char.
variable-length character string. Often called varchar. Almost always has a limit
of 255 chars.
true or false. Sometimes called bool or bit.
time of day
date and time
There are variants on time and timestamp, with timezone. Other types widely implemented
are text and blob, for large blocks of text and binary data, respectively.
The more comprehensive of the R interface packages hide the type conversion issues from the
4.3 R interface packages
There are several packages available on
to help R communicate with DBMSs. They
provide diﬀerent levels of abstraction. Some provide means to copy whole data frames to and
from databases. All have functions to select data within the database via
queries, and to
retrieve the result as a whole as a data frame or in pieces (usually as groups of rows).
All except RODBC (https: //CRAN.R-project.org/package=RODBC) are tied to one
DBMS, but there has been a proposal for a uniﬁed ‘front-end’ package DBI (https://CRAN.
R-project.org/package=DBI) (https://developer.r-project.org/db) in conjunction with
a ‘back-end’, the most developed of which is RMySQL (https://CRAN.R-project.org/
package=RMySQL). Also on
are the back-ends ROracle (https://CRAN.R-project.org/
package=ROracle), RPostgreSQL (https://CRAN.R-project.org/package=RPostgreSQL)
and RSQLite (https://CRAN.R-project.org/package=RSQLite) (which works with the bun-
dled DBMS SQLite, https://www.sqlite.org), RJDBC (https://CRAN.R-project.org/
package=RJDBC) (which uses Java and can connect to any DBMS that has a JDBC driver)
and RpgSQL (https://CRAN.R-project.org/package=RpgSQL) (a specialist interface to Post-
greSQL built on top of RJDBC (https://CRAN.R-project.org/package=RJDBC)).
The BioConductor project has updated RdbiPgSQL (formerly on
ca 2000), a ﬁrst-
generation interface to PostgreSQL.
PL/R (http://www.joeconway.com/plr/ (http://www.joeconway.com/plr/)) is a project
to embed R into PostgreSQL.
Chapter 4: Relational databases
Package RMongo (https://CRAN.R-project.org/package=RMongo) provides an R interface
to a Java client for ‘MongoDB’ (https://en.wikipedia.org/wiki/MongoDB) databases, which
org/package=rmongodb) is another client using mongodb’s C driver.
4.3.1 Packages using DBI
Package RMySQL (https://CRAN.R-project.org/package=RMySQL) on
interface to the MySQL database system (see https://www.mysql.com and Dubois, 2000)
or its fork MariaDB (see https://mariadb.org/). The description here applies to versions
0.5-0 and later: earlier versions had a substantially diﬀerent interface. The current version
requires the DBI (https://CRAN.R-project.org/package=DBI) package, and this description
will apply with minor changes to all the other back-ends to DBI (https://CRAN.R-project.
MySQL exists on Unix/Linux/OS X and Windows: there is a ‘Community Edition’ released
under GPL but commercial licenses are also available. MySQL was originally a ‘light and lean’
database. (It preserves the case of names where the operating ﬁle system is case-sensitive, so
not on Windows.)
The call dbDriver("MySQL") returns a database connection manager object, and then a
call to dbConnect opens a database connection which can subsequently be closed by a call
to the generic function dbDisconnect. Use dbDriver("Oracle"), dbDriver("PostgreSQL")
or dbDriver("SQLite") with those DBMSs and packages ROracle (https: / / CRAN .
R-project . org / package=ROracle), RPostgreSQL (https: / / CRAN . R-project . org /
package=RPostgreSQL) or RSQLite (https:/ / CRAN .R-project . org / package=RSQLite)
queries can be sent by either dbSendQuery or dbGetQuery. dbGetquery sends the query
and retrieves the results as a data frame. dbSendQuery sends the query and returns an object of
class inheriting from "DBIResult" which can be used to retrieve the results, and subsequently
used in a call to dbClearResult to remove the result.
Function fetch is used to retrieve some or all of the rows in the query result, as a list.
The function dbHasCompleted indicates if all the rows have been fetched, and dbGetRowCount
returns the number of rows in the result.
These are convenient interfaces to read/write/test/delete tables in the database.
dbReadTable and dbWriteTable copy to and from an R data frame, mapping the row names
of the data frame to the ﬁeld row_names in the MySQL table.
> library(RMySQL) # will load DBI as well
## open a connection to a MySQL database
> con <- dbConnect(dbDriver("MySQL"), dbname = "test")
## list the tables in the database
## load a data frame into the database, deleting any existing copy
> dbWriteTable(con, "arrests", USArrests, overwrite = TRUE)
## get the whole table
> dbReadTable(con, "arrests")
Murder Assault UrbanPop Rape
## Select from the loaded table
> dbGetQuery(con, paste("select row_names, Murder from arrests",
Chapter 4: Relational databases
"where Rape > 30 order by Murder"))
5 New Mexico
> dbRemoveTable(con, "arrests")
4.3.2 Package RODBC
Package RODBC (https://CRAN.R-project.org/package=RODBC) on
provides an inter-
face to database sources supporting an
interface. This is very widely available, and allows
the same R code to access diﬀerent database systems. RODBC (https://CRAN.R-project.
org/package=RODBC) runs on Unix/Linux, Windows and OS X,and almost all database systems
provide support for
. We have tested Microsoft SQL Server, Access, MySQL, PostgreSQL,
Oracle and IBM DB2 on Windows and MySQL, MariaDB, Oracle, PostgreSQL and SQLite on
ODBC is a client-server system, and we have happily connected to a DBMS running on a
Unix server from a Windows client, and vice versa.
On Windows ODBC support is part of the OS. On Unix/Linux you will need an
Manager such as unixODBC (http://www.unixODBC.org) or iOBDC (http://www.iODBC.
org: this is pre-installed in OS X) and an installed driver for your database system.
Windows provides drivers not just for DBMSs but also for Excel (.xls) spreadsheets, DBase
(.dbf) ﬁles and even text ﬁles. (The named applications do not need to be installed. Which
ﬁle formats are supported depends on the versions of the drivers.) There are versions for Excel
and Access 2007/2010 (go to https://www.microsoft.com/en-us/download/default.aspx,
and search for ‘Oﬃce ODBC’, which will lead to AccessDatabaseEngine.exe), the ‘2007 Oﬃce
System Driver’ (the latter has a version for 64-bit Windows, and that will also read earlier
On OS X the Actual Technologies (https://www.actualtech.com/product_access.php)
drivers provide ODBC interfaces to Access databases (including Access 2007/2010) and to Excel
spreadsheets (not including Excel 2007/2010).
Many simultaneous connections are possible. A connectionis openedby acallto odbcConnect
or odbcDriverConnect (which on the Windows GUI allows a database to be selected via dialog
boxes) which returns a handle used for subsequent access to the database. Printing a connection
will provide some details of the ODBC connection, and calling odbcGetInfo will give details on
the client and server.
Aconnection is closed by a call to close or odbcClose, and also (with a warning) when not
Robject refers to it and at the end of an R session.
Details of the tables on a connection can be found using sqlTables.
Function sqlSave copies an R data frame to a table in the database, and sqlFetch copies a
table in the database to an R data frame.
query can be sent to the database by a call to sqlQuery. This returns the result in
an R data frame. (sqlCopy sends a query to the database and saves the result as a table in the
database.) A ﬁner level of controlis attained by ﬁrst calling odbcQuery and thensqlGetResults
to fetch the results. The latter can be used within a loop to retrieve a limited number of rows
at a time, as can function sqlFetchMore.
Here is an example using PostgreSQL, for which the
driver maps column and data
frame names to lower case. We use a database testdb we created earlier, and had the DSN
(data source name) set up in ~/.odbc.ini under unixODBC. Exactly the same code worked
using MyODBC to access a MySQL database under Linux or Windows (where MySQL also
maps names to lowercase). Under Windows,
sare set up in the
applet in the Control
Panel (‘Data Sources (ODBC)’ in the ‘Administrative Tools’ section).
## tell it to map names to l/case
> channel <- odbcConnect("testdb", uid="ripley", case="tolower")
## load a data frame into the database
> sqlSave(channel, USArrests, rownames = "state", addPK = TRUE)
## list the tables in the database
TABLE_QUALIFIER TABLE_OWNER TABLE_NAME TABLE_TYPE REMARKS
## list it
> sqlFetch(channel, "USArrests", rownames = "state")
murder assault urbanpop rape
## an SQL query, originally on one line
> sqlQuery(channel, "select state, murder from USArrests
where rape > 30 order by murder")
5 New Mexico
## remove the table
> sqlDrop(channel, "USArrests")
## close the connection
As a simple example of using
under Windows with a Excel spreadsheet, we can read
from a spreadsheet by
> channel <- odbcConnectExcel("bdr.xls")
## list the spreadsheets
Sheet1$ SYSTEM TABLE
Sheet2$ SYSTEM TABLE
Sheet3$ SYSTEM TABLE
## retrieve the contents of sheet 1, by either of
> sh1 <- sqlFetch(channel, "Sheet1")
> sh1 <- sqlQuery(channel, "select * from [Sheet1$]")
Notice that the speciﬁcation of the table is diﬀerent from the name returned by sqlTables:
sqlFetch is able to map the diﬀerences.
C# HTML5 PDF Viewer SDK deployment on Visual Studio .NET
C#.NET rotate PDF pages, C#.NET search text in PDF, C# Unzip the download package and you can find a project Once done debugging with x86 dlls, replace the x86 how to make a pdf document text searchable; find text in pdf files
5 Binary ﬁles
Binary connections (Chapter 7 [Connections], page 24) are now the preferred way to handle
5.1 Binary data formats
Packages hdf5 (https: / / CRAN . R-project . org / package=hdf5), h5r (https: / / CRAN .
R-project.org/package=h5r), Bioconductor’s rhdf5, RNetCDF (https://CRAN.R-project.
org/package=RNetCDF), ncdf (https://CRAN.R-project.org/package=ncdf) and ncdf4
provide interfaces to
(Hierarchical Data Format, see https://www.hdfgroup.org/HDF5/) and to UCAR’s netCDF
data ﬁles (network Common Data Form, see http://www.unidata.ucar.edu/software/
Both of these are systems to store scientiﬁc data in array-oriented ways, including descrip-
tions, labels, formats, units, . ... HDF5 also allows groups of arrays, and the R interface maps
lists to HDF5 groups, and can write numeric and character vectors and matrices.
NetCDF’s version 4 format (confusingly, implemented in netCDF 4.1.1 and later, but not in
4.0.1) includes the use of various HDF5 formats. This is handled by package ncdf4 (https://
CRAN.R-project.org/package=ncdf4) whereas RNetCDF (https://CRAN.R-project.org/
package=RNetCDF) and ncdf (https://CRAN.R-project.org/package=ncdf) handle version 3
The availability of software to support these formats is somewhat limited by platform, espe-
cially on Windows.
5.2 dBase ﬁles (DBF)
dBase was a DOS program written by Ashton-Tate and later owned by Borland which has a
binary ﬂat-ﬁle format that became popular, with ﬁle extension .dbf. It has been adopted for the
’Xbase’ family of databases, covering dBase, Clipper, FoxPro and their Windows equivalents Vi-
sual dBase, Visual Objects and Visual FoxPro (see http://www.e-bachmann.dk/docs/xbase.
htm). A dBase ﬁle contains a header andthen a series of ﬁelds and so is most similar to an R data
frame. The data itself is stored in text format, and can include character, logical and numeric
ﬁelds, and other types in later versions (see for example http://www.digitalpreservation.
gov/formats/fdd/fdd000325.shtml and http://www.clicketyclick.dk/databases/xbase/
Functions read.dbf and write.dbf provide ways to read and write basic DBF ﬁles on all
R platforms. For Windows users odbcConnectDbase in package RODBC (https://CRAN.
R-project.org/package=RODBC) provides more comprehensive facilities to read DBF ﬁles
via Microsoft’s dBase ODBC driver (and the Visual FoxPro driver can also be used via
6 Image ﬁles
Aparticular class of binary ﬁles are those representing images, and a not uncommon request is
to read such a ﬁle into R as a matrix.
There are many formats for image ﬁles (most with lots of variants), and it may be necessary
to use external conversion software to ﬁrst convert the image into one of the formats for which
apackage currently provides an R reader. A versatile example of such software is ImageMagick
and its fork GraphicsMagick. These provide command-line programs convert and gm convert
to convert images from one format to another: what formats they can input is determined when
they are compiled, and the supported formats can be listed by e.g. convert -list format.
Package pixmap (https://CRAN.R-project.org/package=pixmap) has a function read.pnm
to read ‘portable anymap’ images in PBM (black/white), PGM (grey) and PPM (RGB colour)
formats. These are also known as ‘netpbm’ formats.
Packages bmp (https://CRAN.R-project.org/package=bmp), jpeg (https://CRAN.
R-project.org/package=jpeg) and png (https://CRAN.R-project.org/package=png) read
the formats after which they are named. See also packages biOps (https://CRAN.R-project.
org/package=biOps) and Momocs (https://CRAN.R-project.org/package=Momocs), and
Bioconductor package EBImage.
TIFF is more a meta-format, a wrapper within which a very large variety of image formats
can be embedded. Packages rtiﬀ (https://CRAN.R-project.org/package=rtiff) (orphaned)
and tiﬀ (https://CRAN.R-project.org/package=tiff) can read some of the sub-formats
(depending on the external libtiff software against which they are compiled). There some
facilities for specialized sub-formats, for example in Bioconductor package beadarray.
Raster ﬁles are common in the geographical sciences, and package rgdal (https://CRAN.
R-project.org/package=rgdal) provides an interface to GDAL which provides some facilities
of its own to read raster ﬁles and links to many others. Which formats it supports is determined
when GDAL is compiled: use gdalDrivers() to see what these are for the build you are using.
It can be useful for uncommon formats such as JPEG 2000 (which is a diﬀerent format from
JPEG,andnot currently supportedin the OSX nor Windows binary versions of rgdal (https://
Connections are used in R in the sense of Chambers (1998) and Ripley (2001), a set of functions
to replace the use of ﬁle names by a ﬂexible interface to ﬁle-like objects.
7.1 Types of connections
The most familiar type of connection will be a ﬁle, and ﬁle connections are created by function
file. File connections can (if the OS will allow it for the particular ﬁle) be opened for reading
or writing or appending, in text or binary mode. In fact, ﬁles can be opened for both reading
and writing, and R keeps a separate ﬁle position for reading and writing.
Note that by default a connection is not openedwhenit is created. The rule is that a function
using a connection should open a connection (needed) if the connection is not already open, and
close a connection after use if it opened it. In brief, leave the connection in the state you found
it in. There are generic functions open and close with methods to explicitly open and close
Files compressed via the algorithm used by gzip can be used as connections created by the
function gzfile, whereas ﬁles compressed by bzip2 can be used via bzfile.
Unix programmers are used to dealing with special ﬁles stdin, stdout and stderr. These
exist as terminal connections in R. They may be normal ﬁles, but they might also refer to input
from and output to a GUI console. (Even with the standard Unix R interface, stdin refers to
the lines submitted from readline rather than a ﬁle.)
The three terminal connections are always open, and cannot be opened or closed. stdout
and stderr are conventionally used for normal output and error messages respectively. They
may normally go to the same place, but whereas normal output can be re-directed by a call
to sink, error output is sent to stderr unless re-directed by sink, type="message"). Note
carefully the language used here: the connections cannot be re-directed, but output can be sent
to other connections.
Text connections are another source of input. They allow R character vectors to be read as
if the lines were being read from a text ﬁle. A text connection is created and opened by a call to
textConnection, which copies the current contents of the character vector to an internal buﬀer
at the time of creation.
Text connections canalso beusedtocaptureR output to a character vector. textConnection
can be asked to create a new character object or append to an existing one, in both cases in the
user’s workspace. The connection is opened by the call to textConnection, and at all times the
complete lines output to the connection are available in the R object. Closing the connection
writes any remaining output to a ﬁnal element of the character vector.
Pipes are a special form of ﬁle that connects to another process, and pipe connections are
created by the function pipe. Opening a pipe connection for writing (it makes no sense to
append to a pipe) runs an OS command, and connects its standard input to whatever R then
writes to that connection. Conversely, opening a pipe connection for input runs an OS command
and makes its standard output available for R input from that connection.
sof types ‘http://’, ‘ftp://’ and ‘file://’ can be read from using the function url.
For convenience, file will also accept these as the ﬁle speciﬁcation and call url. On most
platforms ‘https://’ are also accepted.
Sockets can also be used as connections via function socketConnection on platforms which
support Berkeley-like sockets (most Unix systems, Linux and Windows). Sockets can be written
to or read from, and both client and server sockets can be used.
Chapter 7: Connections
7.2 Output to connections
We have described functions cat, write, write.table and sink as writing to a ﬁle, possibly
appending to a ﬁle if argument append = TRUE, and this is what they did prior to R version
The current behaviour is equivalent, but what actually happens is that when the file ar-
gument is a character string, a ﬁle connection is opened (for writing or appending) and closed
again at the end of the function call. If we want to repeatedly write to the same ﬁle, it is more
eﬃcient to explicitly declare and open the connection, and pass the connection object to each
call to an output function. This also makes it possible to write to pipes, which was implemented
earlier in a limited way via the syntax file = "|cmd" (which can still be used).
There is a function writeLines to write complete text lines to a connection.
Some simple examples are
zz <- file("ex.data", "w") # open an output file connection
cat("TITLE extra line", "2 3 5 7", "", "11 13 17",
file = zz, sep = "\n")
cat("One more line\n", file = zz)
## convert decimal point to comma in output, using a pipe (Unix)
## both R strings and (probably) the shell need \ doubled
zz <- pipe(paste("sed s/\\\\./,/ >", "outfile"), "w")
cat(format(round(rnorm(100), 4)), sep = "\n", file = zz)
## now look at the output file:
file.show("outfile", delete.file = TRUE)
## capture R output: use examples from help(lm)
zz <- textConnection("ex.lm.out", "w")
example(lm, prompt.echo = "> ")
## now ‘ex.lm.out’ contains the output for futher processing.
## Look at it by, e.g.,
cat(ex.lm.out, sep = "\n")
7.3 Input from connections
The basic functions to read from connections are scan and readLines. These take a character
string argument and open a ﬁle connection for the duration of the function call, but explicitly
opening a ﬁle connection allows a ﬁle to be read sequentially in diﬀerent formats.
Other functions that call scan can also make use of connections, in particular read.table.
Some simple examples are
## read in file created in last examples
## read listing of current directory (Unix)
Chapter 7: Connections
# remove trailing commas from an input file.
# Suppose we are given a file ‘data’ containing
450, 390, 467, 654, 30, 542, 334, 432, 421,
357, 497, 493, 550, 549, 467, 575, 578, 342,
446, 547, 534, 495, 979, 479
# Then read this by
scan(pipe("sed -e s/,$// data"), sep=",")
For convenience, if the file argument speciﬁes a FTP or HTTP
is opened for
reading via url. Specifying ﬁles via ‘file://foo.bar’ is also allowed.
Cprogrammers may be familiar with the ungetc function to push back a character onto a text
input stream. R connections have the same idea in a more powerful way, in that an (essentially)
arbitrary number of lines of text can be pushed back onto a connection via a call to pushBack.
Pushbacks operate as a stack, so a read request ﬁrst uses each line from the most recently
pushbacked text, then those from earlier pushbacks and ﬁnally reads from the connection itself.
Once a pushbacked line is read completely, it is cleared. The number of pending lines pushed
back can be found via a call to pushBackLength.
Asimple example will show the idea.
> zz <- textConnection(LETTERS)
> readLines(zz, 2)
 "A" "B"
> scan(zz, "", 4)
Read 4 items
 "C" "D" "E" "F"
> pushBack(c("aa", "bb"), zz)
> scan(zz, "", 4)
Read 4 items
 "aa" "bb" "G" "H"
Pushback is only available for connections opened for input in text mode.
7.4 Listing and manipulating connections
A summary of all the connections currently opened by the user can be found by
showConnections(), and a summary of all connections, including closed and terminal
connections, by showConnections(all = TRUE)
The generic function seek can be used to read and (on some connections) reset the current
position for reading or writing. Unfortunately it depends onOSfacilities which may be unreliable
(e.g. with text ﬁles under Windows). Function isSeekable reports if seek can change the
position on the connection given by its argument.
The function truncate canbeusedtotruncate a ﬁleopened for writing at its current position.
It works only for file connections, and is not implemented on all platforms.
7.5 Binary connections
Functions readBin and writeBin read to and write from binary connections. A connection is
opened in binary mode by appending "b" to the mode speciﬁcation, that is using mode "rb" for
reading, and mode "wb" or "ab" (where appropriate) for writing. The functions have arguments
readBin(con, what, n = 1, size = NA, endian = .Platform$endian)
writeBin(object, con, size = NA, endian = .Platform$endian)
Documents you may be interested
Documents you may be interested