45
24
Flexible Generation of E-Learning Exams in R
The approach taken in exams2qti12() is essentially analogous to that of the Moodle inter-
face. It also has separate num, mchoice, schoice, string, and cloze XML transformation
functions, each of which is by default generated by make_itembody_qti12() (as exercises are
called items in OLAT), possibly supplying further arguments for customization.
exams2qti12(file, n = 1L, nsamp = NULL, dir = ".", verbose = FALSE,
template = "qti12", name = NULL, edir = NULL, tdir = NULL, sdir = NULL,
quiet = TRUE, resolution = 100, width = 4, height = 4,
encoding = "", num = NULL, mchoice = NULL, schoice = mchoice,
string = NULL, cloze = NULL, duration = NULL,
stitle = "Exercise", ititle = "Question",
adescription = "Please solve the following exercises.",
sdescription = "Please answer the following question.",
maxattempts = 1, cutvalue = 0, solutionswitch = TRUE, zip = TRUE,
points = NULL, eval = list(partial = TRUE, negative = FALSE), ...)
make_itembody_qti12(rtiming = FALSE, shuffle = FALSE, rshuffle = shuffle,
minnumber = NULL, maxnumber = NULL, defaultval = NULL, minvalue = NULL,
maxvalue = NULL, cutvalue = NULL, enumerate = TRUE, digits = NULL,
tolerance = is.null(digits), maxchars = 12,
eval = list(partial = TRUE, negative = FALSE), fix_num = TRUE)
For details about the arguments see ?exams2qti12. The main difference between the Moodle
XML and QTI 1.2 XML specifications is that the former just provides some control over
the individual exercises (or questions, items) whereas the latter also has control options for
the whole exam (or assessment). Therefore, the XML specification is somewhat more com-
plex. Hence, exams2qti12() also takes a template argument that is by default set to the
qti12.xml file provided within exams. The template must contain exactly one <section>
with exactly one <item> with a placeholder ##ItemBody##. Then, exams2qti12() reads the
template, replicates the <section> for each exercise, replicates the <item> n times within
each <section>, and then inserts the ##ItemBody## with the XML transformation functions
for num, mchoice, etc.
Similar to exams2moodle(), one can specify the points for each exercise (again overruling
the \expoints metainformation) or specify an eval argument that describes the evaluation
policy employed (see AppendixB for details).
One notable feature of the QTI 1.2 interfaces should be briefly explained: The fix_num
argument is provided to work around an error in OLAT. While numeric exercises/items are
not officially supported, they actually work correctly except for the display of the correct
answer in the final summary. This is fixed (by default) by extending the processing of the
answer (without changing the result). Another optional route for numeric exercises is to
process them up to a certain number of digits only (e.g., digits = 2) and store them as a
string. However, in that case, the tolerance cannot be employed.
In summary, most end users should just have to call the main interfaces exams2moodle() or
exams2qti12() and customize by setting options for num, mchoice, etc. as some list(...).
If this is not sufficient, though, the users could program their own XML transformation
functions for num, mchoice, etc. And finally, for QTI 1.2, a different template could be used.
46
Achim Zeileis, Nikolaus Umlauf, Friedrich Leisch
25
4. Extending the exams toolbox and writing new drivers
In some cases it is not sufficient to use the arguments of the existing exams2xyz() functions or
to provide alternative templates to them. In particular, when a completely different output
format is required (e.g., a different XML format), it might be necessary to develop new drivers
for the xexams() toolbox. One example for such a situation is the software product that is
currently employed for generating printed large-lecture exams at Universit
¨
at Innsbruck. This
allows for
specification of (static) single/multiple-choice exercises in a browser interface,
production of so-called“scrambled”PDF exams from it (where the static questions and
solutions are simply shuffled),
optical character recognition (OCR) of scans from the exams’ title pages,
computation of the points/marks achieved by the students.
Although, the exams package can also generate PDF exams directly, some courses still prefer
the interface that have been used previously.
Fortunately, this so-called LOPS exam server (developed by a spin-off company of WU Wien)
also employs an XML specification for importing/exporting its exams. Therefore, it was easily
possible for us to establish a new exams2lops() interface that produces one ZIP file for each
exam, including the XML plus supplementary graphics. A corresponding write driver gener-
ator make_exams_write_lops() is also supplied in the package. Its details are not discussed
here because the XML format adopted is specific to this WU-developed software which is not
widely used. The exams2lops() interface then essentially proceeds in the following manner:
htmltransform <- make_exercise_transform_html(converter = "tex2image",
base64 = FALSE)
lopswrite <- make_exams_write_lops(...)
xexams(file, n, nsamp, driver = list(
sweave = list(quiet = TRUE, pdf = FALSE, png = TRUE, ...),
read = NULL,
transform = htmltransform,
write = lopswrite),
...)
First, an HTML transform driver is set up which uses the "tex2image" converter because
the LOPS server does not support MathML. Then, it sets up the custom write driver using
acouple of extra arguments (...) whose details are suppressed here for simplicity. Finally,
xexams() is called with (1) the default sweave driver Sweave() with options set to producing
PNG but not PDF graphics, (2) the default read driver, (3) the tex2image-based T
E
X-to-
HTML transform driver, (4) the custom write driver.
Of course, the part that involves a certain amount of coding is to program the write driver
(or driver generator, as here). However, the building blocks for the weave/read/transform
steps can be easily recycled. Also, if readers of this manuscript need to code their own driver
generator, we recommend to use the drivers from the exams package for inspiration. Last
48
26
Flexible Generation of E-Learning Exams in R
but not least, the exams package is hosted and R-Forge (TheußlandZeileis2009) and also
provides a forum for support and discussions of e-learning exams in R athttp://R-Forge.
R-project.org/forum/?group_id=1337.
5. Summary and discussion
Summary
Motivated by the need for automatic generation of exams (or quizzes, tests, assessments)
for learning management systems, the exams package is turned into an extensible toolbox for
exam generation. While previous versions of the package just supported generation of random
replications of exams in PDF format, the new version of the package provides interfaces for
various output formats, such as PDF, HTML, or XML specifications for Moodle or OLAT.
All exam output formats are based on the same specification of exercise Sweave files whose
format was only slightly extended compared to previous versions. The flexibility of producing
different output formats is accomplished by adopting a new extensible framework that consists
of the following modular steps: (1) weaving a single exercise, (2) reading the resulting LAT
E
X
text and metainformation into R, (3) transforming the text (if necessary, e.g., from LAT
E
X
to HTML), (4) writing the text into output files such as LAT
E
X, HTML, or XML templates.
Flexible building blocks are available for each of the steps that can either be customized for
the existing output formats or reused for generating new output formats.
Infrastructure vs. content
As emphasized in the discussion of version 1 of exams (Gr¨unandZeileis2009), the objective of
the package is to provide the technological infrastructure for automatic generation of exams,
especially for large-lecture courses. Thus, users of exams should not have to worry about
implementation details and can focus on the content of their exams when they build up
a pool of exercises accompanying a particular course. Creating “good” exercises from an
educational (rather than computational) point of view is not a trivial task but guidelines for
this are beyond the scope of the exams package and this manuscript. Hence, we just provide
afew references to the relevant literature on statistical education and assessment: : Galand
Garfield(1997) and Garfield and Chance(2000) discuss issues s such as s topics covered d and
skills developed in statistics courses as well as suitable ways of assessment. Strategies for
good multiple-choice questions, especially if they are also used for self-study materials, are
suggested byKlinke(2004).
Strategies for setting up exercises
When switching a course to the exams infrastructure, clearly the most work has to go into the
generation of the content, i.e., the Sweave exercises. However, by the modular design of the
package it is easy to distribute the workload among a large team of contributors. Each person
can just work on stand-alone .Rnw files, e.g., for a particular exercises type or for the exercises
pertaining to a specific chapter from the lecture etc. Depending on the output formats, it
is typically a good idea to make sure that the exercise, foo.Rnw say, works as desired by
running exams2pdf("foo.Rnw") and exams2html("foo.Rnw") to make sure that it can be
appropriately rendered in both PDF and HTML. To check that the solution is correctly
56
Achim Zeileis, Nikolaus Umlauf, Friedrich Leisch
27
entered in the metainformation, it helps to run exams_metainfo(exams2html("foo.Rnw"))
(or analogously for exams2pdf()).
When the pool of exercises is ready, then it is typically useful to set up a convenience wrapper
function that (a) selects the desired exercises from this pool and (b) produces the desired
output format(s) for them. For the latter step, it may just be necessary to set the arguments
of one of the exams2xyz() functions appropriately or maybe to write a custom template that
can be plugged into the function. However, the customization of such a wrapper function is
typically not a lot of work and can be performed by a single person, e.g., the team member
with some more experience in the technologies involved (R, HTML, XML, ...). A useful
starting point for setting up such a wrapper can be generated with the exams_skeleton()
function, based on which different interfaces and templates can be easily explored.
Experiences at Universit
¨
at Innsbruck
In 2012, the Department of Statistics at Universit
¨
at Innsbruck built up infrastructure for a
new “Mathematics 101” course. The team included seven professors and lecturers, and six
student assistants. All professors and lecturers were previously familiar with R and LAT
E
X(but
not necessarily with HTML or XML) while several of the student assistants had experience
in neither. The workload was then split up so that the professors and lecturers designed the
content of the exercises and programmed prototypes. The student assistants then typically
performed tasks such as checking the correctness of the exercises, testing out the random data
generation or making it more flexible, and creating variations of existing exercises by making
small modifications in the underlying“stories”or changing the data generating process. Even
though many of the student assistants had no prior knowledge of R and LAT
E
X, they were
rather quickly able to work on the exercise Sweave files (with all the usual small problems
that often occur when learning R/LAT
E
X).
The resulting pool of exercises is maintained in a Subversion repository (SVN,Pilato,Collins-
Sussman, andFitzpatrick 2004)forversioncontrolsothatallteammemberscaneasilyobtain
the latest version or contribute fixes/improvements. In combination with the exams package
this approach proved to be rather successful in addressing the needs of multi-author and
cross-platform development.
After having the pool of exercises established, just one team member is concerned with run-
ning exams2qti12() and uploading the resulting ZIP file into OLAT for the biweekly online
tests. And for creating the printed tests at the end of the semester the exams2lops() or
exams2pdf() interfaces are employed.
9
Experiences at BOKU Wien
At BOKU a web-based online exercise system had been in place for several years (Moder
2011). The e old system used Fortran to generate data and a standalone web-interface e for
students to enter results and get feedback. The storylines of the old system were transfered
into Sweave files, the Fortran code was re-programmed in R for more flexibility in random
number generation. Workload management was similar as in Innsbruck: Professors and
lecturers supervised the effort, but programming of examples was done by a team of PhD
9
Meanwhile, further OCR infrastructure for scanning multiple/single-choice exams within R has been de-
veloped. This is not part of exams, yet, but interested readers are welcome to contact the authors.
43
28
Flexible Generation of E-Learning Exams in R
students, two of which were new at the department and had only limited prior experience
with R (but all had some experience in L
A
T
E
X).
There are three types of exams-generated exercises used throughout three “Statistics 101”
courses totalling more than 1200 students:
Online exercises where students get immediate feedback on correctness of results,
homework exercises where students create a small report as PDF and upload this to the
server, and
pen-and-paper multiple-choice exams as in package exams version 1.
For the pen-and-paper multiple-choice tests a mixture of exams2pdf() and exams2moodle()
is currently used: Moodle in principle generates exercise and answer sheets for multiple-choice
exams. However, in the current implementation the questiontext may not contain figures and
mathematical equations are really ugly. So question sheets are generated directly as PDF,
but the XML for Moodle is also created, which generates then in turn the answer sheets and
is used for automatic scanning and grading. A much more detailed description of contents
and example types has been written as a seperate manuscript and is currently under review.
The focus of this paper is to be a technical manual of the new features of package exams.
Outlook
In the current version, exams already provides a wide variety of different output formats,
some additional formats may be desirable for future developments though. For example,
QTI 2.0/2.1 is likely to become more widely adopted – and is already currently employed
by some programs such as ONYX. The current version of exams already provides a new
exams2qti21() function which so far has only been tested with ONYX. Hence, the function
will probably be changed and improved somewhat in future versions of the package.
Furthermore, we have started to explore support for the proprietary Blackboard system.
While this is, in principle, based on QTI 1.2, Blackboard employs its own and somewhat
special flavor of QTI 1.2.
Finally, users may be interested in extensions/adaptions of existing e-learning formats. A
forum for support and discussions of such issues is available on R-Forge athttp://R-Forge.
R-project.org/forum/?group_id=1337.
Acknowledgments
We are indebted to all our colleagues in the“Mathematics”team at the Department of Statis-
tics at Universit¨at Innsbruck for testing and challenging the code and making suggestions for
improvement. Furthermore, code patches, discussions, and feedback by Dominik Ernst (Uni-
versit¨at f¨ur Bodenkultur Wien) and Niels Smits (Vrije Universiteit Amsterdam) are gratefully
acknowledged. This project was also partially supported by an e-learning grant (#2011.241)
of the Universit¨at Innsbruck.
References
49
Achim Zeileis, Nikolaus Umlauf, Friedrich Leisch
29
Agea
´
A,Crespo Garc´ıaRM, Delgado Kloos C, Guti´errez I,Leony D,Pardo A (2009).“Analysis
of Existing Specifications and Standards for Assessment and Evaluation and Their Usage
in Europe.”Deliverable D6.1, ICOPER Network for Interoperable Content for Performance
in a Competency-Driven Society. URLhttp://www.icoper.org/results/deliverables/
D6-1.
Blackboard Inc (2010). Blackboard Learn 9.1. Washington, DC. URL L http://www.
blackboard.com/.
Blesius CR, Moreno-Ger P, Neumann G, Raffenne E, Gonzalez Boticario J, Delgado Kloos
C(2007). “.LRN: E-Learning Inside and Outside the Classroom – Supporting Collabora-
tive Learning Communities Using a Web Application Toolkit.” In B Fern´andez-Manj´on,
JM S´anchez-P´erez, JA G´omez-Pulido, MA Vega-Rodr´ıguez, J Bravo-Rodr´ıguez (eds.),
Computers and Education: E-Learning, From Theory to Practice. Springer-Verlag, Dor-
drecht.
BPS Bildungsportal Sachsen GmbH (2014). ONYX Testsuite. Chemnitz, Germany. URL
http://onyx.bps-system.de/.
de Leeuw J(2001).“Reproducible Research: The Bottom Line.”Technical Report 2001031101,
Department of Statistics Papers, University of California, Los Angeles. URL L http://
repositories.cdlib.org/uclastat/papers/2001031101/.
Design Science (2013a). MathJax 2.3 Documentation. Long Beach, CA. URLhttp://www.
MathJax.org/.
Design Science (2013b). MathPlayer: Display MathML in Your Browser. Long Beach, CA.
URLhttp://www.dessci.com/en/products/mathplayer/.
Dougiamas M, et al. (2014). Moodle, Version 2.6. URLhttp://moodle.org/.
frentix GmbH (2014). OpenOLAT 9.4 – User Manual. Z
¨
urich, Switzerland. URLhttp:
//www.openolat.org/.
Gal I, Garfield JB (eds.) (1997). The Assessment Challenge in Statistics Education. IOS
Press, Netherlands.
Garfield JB, Chance B (2000). “Assessment in Statistics Education: Issues and Challenges.”
Mathematical Thinking and Learning, 2(1/2), 99–125.
Gr
¨
un B, Zeileis A (2009). “Automatic Generation of Exams in R.” Journal of Statistical
Software, 29(10), 1–14. URLhttp://www.jstatsoft.org/v29/i10/.
Hutchinson IH (2012). TtH: A ‘T
E
X to HTML’ Translator. C code version 4.03, URL
http://hutchinson.belmont.ma.us/tth/.
Hutchinson IH, Leisch F, Zeileis A (2013). tth: T
E
Xto HTML/MathML Translators tth/ttm.
Rpackage version 4.3-1, URLhttp://CRAN.R-project.org/package=tth.
ImageMagick Studio LLC (2014). ImageMagick: Convert, Edit, and Compose Images.
Version 6.8.9-1, URLhttp://www.ImageMagick.org/.
57
30
Flexible Generation of E-Learning Exams in R
IMS Global Learning Consortium, Inc (2002). IMS Question & Test Interoperability: ASI
XML Binding Specification Final Specification Version 1.2. Lake Mary, FL. URLhttp:
//www.imsglobal.org/question/qtiv1p2/imsqti_asi_bindv1p2.html.
Klinke S (2004). “Q&A – Variable Multiple Choice Exercises with Commented Answers.” In
JAntoch (ed.), COMPSTAT 2004 – Proceedings in Computational Statistics, pp. 1323–
1328. Physica Verlag, Heidelberg.
Knuth DE (1984). The T
E
Xbook, volume A of Computers and Typesetting. Addison-Wesley,
Reading, Massachusetts.
Knuth DE (1992). Literate Programming, volume 27 of CSLI Lecture Notes. Center for the
Study of Language and Information, Stanford, California.
Kuhn M (2014). “CRAN Task View: Reproducible Research.” Version 2014-01-20, URL
http://CRAN.R-project.org/view=ReproducibleResearch.
Lamport L (1994). L
A
T
E
X: A Document Preparation System. 2nd edition. Addison-Wesley,
Reading, Massachusetts.
Leisch F (2002). “Dynamic Generation of Statistical Reports Using Literate Data Analysis.”
In W H
¨
ardle, B R
¨
onz (eds.), COMPSTAT 2002 – Proceedings in Computational Statistics,
pp. 575–580. Physica Verlag, Heidelberg.
Leisch F (2012a). “Sweave FAQ.” URL L http://www.stat.uni-muenchen.de/~leisch/
Sweave/.
Leisch F (2012b). Sweave User Manual. URL L http://www.stat.uni-muenchen.de/
~leisch/Sweave/.
Leisch F, Rossini AJ (2003). “Reproducible Statistical Research.” Chance, 16(2), 46–50.
Moder K (2011). “Statistical Education at the University of Natural Resources And Life
Sciences inVienna.”In BShishkov (ed.), Proceedings of the Second International Conference
on Innovative Developments in ICT (Information and Communication Technology),pp.77–
81.
Pilato CM, Collins-Sussman B, Fitzpatrick BW (2004). Version Control with Subversion.
O’Reilly. Full book available online athttp://svnbook.red-bean.com/.
RCore Team (2014). R: A Language and Environment for Statistical Computing. R Founda-
tion for Statistical Computing, Vienna, Austria. URLhttp://www.R-project.org/.
RStudio Team (2014). RStudio: Integrated Development for R. RStudio, Inc., Boston, MA.
URLhttp://www.RStudio.com/ide/.
Theußl S, Zeileis A (2009). “Collaborative Software Development Using R-Forge.” The R
Journal, 1(1), 9–14. URLhttp://journal.R-project.org/2009-1/RJournal_2009-1_
Theussl+Zeileis.pdf.
Universit
¨
at Z
¨
urich (2012). OLAT 7.6 – User Manual. IT Services, Universit
¨
at Z
¨
urich,
Switzerland. URLhttp://www.olat.org/.
Documents you may be interested
Documents you may be interested
