DeepLearningTutorial,Release0.1
12.5 RecurrentNeuralNetworkModel
12.5.1 Rawinputencoding
Atokencorrespondstoaword.EachtokenintheATISvocabularyisassociatedtoanindex.Eachsentence
isaarrayofindexes(int32). Then,eachset(train,valid,test)isalistofarraysofindexes. Apython
dictionaryisdefinedformappingthespaceofindexestothespaceofwords.
>>> sentence
array([383, 189, , 13, , 193, 208, 307, 195, 502, 260, , 539,
7,
60, 72, , 8, 350, 384], dtype=int32)
>>> map(lambda x: index2word[x], , sentence)
[’please’, ’find’, ’a’, ’flight’, ’from’, ’miami’, , ’florida’,
’to’, ’las’, ’vegas’, ’<UNK>’, ’arriving’, , ’before’, , ’DIGIT’, "o’clock", ’pm’]
Samethingforlabelscorrespondingtothisparticularsentence.
>>> labels
array([126, 126, , 126, , 126, 126,
48,
50, 126,
78, 123,
81, 126,
15,
14,
89, 89], , dtype=int32)
>>> map(lambda x: index2label[x], , labels)
[’O’, ’O’, ’O’, ’O’, ’O’, ’B-fromloc.city_name’, ’B-fromloc.state_name’,
’O’, ’B-toloc.city_name’, ’I-toloc.city_name’, ’B-toloc.state_name’,
’O’, ’B-arrive_time.time_relative’, ’B-arrive_time.time’,
’I-arrive_time.time’, ’I-arrive_time.time’]
12.5.2 Contextwindow
Givenasentencei.e.anarrayofindexes,andawindowsizei.e.1,3,5,...,weneedtoconverteachwordin
thesentencetoacontextwindowsurroundingthisparticularword.Indetails,wehave:
def contextwin(l, win):
’’’
win :: int corresponding to the size of the window
given a list t of f indexes composing a sentence
l :: array containing the word indexes
it will return a list of list of indexes corresponding
to context windows surrounding each word in the e sentence
’’’
assert (win 2== 1
assert win >= 1
list(l)
lpadded win // 2
*
[-1win // 2
*
[-1]
out [lpadded[i:(i win)] for in range(len(l))]
assert len(out) == len(l)
return out
12.5. RecurrentNeuralNetworkModel
135
Convert pdf file to jpg - Convert PDF to JPEG images in C#.net, ASP.NET MVC, WinForms, WPF project
How to convert PDF to JPEG using C#.NET PDF to JPEG conversion / converter library control SDK
convert multi page pdf to single jpg; change pdf file to jpg file
Convert pdf file to jpg - VB.NET PDF Convert to Jpeg SDK: Convert PDF to JPEG images in vb.net, ASP.NET MVC, WinForms, WPF project
Online Tutorial for PDF to JPEG (JPG) Conversion in VB.NET Image Application
change file from pdf to jpg on; change pdf to jpg online
DeepLearningTutorial,Release0.1
Theindex-1correspondstothePADDINGindexweinsertatthebeginning/endofthesentence.
Hereisasample:
>>> x
array([0, 1, 2, 3, 4], dtype=int32)
>>> contextwin(x, 3)
[[-1, 0, 1],
[ 0, 1, 2],
[ 1, 2, 3],
[ 2, 3, 4],
[ 3, 4,-1]]
>>> contextwin(x, 7)
[[-1, -1, -1, 0, , 1, , 2, 3],
[-1, -1,
0, 1, , 2, , 3, 4],
[-1,
0,
1, 2, , 3, , 4,-1],
[ 0,
1,
2, 3, , 4,-1,-1],
[ 1,
2,
3, 4,-1,-1,-1]]
Tosummarize,westartedwithanarrayofindexesandendedwithamatrixofindexes.Eachlinecorresponds
tothecontextwindowsurroundingthisword.
12.5.3 Wordembeddings
Oncewehavethesentenceconvertedtocontextwindowsi.e.amatrixofindexes,wehavetoassociatethese
indexestotheembeddings(real-valuedvectorassociatedtoeachword).UsingTheano,itgives:
import theanonumpy
from theano import tensor as T
# nv :: size of our vocabulary
# de :: dimension n of f the embedding space
# cs :: context window size
nv, de, cs 1000505
embeddings theano.shared(0.2
*
numpy.random.uniform(-1.01.0, \
(nv+1, de)).astype(theano.config.floatX)) # add d one e for PADDING at the end
idxs T.imatrix() # as many columns as words in the context window and as many lines as words in the sentence
x
self.emb[idxs].reshape((idxs.shape[0], de
*
cs))
Thexsymbolicvariablecorrespondstoamatrixofshape(numberofwordsinthesentences,dimensionof
theembeddingspaceXcontextwindowsize).
Let’scompileatheanofunctiontodoso
>>> sample
array([0, 1, 2, 3, 4], dtype=int32)
>>> csample contextwin(sample, 7)
[[-1, -1, -1, 0, , 1, , 2, 3],
[-1, -1,
0, 1, , 2, , 3, 4],
[-1,
0,
1, 2, , 3, , 4,-1],
[ 0,
1,
2, 3, , 4,-1,-1],
136
Chapter12. RecurrentNeuralNetworkswithWordEmbeddings
Online Convert Jpeg to PDF file. Best free online export Jpg image
Convert a JPG to PDF. You can drag and drop your JPG file in the box, and then start immediately to sort the files, try out some settings and then create the
convert pdf file to jpg on; convert pdf to jpg
Online Convert PDF to Jpeg images. Best free online PDF JPEG
Online PDF to JPEG Converter. Download Free Trial. Convert a PDF File to JPG. Drag and drop your PDF in the box above and we'll convert the files for you.
change pdf to jpg image; bulk pdf to jpg
DeepLearningTutorial,Release0.1
[ 1,
2,
3, 4,-1,-1,-1]]
>>> theano.function(inputs=[idxs], outputs=x)
>>> f(csample)
array([[-0.08088442,
0.08458307,
0.05064092, ..., , 0.06876887,
-0.06648078, -0.15192257],
[-0.08088442,
0.08458307,
0.05064092, ..., , 0.11192625,
0.08745284,
0.04381778],
[-0.08088442,
0.08458307,
0.05064092, ..., , -0.00937143,
0.10804889,
0.1247109 ],
[ 0.11038255, -0.10563177, -0.18760249, ..., , -0.00937143,
0.10804889,
0.1247109 ],
[ 0.18738101,
0.14727569, -0.069544
, ..., , -0.00937143,
0.10804889,
0.1247109 ]], dtype=float32)
>>> f(csample).shape
(5, 350)
Wenowhaveasequence(oflength5whichiscorrespondstothelengthofthesentence)ofcontextwindow
wordembeddingswhichiseasytofeedtoasimplerecurrentneuralnetworktoiteratewith.
12.5.4 Elmanrecurrentneuralnetwork
Thefollowin(Elman)recurrentneuralnetwork(E-RNN)takesasinputthecurrentinput(timet)andthe
previoushiddentstate(timet-1).Thenititerates.
Intheprevioussection,weprocessedtheinputtofitthissequential/temporalstructure.Itconsistsinamatrix
wheretherow0correspondstothetimestept=0,therow1correspondstothetimestept=1,etc.
TheparametersoftheE-RNNtobelearnedare:
• thewordembeddings(real-valuedmatrix)
• theinitialhiddenstate(real-valuevector)
• twomatricesforthelinearprojectionoftheinputtandtheprevioushiddenlayerstatet-1
• (optional)bias.Recommendation:don’tuseit.
• softmaxclassificationlayerontop
Thehyperparametersdefinethewholearchitecture:
• dimensionofthewordembedding
• sizeofthevocabulary
• numberofhiddenunits
• numberofclasses
• randomseed+waytoinitializethemodel
Itgivesthefollowingcode:
class RNNSLU(object):
’’’ elman neural net model ’’’
def __init__(self, nh, nc, ne, de, cs):
12.5. RecurrentNeuralNetworkModel
137
C# Image Convert: How to Convert Adobe PDF to Jpeg, Png, Bmp, &
C# sample code for PDF to jpg image conversion. This demo code convert PDF file all pages to jpg images. // Define input and output files path.
change from pdf to jpg on; convert .pdf to .jpg online
C# Image Convert: How to Convert Dicom Image File to Raster Images
RasterEdge.XDoc.Office.Inner.Office03.dll. RasterEdge.XDoc.PDF.dll. This demo code convert dicom file all pages to jpg images.
convert pdf to jpg batch; convert pdf file into jpg format
DeepLearningTutorial,Release0.1
’’’
nh :: dimension of the hidden layer
nc :: number of classes
ne :: number of word embeddings in the vocabulary
de :: dimension of the word embeddings
cs :: word window context size
’’’
# parameters of the model
self.emb theano.shared(name=’embeddings’,
value=0.2
*
numpy.random.uniform(-1.01.0,
(ne+1, de))
# add one for padding at the end
.astype(theano.config.floatX))
self.wx theano.shared(name=’wx’,
value=0.2
*
numpy.random.uniform(-1.01.0,
(de
*
cs, nh))
.astype(theano.config.floatX))
self.wh theano.shared(name=’wh’,
value=0.2
*
numpy.random.uniform(-1.01.0,
(nh, nh))
.astype(theano.config.floatX))
self.theano.shared(name=’w’,
value=0.2
*
numpy.random.uniform(-1.01.0,
(nh, nc))
.astype(theano.config.floatX))
self.bh theano.shared(name=’bh’,
value=numpy.zeros(nh,
dtype=theano.config.floatX))
self.theano.shared(name=’b’,
value=numpy.zeros(nc,
dtype=theano.config.floatX))
self.h0 theano.shared(name=’h0’,
value=numpy.zeros(nh,
dtype=theano.config.floatX))
# bundle
self.params [self.emb, self.wx, self.wh, self.w,
self.bh, self.b, self.h0]
Thenweintegratethewaytobuildtheinputfromtheembeddingmatrix:
idxs T.imatrix()
self.emb[idxs].reshape((idxs.shape[0], , de
*
cs))
y_sentence T.ivector(’y_sentence’)
# labels
Weusethescanoperatortoconstructtherecursion,workslikeacharm:
def recurrence(x_t, h_tm1):
h_t T.nnet.sigmoid(T.dot(x_t, self.wx)
T.dot(h_tm1, self.wh) self.bh)
s_t T.nnet.softmax(T.dot(h_t, self.w) self.b)
return [h_t, s_t]
[h, s], _ theano.scan(fn=recurrence,
138
Chapter12. RecurrentNeuralNetworkswithWordEmbeddings
C# Create PDF from images Library to convert Jpeg, png images to
C# Create PDF from Raster Images, .NET Graphics and REImage File with XDoc Batch convert PDF documents from multiple image formats, including Jpg, Png, Bmp
batch pdf to jpg; convert pdf file into jpg
VB.NET PDF Convert to Images SDK: Convert PDF to png, gif images
Convert PDF documents to multiple image formats, including Jpg, Png, Bmp, Gif, Tiff, Bitmap, .NET Graphics, and REImage. Turn multipage PDF file into image
batch pdf to jpg converter; .net convert pdf to jpg
DeepLearningTutorial,Release0.1
sequences=x,
outputs_info=[self.h0, None],
n_steps=x.shape[0])
p_y_given_x_sentence s[:, 0, :]
y_pred T.argmax(p_y_given_x_sentence, axis=1)
Theanowillthencomputeallthegradientsautomaticallytomaximizethelog-likelihood:
lr T.scalar(’lr’)
sentence_nll = -T.mean(T.log(p_y_given_x_sentence)
[T.arange(x.shape[0]), y_sentence])
sentence_gradients T.grad(sentence_nll, self.params)
sentence_updates OrderedDict((p, p lr
*
g)
for p, g in
zip(self.params, sentence_gradients))
Nextcompilethosefunctions:
self.classify theano.function(inputs=[idxs], outputs=y_pred)
self.sentence_train theano.function(inputs=[idxs, y_sentence, lr],
outputs=sentence_nll,
updates=sentence_updates)
Wekeepthewordembeddingsontheunitspherebynormalizingthemaftereachupdate:
self.normalize theano.function(inputs=[],
updates={self.emb:
self.emb /
T.sqrt((self.emb
**
2)
.sum(axis=1))
.dimshuffle(0’x’)})
Andthat’sit!
12.6 Evaluation
Withthepreviousdefinedfunctions,youcancomparethepredictedlabelswiththetruelabelsandcompute
somemetrics.Inthisrepo,webuildawrapperaroundtheconllevalPERLscript.It’snottrivialtocompute
thosemetricsduetotheInsideOutsideBeginning(IOB)representationi.e.apredictionisconsideredcorrect
iftheword-beginningandtheword-insideandtheword-outsidepredictionsareallcorrect. Notethatthe
extensionistxtandyouwillhavetochangeittopl.
12.6. Evaluation
139
C# WPF PDF Viewer SDK to convert and export PDF document to other
Convert PDF to image file formats with high quality, support converting PDF to PNG, JPG, BMP and GIF. C#.NET WPF PDF Viewer Tool: Convert and Export PDF.
convert pdf picture to jpg; best pdf to jpg converter online
C# TIFF: C#.NET Code to Convert JPEG Images to TIFF
demo1.jpg", @"C:\demo2.jpg", @"C:\demo3.jpg" }; // Construct List in imagePaths) { Bitmap tmpBmp = new Bitmap(file); if (null Use C# Code to Convert Png to Tiff.
convert pdf to jpeg; batch pdf to jpg converter online
DeepLearningTutorial,Release0.1
12.7 Training
12.7.1 Updates
Forstochasticgradientdescent(SGD)update,weconsiderthewholesentenceasamini-batchandperform
oneupdatepersentence.ItispossibletoperformapureSGD(contrarytomini-batch)wheretheupdateis
doneononlyonesinglewordatatime.
Aftereachiteration/update,wenormalizethewordembeddingstokeepthemonaunitsphere.
12.7.2 StoppingCriterion
Early-stoppingonavalidationsetisourregularizationtechnique:thetrainingisrunforagivennumberof
epochs(asinglepassthroughthewholedataset)andkeepthebestmodelalongwithrespecttotheF1score
computedonthevalidationsetaftereachepoch.
12.7.3 Hyper-ParameterSelection
Althoughthereisinterestingresearch/codeonthetopicofautomatichyper-parameterselection,weusethe
KISSrandomsearch.
Thefollowingintervalscangiveyousomestartingpoint:
• learningrate:uniform([0.05,0.01])
• windowsize:randomvaluefrom{3,...,19}
• numberofhiddenunits:randomvaluefrom{100,200}
• embeddingdimension:randomvaluefrom{50,100}
12.8 RunningtheCode
Afterdownloadingthedatausingdownload:sh,theusercanthenrunthecodebycalling:
python code/rnnslu.py
(’NEW BEST: epoch’, 25, ’valid F1’, 96.84, ’best test F1’, 93.79)
[learningepoch 26 6 >> 100.00% completed in 28.76 (sec<<
[learningepoch 27 7 >> 100.00% completed in 28.76 (sec<<
...
(’BEST RESULT: epoch’, 57, ’valid F1’, 97.23, ’best test t F1’, 94.2, ’with the model’’rnnslu’)
12.8.1 Timing
RunningexperimentsonATISusingthisrepositorywillrunoneepochinlessthan40secondsoni7CPU
950@3.07GHzusinglessthan200MoofRAM:
140
Chapter12. RecurrentNeuralNetworkswithWordEmbeddings
DeepLearningTutorial,Release0.1
[learning] epoch h 0 0 >> 100.00% completed in 34.48 (sec) <<
Afterafewepochs,youobtaindecentperformance94.48%ofF1score.:
NEW BEST: epoch 28 valid F1 96.61 best test F1 94.19
NEW BEST: epoch 29 valid F1 96.63 best test F1 94.42
[learning] epoch h 30 0 >> 100.00% completed in 35.04 (sec) <<
[learning] epoch h 31 1 >> 100.00% completed in 34.80 (sec) <<
[...]
NEW BEST: epoch 40 valid F1 97.25 best test F1 94.34
[learning] epoch h 41 1 >> 100.00% completed in 35.18 (sec) <<
NEW BEST: epoch 42 valid F1 97.33 best test F1 94.48
[learning] epoch h 43 3 >> 100.00% completed in 35.39 (sec) <<
[learning] epoch h 44 4 >> 100.00% completed in 35.31 (sec) <<
[...]
12.8.2 WordEmbeddingNearestNeighbors
Wecancheckthek-nearestneighborsofthelearnedembeddings. L2andcosinedistancegavethesame
resultssoweplotthemforthecosinedistance.
atlanta
back
ap80
but
aircraft
business
a
august
actually
cheap
phoenix
live
ap57
if
plane
coach
people
september
provide
weekday
denver
lives
ap
up
service
first
do
january
prices
weekdays
tacoma
both
connections
a
airplane
fourth
but
june
stop
am
columbus
how
tomorrow
now
seating
thrift
numbers
december
number
early
seattle
me
before
amount
stand
tenth
abbreviation
november
flight
sfo
minneapolis
out
earliest
more
that
second
if
april
there
milwaukee
pittsburgh
other
connect
abbreviation
on
fifth
up
july
serving
jfk
ontario
plane
thrift
restrictions
turboprop
third
serve
jfk
thank
shortest
montreal
service
coach
mean
mean
twelfth
database
october
ticket
bwi
philadelphia
fare
today
interested
amount
sixth
passengers
may
are
lastest
Asyoucanjudge,thelimitedsizeofthevocabulary(about500words)givesusmitigatedperformance.
Accordingtohumanjudgement:somearegood,somearebad.
12.8. RunningtheCode
141
DeepLearningTutorial,Release0.1
142
Chapter12. RecurrentNeuralNetworkswithWordEmbeddings
CHAPTER
THIRTEEN
LSTMNETWORKSFORSENTIMENTANALYSIS
13.1 Summary
ThistutorialaimstoprovideanexampleofhowaRecurrentNeuralNetwork(RNN)usingtheLongShort
TermMemory(LSTM)architecturecanbeimplementedusingTheano. Inthistutorial,thismodelisused
toperformsentimentanalysisonmoviereviewsfromtheLargeMovieReviewDataset,sometimesknown
astheIMDBdataset.
Inthistask,givenamoviereview,themodelattemptstopredictwhetheritispositiveornegative.Thisisa
binaryclassificationtask.
13.2 Data
Aspreviouslymentioned,theprovidedscriptsareusedtotrainaLSTMrecurrentneuralnetworkonthe
LargeMovieReviewDatasetdataset.
Whilethedatasetispublic,inthistutorialweprovideacopyofthedatasetthathaspreviouslybeenpre-
processedaccordingtotheneedsofthisLSTMimplementation.Runningthecodeprovidedinthistutorial
willautomaticallydownloadthedatatothelocaldirectory. Inordertouseyourowndata,pleaseusea
(preprocessingscript)providedasapartofthistutorial.
Once the model is trained, , you u can test t it t with your r own n corpus using the e word-index x dictionary
(imdb.dict.pkl.gz)providedasapartofthistutorial.
13.3 Model
13.3.1 LSTM
Inatraditionalrecurrentneuralnetwork,duringthegradientback-propagationphase,thegradientsignal
canendupbeingmultipliedalargenumberoftimes(asmanyasthenumberoftimesteps)bytheweight
matrixassociatedwiththeconnectionsbetweentheneuronsoftherecurrenthiddenlayer.Thismeansthat,
themagnitudeofweightsinthetransitionmatrixcanhaveastrongimpactonthelearningprocess.
Iftheweightsinthismatrixaresmall(or,moreformally,iftheleadingeigenvalueoftheweightmatrix
issmallerthan1.0),itcanleadtoasituationcalledvanishinggradientswherethegradientsignalgetsso
143
DeepLearningTutorial,Release0.1
smallthatlearningeitherbecomesverysloworstopsworkingaltogether. Itcanalsomakemoredifficult
thetaskoflearninglong-termdependenciesinthedata.Conversely,iftheweightsinthismatrixarelarge
(or,again,moreformally,iftheleadingeigenvalueoftheweightmatrixislargerthan1.0),itcanleadtoa
situationwherethegradientsignalissolargethatitcancauselearningtodiverge.Thisisoftenreferredto
asexplodinggradients.
TheseissuesarethemainmotivationbehindtheLSTMmodelwhichintroducesanewstructurecalleda
memorycell(seeFigure1below). Amemorycelliscomposedoffourmainelements: : aninputgate,a
neuronwithaself-recurrentconnection(aconnectiontoitself),aforgetgateandanoutputgate.Theself-
recurrentconnectionhasaweightof1.0andensuresthat,barringanyoutsideinterference,thestateofa
memorycellcanremainconstantfromonetimesteptoanother.Thegatesservetomodulatetheinteractions
betweenthememorycellitselfanditsenvironment. Theinputgatecanallowincomingsignaltoalterthe
stateofthememorycellorblockit. Ontheotherhand,theoutputgatecanallowthestateofthememory
celltohaveaneffectonotherneuronsorpreventit.Finally,theforgetgatecanmodulatethememorycell’s
self-recurrentconnection,allowingthecelltorememberorforgetitspreviousstate,asneeded.
Figure13.1:Figure1:IllustrationofanLSTMmemorycell.
Theequationsbelowdescribehowalayerofmemorycellsisupdatedateverytimestept.Intheseequations
:
• x
t
istheinputtothememorycelllayerattimet
• W
i
,W
f
,W
c
,W
o
,U
i
,U
f
,U
c
,U
o
andV
o
areweightmatrices
• b
i
,b
f
,b
c
andb
o
arebiasvectors
First,wecomputethevaluesfori
t
,theinputgate,and
f
C
t
thecandidatevalueforthestatesofthememory
cellsattimet:
i
t
=(W
i
x
t
+U
i
h
t 1
+b
i
)
(13.1)
f
C
t
=tanh(W
c
x
t
+U
c
h
t 1
+b
c
)
(13.2)
144
Chapter13. LSTMNetworksforSentimentAnalysis
Documents you may be interested
Documents you may be interested