DeepLearningTutorial,Release0.1
Second,wecomputethevalueforf
t
,theactivationofthememorycells’forgetgatesattimet:
f
t
=(W
f
x
t
+U
f
h
t 1
+b
f
)
(13.3)
Giventhevalueoftheinputgateactivationi
t
,theforgetgateactivationf
t
andthecandidatestatevalue
f
C
t
,
wecancomputeC
t
thememorycells’newstateattimet:
C
t
=i
t
f
C
t
+f
t
C
t 1
(13.4)
Withthenewstateofthememorycells,wecancomputethevalueoftheiroutputgatesand,subsequently,
theiroutputs:
o
t
=(W
o
x
t
+U
o
h
t 1
+V
o
C
t
+b
o
)
(13.5)
h
t
=o
t
tanh(C
t
)
(13.6)
13.3.2 Ourmodel
ThemodelweusedinthistutorialisavariationofthestandardLSTMmodel.Inthisvariant,theactivation
ofacell’soutputgatedoesnotdependonthememorycell’sstateC
t
. Thisallowsustoperformpartof
thecomputationmoreefficiently(seetheimplementationnote,below,fordetails). Thismeansthat,inthe
variantwehaveimplemented,thereisnomatrixV
o
andequation(13.5)isreplacedbyequation(13.7):
o
t
=(W
o
x
t
+U
o
h
t 1
+b
o
)
(13.7)
OurmodeliscomposedofasingleLSTMlayerfollowedbyanaveragepoolingandalogisticregression
layerasillustratedinFigure2below. Thus,fromaninputsequencex
0
;x
1
;x
2
;:::;x
n
,thememorycellsin
theLSTMlayerwillproducearepresentationsequenceh
0
;h
1
;h
2
;:::;h
n
. Thisrepresentationsequenceis
thenaveragedoveralltimestepsresultinginrepresentationh.Finally,thisrepresentationisfedtoalogistic
regressionlayerwhosetargetistheclasslabelassociatedwiththeinputsequence.
Implementationnote: Inthecodeincludedthistutorial,theequations(13.1),(13.2),(13.3)and(13.7)
areperformedinparalleltomakethecomputationmoreefficient. Thisispossiblebecausenoneofthese
equationsrelyonaresultproducedbytheotherones.ItisachievedbyconcatenatingthefourmatricesW
intoasingleweightmatrixWandperformingthesameconcatenationontheweightmatricesU
toproduce
thematrixUandthebiasvectorsb
toproducethevectorb. Then,thepre-nonlinearityactivationscanbe
computedwith:
z=(Wx
t
+Uh
t 1
+b)
Theresultisthenslicedtoobtainthepre-nonlinearityactivationsfori,f,
f
C
t
,andoandthenon-linearities
arethenappliedindependentlyforeach.
13.4 Code-Citations-Contact
13.4.1 Code
TheLSTMimplementationcanbefoundinthetwofollowingfiles:
13.4. Code-Citations-Contact
145
Change pdf into jpg - Convert PDF to JPEG images in C#.net, ASP.NET MVC, WinForms, WPF project
How to convert PDF to JPEG using C#.NET PDF to JPEG conversion / converter library control SDK
changing pdf to jpg file; convert pdf file to jpg format
Change pdf into jpg - VB.NET PDF Convert to Jpeg SDK: Convert PDF to JPEG images in vb.net, ASP.NET MVC, WinForms, WPF project
Online Tutorial for PDF to JPEG (JPG) Conversion in VB.NET Image Application
convert multi page pdf to jpg; changing file from pdf to jpg
DeepLearningTutorial,Release0.1
Figure13.2:Figure2:Illustrationofthemodelusedinthistutorial.ItiscomposedofasingleLSTMlayer
followedbymeanpoolingovertimeandlogisticregression.
146
Chapter13. LSTMNetworksforSentimentAnalysis
Online Convert Jpeg to PDF file. Best free online export Jpg image
Web Security. All your JPG and PDF files will be permanently erased from our servers after one hour. If you want to turn PDF file into image file format in
bulk pdf to jpg converter; reader convert pdf to jpg
C# Image Convert: How to Convert Adobe PDF to Jpeg, Png, Bmp, &
PDF document can be easily loaded into your C# String inputFilePath = @"C:\input.pdf"; String outputFilePath = @"C:\output.jpg"; // Convert PDF
convert pdf pictures to jpg; convert pdf photo to jpg
DeepLearningTutorial,Release0.1
• lstm.py:Mainscript.Definesandtrainthemodel.
• imdb.py:Secondaryscript.HandlestheloadingandpreprocessingoftheIMDBdataset.
Afterdownloadingbothscriptsandputtingbothinthesamefolder,theusercanrunthecodebycalling:
THEANO_FLAGS="floatX=float32" python lstm.py
Thescriptwillautomaticallydownloadthedataanddecompressit.
Note:TheprovidedcodesupportstheStochasticGradientDescent(SGD),AdaDeltaandRMSPropopti-
mizationmethods.YouareadvisedtouseAdaDeltaorRMSPropbecauseSGDappearstoperformspoorly
onthistaskwiththisparticularmodel.
13.4.2 Papers
Ifyouusethistutorial,pleasecitethefollowingpapers.
IntroductionoftheLSTMmodel:
• [pdf]Hochreiter,S.,&Schmidhuber,J.(1997).Longshort-termmemory.Neuralcomputation,9(8),
1735-1780.
AdditionoftheforgetgatetotheLSTMmodel:
• [pdf]Gers,F.A.,Schmidhuber,J.,&Cummins,F.(2000). . Learningtoforget:Continualprediction
withLSTM.Neuralcomputation,12(10),2451-2471.
MorerecentLSTMpaper:
• [pdf]Graves,Alex.Supervisedsequencelabellingwithrecurrentneuralnetworks.Vol.385.Springer,
2012.
PapersrelatedtoTheano:
• [pdf]Bastien,Frédéric,Lamblin,Pascal,Pascanu,Razvan,Bergstra,James,Goodfellow,Ian,Berg-
eron,Arnaud,Bouchard,Nicolas,andBengio,Yoshua. Theano: : newfeaturesandspeedimprove-
ments.NIPSWorkshoponDeepLearningandUnsupervisedFeatureLearning,2012.
• [pdf]Bergstra,James,Breuleux,Olivier,Bastien,Frédéric,Lamblin,Pascal,Pascanu,Razvan,Des-
jardins,Guillaume,Turian,Joseph,Warde-Farley,David,andBengio,Yoshua. Theano:aCPUand
GPUmathexpressioncompiler. InProceedingsofthePythonforScientificComputingConference
(SciPy),June2010.
Thankyou!
13.4.3 Contact
PleaseemailPierreLucCarrierorKyunghyunChoforanyproblemreportorfeedback.Wewillbegladto
hearfromyou.
13.4. Code-Citations-Contact
147
VB.NET PDF Convert to Images SDK: Convert PDF to png, gif images
Convert PDF to Jpg, Png, Bmp, Gif, Tiff and Bitmap in from local file or stream and convert it into BMP, GIF Or directly change PDF to Gif image file in VB.NET
convert online pdf to jpg; conversion pdf to jpg
C# Create PDF from images Library to convert Jpeg, png images to
convert PDF documents from multiple image formats, including Jpg, Png, Bmp Component for combining multiple image formats into one or multiple PDF file in C#
.pdf to jpg; .net convert pdf to jpg
DeepLearningTutorial,Release0.1
13.5 References
• Hochreiter,S.,&Schmidhuber,J.(1997).Longshort-termmemory.Neuralcomputation,9(8),1735-
1780.
• Gers,F.A.,Schmidhuber,J.,&Cummins,F.(2000). . Learningtoforget:Continualpredictionwith
LSTM.Neuralcomputation,12(10),2451-2471.
• Graves,A.(2012).Supervisedsequencelabellingwithrecurrentneuralnetworks(Vol.385).Springer.
• Hochreiter,S.,Bengio,Y.,Frasconi,P.,&Schmidhuber,J.(2001). . Gradientflowinrecurrentnets:
thedifficultyoflearninglong-termdependencies.
• Bengio,Y.,Simard,P.,&Frasconi,P.(1994).Learninglong-termdependencieswithgradientdescent
isdifficult.NeuralNetworks,IEEETransactionson,5(2),157-166.
• Maas,A.L.,Daly,R.E.,Pham,P.T., , Huang,D.,Ng, A.Y.,&Potts,C.(2011,June). . Learning
wordvectorsforsentimentanalysis. InProceedingsofthe49thAnnualMeetingoftheAssociation
forComputationalLinguistics:HumanLanguageTechnologies-Volume1(pp.142-150).Association
forComputationalLinguistics.
148
Chapter13. LSTMNetworksforSentimentAnalysis
C# PDF insert image Library: insert images into PDF in C#.net, ASP
Support various image formats, like Jpeg or Jpg, Png, Gif, Bmp, Tiff and other Import graphic picture, digital photo, signature and logo into PDF document.
convert pdf to high quality jpg; change pdf file to jpg online
VB.NET PDF insert image library: insert images into PDF in vb.net
Support various image formats, like Jpeg or Jpg, Png, Gif, Bmp, Tiff and other Import graphic picture, digital photo, signature and logo into PDF document.
convert from pdf to jpg; convert .pdf to .jpg online
CHAPTER
FOURTEEN
MODELINGANDGENERATINGSEQUENCESOFPOLYPHONIC
MUSICWITHTHERNN-RBM
Note: ThistutorialdemonstratesabasicimplementationoftheRNN-RBMasdescribedin[Boulanger-
Lewandowski12](pdf). Weassumethereaderisfamiliarwithrecurrentneuralnetworksusingthescanop
andrestrictedBoltzmannmachines(RBM).
Note: Thecodeforthissectionisavailablefordownloadhere:rnnrbm.py.
YouwillneedthemodifiedPythonMIDIpackage(GPLlicense)inyour$PYTHONPATHorintheworking
directoryinordertoconvertMIDIfilestoandfrompiano-rolls. Thescriptalsoassumesthatthecontent
oftheNottinghamDatabaseoffolktuneshasbeenextractedinthe../datadirectory.AlternativeMIDI
datasetsareavailablehere.
Notethatbothdependenciesabovecanbesetupautomaticallybyrunningthedownload.shscriptinthe
../datadirectory.
Caution: NeedTheano0.6ormorerecent.
14.1 TheRNN-RBM
TheRNN-RBMisanenergy-basedmodelfordensityestimationoftemporalsequences,wherethefeature
vectorv
(t)
attimesteptmaybehigh-dimensional.Itallowstodescribemultimodalconditionaldistributions
ofv
(t)
jA
(t)
,whereA
(t)
fv
j <tgdenotesthesequencehistoryattimet,viaaseriesofconditional
RBMs(oneaeachtimestep)whoseparametersb
(t)
v
;b
(t)
h
dependontheoutputofadeterministicRNNwith
hiddenunitsu
(t)
:
b
(t)
v
=b
v
+W
uv
u
(t 1)
(14.1)
b
(t)
h
=b
h
+W
uh
u
(t 1)
(14.2)
andthesingle-layerRNNrecurrencerelationisdefinedby:
u
(t)
=tanh(b
u
+W
uu
u
(t 1)
+W
vu
v
(t)
)
(14.3)
149
VB.NET Create PDF from images Library to convert Jpeg, png images
Support create PDF from multiple image formats in VB.NET, including Jpg, Png, Bmp, Gif, Tiff Turn multiple image formats into one or multiple PDF file.
convert multiple page pdf to jpg; best convert pdf to jpg
C# TIFF: C#.NET Code to Convert JPEG Images to TIFF
XDoc.Tiff for .NET, which can be stably integrated into C#.NET string[] imagePaths = { @"C:\demo1.jpg", @"C:\demo2.jpg", @"C:\demo3.jpg" }; // Construct List
convert pdf into jpg online; convert pdf to gif or jpg
DeepLearningTutorial,Release0.1
Theresultingmodelisunrolledintimeinthefollowingfigure:
TheoverallprobabilitydistributionisgivenbythesumovertheT timestepsinagivensequence:
P(fv
(t)
g)=
T
X
t=1
P(v
(t)
jA
(t)
)
(14.4)
wheretheright-handsidemultiplicandisthemarginalizedprobabilityofthet
th
RBM.
Notethatforclarityoftheimplementation,contrarilyto[BoulangerLewandowski12],weusetheobvious
namingconventionforweightmatricesandweuseu
(t)
insteadof
^
h
(t)
fortherecurrenthiddenunits.
14.2 Implementation
WewishtoconstructtwoTheanofunctions: onetotraintheRNN-RBM,andonetogeneratesamplese-
quencesfromit.
Fortraining,i.e. givenfv
(t)
g,theRNNhiddenstatefu
(t)
gandtheassociatedfb
(t)
v
;b
(t)
h
gparametersare
deterministicandcanbereadilycomputedforeachtrainingsequence.Astochasticgradientdescent(SGD)
updateontheparameterscanthenbeestimatedviacontrastivedivergence(CD)ontheindividualtimesteps
ofasequenceinthesamewaythatindividualtrainingexamplesaretreatedinamini-batchforregular
RBMs.
Sequencegenerationissimilarexceptthatthev
(t)
mustbesampledsequentiallyateachtimestepwitha
separate(non-batch)Gibbschainbeforebeingpasseddowntotherecurrenceandthesequencehistory.
14.2.1 TheRBMlayer
Thebuild_rbmfunctionshownbelowbuildsaGibbschainfromaninputmini-batch(abinarymatrix)
viatheCDapproximation.Notethatitalsosupportsasingleframe(abinaryvector)inthenon-batchcase.
1C50hapter14. ModelingandgeneratingsequencesofpolyphonicmusicwiththeRNN-RBM
DeepLearningTutorial,Release0.1
def build_rbm(v, W, , bv, bh, k):
’’’Construct a a k-step Gibbs chain starting at v v for r an RBM.
v : Theano vector or matrix
If a matrix, multiple chains will be run in n parallel l (batch).
W : Theano matrix
Weight matrix of the RBM.
bv : Theano vector
Visible bias vector of the RBM.
bh : Theano vector
Hidden bias vector of the RBM.
k : scalar or r Theano o scalar
Length of f the e Gibbs chain.
Return a (v_sample, cost, monitor, updates) tuple:
v_sample : Theano vector or matrix with the same shape as ‘v‘
Corresponds to the generated sample(s).
cost : Theano o scalar
Expression whose gradient with respect to W, bv, bh is the CD-k
approximation to the log-likelihood of ‘v‘ ‘ (training g example) under the
RBM. The e cost t is averaged in the batch case.
monitor: Theano scalar
Pseudo log-likelihood (also averaged in the e batch h case).
updates: dictionary of Theano variable -> Theano variable
The ‘updates‘ object returned by scan.’’’
def gibbs_step(v):
mean_h T.nnet.sigmoid(T.dot(v, W) bh)
rng.binomial(size=mean_h.shape, n=1, p=mean_h,
dtype=theano.config.floatX)
mean_v T.nnet.sigmoid(T.dot(h, W.T) bv)
rng.binomial(size=mean_v.shape, n=1, p=mean_v,
dtype=theano.config.floatX)
return mean_v, v
chain, updates theano.scan(lambda v: gibbs_step(v)[1], outputs_info=[v],
n_steps=k)
v_sample chain[-1]
mean_v gibbs_step(v_sample)[0]
monitor T.xlogx.xlogy0(v, mean_v) T.xlogx.xlogy0(v, mean_v)
monitor monitor.sum() v.shape[0]
def free_energy(v):
return -(v
*
bv).sum() T.log(T.exp(T.dot(v, W) bh)).sum()
cost (free_energy(v) free_energy(v_sample)) v.shape[0]
return v_sample, cost, monitor, updates
14.2. Implementation
151
DeepLearningTutorial,Release0.1
14.2.2 TheRNNlayer
Thebuild_rnnrbmfunctiondefinestheRNNrecurrencerelationtoobtaintheRBMparameters;the
recurrencefunctionisflexibleenoughtoservebothinthetrainingscenariowherev
(t)
isgivenandthe
“batch”RBMisconstructedattheendonthewholesequenceatonce,andinthegenerationscenariowhere
v
(t)
issampledseparatelyateachtimestepusingtheGibbschaindefinedabove.
def build_rnnrbm(n_visible, n_hidden, n_hidden_recurrent):
’’’Construct a a symbolic RNN-RBM and initialize e parameters.
n_visible : integer
Number of f visible e units.
n_hidden : integer
Number of f hidden n units of the conditional RBMs.
n_hidden_recurrent : integer
Number of f hidden n units of the RNN.
Return a (v, , v_sample, , cost, monitor, params, updates_train, v_t,
updates_generate) tuple:
v : Theano matrix
Symbolic variable e holding an input sequence e (used d during training)
v_sample : Theano matrix
Symbolic variable e holding the negative particles for CD log-likelihood
gradient estimation n (used during training)
cost : Theano o scalar
Expression whose gradient (considering v_sample constant) corresponds
to the LL L gradient t of the RNN-RBM (used during training)
monitor : Theano scalar
Frame-level pseudo-likelihood (useful for monitoring during training)
params : tuple of Theano shared variables
The parameters of the model to be optimized d during g training.
updates_train : : dictionary of Theano variable -> Theano variable
Update object that should be passed to theano.function when compiling
the training function.
v_t : Theano o matrix
Symbolic variable e holding a generated sequence (used during sampling)
updates_generate : dictionary of Theano variable -> Theano variable
Update object that should be passed to theano.function when compiling
the generation function.’’’
shared_normal(n_visible, n_hidden, 0.01)
bv shared_zeros(n_visible)
bh shared_zeros(n_hidden)
Wuh shared_normal(n_hidden_recurrent, n_hidden, 0.0001)
Wuv shared_normal(n_hidden_recurrent, n_visible, 0.0001)
Wvu shared_normal(n_visible, n_hidden_recurrent, 0.0001)
Wuu shared_normal(n_hidden_recurrent, n_hidden_recurrent, 0.0001)
bu shared_zeros(n_hidden_recurrent)
params W, bv, bh, Wuh, Wuv, Wvu, Wuu, bu
# learned parameters as shared
# variables
1C52hapter14. ModelingandgeneratingsequencesofpolyphonicmusicwiththeRNN-RBM
DeepLearningTutorial,Release0.1
T.matrix()
# a training sequence
u0 T.zeros((n_hidden_recurrent,))
# initial l value e for the RNN hidden
# units
# If ‘v_t‘ is s given, , deterministic recurrence to compute the variable
# biases bv_t, bh_t at each time step. If ‘v_t‘ ‘ is s None, same recurrence
# but with a a separate e Gibbs chain at each time e step p to sample (generate)
# from the RNN-RBM. The resulting sample v_t is s returned d in order to be
# passed down n to o the sequence history.
def recurrence(v_t, u_tm1):
bv_t bv T.dot(u_tm1, , Wuv)
bh_t bh T.dot(u_tm1, , Wuh)
generate v_t is None
if generate:
v_t, _, , _, updates build_rbm(T.zeros((n_visible,)), W, bv_t,
bh_t, k=25)
u_t T.tanh(bu T.dot(v_t, Wvu) T.dot(u_tm1, Wuu))
return ([v_t, u_t], updates) if generate else [u_t, bv_t, bh_t]
# For training, the deterministic recurrence is s used d to compute all the
# {bv_t, bh_t, 1 <= t <= T} given v. Conditional RBMs can then be trained
# in batches s using g those parameters.
(u_t, bv_t, bh_t), updates_train theano.scan(
lambda v_t, u_tm1,
*
_: recurrence(v_t, u_tm1),
sequences=v, outputs_info=[u0, NoneNone], , non_sequences=params)
v_sample, cost, monitor, updates_rbm build_rbm(v, W, bv_t[:], bh_t[:],
k=15)
updates_train.update(updates_rbm)
# symbolic loop for sequence generation
(v_t, u_t), updates_generate theano.scan(
lambda u_tm1,
*
_: recurrence(None, u_tm1),
outputs_info=[None, u0], non_sequences=params, n_steps=200)
return (v, v_sample, cost, monitor, params, updates_train, v_t,
updates_generate)
14.2.3 Puttingitalltogether
Wenowhaveallthenecessaryingredients tostarttrainingournetworkonrealsymbolicsequencesof
polyphonicmusic.
class RnnRbm:
’’’Simple class to train an RNN-RBM from MIDI files and to generate sample
sequences.’’’
def __init__(
self,
n_hidden=150,
n_hidden_recurrent=100,
lr=0.001,
r=(21109),
14.2. Implementation
153
DeepLearningTutorial,Release0.1
dt=0.3
):
’’’Constructs and compiles Theano functions s for r training and sequence
generation.
n_hidden : : integer
Number of hidden units of the conditional RBMs.
n_hidden_recurrent : integer
Number of hidden units of the RNN.
lr : float
Learning rate
r : (integer, integer) tuple
Specifies the pitch range of the piano-roll in MIDI note numbers,
including r[0] but not r[1], such that t r[1]-r[0] ] is the number of
visible units of the RBM at a given time step. The default (21,
109) corresponds s to the full range of piano (88 notes).
dt : float
Sampling period when converting the MIDI files into piano-rolls, or
equivalently the time difference between consecutive time steps.’’’
self.r
self.dt dt
(v, v_sample, cost, monitor, params, updates_train, v_t,
updates_generate) build_rnnrbm(
r[1r[0],
n_hidden,
n_hidden_recurrent
)
gradient T.grad(cost, , params, consider_constant=[v_sample])
updates_train.update(
((p, p lr
*
g) for p, g in zip(params, gradient))
)
self.train_function theano.function(
[v],
monitor,
updates=updates_train
)
self.generate_function theano.function(
[],
v_t,
updates=updates_generate
)
def train(self, files, batch_size=100, num_epochs=200):
’’’Train the e RNN-RBM via stochastic gradient descent (SGD) using MIDI
files converted to piano-rolls.
files : list of strings
List of f MIDI files that will be loaded d as s piano-rolls for training.
batch_size : integer
Training sequences will be split into subsequences of at most this
size before e applying the SGD updates.
1C54hapter14. ModelingandgeneratingsequencesofpolyphonicmusicwiththeRNN-RBM
Documents you may be interested
Documents you may be interested