DeepLearningTutorial,Release0.1
index T.lscalar(’index’)
# index to a minibatch
Tobeabletochangethecorruptionlevelorthelearningrateduringtraining,weassociateTheanovariables
withthem.
corruption_level T.scalar(’corruption’# % of corruption to use
learning_rate T.scalar(’lr’)
# learning g rate e to use
# begining of a batch, given ‘index‘
batch_begin index
*
batch_size
# ending g of f a batch given ‘index‘
batch_end batch_begin batch_size
pretrain_fns []
for dA in self.dA_layers:
# get t the e cost and the updates list
cost, updates dA.get_cost_updates(corruption_level,
learning_rate)
# compile the theano function
fn theano.function(
inputs=[
index,
theano.Param(corruption_level, default=0.2),
theano.Param(learning_rate, default=0.1)
],
outputs=cost,
updates=updates,
givens={
self.x: train_set_x[batch_begin: batch_end]
}
)
# append ‘fn‘ to the list of functions
pretrain_fns.append(fn)
return pretrain_fns
Nowanyfunctionpretrain_fns[i]takesasargumentsindexandoptionallycorruption—the
corruptionlevelorlr—thelearningrate.Notethatthenamesoftheparametersarethenamesgiventothe
Theanovariableswhentheyareconstructed,notthenamesofthePythonvariables(learning_rateor
corruption_level).KeepthisinmindwhenworkingwithTheano.
In the same fashion n we e build a method for constructing the functions s required d during finetuning
(train_fn,valid_scoreandtest_score).
def build_finetune_functions(self, datasets, batch_size, learning_rate):
’’’Generates a function ‘train‘ that implements one step of
finetuning, a function ‘validate‘ that computes the error on
a batch from the validation set, and a function ‘test‘ that
computes the e error on a batch from the testing set
:type datasets: list of pairs of theano.tensor.TensorType
:param datasets: It is a list that contain n all l the datasets;
the has to contain three pairs, ‘train‘,
‘valid‘, ‘test‘ in this order, where each pair
8.1. StackedAutoencoders
85
Batch convert pdf to jpg online - Convert PDF to JPEG images in C#.net, ASP.NET MVC, WinForms, WPF project
How to convert PDF to JPEG using C#.NET PDF to JPEG conversion / converter library control SDK
change pdf into jpg; convert pdf images to jpg
Batch convert pdf to jpg online - VB.NET PDF Convert to Jpeg SDK: Convert PDF to JPEG images in vb.net, ASP.NET MVC, WinForms, WPF project
Online Tutorial for PDF to JPEG (JPG) Conversion in VB.NET Image Application
batch pdf to jpg; bulk pdf to jpg converter
DeepLearningTutorial,Release0.1
is formed of two Theano variables, one for the
datapoints, the other for r the e labels
:type batch_size: int
:param batch_size: size of a minibatch
:type learning_rate: float
:param learning_rate: learning rate used during finetune stage
’’’
(train_set_x, train_set_y) datasets[0]
(valid_set_x, valid_set_y) datasets[1]
(test_set_x, test_set_y) datasets[2]
# compute e number r of minibatches for training, validation and testing
n_valid_batches valid_set_x.get_value(borrow=True).shape[0]
n_valid_batches /= batch_size
n_test_batches test_set_x.get_value(borrow=True).shape[0]
n_test_batches /= batch_size
index T.lscalar(’index’)
# index to a [mini]batch
# compute e the e gradients with respect to the e model l parameters
gparams T.grad(self.finetune_cost, self.params)
# compute e list t of fine-tuning updates
updates [
(param, param gparam
*
learning_rate)
for param, gparam in zip(self.params, gparams)
]
train_fn theano.function(
inputs=[index],
outputs=self.finetune_cost,
updates=updates,
givens={
self.x: train_set_x[
index
*
batch_size: (index 1)
*
batch_size
],
self.y: train_set_y[
index
*
batch_size: (index 1)
*
batch_size
]
},
name=’train’
)
test_score_i theano.function(
[index],
self.errors,
givens={
self.x: test_set_x[
index
*
batch_size: (index 1)
*
batch_size
],
86
Chapter8. StackedDenoisingAutoencoders(SdA)
C# Create PDF from images Library to convert Jpeg, png images to
Batch convert PDF documents from multiple image formats, including Jpg, Png, Bmp, Gif, Tiff, Bitmap, .NET Graphics, and REImage.
best convert pdf to jpg; batch pdf to jpg converter
VB.NET PDF Convert to Images SDK: Convert PDF to png, gif images
VB.NET components for batch convert high resolution images from PDF. Convert PDF documents to multiple image formats, including Jpg, Png, Bmp, Gif, Tiff, Bitmap
best program to convert pdf to jpg; bulk pdf to jpg
DeepLearningTutorial,Release0.1
self.y: test_set_y[
index
*
batch_size: (index 1)
*
batch_size
]
},
name=’test’
)
valid_score_i theano.function(
[index],
self.errors,
givens={
self.x: valid_set_x[
index
*
batch_size: (index 1)
*
batch_size
],
self.y: valid_set_y[
index
*
batch_size: (index 1)
*
batch_size
]
},
name=’valid’
)
# Create e a a function that scans the entire validation set
def valid_score():
return [valid_score_i(i) for in xrange(n_valid_batches)]
# Create e a a function that scans the entire test set
def test_score():
return [test_score_i(i) for in xrange(n_test_batches)]
return train_fn, valid_score, test_score
Notethatvalid_scoreandtest_scorearenotTheanofunctions,butratherPythonfunctionsthat
loopovertheentirevalidationsetandtheentiretestset,respectively,producingalistofthelossesover
thesesets.
8.2 Puttingitalltogether
Thefewlinesofcodebelowconstructthestackeddenoisingautoencoder:
numpy_rng numpy.random.RandomState(89677)
print ’... building the model’
# construct the stacked denoising autoencoder class
sda SdA(
numpy_rng=numpy_rng,
n_ins=28
*
28,
hidden_layers_sizes=[100010001000],
n_outs=10
)
Therearetwostagesoftrainingforthisnetwork:layer-wisepre-trainingfollowedbyfine-tuning.
Forthepre-trainingstage,wewillloopoverallthelayersofthenetwork. Foreachlayerwewillusethe
8.2. Puttingitalltogether
87
VB.NET Create PDF from images Library to convert Jpeg, png images
Components to batch convert PDF documents in Visual Basic .NET class. Support create PDF from multiple image formats in VB.NET, including Jpg, Png, Bmp, Gif
convert pdf to jpg file; c# pdf to jpg
C# PDF Convert to Tiff SDK: Convert PDF to tiff images in C#.net
Powerful .NET control to batch convert PDF documents to tiff format in Visual C# .NET program. Also supports convert PDF files to jpg, jpeg images.
convert pdf to 300 dpi jpg; convert multiple pdf to jpg
DeepLearningTutorial,Release0.1
compiledTheanofunctionthatimplementsaSGDsteptowardsoptimizingtheweightsforreducingthe
reconstructioncostofthatlayer. Thisfunctionwillbeappliedtothetrainingsetforafixednumberof
epochsgivenbypretraining_epochs.
#########################
# PRETRAINING G THE E MODEL #
#########################
print ’... getting the pretraining functions’
pretraining_fns sda.pretraining_functions(train_set_x=train_set_x,
batch_size=batch_size)
print ’... pre-training the model’
start_time timeit.default_timer()
## Pre-train n layer-wise
corruption_levels [.1.2.3]
for in xrange(sda.n_layers):
# go through pretraining epochs
for epoch in xrange(pretraining_epochs):
# go o through h the training set
[]
for batch_index in xrange(n_train_batches):
c.append(pretraining_fns[i](index=batch_index,
corruption=corruption_levels[i],
lr=pretrain_lr))
print ’Pre-training layer %i, epoch %d, cost ’ (i, epoch),
print numpy.mean(c)
end_time timeit.default_timer()
print >> sys.stderr, (’The pretraining code for r file e ’ +
os.path.split(__file__)[1+
’ ran for %.2fm’ ((end_time start_time) 60.))
Thefine-tuningloopisverysimilartotheoneintheMultilayerPerceptron. Theonlydifferenceisthatit
usesthefunctionsgivenbybuild_finetune_functions.
8.3 RunningtheCode
Theusercanrunthecodebycalling:
python code/SdA.py
Bydefaultthecoderuns15pre-trainingepochsforeachlayer,withabatchsizeof1.Thecorruptionlevels
are0.1forthefirstlayer,0.2forthesecond,and0.3forthethird. Thepretraininglearningrateis0.001
andthefinetuninglearningrateis0.1.Pre-trainingtakes585.01minutes,withanaverageof13minutesper
epoch. Fine-tuningiscompletedafter36epochsin444.2minutes,withanaverageof12.34minutesper
epoch. Thefinalvalidationscoreis1.39%withatestingscoreof1.3%. . Theseresultswereobtainedona
machinewithanIntelXeonE5430@2.66GHzCPU,withasingle-threadedGotoBLAS.
88
Chapter8. StackedDenoisingAutoencoders(SdA)
JPEG to PDF Converter | Convert JPEG to PDF, Convert PDF to JPEG
and Benefits. Powerful image converter to convert images of Support a batch conversion of JPG to PDF with amazingly high speed; Get a compressed PDF file after
convert pdf to jpg 300 dpi; convert pdf into jpg
JPG to GIF Converter | Convert JPEG to GIF, Convert GIF to JPG
speed JPEG to GIF Converter, faster than other JPG Converters; Automatically sort the file name when you convert the files in batch; Storing conversion
batch convert pdf to jpg online; conversion of pdf to jpg
DeepLearningTutorial,Release0.1
8.4 TipsandTricks
Onewaytoimprovetherunningtimeofyourcode(assumingyouhavesufficientmemoryavailable),isto
computehowthenetwork,uptolayerk 1,transformsyourdata.Namely,youstartbytrainingyourfirst
layerdA.Onceitistrained,youcancomputethehiddenunitsvaluesforeverydatapointinyourdataset
andstorethisasanewdatasetthatyouwillusetotrainthedAcorrespondingtolayer2. Onceyouhave
trainedthedAforlayer2,youcompute,inasimilarfashion,thedatasetforlayer3andsoon.Youcansee
now,thatatthispoint,thedAsaretrainedindividually,andtheyjustprovide(onetotheother)anon-linear
transformationoftheinput.OncealldAsaretrained,youcanstartfine-tuningthemodel.
8.4. TipsandTricks
89
C# PDF Convert to Images SDK: Convert PDF to png, gif images in C#
An advanced .NET control able to batch convert PDF documents to image formats in C# Support exporting PDF to multiple image forms, including Jpg, Png, Bmp
convert pdf document to jpg; batch pdf to jpg online
JPG to DICOM Converter | Convert JPEG to DICOM, Convert DICOM to
Select "Convert to DICOM"; Select "Start" to start conversion How to Start Batch JPEG Conversion to DICOM. JPEG to DICOM Converter first; Load JPG images from
pdf to jpeg; pdf to jpg converter
DeepLearningTutorial,Release0.1
90
Chapter8. StackedDenoisingAutoencoders(SdA)
CHAPTER
NINE
RESTRICTEDBOLTZMANNMACHINES(RBM)
Note: ThissectionassumesthereaderhasalreadyreadthroughClassifyingMNISTdigitsusingLogistic
RegressionandMultilayerPerceptron. AdditionallyitusesthefollowingTheanofunctionsandconcepts:
T.tanh,sharedvariables,basicarithmeticops,T.grad,Randomnumbers,floatXandscan. Ifyouintendto
runthecodeonGPUalsoreadGPU.
Note: Thecodeforthissectionisavailablefordownloadhere.
9.1 Energy-BasedModels(EBM)
Energy-basedmodelsassociateascalarenergytoeachconfigurationofthevariablesofinterest.Learning
correspondstomodifyingthatenergyfunctionsothatitsshapehasdesirableproperties. Forexample,we
wouldlikeplausibleordesirableconfigurationstohavelowenergy. Energy-basedprobabilisticmodels
defineaprobabilitydistributionthroughanenergyfunction,asfollows:
p(x)=
e
E(x)
Z
:
(9.1)
ThenormalizingfactorZiscalledthepartitionfunctionbyanalogywithphysicalsystems.
Z=
X
x
e
E(x)
Anenergy-basedmodelcanbelearntbyperforming(stochastic)gradientdescentontheempiricalnegative
log-likelihoodofthetrainingdata.Asforthelogisticregressionwewillfirstdefinethelog-likelihoodand
thenthelossfunctionasbeingthenegativelog-likelihood.
L(;D)=
1
N
X
x(i)2D
logp(x
(i)
)
‘(;D)= L(;D)
usingthestochasticgradient 
@logp(x
(i)
)
@
,wherearetheparametersofthemodel.
91
DeepLearningTutorial,Release0.1
EBMswithHiddenUnits
Inmanycasesofinterest,wedonotobservetheexamplexfully,orwewanttointroducesomenon-observed
variablestoincreasetheexpressivepowerofthemodel. Soweconsideranobservedpart(stilldenotedx
here)andahiddenparth.Wecanthenwrite:
P(x)=
X
h
P(x;h)=
X
h
e
E(x;h)
Z
:
(9.2)
Insuchcases,tomapthisformulationtoonesimilartoEq.(9.1),weintroducethenotation(inspiredfrom
physics)offreeenergy,definedasfollows:
F(x)= log
X
h
e
E(x;h)
(9.3)
whichallowsustowrite,
P(x)=
e
F(x)
Z
withZ=
X
x
e
F(x)
:
Thedatanegativelog-likelihoodgradientthenhasaparticularlyinterestingform.
@logp(x)
@
=
@F(x)
@
X
~x
p(~x)
@F(~x)
@
:
(9.4)
Noticethattheabovegradientcontainstwoterms,whicharereferredtoasthepositiveandnegativephase.
Thetermspositiveandnegativedonotrefertothesignofeachtermintheequation,butratherreflecttheir
effectontheprobabilitydensitydefinedbythemodel. Thefirsttermincreasestheprobabilityoftraining
data(byreducingthecorrespondingfreeenergy),whilethesecondtermdecreasestheprobabilityofsamples
generatedbythemodel.
Itisusuallydifficulttodeterminethisgradientanalytically,asitinvolvesthecomputationofE
P
[
@F(x)
@
].
Thisisnothinglessthananexpectationoverallpossibleconfigurationsoftheinputx(underthedistribution
Pformedbythemodel)!
Thefirststepinmakingthiscomputationtractableistoestimatetheexpectationusingafixednumberof
modelsamples.Samplesusedtoestimatethenegativephasegradientarereferredtoasnegativeparticles,
whicharedenotedasN.Thegradientcanthenbewrittenas:
@logp(x)
@
@F(x)
@
1
jNj
X
~x2N
@F(~x)
@
:
(9.5)
wherewewouldideallylikeelements~xofN tobesampledaccordingtoP (i.e. . wearedoingMonte-
Carlo). Withtheaboveformula,wealmosthaveapratical,stochasticalgorithmforlearninganEBM.The
onlymissingingredientishowtoextractthesenegativeparticlesN.Whilethestatisticalliteratureabounds
withsamplingmethods,MarkovChainMonteCarlomethodsareespeciallywellsuitedformodelssuchas
theRestrictedBoltzmannMachines(RBM),aspecifictypeofEBM.
92
Chapter9. RestrictedBoltzmannMachines(RBM)
DeepLearningTutorial,Release0.1
9.2 RestrictedBoltzmannMachines(RBM)
BoltzmannMachines(BMs)areaparticularformoflog-linearMarkovRandomField(MRF),i.e., for
whichtheenergyfunctionislinearinitsfreeparameters. Tomakethempowerfulenoughtorepresent
complicateddistributions(i.e.,gofromthelimitedparametricsettingtoanon-parametricone),weconsider
thatsomeofthevariablesareneverobserved(theyarecalledhidden).Byhavingmorehiddenvariables(also
calledhiddenunits),wecanincreasethemodelingcapacityoftheBoltzmannMachine(BM).Restricted
BoltzmannMachinesfurtherrestrictBMstothosewithoutvisible-visibleandhidden-hiddenconnections.
AgraphicaldepictionofanRBMisshownbelow.
TheenergyfunctionE(v;h)ofanRBMisdefinedas:
E(v;h)= b
0
v c
0
h h
0
Wv
(9.6)
whereW representstheweightsconnectinghiddenandvisibleunitsandb,caretheoffsetsofthevisible
andhiddenlayersrespectively.
Thistranslatesdirectlytothefollowingfreeenergyformula:
F(v)= b
0
X
i
log
X
h
i
e
h
i
(c
i
+W
i
v)
:
BecauseofthespecificstructureofRBMs,visibleandhiddenunitsareconditionallyindependentgiven
one-another.Usingthisproperty,wecanwrite:
p(hjv)=
Y
i
p(h
i
jv)
p(vjh)=
Y
j
p(v
j
jh):
RBMswithbinaryunits
Inthecommonlystudiedcaseofusingbinaryunits(wherev
j
andh
i
2f0;1g),weobtainfromEq. (9.6)
and(9.2),aprobabilisticversionoftheusualneuronactivationfunction:
P(h
i
=1jv)=sigm(c
i
+W
i
v)
(9.7)
P(v
j
=1jh)=sigm(b
j
+W
0
j
h)
(9.8)
ThefreeenergyofanRBMwithbinaryunitsfurthersimplifiesto:
F(v)= b
0
X
i
log(1+e
(c
i
+W
i
v)
):
(9.9)
9.2. RestrictedBoltzmannMachines(RBM)
93
DeepLearningTutorial,Release0.1
UpdateEquationswithBinaryUnits
CombiningEqs.(9.5)with(9.9),weobtainthefollowinglog-likelihoodgradientsforanRBMwithbinary
units:
@logp(v)
@W
ij
=E
v
[p(h
i
jv)v
j
] v
(i)
j
sigm(W
i
v
(i)
+c
i
)
@logp(v)
@c
i
=E
v
[p(h
i
jv)] sigm(W
i
v
(i)
)
@logp(v)
@b
j
=E
v
[p(v
j
jh)] v
(i)
j
(9.10)
Foramoredetailedderivationoftheseequations,wereferthereadertothefollowingpage,ortosection5
ofLearningDeepArchitecturesforAI.Wewillhowevernotusetheseformulas,butrathergetthegradient
usingTheanoT.gradfromequation(9.4).
9.3 SamplinginanRBM
Samplesofp(x)canbeobtainedbyrunningaMarkovchaintoconvergence,usingGibbssamplingasthe
transitionoperator.
GibbssamplingofthejointofNrandomvariablesS = (S
1
;:::;S
N
)isdonethroughasequenceofN
samplingsub-stepsoftheformS
i
p(S
i
jS
i
)whereS
i
containstheN 1otherrandomvariablesinS
excludingS
i
.
ForRBMs,Sconsistsofthesetofvisibleandhiddenunits.However,sincetheyareconditionallyindepen-
dent,onecanperformblockGibbssampling.Inthissetting,visibleunitsaresampledsimultaneouslygiven
fixedvaluesofthehiddenunits. Similarly,hiddenunitsaresampledsimultaneouslygiventhevisibles. A
stepintheMarkovchainisthustakenasfollows:
h
(n+1)
sigm(W
0
v
(n)
+c)
v
(n+1)
sigm(Wh
(n+1)
+b);
whereh
(n)
referstothesetofallhiddenunitsatthen-thstepoftheMarkovchain.Whatitmeansisthat,for
example,h
(n+1)
i
israndomlychosentobe1(versus0)withprobabilitysigm(W
0
i
v
(n)
+c
i
),andsimilarly,
v
(n+1)
j
israndomlychosentobe1(versus0)withprobabilitysigm(W
:j
h
(n+1)
+b
j
).
Thiscanbeillustratedgraphically:
94
Chapter9. RestrictedBoltzmannMachines(RBM)
Documents you may be interested
Documents you may be interested