DeepLearningTutorial,Release0.1
plot_every 1000
# define one e step p of Gibbs sampling (mf = mean-field) define a
# function that does ‘plot_every‘ steps before e returning g the
# sample for r plotting
(
[
presig_hids,
hid_mfs,
hid_samples,
presig_vis,
vis_mfs,
vis_samples
],
updates
theano.scan(
rbm.gibbs_vhv,
outputs_info=[NoneNoneNoneNoneNone, persistent_vis_chain],
n_steps=plot_every
)
# add to updates the shared variable that takes s care e of our persistent
# chain :.
updates.update({persistent_vis_chain: vis_samples[-1]})
# construct the function that implements our persistent chain.
# we generate e the e "mean field" activations for r plotting g and the actual
# samples for r reinitializing g the state of our persistent chain
sample_fn theano.function(
[],
[
vis_mfs[-1],
vis_samples[-1]
],
updates=updates,
name=’sample_fn’
)
# create a space to store the image for plotting ( we need to leave
# room for the tile_spacing as well)
image_data numpy.zeros(
(29
*
n_samples 129
*
n_chains 1),
dtype=’uint8’
)
for idx in xrange(n_samples):
# generate ‘plot_every‘ intermediate samples that we discard,
# because e successive e samples in the chain are too correlated
vis_mf, vis_sample sample_fn()
print ’ ... plotting sample ’, idx
image_data[29
*
idx:29
*
idx 28, :] tile_raster_images(
X=vis_mf,
img_shape=(2828),
tile_shape=(1, n_chains),
tile_spacing=(11)
)
9.4. Implementation
105
Convert pdf to high quality jpg - Convert PDF to JPEG images in C#.net, ASP.NET MVC, WinForms, WPF project
How to convert PDF to JPEG using C#.NET PDF to JPEG conversion / converter library control SDK
convert pdf picture to jpg; .pdf to .jpg online
Convert pdf to high quality jpg - VB.NET PDF Convert to Jpeg SDK: Convert PDF to JPEG images in vb.net, ASP.NET MVC, WinForms, WPF project
Online Tutorial for PDF to JPEG (JPG) Conversion in VB.NET Image Application
convert pdf to jpg for online; convert multiple page pdf to jpg
DeepLearningTutorial,Release0.1
# construct image
image Image.fromarray(image_data)
image.save(’samples.png’)
9.5 Results
WeranthecodewithPCD-15,learningrateof0.1andabatchsizeof20,for15epochs.Trainingthemodel
takes122.466minutesonaIntelXeonE5430@2.66GHzCPU,withasingle-threadedGotoBLAS.
Theoutputwasthefollowing:
... loading data
Training epoch 0, , cost t is
-90.6507246003
Training epoch 1, , cost t is
-81.235857373
Training epoch 2, , cost t is
-74.9120966945
Training epoch 3, , cost t is
-73.0213216101
Training epoch 4, , cost t is
-68.4098570497
Training epoch 5, , cost t is
-63.2693021647
Training epoch 6, , cost t is
-65.99578971
Training epoch 7, , cost t is
-68.1236650015
Training epoch 8, , cost t is
-68.3207365087
Training epoch 9, , cost t is
-64.2949797113
Training epoch 10, cost is
-61.5194867893
Training epoch 11, cost is
-61.6539369402
Training epoch 12, cost is
-63.5465278086
Training epoch 13, cost is
-63.3787093527
Training epoch 14, cost is
-62.755739271
Training took 122.466000 minutes
... plotting sample
0
... plotting sample
1
... plotting sample
2
... plotting sample
3
... plotting sample
4
... plotting sample
5
... plotting sample
6
... plotting sample
7
... plotting sample
8
... plotting sample
9
Thepicturesbelowshowthefiltersafter15epochs:
HerearethesamplesgeneratedbytheRBMaftertraining. Eachrowrepresentsamini-batchofnegative
particles(samplesfromindependentGibbschains).1000stepsofGibbssamplingweretakenbetweeneach
ofthoserows.
106
Chapter9. RestrictedBoltzmannMachines(RBM)
JPG to GIF Converter | Convert JPEG to GIF, Convert GIF to JPG
What you need to do is just click the convert button, sort the imported file, and then JPEG to GIF Converter will give you JPG files with high good quality.
convert pdf into jpg; change pdf to jpg image
JPEG Image Viewer| What is JPEG
RasterEdge Image products support for high-quality JPEG, JPEG interface enabling you to quickly convert your JPEG including Bitmap, Png, Gif, Tiff, PDF, MS-Word
change pdf to jpg online; convert pdf pictures to jpg
DeepLearningTutorial,Release0.1
Figure9.1:Filtersobtainedafter15epochs.
9.5. Results
107
C# Create PDF from images Library to convert Jpeg, png images to
Batch convert PDF documents from multiple image formats Create PDF from images in both .NET NET converter control for exporting high quality PDF from images in
convert pdf file to jpg online; convert pdf image to jpg online
C# PDF Convert to Tiff SDK: Convert PDF to tiff images in C#.net
Tiff files are exported with high resolution and no loss in quality in .NET framework. Also supports convert PDF files to jpg, jpeg images.
changing pdf to jpg; convert multi page pdf to single jpg
DeepLearningTutorial,Release0.1
108
Chapter9. RestrictedBoltzmannMachines(RBM)
VB.NET Imaging - Generate Barcode Image in VB.NET
Create high-quality ITF-14 valid for scanner reading on any pages in a PDF or TIFF documents as well as common image files such as png and jpg.
convert pdf into jpg format; change from pdf to jpg
C# Image Convert: How to Convert Tiff Image to Jpeg, Png, Bmp, &
RasterEdge.XDoc.Office.Inner.Office03.dll. RasterEdge.XDoc.PDF.dll. C# Image Convert: Tiff to Gif. How to achieve high quality Tiff to Gif image file conversion.
pdf to jpeg converter; convert pdf to jpg 300 dpi
CHAPTER
TEN
DEEPBELIEFNETWORKS
Note: ThissectionassumesthereaderhasalreadyreadthroughClassifyingMNISTdigitsusingLogistic
RegressionandMultilayerPerceptronandRestrictedBoltzmannMachines(RBM).Additionallyitusesthe
followingTheanofunctionsandconcepts:T.tanh,sharedvariables,basicarithmeticops,T.grad,Random
numbers,floatX.IfyouintendtorunthecodeonGPUalsoreadGPU.
Note: Thecodeforthissectionisavailablefordownloadhere.
10.1 DeepBeliefNetworks
[Hinton06]showedthatRBMscanbestackedandtrainedinagreedymannertoformso-calledDeepBelief
Networks(DBN).DBNsaregraphicalmodelswhichlearntoextractadeephierarchicalrepresentationof
thetrainingdata.Theymodelthejointdistributionbetweenobservedvectorxandthe‘hiddenlayersh
k
as
follows:
P(x;h
1
;:::;h
)=
‘ 2
k=0
P(h
k
jh
k+1
)
!
P(h
‘ 1
;h
)
(10.1)
wherex=h
0
,P(h
k 1
jh
k
)isaconditionaldistributionforthevisibleunitsconditionedonthehiddenunits
oftheRBMatlevelk,andP(h
‘ 1
;h
)isthevisible-hiddenjointdistributioninthetop-levelRBM.Thisis
illustratedinthefigurebelow.
Theprincipleofgreedylayer-wiseunsupervisedtrainingcanbeappliedtoDBNswithRBMsasthebuilding
blocksforeachlayer[Hinton06],[Bengio07].Theprocessisasfollows:
1.TrainthefirstlayerasanRBMthatmodelstherawinputx=h
(0)
asitsvisiblelayer.
2.Usethatfirstlayertoobtainarepresentationoftheinputthatwillbeusedasdataforthesecondlayer.Two
commonsolutionsexist.Thisrepresentationcanbechosenasbeingthemeanactivationsp(h
(1)
=1jh
(0)
)
orsamplesofp(h
(1)
jh
(0)
).
3.TrainthesecondlayerasanRBM,takingthetransformeddata(samplesormeanactivations)astraining
examples(forthevisiblelayerofthatRBM).
4.Iterate(2and3)forthedesirednumberoflayers,eachtimepropagatingupwardeithersamplesormean
values.
109
VB.NET PDF - Convert PDF with VB.NET WPF PDF Viewer
Convert PDF to image file formats with high quality, support converting PDF to PNG, JPG, BMP and GIF. VB.NET WPF PDF Viewer: Convert and Export PDF.
.pdf to jpg; convert pdf to jpg batch
C# Imaging - Generate Barcode Image in C#.NET
Create high-quality ITF-14 valid for scanner reading on any pages in a PDF or TIFF documents as well as common image files such as png and jpg.
change pdf to jpg format; convert pdf to jpeg
DeepLearningTutorial,Release0.1
5.Fine-tunealltheparametersofthisdeeparchitecturewithrespecttoaproxyfortheDBNlog-likelihood,
orwithrespecttoasupervisedtrainingcriterion(afteraddingextralearningmachinerytoconvertthelearned
representationintosupervisedpredictions,e.g.alinearclassifier).
Inthistutorial,wefocusonfine-tuningviasupervisedgradientdescent. Specifically,weusealogistic
regressionclassifiertoclassifytheinputxbasedontheoutputofthelasthiddenlayerh
(l)
oftheDBN.
Fine-tuningisthenperformedviasupervisedgradientdescentofthenegativelog-likelihoodcostfunction.
Sincethesupervisedgradientisonlynon-nullfortheweightsandhiddenlayerbiasesofeachlayer(i.e.
nullforthevisiblebiasesofeachRBM),thisprocedureisequivalenttoinitializingtheparametersofadeep
MLPwiththeweightsandhiddenlayerbiasesobtainedwiththeunsupervisedtrainingstrategy.
10.2 JustifyingGreedy-LayerWisePre-Training
Whydoessuchanalgorithmwork?Takingasexamplea2-layerDBNwithhiddenlayersh
(1)
andh
(2)
(with
respectiveweightparametersW
(1)
andW
(2)
),[Hinton06]established(seealsoBengio09]_foradetailed
derivation)thatlogp(x)canberewrittenas,
logp(x)=KL(Q(h
(1)
jx)jjp(h
(1)
jx))+H
Q(h(1)jx)
+
X
h
Q(h
(1)
jx)(logp(h
(1)
)+logp(xjh
(1)
)):
(10.2)
KL(Q(h
(1)
jx)jjp(h
(1)
jx))representstheKLdivergencebetweentheposteriorQ(h
(1)
jx)ofthefirstRBM
ifitwerestandalone,andtheprobabilityp(h
(1)
jx)forthesamelayerbutdefinedbytheentireDBN(i.e.
takingintoaccountthepriorp(h
(1)
;h
(2)
)definedbythetop-levelRBM).H
Q(h(1)jx)
istheentropyofthe
distributionQ(h
(1)
jx).
ItcanbeshownthatifweinitializebothhiddenlayerssuchthatW
(2)
=W
(1)
T
,Q(h
(1)
jx) =p(h
(1)
jx)
andtheKLdivergencetermisnull.IfwelearnthefirstlevelRBMandthenkeepitsparametersW
(1)
fixed,
optimizingEq.(10.2)withrespecttoW
(2)
canthusonlyincreasethelikelihoodp(x).
110
Chapter10. DeepBeliefNetworks
DeepLearningTutorial,Release0.1
Also,noticethatifweisolatethetermswhichdependonlyonW
(2)
,weget:
X
h
Q(h
(1)
jx)p(h
(1)
)
OptimizingthiswithrespecttoW
(2)
amountstotrainingasecond-stageRBM,usingtheoutputofQ(h
(1)
jx)
asthetrainingdistribution,whenxissampledfromthetrainingdistributionforthefirstRBM.
10.3 Implementation
ToimplementDBNsinTheano,wewillusetheclassdefinedintheRestrictedBoltzmannMachines(RBM)
tutorial.OnecanalsoobservethatthecodefortheDBNisverysimilarwiththeoneforSdA,becauseboth
involvetheprincipleofunsupervisedlayer-wisepre-trainingfollowedbysupervisedfine-tuningasadeep
MLP.ThemaindifferenceisthatweusetheRBMclassinsteadofthedAclass.
WestartoffbydefiningtheDBNclasswhichwillstorethelayersoftheMLP,alongwiththeirassociated
RBMs. SincewetaketheviewpointofusingtheRBMstoinitializeanMLP,thecodewillreflectthisby
seperatingasmuchaspossibletheRBMsusedtoinitializethenetworkandtheMLPusedforclassification.
class DBN(object):
"""Deep Belief Network
A deep belief f network k is obtained by stacking several RBMs on top of each
other. The hidden layer of the RBM at layer ‘i‘ ‘ becomes s the input of the
RBM at layer r ‘i+1‘. . The first layer RBM gets as s input t the input of the
network, and d the e hidden layer of the last RBM represents the output. When
used for classification, the DBN is treated as s a a MLP, by adding a logistic
regression layer on top.
"""
def __init__(self, numpy_rng, theano_rng=None, n_ins=784,
hidden_layers_sizes=[500500], n_outs=10):
"""This class is made to support a variable e number r of layers.
:type numpy_rng: numpy.random.RandomState
:param numpy_rng: numpy random number generator used to draw initial
weights
:type theano_rng: theano.tensor.shared_randomstreams.RandomStreams
:param theano_rng: Theano random generator; ; if f None is given one is
generated based on a seed drawn from ‘rng‘
:type n_ins: int
:param n_ins: dimension of the input to the e DBN
:type hidden_layers_sizes: list of ints
:param hidden_layers_sizes: intermediate layers size, must contain
at least one value
:type n_outs: int
10.3. Implementation
111
DeepLearningTutorial,Release0.1
:param n_outs: dimension of the output of the network
"""
self.sigmoid_layers []
self.rbm_layers []
self.params []
self.n_layers len(hidden_layers_sizes)
assert self.n_layers 0
if not theano_rng:
theano_rng MRG_RandomStreams(numpy_rng.randint(2
**
30))
# allocate symbolic variables for the data
self.T.matrix(’x’)
# the data is presented as rasterized images
self.T.ivector(’y’)
# the labels are presented as 1D vector
# of [int] labels
self.sigmoid_layers will store the feed-forward d graphs s which h together form m the MLP, while
self.rbm_layerswillstoretheRBMsusedtopretraineachlayeroftheMLP.
Nextstep,weconstructn_layerssigmoidlayers(weusetheHiddenLayerclassintroducedinMul-
tilayerPerceptron,withtheonlymodificationthatwereplacedthenon-linearityfromtanhtothelogistic
functions(x)=
1
1+e x
)andn_layersRBMs,wheren_layersisthedepthofourmodel.Welinkthe
sigmoidlayerssuchthattheyformanMLP,andconstructeachRBMsuchthattheysharetheweightmatrix
andthehiddenbiaswithitscorrespondingsigmoidlayer.
for in xrange(self.n_layers):
# construct the sigmoidal layer
# the e size e of the input is either the number of hidden
# units of the layer below or the input t size e if we are on
# the e first t layer
if == 0:
input_size n_ins
else:
input_size hidden_layers_sizes[i 1]
# the e input t to this layer is either the e activation n of the
# hidden layer below or the input of the DBN if you are on
# the e first t layer
if == 0:
layer_input self.x
else:
layer_input self.sigmoid_layers[-1].output
sigmoid_layer HiddenLayer(rng=numpy_rng,
input=layer_input,
n_in=input_size,
n_out=hidden_layers_sizes[i],
activation=T.nnet.sigmoid)
# add d the e layer to our list of layers
112
Chapter10. DeepBeliefNetworks
DeepLearningTutorial,Release0.1
self.sigmoid_layers.append(sigmoid_layer)
# its s arguably y a philosophical question...
but we are
# going to only declare that the parameters of the
# sigmoid_layers are parameters of the e DBN. . The visible
# biases in the RBM are parameters of those RBMs, but not
# of f the e DBN.
self.params.extend(sigmoid_layer.params)
# Construct an RBM that shared weights s with h this layer
rbm_layer RBM(numpy_rng=numpy_rng,
theano_rng=theano_rng,
input=layer_input,
n_visible=input_size,
n_hidden=hidden_layers_sizes[i],
W=sigmoid_layer.W,
hbias=sigmoid_layer.b)
self.rbm_layers.append(rbm_layer)
Allthatis leftis tostackonelastlogistic regressionlayerin orderto formanMLP.Wewillusethe
LogisticRegressionclassintroducedinClassifyingMNISTdigitsusingLogisticRegression.
self.logLayer LogisticRegression(
input=self.sigmoid_layers[-1].output,
n_in=hidden_layers_sizes[-1],
n_out=n_outs)
self.params.extend(self.logLayer.params)
# compute e the e cost for second phase of training, defined as the
# negative log likelihood of the logistic regression (output) layer
self.finetune_cost self.logLayer.negative_log_likelihood(self.y)
# compute e the e gradients with respect to the e model l parameters
# symbolic variable that points to the number of errors made on the
# minibatch given by self.x and self.y
self.errors self.logLayer.errors(self.y)
TheclassalsoprovidesamethodwhichgeneratestrainingfunctionsforeachoftheRBMs.Theyarereturned
asalist,whereelementiisafunctionwhichimplementsonestepoftrainingfortheRBMatlayeri.
def pretraining_functions(self, train_set_x, batch_size, k):
’’’Generates a list of functions, for performing one step of
gradient descent t at a given layer. The function will require
as input t the e minibatch index, and to train n an n RBM you just
need to iterate, calling the corresponding g function n on all
minibatch indexes.
:type train_set_x: theano.tensor.TensorType
:param train_set_x: Shared var. that contains all datapoints used
for training the RBM
:type batch_size: int
:param batch_size: size of a [mini]batch
:param k: : number r of Gibbs steps to do in CD-k / PCD-k
10.3. Implementation
113
DeepLearningTutorial,Release0.1
’’’
# index to a [mini]batch
index T.lscalar(’index’)
# index to a minibatch
Inordertobeabletochangethelearningrateduringtraining,weassociateaTheanovariabletoitthathas
adefaultvalue.
learning_rate T.scalar(’lr’)
# learning g rate e to use
# number r of f batches
n_batches train_set_x.get_value(borrow=True).shape[0batch_size
# begining of a batch, given ‘index‘
batch_begin index
*
batch_size
# ending g of f a batch given ‘index‘
batch_end batch_begin batch_size
pretrain_fns []
for rbm in self.rbm_layers:
# get t the e cost and the updates list
# using CD-k here (persisent=None) for r training g each RBM.
# TODO: change cost function to reconstruction error
cost, updates rbm.get_cost_updates(learning_rate,
persistent=None, k=k)
# compile the theano function
fn theano.function(
inputs=[index, theano.Param(learning_rate, default=0.1)],
outputs=cost,
updates=updates,
givens={
self.x: train_set_x[batch_begin:batch_end]
}
)
# append ‘fn‘ to the list of functions
pretrain_fns.append(fn)
return pretrain_fns
Nowanyfunctionpretrain_fns[i]takesasargumentsindexandoptionallylr–thelearningrate.
NotethatthenamesoftheparametersarethenamesgiventotheTheanovariables(e.g.lr)whentheyare
constructedandnotthenameofthepythonvariables(e.g. learning_rate). . Keepthisinmindwhen
workingwithTheano. Optionally,ifyouprovidek(thenumberofGibbsstepstoperforminCDorPCD)
thiswillalsobecomeanargumentofyourfunction.
Inthesamefashion,theDBNclassincludesamethodforbuildingthefunctionsrequiredforfinetuning(a
train_model,avalidate_modelandatest_modelfunction).
def build_finetune_functions(self, datasets, batch_size, learning_rate):
’’’Generates a function ‘train‘ that implements one step of
finetuning, a function ‘validate‘ that computes the error on a
batch from the validation set, and a function ‘test‘ that
114
Chapter10. DeepBeliefNetworks
Documents you may be interested
Documents you may be interested