DeepLearningTutorial,Release0.1
http://www.iro.umontreal.ca/~lisa/deep/data/mnist/mnist.pkl.gz
"""
datasets load_data(dataset)
train_set_x, train_set_y datasets[0]
valid_set_x, valid_set_y datasets[1]
test_set_x, test_set_y datasets[2]
# compute number of minibatches for training, validation and testing
n_train_batches train_set_x.get_value(borrow=True).shape[0batch_size
n_valid_batches valid_set_x.get_value(borrow=True).shape[0batch_size
n_test_batches test_set_x.get_value(borrow=True).shape[0batch_size
######################
# BUILD ACTUAL MODEL #
######################
print ’... building the model’
# allocate symbolic variables for the data
index T.lscalar()
# index to a [mini]batch
T.matrix(’x’)
# the data is presented as rasterized images
T.ivector(’y’)
# the labels are presented d as s 1D vector of
# [int] labels
rng numpy.random.RandomState(1234)
# construct the MLP class
classifier MLP(
rng=rng,
input=x,
n_in=28
*
28,
n_hidden=n_hidden,
n_out=10
)
# start-snippet-4
# the cost we e minimize e during training is the negative log likelihood of
# the model plus the regularization terms (L1 and L2); cost is expressed
# here symbolically
cost (
classifier.negative_log_likelihood(y)
L1_reg
*
classifier.L1
L2_reg
*
classifier.L2_sqr
)
# end-snippet-4
# compiling a a Theano o function that computes the e mistakes s that are made
# by the model on a minibatch
test_model theano.function(
inputs=[index],
outputs=classifier.errors(y),
5.3. PuttingitAllTogether
45
Change from pdf to jpg on - Convert PDF to JPEG images in C#.net, ASP.NET MVC, WinForms, WPF project
How to convert PDF to JPEG using C#.NET PDF to JPEG conversion / converter library control SDK
conversion of pdf to jpg; convert pdf file into jpg
Change from pdf to jpg on - VB.NET PDF Convert to Jpeg SDK: Convert PDF to JPEG images in vb.net, ASP.NET MVC, WinForms, WPF project
Online Tutorial for PDF to JPEG (JPG) Conversion in VB.NET Image Application
c# pdf to jpg; change pdf file to jpg online
DeepLearningTutorial,Release0.1
givens={
x: test_set_x[index
*
batch_size:(index 1)
*
batch_size],
y: test_set_y[index
*
batch_size:(index 1)
*
batch_size]
}
)
validate_model theano.function(
inputs=[index],
outputs=classifier.errors(y),
givens={
x: valid_set_x[index
*
batch_size:(index 1)
*
batch_size],
y: valid_set_y[index
*
batch_size:(index 1)
*
batch_size]
}
)
# start-snippet-5
# compute the e gradient t of cost with respect to o theta a (sotred in params)
# the resulting gradients will be stored in a list gparams
gparams [T.grad(cost, param) for param in classifier.params]
# specify how w to o update the parameters of the model as a list of
# (variable, , update e expression) pairs
# given two lists of the same length, A = [a1, , a2, , a3, a4] and
# B = [b1, b2, b3, b4], zip generates a list C C of f same size, where each
# element is s a a pair formed from the two lists :
#
C = [(a1, b1), (a2, b2), (a3, b3), (a4, b4)]
updates [
(param, param learning_rate
*
gparam)
for param, gparam in zip(classifier.params, gparams)
]
# compiling a a Theano o function ‘train_model‘ that returns the cost, but
# in the same e time e updates the parameter of the e model l based on the rules
# defined in n ‘updates‘
train_model theano.function(
inputs=[index],
outputs=cost,
updates=updates,
givens={
x: train_set_x[index
*
batch_size: (index 1)
*
batch_size],
y: train_set_y[index
*
batch_size: (index 1)
*
batch_size]
}
)
# end-snippet-5
###############
# TRAIN MODEL L #
###############
print ’... training’
# early-stopping parameters
patience 10000
# look as this many examples s regardless
46
Chapter5. MultilayerPerceptron
Online Convert Jpeg to PDF file. Best free online export Jpg image
Download Free Trial. Convert a JPG to PDF. Web Security. All your JPG and PDF files will be permanently erased from our servers after one hour.
convert pdf file to jpg online; convert online pdf to jpg
JPG to PNG Converter | Convert JPEG to PNG, Convert PNG to JPG
Allow to change converting image with adjusted width & height; Change image resolution Open JPEG to PNG Converter first; Load JPG images from local folders in
convert pdf file to jpg; convert .pdf to .jpg online
DeepLearningTutorial,Release0.1
patience_increase 2
# wait this much longer r when n a new best is
# found
improvement_threshold 0.995
# a relative improvement of this much is
# considered significant
validation_frequency min(n_train_batches, patience 2)
# go through this s many
# minibatche before checking the network
# on the validation set; in this case we
# check every epoch
best_validation_loss numpy.inf
best_iter 0
test_score 0.
start_time timeit.default_timer()
epoch 0
done_looping False
while (epoch n_epochs) and (not done_looping):
epoch epoch 1
for minibatch_index in xrange(n_train_batches):
minibatch_avg_cost train_model(minibatch_index)
# iteration number
iter (epoch 1)
*
n_train_batches minibatch_index
if (iter 1validation_frequency == 0:
# compute e zero-one loss on validation set
validation_losses [validate_model(i) for i
in xrange(n_valid_batches)]
this_validation_loss numpy.mean(validation_losses)
print(
’epoch %i, minibatch %i/%i, validation error %f %%’ %
(
epoch,
minibatch_index 1,
n_train_batches,
this_validation_loss
*
100.
)
)
# if f we got the best validation score until now
if this_validation_loss best_validation_loss:
#improve patience if loss improvement is good enough
if (
this_validation_loss best_validation_loss
*
improvement_threshold
):
patience max(patience, iter
*
patience_increase)
best_validation_loss this_validation_loss
best_iter iter
5.3. PuttingitAllTogether
47
Online Convert PDF to Jpeg images. Best free online PDF JPEG
Download Free Trial. Convert a PDF File to JPG. Web Security. Your PDF and JPG files will be deleted from our servers an hour after the conversion.
convert pdf pictures to jpg; best pdf to jpg converter for
C# Image Convert: How to Convert Tiff Image to Jpeg, Png, Bmp, &
RasterEdge.XDoc.PDF.dll. C:\input.tif"; String outputDirectory = @"C:\output\"; // Convert tiff to jpg and show How to change Tiff image to Bmp image in your C#
.pdf to .jpg converter online; convert pdf to high quality jpg
DeepLearningTutorial,Release0.1
# test it on the test set
test_losses [test_model(i) for i
in xrange(n_test_batches)]
test_score numpy.mean(test_losses)
print((
epoch %i, minibatch %i/%i, test error of ’
’best model %f %%%
(epoch, minibatch_index 1, n_train_batches,
test_score
*
100.))
if patience <= iter:
done_looping True
break
end_time timeit.default_timer()
print((’Optimization complete. Best validation n score e of %f %% 
’obtained at iteration %i, with test performance %f %%%
(best_validation_loss
*
100., best_iter 1, test_score
*
100.))
print >> sys.stderr, (’The code for file ’ +
os.path.split(__file__)[1+
’ ran for %.2fm’ ((end_time start_time) 60.))
if __name__ == ’__main__’:
test_mlp()
Theusercanthenrunthecodebycalling:
python code/mlp.py
Theoutputoneshouldexpectisoftheform:
Optimization complete. Best validation score of 1.690000 % obtained at iteration 2070000, with test performance 1.650000 %
The code for file e mlp.py y ran for 97.34m
OnanIntel(R)Core(TM)i7-2600KCPU@3.40GHzthecoderunswithapproximately10.3epoch/minute
andittook828epochstoreachatesterrorof1.65%.
Toputthisintoperspective,wereferthereadertotheresultssectionofthispage.
5.4 TipsandTricksfortrainingMLPs
Thereareseveralhyper-parametersintheabovecode,whicharenot(and,generallyspeaking,cannotbe)
optimizedbygradientdescent.Strictlyspeaking,findinganoptimalsetofvaluesforthesehyper-parameters
isnotafeasibleproblem. First,wecan’tsimplyoptimizeeachofthemindependently.Second,wecannot
readilyapplygradienttechniquesthatwedescribedpreviously(partlybecausesomeparametersarediscrete
valuesandothersarereal-valued). Third, , theoptimizationproblemisnotconvexandfindinga(local)
minimumwouldinvolveanon-trivialamountofwork.
Thegoodnewsisthatoverthelast25years,researchershavedevisedvariousrulesofthumbforchoosing
hyper-parametersinaneuralnetwork. AverygoodoverviewofthesetrickscanbefoundinEfficient
48
Chapter5. MultilayerPerceptron
C# Image Convert: How to Convert Adobe PDF to Jpeg, Png, Bmp, &
String inputFilePath = @"C:\input.pdf"; String outputFilePath = @"C:\output.jpg"; // Convert PDF to jpg. C# sample code for PDF to jpg image conversion.
convert pdf to jpeg; changing pdf to jpg file
JPG to JBIG2 Converter | Convert JPEG to JBIG2, Convert JBIG2 to
Users may easily change image size, rotate image angle, set image rotation in dpi Covert JPG & JBIG2 image with high-quality; Provide user-friendly interface
batch pdf to jpg converter online; bulk pdf to jpg converter online
DeepLearningTutorial,Release0.1
BackPropbyYannLeCun,LeonBottou,GenevieveOrr,andKlaus-RobertMueller.Inhere,wesummarize
thesameissues,withanemphasisontheparametersandtechniquesthatweactuallyusedinourcode.
5.4.1 Nonlinearity
Twoofthemostcommononesarethesigmoidandthetanhfunction.ForreasonsexplainedinSection4.4,
nonlinearitiesthataresymmetricaroundtheoriginarepreferredbecausetheytendtoproducezero-mean
inputstothenextlayer(whichisadesirableproperty). Empirically,wehaveobservedthatthetanhhas
betterconvergenceproperties.
5.4.2 Weightinitialization
Atinitializationwewanttheweightstobesmallenougharoundtheoriginsothattheactivationfunc-
tionoperatesinitslinearregime,wheregradientsarethelargest. Otherdesirableproperties,especially
fordeepnetworks,aretoconservevarianceoftheactivationaswellasvarianceofback-propagatedgra-
dientsfromlayertolayer. Thisallowsinformationtoflowwellupwardanddownwardinthenetwork
andreduces discrepancies betweenlayers. . Undersomeassumptions,acompromisebetweenthesetwo
constraintsleadstothefollowinginitialization: uniform[ 
p
6
p
fan
in
+fan
out
;
p
6
p
fan
in
+fan
out
]fortanhand
uniform[ 4
p
6
p
fan
in
+fan
out
;4
p
6
p
fan
in
+fan
out
]forsigmoid. Wherefan
in
isthenumberofinputsand
fan
out
thenumberofhiddenunits.Formathematicalconsiderationspleasereferto[Xavier10].
5.4.3 Learningrate
Thereisagreatdealofliteratureonchoosingagoodlearningrate.Thesimplestsolutionistosimplyhave
aconstantrate.Ruleofthumb:tryseverallog-spacedvalues(10
1
;10
2
;:::)andnarrowthe(logarithmic)
gridsearchtotheregionwhereyouobtainthelowestvalidationerror.
Decreasingthelearningrateovertimeissometimesagoodidea. Onesimplerulefordoingthatis
0
1+dt
where
0
istheinitialrate(chosen,perhaps,usingthegridsearchtechniqueexplainedabove),disaso-
called“decreaseconstant”whichcontrolstherateatwhichthelearningratedecreases(typically,asmaller
positivenumber,10
3
andsmaller)andtistheepoch/stage.
Section4.7detailsproceduresforchoosingalearningrateforeachparameter(weight)inournetworkand
forchoosingthemadaptivelybasedontheerroroftheclassifier.
5.4.4 Numberofhiddenunits
Thishyper-parameterisverymuchdataset-dependent. Vaguelyspeaking,themorecomplicatedtheinput
distributionis, themorecapacitythenetworkwillrequiretomodelit,andsothelargerthenumberof
hiddenunitsthatwillbeneeded(notethatthenumberofweightsinalayer,perhapsamoredirectmeasure
ofcapacity,isDD
h
(recallDisthenumberofinputsandD
h
isthenumberofhiddenunits).
Unlessweemploysomeregularizationscheme(earlystoppingorL1/L2penalties), atypicalnumberof
hiddenunitsvs.generalizationperformancegraphwillbeU-shaped.
5.4. TipsandTricksfortrainingMLPs
49
VB.NET PDF Convert to Images SDK: Convert PDF to png, gif images
Convert PDF to Jpg, Png, Bmp, Gif, Tiff and Bitmap in ASP.NET. Or directly change PDF to Gif image file in VB.NET program with this demo code.
convert pdf file to jpg online; .net pdf to jpg
JPEG to PDF Converter | Convert JPEG to PDF, Convert PDF to JPEG
similar software; Support a batch conversion of JPG to PDF with amazingly high speed; Get a compressed PDF file after conversion; Support
best pdf to jpg converter; change file from pdf to jpg
DeepLearningTutorial,Release0.1
5.4.5 Regularizationparameter
TypicalvaluestotryfortheL1/L2regularizationparameterare10
2
;10
3
;:::. Intheframeworkthat
wedescribedsofar,optimizingthisparameterwillnotleadtosignificantlybettersolutions,butisworth
exploringnonetheless.
50
Chapter5. MultilayerPerceptron
CHAPTER
SIX
CONVOLUTIONALNEURALNETWORKS(LENET)
Note: ThissectionassumesthereaderhasalreadyreadthroughClassifyingMNISTdigitsusingLogistic
RegressionandMultilayerPerceptron. Additionally,itusesthefollowingnewTheanofunctionsandcon-
cepts:T.tanh,sharedvariables,basicarithmeticops,T.grad,floatX,downsample,conv2d,dimshuffle. If
youintendtorunthecodeonGPUalsoreadGPU.
TorunthisexampleonaGPU,youneedagoodGPU.Itneedsatleast1GBofGPURAM.Moremaybe
requiredifyourmonitorisconnectedtotheGPU.
WhentheGPUisconnectedtothemonitor,thereisalimitofafewsecondsforeachGPUfunctioncall.
ThisisneededascurrentGPUscan’tbeusedforthemonitorwhiledoingcomputation.Withoutthislimit,
thescreenwouldfreezefortoolongandmakeitlookasifthecomputerfroze.Thisexamplehitsthislimit
withmedium-qualityGPUs. WhentheGPUisn’tconnectedtoamonitor,thereisnotimelimit. . Youcan
lowerthebatchsizetofixthetimeoutproblem.
Note: Thecodeforthissectionisavailablefordownloadhereandthe3wolfmoonimage
6.1 Motivation
Convolutional NeuralNetworks (CNN) are biologically-inspired variants ofMLPs. . From m Hubel and
Wiesel’searlyworkonthecat’svisualcortex[Hubel68],weknowthevisualcortexcontainsacomplex
arrangementofcells. Thesecellsaresensitivetosmallsub-regionsofthevisualfield,calledareceptive
field.Thesub-regionsaretiledtocovertheentirevisualfield.Thesecellsactaslocalfiltersovertheinput
spaceandarewell-suitedtoexploitthestrongspatiallylocalcorrelationpresentinnaturalimages.
Additionally,twobasiccelltypeshavebeenidentified:Simplecellsrespondmaximallytospecificedge-like
patternswithintheirreceptivefield. Complexcellshavelargerreceptivefieldsandarelocallyinvariantto
theexactpositionofthepattern.
Theanimalvisualcortexbeingthemostpowerfulvisualprocessingsysteminexistence,itseemsnaturalto
emulateitsbehavior.Hence,manyneurally-inspiredmodelscanbefoundintheliterature.Tonameafew:
theNeoCognitron[Fukushima],HMAX[Serre07]andLeNet-5[LeCun98],whichwillbethefocusofthis
tutorial.
51
DeepLearningTutorial,Release0.1
6.2 SparseConnectivity
CNNsexploitspatially-localcorrelationbyenforcingalocalconnectivitypatternbetweenneuronsofadja-
centlayers. Inotherwords,theinputsofhiddenunitsinlayermarefromasubsetofunitsinlayerm-1,
unitsthathavespatiallycontiguousreceptivefields.Wecanillustratethisgraphicallyasfollows:
Imaginethatlayerm-1istheinputretina.Intheabovefigure,unitsinlayermhavereceptivefieldsofwidth
3intheinputretinaandarethusonlyconnectedto3adjacentneuronsintheretinalayer.Unitsinlayerm+1
haveasimilarconnectivitywiththelayerbelow.Wesaythattheirreceptivefieldwithrespecttothelayer
belowisalso3,buttheirreceptivefieldwithrespecttotheinputislarger(5).Eachunitisunresponsiveto
variationsoutsideofitsreceptivefieldwithrespecttotheretina.Thearchitecturethusensuresthatthelearnt
“filters”producethestrongestresponsetoaspatiallylocalinputpattern.
However,asshownabove,stackingmanysuchlayersleadsto(non-linear)“filters”thatbecomeincreasingly
“global”(i.e. responsivetoalargerregionofpixelspace).Forexample,theunitinhiddenlayerm+1can
encodeanon-linearfeatureofwidth5(intermsofpixelspace).
6.3 SharedWeights
Inaddition,inCNNs,eachfilterh
i
isreplicatedacrosstheentirevisualfield.Thesereplicatedunitsshare
thesameparameterization(weightvectorandbias)andformafeaturemap.
Intheabovefigure,weshow3hiddenunitsbelongingtothesamefeaturemap.Weightsofthesamecolor
areshared—constrainedtobeidentical.Gradientdescentcanstillbeusedtolearnsuchsharedparameters,
withonlyasmallchangetotheoriginalalgorithm.Thegradientofasharedweightissimplythesumofthe
gradientsoftheparametersbeingshared.
Replicatingunitsinthiswayallowsforfeaturestobedetectedregardlessoftheirpositioninthevisualfield.
Additionally,weightsharingincreaseslearningefficiencybygreatlyreducingthenumberoffreeparameters
beinglearnt.TheconstraintsonthemodelenableCNNstoachievebettergeneralizationonvisionproblems.
52
Chapter6. ConvolutionalNeuralNetworks(LeNet)
DeepLearningTutorial,Release0.1
6.4 DetailsandNotation
Afeaturemapisobtainedbyrepeatedapplicationofafunctionacrosssub-regionsoftheentireimage,in
otherwords,byconvolutionoftheinputimagewithalinearfilter,addingabiastermandthenapplyinga
non-linearfunction. Ifwedenotethek-thfeaturemapatagivenlayerash
k
,whosefiltersaredetermined
bytheweightsW
k
andbiasb
k
,thenthefeaturemaph
k
isobtainedasfollows(fortanhnon-linearities):
h
k
ij
=tanh((W
k
x)
ij
+b
k
):
Note:
Recall the following definition of convolution for r a 1D signal. . o[n] ] = = f[n]  g[n] ] =
P
1
u= 1
f[u]g[n u]=
P
1
u= 1
f[n u]g[u].
Thiscanbeextendedto2Dasfollows: o[m;n]= f[m;n]g[m;n]=
P
1
u= 1
P
1
v= 1
f[u;v]g[m 
u;n v].
Toform a richerrepresentation ofthe data, , each h hidden layeris composed ofmultiple feature maps,
fh
(k)
;k = = 0::Kg. . TheweightsW W ofahiddenlayercanberepresentedina4Dtensorcontainingele-
mentsforeverycombinationofdestinationfeaturemap,sourcefeaturemap,sourceverticalposition,and
sourcehorizontalposition. Thebiasesbcanberepresentedasavectorcontainingoneelementforevery
destinationfeaturemap.Weillustratethisgraphicallyasfollows:
Figure6.1:Figure1:exampleofaconvolutionallayer
ThefigureshowstwolayersofaCNN.Layerm-1containsfourfeaturemaps. Hiddenlayermcontains
twofeaturemaps(h
0
andh
1
). Pixels(neuronoutputs)inh
0
andh
1
(outlinedasblueandredsquares)are
computedfrompixelsoflayer(m-1)whichfallwithintheir2x2receptivefieldinthelayerbelow(shownas
coloredrectangles). Noticehowthereceptivefieldspansallfourinputfeaturemaps.TheweightsW
0
and
6.4. DetailsandNotation
53
DeepLearningTutorial,Release0.1
W
1
ofh
0
andh
1
arethus3Dweighttensors.Theleadingdimensionindexestheinputfeaturemaps,while
theothertworefertothepixelcoordinates.
Puttingitalltogether,W
kl
ij
denotestheweightconnectingeachpixelofthek-thfeaturemapatlayerm,with
thepixelatcoordinates(i,j)ofthel-thfeaturemapoflayer(m-1).
6.5 TheConvolutionOperator
ConvOpisthemainworkhorseforimplementingaconvolutionallayerinTheano. ConvOpisusedby
theano.tensor.signal.conv2d,whichtakestwosymbolicinputs:
• a4Dtensorcorrespondingtoamini-batchofinputimages. . Theshapeofthetensorisasfollows:
[mini-batchsize,numberofinputfeaturemaps,imageheight,imagewidth].
• a4DtensorcorrespondingtotheweightmatrixW. Theshapeofthetensoris:[numberoffeature
mapsatlayerm,numberoffeaturemapsatlayerm-1,filterheight,filterwidth]
BelowistheTheanocodeforimplementingaconvolutionallayersimilartotheoneofFigure1.Theinput
consistsof3featuresmaps(anRGBcolorimage)ofsize120x160. Weusetwoconvolutionalfilterswith
9x9receptivefields.
import theano
from theano import tensor as T
from theano.tensor.nnet import conv
import numpy
rng numpy.random.RandomState(23455)
# instantiate 4D D tensor r for input
input T.tensor4(name=’input’)
# initialize shared variable for weights.
w_shp (2399)
w_bound numpy.sqrt(3
*
9
*
9)
theano.shared( numpy.asarray(
rng.uniform(
low=-1.0 w_bound,
high=1.0 w_bound,
size=w_shp),
dtype=input.dtype), name =’W’)
# initialize shared variable for bias (1D tensor) with random values
# IMPORTANT: biases are usually initialized to zero. However in this
# particular application, we simply apply the convolutional layer to
# an image without learning the parameters. We therefore initialize
# them to random m values s to "simulate" learning.
b_shp (2,)
theano.shared(numpy.asarray(
rng.uniform(low=-.5, high=.5, size=b_shp),
dtype=input.dtype), name =’b’)
54
Chapter6. ConvolutionalNeuralNetworks(LeNet)
Documents you may be interested
Documents you may be interested