﻿

# print mvc view to pdf : Adding pdf to html page SDK control project wpf web page winforms UWP LM6-07-07-100-part1060

On the optimality of the aggregate with exponential
weights for low temperatures
Guillaume Lecue
1;3
Shahar Mendelson
2;4
Abstract
In this article, we study the optimality of the aggregate with exponential
weights (AEW) in the regression model with random design, and in the low
temperature regime. We prove three propertiesof AEW. First, that AEW isa
suboptimal aggregationprocedure inexpectationwithrespect to the quadratic
riskwhenT c
1
wherec
1
isanabsolutepositiveconstant(thelowtemperature
regime), and that it is suboptimal in probability even for high temperatures.
Second, we show that asthe cardinality of the dictionary grows, the behavior
ofAEWmightdeteriorate,namely,that inthelowtemperatureregimeit might
concentrate withhigh probability aroundelementsinthe dictionarywhose risk
islarger thanthe risk of the bestfunctioninthedictionaryby atleastorderof
1=
p
n. Ontheotherhand,weprovethatifoneassumesageometricconditionon
the dictionary (theso-calledBernsteincondition),thenAEWisindeedoptimal
bothinhighprobabilityandinexpectationinthelowtemperatureregime. More-
over,under that assumptionthe complexity termisessentially the logarithmof
the cardinality of the set of \almost minimizers" rather than the logarithm of
the cardinality oftheentire dictionary. Thisresult holdsfor small valuesof the
temperature parameter, thuscompleting an analogous result for hightempera-
tures.
1 Introduction and main results
In this note we study the problem concerning the optimality of the AEW in the
regressionmodelwithrandomdesign. Toformulatetheproblemweneedtointroduce
several denitions.
LetZandX betwomeasurespacesandsetZ andZ
1
;:::;Z
n
toben+1i.i.d. ran-
domvariableswithvalues inZ. From the statistical pointofview, D=(Z
1
;:::;Z
n
)
1
CNRS,LAMA,Marne-la-valle,77454France.
2
DepartmentofMathematics, Technion,I.I.T,Haifa 32000,Israel, andCentreforMathematics
anditsApplications,TheAustralianNationalUniversity,Canberra,ACT0200,Australia.
3
Email: guillaume.lecue@univ-mlv.fr
4
Email: shahar.mendelson@anu.edu.au
1
Adding pdf to html page - Library SDK class:C# PDF Convert to HTML SDK: Convert PDF to html files in C#.net, ASP.NET MVC, WinForms, WPF application
How to Convert PDF to HTML Webpage with C# PDF Conversion SDK
www.rasteredge.com
Adding pdf to html page - Library SDK class:VB.NET PDF Convert to HTML SDK: Convert PDF to html files in vb.net, ASP.NET MVC, WinForms, WPF application
PDF to HTML Webpage Converter SDK for VB.NET PDF to HTML Conversion
www.rasteredge.com
is theset of given dataatourdisposal. The risk ofa real-valued function f dened
on X isgiven by
R(f)=EQ(Z;f);
where Q : ZX 7!R is a nonnegative function, called the loss function. If
b
f is a
b
f is the random variable
R(
b
f)=E
h
Q(Z;
b
f)jD
i
:
Throughout thisarticlewewillrestrictourselvestofunctions f,lossfunctions Qand
random variables Z for which jQ(Z;f)j  b almost surely (note that some results
havebeen obtainedinthesamesetup forunbounded lossfunctionsin[7], [29], [13]or
[4]). The lossfunction wewill focuson through most of thisarticle is thequadratic
loss function, dened by Q((X;Y);f)=(Y  f(X))
2
.
In theaggregation framework, oneis given aniteset F ofreal-valued functions
dened on X, usually called a dictionary. The problem of aggregation (see, for ex-
ample, [10], [7] and [28])is toconstruct aprocedure thatproducesafunction whose
riskis as close as possibletotheriskof thebestelement in F. Havingthisin mind,
onecan denetheoptimalrate ofaggregation [24,15], which isthesmallestprice, as
afunction ofthecardinalityofthedictionaryM andthesamplesizen,thatonehas
to pay toconstruct a function whose risk is as close as possible to that of the best
elementin thedictionary. Werecall herethedenitionforthe\expectation case". A
similar denition forthe\probabilitycase"can alsobeformulated(see,forexample,
[15]).
Denition 1.1 ([24]) Letb>0. Wesay that (
n
(M))
n;M2N
is the optimal rate of
aggregation in expectation whenthereexisttwopositiveconstants c
0
andc
1
depending
only on bfor whichthe following holds for any n2N
and M 2N
:
1. there exists an aggregation procedure
~
f
n
such that for any dictionary F of car-
dinality M and any random variable Z satisfying jQ(Z;f)j  b almost surely
for all f 2F, one has
ER(
~
f
n
)min
f2F
R(f)+c
0
n
(M);
(1.1)
2. for any aggregation procedure
f
n
there exists a dictionary F of size M and a
random variable Z suchthat jQ(Z;f)jba.s. for all f 2F and
ER(
f
n
)min
f2F
R(f)+c
1
n
(M):
2
Library SDK class:VB.NET PDF Page Insert Library: insert pages into PDF file in vb.
Support adding PDF page number. Offer PDF page break inserting function. DLLs for Adding Page into PDF Document in VB.NET Class. Add necessary references:
www.rasteredge.com
Library SDK class:C# PDF Page Insert Library: insert pages into PDF file in C#.net
By using reliable APIs, C# programmers are capable of adding and inserting (empty) PDF page or pages from various file formats, such as PDF, Tiff, Word, Excel
www.rasteredge.com
In our setup, onecan show(cf. [24])that,in general, theoptimalrateofaggregation
(in the sense of [24] { optimality in expectation and of [15] { optimality in proba-
bility) is lowerbounded by (logM)=n. Hence, procedures satisfyingan exactoracle
inequality like (1.1) with the residual term
n
(M)=(logM)=n are said to be opti-
mal. Thereare only afew aggregation procedures that have been proved to achieve
this optimal rate. The rst resultsdealing with optimal aggregation procedures can
befound in[7]and[27](and forasurveyonoptimalaggregation procedureswerefer
Ourmainfocushereistheproblem oftheoptimalityoftheaggregationprocedure
with exponential weights(AEW). Theorigin ofthis procedurecomes from the ther-
modynamical point of view oflearningtheory (see[8] for thestateof theart in this
direction). AEW can be seen as arelaxed version ofthetrivial aggregation scheme,
which istominimizetheempirical risk
R
n
(f)=
1
n
Xn
i=1
Q(Z
i
;f)
(1.2)
in thedictionaryF.
Aprocedure that minimizes (1.2) is called empirical risk minimization (ERM),
and it is well known that ERM cannot, in general, achieve the optimal rate of
(logM)=n, unless one assumes that the given class F has certain geometric prop-
erties which will be discussed below (see also [16,19,13]). To have any chance of
obtaining better rates, one has to consider aggregation procedures that are taking
values in larger subsets than F, and the most natural set is the convex hull of F.
AEW has been a very popular candidate for an optimal procedure, and it was one
of the rst procedures to be studied in the context of the aggregation framework
[13,4,14,18,7,2,28,9]. It is dened by the followingconvex sum
~
f
AEW
=
M
X
j=1
b
j
f
j
where
b
j
=
exp
n
T
R
n
(f
j
)
P
M
k=1
exp
n
T
R
n
(f
k
)
(1.3)
forthedictionaryF =ff
1
;:::;f
M
g. TheparameterT >0iscalledthetemperature
1
.
Despiteitslonghistory, theoptimalityofAEWremained open. In thiswork, we
studythefollowing question:
Question 1.2 Is the AEW an optimal aggregation procedure in expectation or in
probability in the regression model with random design?
Wewill showthat the answer toQuestion1.2 is:
1
ThisterminologycomesfromThermodynamics,sincetheweights(
b
1
;:::;
b
M
)canbeseenasa
GibbsmeasurewithtemperatureT onthedictionaryF.
3
Library SDK class:VB.NET PDF Library SDK to view, edit, convert, process PDF file
Perform annotation capabilities to mark, draw, and visualize objects on PDF document page. Capable of adding PDF file navigation features to your VB.NET program
www.rasteredge.com
Library SDK class:C# PDF Library SDK to view, edit, convert, process PDF file for C#
Capable of adding PDF file navigation features to your C# program. Perform annotation capabilities to mark, draw, and visualize objects on PDF document page.
www.rasteredge.com
- negativeforlowtemperaturesT c
1
(wherec
1
isanabsolutepositiveconstant),
both in expectation and in probability, for the quadratic loss function and a
dictionary ofcardinality 2(Theorem A);
- negativein probability for somelarge dictionariesand small temperatures T 
c
1
(Theorem B);
- positive for low temperatures under a geometric condition on the dictionary
(TheoremC); Together with thehightemperatureresultof[1], [2] and [8], this
proves that the temperature parameter has almost no impact on the perfor-
mance of the AEW under this condition, with a residual term of the order of
((T +1)logM)=n forevery T.
TheoremA.Thereexistsabsolutepositiveconstantsc
0
;:::;c
5
forwhichthefollowing
holds. For any integer n  c
0
,there are random variables (X;Y) and a dictionary
F = ff
1
;f
2
gsuch that (Y  f
i
(X))
2
 1 almost surely for i = 1;2, for which the
quadratic risk of the AEW satises
1. if T c
1
and n is odd then
ER(
~
f
AEW
)min
f2F
R(f)+
c
2
p
n
;
2. if T c
3
p
n=logn,then withprobability greater than c
4
,
R(
~
f
AEW
)min
f2F
R(f)+
c
5
p
n
:
TheoremAprovesthatAEWissuboptimalinexpectationinthelowtemperature
regime, and suboptimal in probabilityin bothlow and high temperatureregimes.
It should be mentioned thatsuboptimality in probability does not imply subop-
timality in expectation for the aggregation problem, nor vice-versa. This property
of the aggregation problem was rst noticed in [3] where an aggregation procedure
called the progressive mixture rule was proved to be suboptimal in probability for
dictionaries of cardinality two, whereas it was known to be optimal in expectation
(cf. [7],[27], [29] or[13]).
The proof of Theorem A shows that a dictionary consisting of two functions is
enough togivethelowerbound inexpectationin thelowtemperature regimeandin
probability in both regimes. And, although the second part is to be expected, the
rstoneissurprising, as wewillexplain below.
4
Library SDK class:C# PDF insert image Library: insert images into PDF in C#.net, ASP
image adding library control for PDF document, you can easily and quickly add an image, picture or logo to any position of specified PDF document file page.
www.rasteredge.com
Library SDK class:C# PDF File & Page Process Library SDK for C#.net, ASP.NET, MVC
PDF in VB.NET, VB.NET convert PDF to HTML, VB.NET convert PDF to Word Provides you with examples for adding an (empty) page to a PDF and adding empty pages
www.rasteredge.com
In Theorem B we study the behavior of AEW for larger dictionaries. To our
knowledge,negativeresultson thebehaviorofexponentialweightsbasedaggregation
proceduresarenotknownfordictionarieswithmorethantwofunctions,andwhatwe
showis thatthebehaviorofthe AEW deteriorates,in some sense, as the cardinality
of the dictionary grows.
TheoremB.Thereexistsanintegern
0
andabsoluteconstantsc
1
andc
2
forwhichthe
followingholds. Foreverynn
0
F =ff
1
;:::;f
M
gofcardinality M =c
1
p
nlogn for which the quadratic loss function
of any element in F is bounded by 2 almost surely, and for every 0 <   1=2, if
T c
2
, then with probability at least 1 c
3
()n
 1=2
,
R(
~
f
AEW
)min
f2F
R(f)+c
4
()
r
logM
n
:
Moreover, if f
F
2F denotes the optimal function in F with respect to the quadratic
loss (the oracle), then there exists f
j
6=f
F
whose excess risk larger than c
5
()n
1=2
andfor which the weight of f
j
in the AEWprocedure satises
b
j
1
1
nc
6
()=T
:
Theorem Bimplies thattheAEW procedure mightcausetheweights toconcen-
trate around a \bad" element in the dictionary (that is, an element whose risk is
larger than the best in the classby at least n
1=2
)with high probability. In par-
forlow temperatures.
The analysis of the behavior of AEW for dictionary of cardinality larger than
two is considerably harder than the two-function case, and it requires some results
on rearrangement of independent random variables which are almost Gaussian (see
Proposition5.2below).
Fortunately, not all is lost as far as optimality results for AEW go. Indeed, we
will show thatundersome geometric condition,AEW can beoptimal; infact, it can
Intuitively, a good aggregation scheme should be able to ignore the elements in
the dictionary whose risk is far from the optimal risk in F, or at least the impact
of such elements on the function produced by the aggregation procedure should be
small. Hence, a good procedure is one whose residual term is of the order of  =n,
where   is a complexity measure that is determined only by the complexity of the
set of\almost minimizers"in thedictionary.
5
Library SDK class:VB.NET PDF File & Page Process Library SDK for vb.net, ASP.NET
page modifying page, you will find detailed guidance on creating, loading, merge and splitting PDF pages and Files, adding a page into PDF document, deleting
www.rasteredge.com
Library SDK class:C# PDF insert text Library: insert text into PDF content in C#.net
C#.NET PDF SDK - Insert Text to PDF Document in C#.NET. Providing C# Demo Code for Adding and Inserting Text to PDF File Page with .NET PDF Library.
www.rasteredge.com
Question 1.3 Is it possible to construct an aggregation procedure that adapts to the
real complexity ofthedictionary?
[1],[2]and[8]thatinthehightemperatureregime,AEWsatisestherequirementsof
Question1.3, assumingthat theclass has ageometricproperty, called theBernstein
condition.
Denition 1.4 ([5]) We say that a function class F is a (;B)-Bernstein class
(0<1 and B 1) with respect to Z, if every f 2F satises
E
f
2
(Z)
B(Ef(Z))
:
(1.4)
Thereare many natural situationsin which the Bernstein condition is satised. For
tobelongtoF then theexcess lossfunctions class L
F
=fQ(;f) Q(;f
F
):f 2Fg
satises the Bernstein condition with  =1 (where f
F
2F is the minimizer of the
risk in the class F). Another generic example is when the target function Y is far
fromtheset targetswith \multipleminimizers"in F, and,in which caseaswell,L
F
satises theBernstein condition with  =1(see[19,20] foran exactformulation of
this statementand related results).
TheBernstein conditionisverynaturalinthecontextofERMbecauseithastwo
consequences. Firstly, the empirical excess risk has better concentration properties
around the excess risk, and secondly, the complexity ofthesubset of F consistingof
almostminimizersissmallerunderthisassumption. Asaconsequence,iftheclassL
F
is a (;B)-Bernstein class for 0< 1, then the ERM algorithm can achieve fast
rates (see, for example [5], and references therein). As the results below show, the
sameistrueforAEW.Indeed, underaBernsteinassumptionitwasproved in([1],[2]
or [8])thatifR()isaconvexriskfunctionandifF issuch that jQ(Z;f)jbalmost
surely for any f 2 F then for every T  c
1
maxfb;Bg and x >0, with probability
greater than1 2exp( x),
R(
~
f
AEW
)min
f2F
R(f)+
Tc
2
n
x+log
X
f2F
exp
(n=2T)(R(f) R(f
F
))

: (1.5)
Although thePAC-Bayesian approach can notbeusedtoobtain (1.5)in thelow
temperature regime ( T  c
1
maxfb;Bg), such a result is not surprising. Indeed,
since fast error rates for the ERM are to be expected when the underlying excess
loss functionsclass satisestheBernstein conditionandsinceAEW converges tothe
ERM when the temperature T tends to zero, it is likely that for \small values" of
T, AEW inherits some of the properties of ERM, for example, fast rates under a
6
Library SDK class:VB.NET PDF insert text library: insert text into PDF content in vb
VB.NET PDF - Insert Text to PDF Document in VB.NET. Providing Demo Code for Adding and Inserting Text to PDF File Page in VB.NET Program.
www.rasteredge.com
Library SDK class:C# PDF Annotate Library: Draw, edit PDF annotation, markups in C#.
text comments on PDF page using C# demo code in Visual Stuodio .NET class. C#.NET: Add Text Box to PDF Document. Provide users with examples for adding text box
www.rasteredge.com
Bernstein condition. This is what we show in Theorem C, proving that AEW does
Before formulating Theorem C, let us introduce the following measure of com-
plexity. For everyr>0, let
(r)=log(jff 2F :R(f) R(f
F
)rgj+1)
+
1
X
j=1
2
j
log(jff 2F :2
j 1
r<R(f) R(f
F
)2
j
rgj+1);
wherejAj denotes the cardinalityoftheset A.
Observe that  (r) is a weighted sum of the elements in F that assigns smaller
andsmaller weightstofunctions whose excess risk is relatively large.
Theorem C.There exists absoluteconstants c
0
,c
1
;c
2
and c
3
forwhichthefollowing
holds. Let F be a class of functions bounded by b such that the excess loss class L
F
is a (1;B)-Bernstein class with respect to Z. If the risk function R() is convex and
ifT c
0
maxfb;Bg,then for every x>0, withprobabilityat least1 2exp( x),the
function
~
f
AEW
produced by the AEWalgorithm satises
R(
~
f
AEW
)R(f
F
)+c
1
(b+B)
x+ ()
n
;
where  =c
2
(b+B)(logjFj)=n.
In particular,
ER(
~
f
AEW
)R(f
F
)+c
3
(b+B)
()
n
:
In otherwords, the scaling factor  we useis proportional to(b+B)(logjFj)=n,
andiftheclass is reasonably regular,  ()is roughly the cardinalityoftheelements
in F whoseriskis at most(b+B)(logjFj)=n.
Observe that for every r > 0,  (r)  clogjFj for a suitable absolute constant
c. Therefore, if T is reasonablysmall {below alevel proportional tomaxfB;bg, the
resultingaggregation rate istheoptimal one, proportional to(b+B)(x+logM)=n
with probabilityof1 2exp( x), and proportional to(b+B)(logM)=nin expecta-
tion. Therefore, Theorem C gives a positive answer to Question1.3in the presence
of aBernstein condition and fora low temperature.
Although the residual terms in Theorem C and in (1.5) are not the same, they
are comparable. Indeed, the contribution ofeach element in F in the residual term
dependsexponentially on itsexcess risk.
7
Theorem C together with the result for high temperatures from [1], [2] and [8]
shows that the AEW is an optimal aggregation procedure under the Bernstein con-
dition as long as T =O(1) when M and n tend to innity. In general, the residual
term one obtains isof the order of((T +1)logM)=n.
Finally, a word about the organization of the article. In the next section we
provide some comments about our results. Theproofs of the three theorems follow
in the other sections. Throughout, we denote absolute constants or constants that
depend on other parameters by c
1
, c
2
, etc., (and, of course, we will specify when
a constant is absolute and when it depends on other parameters). The values of
constantsmaychangefromlinetoline. Wewriteabifthereareabsoluteconstants
cand C such thatbcaCb, and a.bif aCb.
Acknowledgments
ThisarticlewaswrittenwhileG.LecuewasvisitingtheDepartmentofMathematics,
Technion,andtheCentre for Mathematicsand its Applications, TheAustralianNa-
tional University. The authors would like to thank both these institutions for their
hospitality. We alsowould like tothank PierreAlquierand Olivier Catoni for useful
discussions.
The research leading to these results has received funding from the European
ResearchCouncil undertheEuropean Community’sSeventh FrameworkProgramme
(FP7/2007-2013) / ERC grant agreement n
o
[203134] and from the Australian Re-
search Council Discovery ProjectDP0986563.
The suboptimality in expectation of the AEW, obtained in Theorem A, is rather
surprising for tworeasons. First ofall, itis known that the progressivemixturerule
is optimal in expectation for T larger than some parameters of the model (see [7],
[27], [29], [13] or[4]). This procedure isdened by
f=
1
n
n
X
k=1
~
f
AEW
k
;
(2.1)
where
~
f
AEW
k
is the function generated by AEW associated with the dictionary F
andconstructedusingtherstk observationsZ
1
;:::;Z
k
. Thus,thisaggregateis the
mean of
~
f
AEW
k
for 1  k  n, where, for every k < n,
~
f
AEW
k
is constructed using
onlythe rst k observations. In particular,
fis the mean ofaggregates that are(or
should be) less \ecient" than
~
f
AEW
n
,since the latter is constructed using all the
8
observationsZ
1
;:::;Z
n
,rather thanasubsetofthegiven observations. Thatis why
one expects the AEW to be an optimal aggregation procedure in expectation { at
least in thehigh temperature regime. Theorem A showsthat, even fortemperature
of the order of a constant,
~
f
AEW
n
might have a very bad behavior, of the order of
(1=
p
n).
Second, theoptimalityin expectation ofAEW was obtainedin [9] fortheregres-
sion model Y
i
=f(x
i
)+
i
with a deterministic design x
1
;:::;x
n
2X with respect
to the risk kg fk
2
n
= n
1
P
n
i=1
(g(x
i
) f(x
i
))
2
(with its empirical version being
R
n
(g) = n
1
P
n
i=1
(Y
i
g(x
i
))
2
); that is, it was shown that for T  cmax(b;
2
)
(where
2
is the varianceofthenoise ),
E
~
f
AEW
f
2
n
min
g2F
kg fk
2
n
+
TlogM
n+1
:
(2.2)
TheoremA showsthat thebehavior oftheAEW isverydierent,atleast inthelow
temperatureregime. Thefactthatthesameprocedure(althoughindierentmodels)
can exhibit such two extremebehaviors- and forroughlythe sametemperaturepa-
rameter is ratherstriking. The 1=
p
nlower bound of Theorem A vs. the 1=n upper
bound derived from the oracle inequality (2.2) can haveone ofthetwofollowingex-
planations. Eitherthatthetwoseeminglysimilarscenariosare,infactverydierent,
or thatAEW exhibits asharp phasetransition at T c. And, if the latter is true,
then an important outcome of Theorem A is that the temperature parameter is of
thehighestimportance with regard totheoptimalityof the AEW in expectation.
Anindicationthataphasetransitionisthelikelyexplanationtothephenomenon
observedinTheorem A,isthatmostoftheoptimalupperboundsonAEWoronthe
progressivemixtureneedT tobelargerthansomeunknownparametersofthemodel
(thevarianceofthenoiseforinstance). Thismeansthatinpractice,AEWislikelyto
beavery\risky"aggregation procedurebecauseofits sensitivitytothetemperature
parameter. Moreover,andtomakethingsevenworse,evenforlargevaluesofT,AEW
is suboptimal with a constant probability for small dictionaries (Part 2 of Theorem
A) and with probabilitythat tends to1 for larger dictionaries (Theorem B). Hence,
given a setofdataand adictionary, AEW is likely tobehavevery poorlyregardless
of what T is. In contrast, Theorem C shows that the choice of the temperature
parameter has no signicant eect on the performance of the AEW (residual term
of the order of T(logM)=n) under the Bernstein condition. To conclude, although
fromatheoreticalpointofviewitremains tobeseenwhetherAEW displaysaphase
transitionatconstanttemperature,andisindeedanoptimalprocedureinexpectation
for high temperatures T c
2
max(b;
2
)as one may conjecture based on theresults
from [7], [27], [29], [13] [4] and [9], from a practical point of view, we believe that
exponentialaggregatingschemessimplyshouldnotbeusedinthesetupofthisarticle.
ThechoiceofT issimplytoo\risky",asindicatedbythelowerboundsinprobability
9