2.11. LQDYNAMICPROGRAMMINGPROBLEMS
211
Introduction
The“linear”partofLQisalinearlawofmotionforthestate,whilethe“quadratic”partrefersto
preferences
Let’sbeginwiththeformer,moveontothelatter,andthenputthemtogetherintoanoptimization
problem
TheLawofMotion Letx
t
beavectordescribingthestateofsomeeconomicsystem
Supposethatx
t
followsalinearlawofmotiongivenby
x
t+1
=Ax
t
+Bu
t
+Cw
t+1
,
t=0,1,2,...
(2.80)
Here
• u
t
isa“control”vector,incorporatingchoicesavailabletoadecisionmakerconfrontingthe
currentstatex
t
• fw
t
gisanuncorrelatedzeromeanshockprocesssatisfyingEw
t
w
0
t
=I,wheretheright-hand
sideistheidentitymatrix
Regardingthedimensions
• x
t
isn1,Aisnn
• u
t
isk1,Bisnk
• w
t
isj1,Cisnj
Example1 Considerahouseholdbudgetconstraintgivenby
a
t+1
+c
t
=(1+r)a
t
+y
t
Herea
t
isassets,risafixedinterestrate,c
t
iscurrentconsumption,andy
t
iscurrentnon-financial
income
Ifwesupposethatfy
t
gisuncorrelatedandN(0,s2),then,takingfw
t
gtobestandardnormal,we
canwritethesystemas
a
t+1
=(1+r)a
t
c
t
+sw
t+1
Thisisclearlyaspecialcaseof(2.80),withassetsbeingthestateandconsumptionbeingthecontrol
Example2 Oneunrealisticfeatureofthepreviousmodelisthatnon-financialincomehasazero
meanandisoftennegative
Thiscaneasilybeovercomebyaddingasufficientlylargemean
Henceinthisexamplewetakey
t
=sw
t+1
+mforsomepositiverealnumberm
Anotheralterationthat’susefultointroduce(we’llseewhysoon)istochangethecontrolvariable
fromconsumptiontothedeviationofconsumptionfromsome“ideal”quantity
¯
c
T
HOMAS
S
ARGENTAND
J
OHN
S
TACHURSKI
April20,2016
Change font size on pdf text box - Compress reduce PDF size in C#.net, ASP.NET, MVC, Ajax, WinForms, WPF
C# Code & .NET API to Compress & Decompress PDF Document
change font size pdf fillable form; adjust size of pdf
Change font size on pdf text box - VB.NET PDF File Compress Library: Compress reduce PDF size in vb.net, ASP.NET, MVC, Ajax, WinForms, WPF
VB.NET PDF Document Compression and Decompression Control SDK
pdf change font size; 300 dpi pdf file size
2.11. LQDYNAMICPROGRAMMINGPROBLEMS
212
(Mostparameterizationswillbesuchthat
¯
cislargerelativetotheamountofconsumptionthatis
attainableineachperiod,andhencethehouseholdwantstoincreaseconsumption)
Forthisreason,wenowtakeourcontroltobeu
t
:=c
t
¯c
Intermsofthesevariables,thebudgetconstrainta
t+1
=(1+r)a
t
c
t
+y
t
becomes
a
t+1
=(1+r)a
t
u
t
¯c+sw
t+1
+m
(2.81)
Howcanwewritethisnewsystemintheformofequation(2.80)?
If,asinthepreviousexample,wetakea
t
asthestate,thenwerunintoaproblem: thelawof
motioncontainssomeconstanttermsontheright-handside
Thismeansthatwearedealingwithanaffinefunction,notalinearone(recallthisdiscussion)
Fortunately,wecaneasilycircumventthisproblembyaddinganextrastatevariable
Inparticular,ifwewrite
a
t+1
1
=
1+¯c+m
0
1

a
t
1
+
1
0
u
t
+
s
0
w
t+1
(2.82)
thenthefirstrowisequivalentto(2.81)
Moreover,themodelisnowlinear,andcanbewrittenintheformof(2.80)bysetting
x
t
:=
a
t
1
A:=
1+¯c+m
0
1
B:=
1
0
C:=
s
0
(2.83)
Ineffect,we’veboughtourselveslinearitybyaddinganotherstate
Preferences IntheLQmodel,theaimistominimizeaflowoflosses,wheretime-tlossisgiven
bythequadraticexpression
x
0
t
Rx
t
+u
0
t
Qu
t
(2.84)
Here
• Risassumedtobenn,symmetricandnonnegativedefinite
• Qisassumedtobekk,symmetricandpositivedefinite
Note: Infact,formanyeconomicproblems,thedefinitenessconditionsonRandQcanberelaxed.
ItissufficientthatcertainsubmatricesofRandQbenonnegativedefinite.See[HS08]fordetails
Example1 Averysimpleexamplethat satisfiestheseassumptionsistotakeandtobe
identitymatrices,sothatcurrentlossis
x
0
t
Ix
t
+u
0
t
Iu
t
=kx
t
k
2
+ku
t
k
2
Thus,forboththestateandthecontrol,lossismeasuredassquareddistancefromtheorigin
(Infactthegeneralcase(2.84)canalsobeunderstoodinthisway,butwithRandQidentifying
other–non-Euclidean–notionsof“distance”fromthezerovector)
Intuitively,wecanoftenthinkofthestatex
t
asrepresentingdeviationfromatarget,suchas
T
HOMAS
S
ARGENTAND
J
OHN
S
TACHURSKI
April20,2016
C# PDF Text Box Edit Library: add, delete, update PDF text box in
Support to change font color in PDF text box. Ability to change text size in PDF text box. Adding text box is another way to add text to PDF page.
change font size pdf; adjust file size of pdf
C# PDF Sticky Note Library: add, delete, update PDF note in C#.net
Allow users to add comments online in ASPX webpage. Able to change font size in PDF comment box. Able to save and print sticky notes in PDF file.
reader compress pdf; pdf font size change
2.11. LQDYNAMICPROGRAMMINGPROBLEMS
213
• deviationofinflationfromsometargetlevel
• deviationofafirm’scapitalstockfromsomedesiredquantity
Theaimistoputthestateclosetothetarget,whileusingcontrolsparsimoniously
Example2 Inthehouseholdproblemstudiedabove,settingR=0andQ=1yieldspreferences
x
0
t
Rx
t
+u
0
t
Qu
t
=u
2
t
=(c
t
¯c)
2
Underthisspecification, thehousehold’scurrent lossisthesquareddeviationofconsumption
fromtheideallevel
¯
c
Optimality– FiniteHorizon
Let’snowbepreciseabouttheoptimizationproblemwewishtoconsider,andlookathowtosolve
it
TheObjective Wewillbeginwiththefinitehorizoncase,withterminaltimeT2N
Inthiscase,theaimistochooseasequenceofcontrolsfu
0
,...,u
1
gtominimizetheobjective
E
(
1
å
t=0
b
t
(x
0
t
Rx
t
+u
0
t
Qu
t
)+b
T
x
0
T
R
f
x
T
)
(2.85)
subjecttothelawofmotion(2.80)andinitialstatex
0
ThenewobjectsintroducedherearebandthematrixR
f
Thescalarbisthediscountfactor,whilex
0
R
f
xgivesterminallossassociatedwithstatex
Comments:
• WeassumeR
f
tobenn,symmetricandnonnegativedefinite
• Weallowb=1,andhenceincludetheundiscountedcase
• x
0
mayitselfberandom,inwhichcasewerequireittobeindependentoftheshocksequence
w
1
,...,w
T
Information There’soneconstraintwe’veneglectedtomentionsofar,whichisthatthedecision
makerwhosolvesthisLQproblemknowsonlythepresentandthepast,notthefuture
Toclarifythispoint,considerthesequenceofcontrolsfu
0
,...,u
1
g
Whenchoosingthesecontrols,thedecisionmakerispermittedtotakeintoaccounttheeffectsof
theshocksfw
1
,...,w
T
gonthesystem
However,itistypicallyassumed—andwillbeassumedhere—thatthetime-tcontrolu
t
canbe
madewithknowledgeofpastandpresentshocksonly
T
HOMAS
S
ARGENTAND
J
OHN
S
TACHURSKI
April20,2016
C# PDF Field Edit Library: insert, delete, update pdf form field
Support to change font size in PDF form. RasterEdge.Imaging.Font.dll. pdf"; String outputFilePath = Program.RootPath + "\\" Output.pdf"; List<BaseFormField
optimize scanned pdf; change font size in fillable pdf form
C# PDF Annotate Library: Draw, edit PDF annotation, markups in C#.
Support to add text, text box, text field and crop marks to PDF document. Able to edit and change PDF annotation properties such as font size or color.
change page size of pdf document; change paper size in pdf document
2.11. LQDYNAMICPROGRAMMINGPROBLEMS
214
Thefancymeasure-theoreticwayofsayingthisisthatu
t
mustbemeasurablewithrespecttothe
s-algebrageneratedbyx
0
,w
1
,w
2
,...,w
t
Thisisinfactequivalenttostatingthatu
t
canbewrittenintheformu
t
=g
t
(x
0
,w
1
,w
2
,...,w
t
)for
someBorelmeasurablefunctiong
t
(Justabouteveryfunctionthat’susefulforapplicationsisBorelmeasurable,so,forthepurposes
ofintuition,youcanreadthatlastphraseas“forsomefunctiong
t
”)
Nownotethatx
t
willultimatelydependontherealizationsofx
0
,w
1
,w
2
,...,w
t
Infactitturnsoutthatx
t
summarizesalltheinformationaboutthesehistoricalshocksthatthe
decisionmakerneedstosetcontrolsoptimally
Moreprecisely,itcanbeshownthatanyoptimalcontrolu
t
canalwaysbewrittenasafunctionof
thecurrentstatealone
Henceinwhatfollowswerestrictattentiontocontrolpolicies(i.e.,functions)oftheformu
t
=
g
t
(x
t
)
Actually,theprecedingdiscussionappliestoallstandarddynamicprogrammingproblems
What’sspecialabouttheLQcaseisthat–asweshallsoonsee—theoptimalu
t
turnsouttobea
linearfunctionofx
t
Solution TosolvethefinitehorizonLQproblemwecanuseadynamicprogrammingstrategy
basedonbackwardsinductionthatisconceptuallysimilartotheapproachadoptedinthislecture
Forreasonsthatwillsoonbecomeclear,wefirstintroducethenotationJ
T
(x):=x0R
f
x
Nowconsidertheproblemofthedecisionmakerinthesecondtolastperiod
Inparticular,letthetimebe1,andsupposethatthestateisx
1
Thedecisionmakermusttradeoffcurrentand(discounted)finallosses,andhencesolves
min
u
fx
0
1
Rx
1
+u
0
Qu+bEJ
T
(Ax
1
+Bu+Cw
T
)g
Atthisstage,itisconvenienttodefinethefunction
J
1
(x):=min
u
fx
0
Rx+u
0
Qu+bEJ
T
(Ax+Bu+Cw
T
)g
(2.86)
ThefunctionJ
1
willbecalledthe1valuefunction,andJ
1
(x)canbethoughtofasrepre-
sentingtotal“loss-to-go”fromstatexattime1whenthedecisionmakerbehavesoptimally
Nowlet’sstepbackto2
Foradecisionmakerat2, thevalueJ
1
(x)playsaroleanalogoustothat playedbythe
terminallossJ
T
(x)=x0R
f
xforthedecisionmakerat1
Thatis,J
1
(x)summarizesthefuturelossassociatedwithmovingtostatex
Thedecisionmakerchooseshercontrolutotradeoffcurrentlossagainstfutureloss,where
• thenextperiodstateisx
1
Ax
2
+Bu+Cw
1
,andhencedependsonthechoiceof
currentcontrol
T
HOMAS
S
ARGENTAND
J
OHN
S
TACHURSKI
April20,2016
C# PDF insert text Library: insert text into PDF content in C#.net
Powerful .NET PDF edit control allows modify existing scanned PDF text. Ability to change text font, color, size and location and output a new PDF document.
pdf form change font size; pdf files optimized
VB.NET PDF insert text library: insert text into PDF content in vb
Save text font, color, size and location changes to Other robust text processing features, like delete and remove PDF text, add PDF text box and field.
best online pdf compressor; pdf paper size
2.11. LQDYNAMICPROGRAMMINGPROBLEMS
215
• the“cost”oflandinginstatex
1
isJ
1
(x
1
)
Herproblemistherefore
min
u
fx
0
2
Rx
2
+u
0
Qu+bEJ
1
(Ax
2
+Bu+Cw
1
)g
Letting
J
2
(x):=min
u
fx
0
Rx+u
0
Qu+bEJ
1
(Ax+Bu+Cw
1
)g
thepatternforbackwardsinductionisnowclear
Inparticular,wedefineasequenceofvaluefunctionsfJ
0
,...,J
T
gvia
J
1
(x)=min
u
fx
0
Rx+u
0
Qu+bEJ
t
(Ax+Bu+Cw
t
)g and J
T
(x)=x
0
R
f
x
ThefirstequalityistheBellmanequationfromdynamicprogrammingtheoryspecializedtothe
finitehorizonLQproblem
NowthatwehavefJ
0
,...,J
T
g,wecanobtaintheoptimalcontrols
Asafirststep,let’sfindoutwhatthevaluefunctionslooklike
ItturnsoutthateveryJ
t
hastheformJ
t
(x) = x
0
P
t
x+d
t
whereP
t
isannmatrixandd
t
isa
constant
Wecanshowthisbyinduction,startingfromP
T
:=R
f
andd
T
=0
Usingthisnotation,(2.86)becomes
J
1
(x):=min
u
fx
0
Rx+u
0
Qu+bE(Ax+Bu+Cw
T
)
0
P
T
(Ax+Bu+Cw
T
)g
(2.87)
Toobtaintheminimizer,wecantakethederivativeofther.h.s.withrespecttouandsetitequal
tozero
Applyingtherelevantrulesofmatrixcalculus,thisgives
u= (Q+bB
0
P
T
B)
1
bB
0
P
T
Ax
(2.88)
Pluggingthisbackinto(2.87)andrearrangingyields
J
1
(x):=x
0
P
1
x+d
1
where
P
1
:=b
2
A
0
P
T
B(Q+bB
0
P
T
B)
1
B
0
P
T
A+bA
0
P
T
A
(2.89)
and
d
1
:=btrace(C
0
P
T
C)
(2.90)
(Thealgebraisagoodexercise—we’llleaveituptoyou)
Ifwecontinueworkingbackwardsinthismanner,itsoonbecomesclearthatJ
t
(x):=x0P
t
x+d
t
asclaimed,wherefP
t
gandfd
t
gsatisfytherecursions
P
1
:=b
2
A
0
P
t
B(Q+bB
0
P
t
B)
1
B
0
P
t
A+bA
0
P
t
with
P
T
=R
f
(2.91)
T
HOMAS
S
ARGENTAND
J
OHN
S
TACHURSKI
April20,2016
C# PDF Convert to Word SDK: Convert PDF to Word library in C#.net
PDF document, keeps the elements (like images, tables and chats) of original PDF file and maintains the original text style (including font, size, color, links
adjusting page size in pdf; change font size fillable pdf
VB.NET PDF delete text library: delete, remove text from PDF file
Functionality to remove text format by modifying text font, size, color, etc. Other PDF edit functionalities, like add PDF text, add PDF text box and field.
batch pdf compression; change page size pdf acrobat
2.11. LQDYNAMICPROGRAMMINGPROBLEMS
216
and
d
1
:=b(d
t
+trace(C
0
P
t
C)) with
d
T
=0
(2.92)
Recalling(2.88),theminimizersfromthesebackwardstepsare
u
t
F
t
x
t
where F
t
:=(Q+bB
0
P
t+1
B)
1
bB
0
P
t+1
A
(2.93)
Thesearethelinearoptimalcontrolpolicieswediscussedabove
Inparticular,thesequenceofcontrolsgivenby(2.93)and(2.80)solvesourfinitehorizonLQprob-
lem
Rephrasingthismoreprecisely,thesequenceu
0
,...,u
1
givenby
u
t
F
t
x
t
with x
t+1
=(A BF
t
)x
t
+Cw
t+1
(2.94)
fort=0,...,1attainstheminimumof(2.85)subjecttoourconstraints
AnApplication EarlyKeynesianmodelsassumedthat householdshaveaconstantmarginal
propensitytoconsumefromcurrentincome
Datacontradictedtheconstancyofthemarginalpropensitytoconsume
Inresponse,MiltonFriedman,FrancoModiglianiandmanyothersbuiltmodelsbasedonacon-
sumer’spreferenceforastableconsumptionstream
(See,forexample,[Fri56]or[MB54])
Onepropertyofthosemodelsisthathouseholdspurchaseandsellfinancialassetstomakecon-
sumptionstreamssmootherthanincomestreams
Thehouseholdsavingsproblemoutlinedabovecapturestheseideas
Theoptimizationproblemforthehouseholdistochooseaconsumptionsequenceinorderto
minimize
E
(
1
å
t=0
b
t
(c
t
¯c)
2
+b
T
qa
2
T
)
(2.95)
subjecttothesequenceofbudgetconstraintsa
t+1
=(1+r)a
t
c
t
+y
t
t0
Hereqisalargepositiveconstant,theroleofwhichistoinducetheconsumertotargetzerodebt
attheendofherlife
(Withoutsuchaconstraint,theoptimalchoiceistochoosec
t
¯cineachperiod,lettingassets
adjustaccordingly)
Asbeforewesety
t
=sw
t+1
+mandu
t
:=c
t
¯c,afterwhichtheconstraintcanbewrittenasin
(2.81)
WesawhowthisconstraintcouldbemanipulatedintotheLQformulationx
t+1
Ax
t
+Bu
t
+
Cw
t+1
bysettingx
t
=(a
t
1)
0
andusingthedefinitionsin(2.83)
Tomatchwiththisstateandcontrol,theobjectivefunction(2.95)canbewrittenintheformof
(2.85)bychoosing
Q:=1, R:=
0 0
0 0
, and R
f
:=
0
0 0
T
HOMAS
S
ARGENTAND
J
OHN
S
TACHURSKI
April20,2016
2.11. LQDYNAMICPROGRAMMINGPROBLEMS
217
NowthattheproblemisexpressedinLQform,wecanproceedtothesolutionbyapplying(2.91)
and(2.93)
Aftergeneratingshocksw
1
,...,w
T
,thedynamicsforassetsandconsumptioncanbesimulated
via(2.94)
Weprovidecodeforalltheseoperationsbelow
Thefollowingfigurewascomputedusingthiscode, with0.05,1/(1+r)2,=
1,s=0.25,T=45andq=10
6
Theshocksfw
t
gweretakentobeiidandstandardnormal
Thetoppanelshowsthetimepathofconsumptionc
t
andincomey
t
inthesimulation
Asanticipatedbythediscussiononconsumptionsmoothing, thetimepathofconsumptionis
muchsmootherthanthatforincome
(Butnotethatconsumptionbecomesmoreirregulartowardstheendoflife,whenthezerofinal
assetrequirementimpingesmoreonconsumptionchoices)
T
HOMAS
S
ARGENTAND
J
OHN
S
TACHURSKI
April20,2016
2.11. LQDYNAMICPROGRAMMINGPROBLEMS
218
Thesecondpanelinthefigureshowsthatthetimepathofassetsa
t
iscloselycorrelatedwith
cumulativeunanticipatedincome,wherethelatterisdefinedas
z
t
:=
t
å
j=0
sw
t
Akeymessageisthatunanticipatedwindfallgainsaresavedratherthanconsumed,whileunan-
ticipatednegativeshocksaremetbyreducingassets
(Again,thisrelationshipbreaksdowntowardstheendoflifeduetothezerofinalassetrequire-
ment)
Theseresultsarerelativelyrobusttochangesinparameters
Forexample,let’sincreasebfrom1/(1+r)0.952to0.96whilekeepingotherparametersfixed
Thisconsumerisslightlymorepatientthanthelastone,andhenceputsrelativelymoreweight
onlaterconsumptionvalues
Asimulationisshownbelow
T
HOMAS
S
ARGENTAND
J
OHN
S
TACHURSKI
April20,2016
2.11. LQDYNAMICPROGRAMMINGPROBLEMS
219
Wenowhaveaslowlyrisingconsumptionstreamandahump-shapedbuildupofassetsinthe
middleperiodstofundrisingconsumption
However,theessentialfeaturesarethesame:consumptionissmoothrelativetoincome,andassets
arestronglypositivelycorrelatedwithcumulativeunanticipatedincome
ExtensionsandComments
Let’snowconsideranumberofstandardextensionstotheLQproblemtreatedabove
NonstationaryParameters InsomesettingsitcanbedesirabletoallowA,B,C,RandQtode-
pendont
Forthesakeofsimplicity,we’vechosennottotreatthisextensioninourimplementationgiven
below
However,thelossofgeneralityisnotaslargeasyoumightfirstimagine
Infact,wecantacklemanynonstationarymodelsfromwithinourimplementationbysuitable
choiceofstatevariables
Oneillustrationisgivenbelow
Forfurtherexamplesandamoresystematictreatment,see[HS13],section2.4
AddingaCross-ProductTerm
InsomeLQproblems,preferencesincludeacross-productterm
u
0
t
Nx
t
,sothattheobjectivefunctionbecomes
E
(
1
å
t=0
b
t
(x
0
t
Rx
t
+u
0
t
Qu
t
+2u
0
t
Nx
t
)+b
T
x
0
T
R
f
x
T
)
(2.96)
Ourresultsextendtothiscaseinastraightforwardway
ThesequencefP
t
gfrom(2.91)becomes
P
1
:=(bB
0
P
t
A+N)
0
(Q+bB
0
P
t
B)
1
(bB
0
P
t
A+N)+bA
0
P
t
A
with
P
T
=R
f
(2.97)
Thepoliciesin(2.93)aremodifiedto
u
t
F
t
x
t
where F
t
:=(Q+bB
0
P
t+1
B)
1
(bB
0
P
t+1
A+N)
(2.98)
Thesequencefd
t
gisunchangedfrom(2.92)
Weleaveinterestedreaderstoconfirmtheseresults(thecalculationsarelongbutnotoverlydiffi-
cult)
T
HOMAS
S
ARGENTAND
J
OHN
S
TACHURSKI
April20,2016
2.11. LQDYNAMICPROGRAMMINGPROBLEMS
220
Infinite Horizon Finally, weconsider r the infinitehorizon n case, , with cross-productterm, un-
changeddynamicsandobjectivefunctiongivenby
E
(
¥
å
t=0
b
t
(x
0
t
Rx
t
+u
0
t
Qu
t
+2u
0
t
Nx
t
)
)
(2.99)
Intheinfinitehorizoncase,optimalpoliciescandependontimeonlyiftimeitselfisacomponent
ofthestatevectorx
t
Inotherwords,thereexistsafixedmatrixFsuchthatu
t
Fx
t
forallt
Thisstationarityisintuitive—afterall,thedecisionmakerfacesthesameinfinitehorizonatevery
stage,withonlythecurrentstatechanging
Notsurprisingly,Panddarealsoconstant
ThestationarymatrixPisgivenbythefixedpointof(2.91)
Equivalently,itisthesolutionPtothediscretetimealgebraicRiccatiequation
P:=(bB
0
PA+N)
0
(Q+bB
0
PB)
1
(bB
0
PA+N)+bA
0
PA
(2.100)
Equation(2.100)isalsocalledtheLQBellmanequation,andthemapthatsendsagivenPintothe
right-handsideof(2.100)iscalledtheLQBellmanoperator
Thestationaryoptimalpolicyforthismodelis
uFx where F:=(Q+bB
0
PB)
1
(bB
0
PA+N)
(2.101)
Thesequencefd
t
gfrom(2.92)isreplacedbytheconstantvalue
d:=trace(C
0
PC)
b
b
(2.102)
Thestateevolvesaccordingtothetime-homogeneousprocessx
t+1
=(A BF)x
t
+Cw
t+1
Anexampleinfinitehorizonproblemistreatedbelow
CertaintyEquivalence Linearquadraticcontrolproblemsoftheclassdiscussedabovehavethe
propertyofcertaintyequivalence
BythiswemeanthattheoptimalpolicyFisnotaffectedbytheparametersinC,whichspecify
theshockprocess
Thiscanbeconfirmedbyinspecting(2.101)or(2.98)
Itfollowsthatwecanignoreuncertaintywhensolvingforoptimalbehavior,andplugitbackin
whenexaminingoptimalstatedynamics
Implementation
Wehaveputtogethersomecodeforsolvingfiniteandinfinitehorizonlinearquadraticcontrol
problems
T
HOMAS
S
ARGENTAND
J
OHN
S
TACHURSKI
April20,2016
Documents you may be interested
Documents you may be interested