pdf viewer in asp net c# : How to add image to pdf form control SDK system web page wpf html console postgresql-9.4-A471-part2996

Chapter 29. Reliability and the Write-Ahead
Log
ThischapterexplainshowtheWrite-AheadLogisusedtoobtainefficient,reliableoperation.
29.1. Reliability
Reliabilityisanimportantpropertyofanyseriousdatabasesystem,andPostgreSQLdoeseverything
possibletoguaranteereliableoperation.Oneaspectofreliableoperationisthatalldatarecordedby
acommittedtransactionshouldbestoredinanonvolatileareathatissafefrompowerloss,operating
systemfailure,andhardwarefailure(exceptfailureofthenonvolatileareaitself,ofcourse).Success-
fullywritingthedatatothecomputer’spermanentstorage(diskdriveorequivalent)ordinarilymeets
thisrequirement.Infact,evenifacomputerisfatallydamaged,ifthediskdrivessurvivetheycanbe
movedtoanothercomputerwithsimilarhardwareandallcommittedtransactionswillremainintact.
Whileforcingdatatothediskplattersperiodicallymightseemlikeasimpleoperation,itisnot.Be-
causediskdrivesaredramaticallyslowerthanmainmemoryandCPUs,severallayersofcaching
existbetweenthecomputer’smainmemoryandthediskplatters.First,thereistheoperatingsystem’s
buffercache,whichcachesfrequentlyrequesteddiskblocksandcombinesdiskwrites.Fortunately,
alloperatingsystemsgiveapplicationsawaytoforcewritesfromthebuffercachetodisk,andPost-
greSQLusesthosefeatures.(Seethewal_sync_methodparametertoadjusthowthisisdone.)
Next,theremightbeacacheinthediskdrivecontroller;thisisparticularlycommononRAIDcon-
trollercards.Someofthesecachesarewrite-through,meaningwritesaresenttothedriveassoonas
theyarrive.Othersarewrite-back,meaningdataissenttothedriveatsomelatertime.Suchcaches
canbeareliabilityhazardbecausethememoryinthediskcontrollercacheisvolatile,andwilllose
itscontentsinapowerfailure.Bettercontrollercardshavebattery-backupunits(BBUs),meaning
thecardhasabatterythatmaintainspowertothecacheincaseofsystempowerloss.Afterpoweris
restoredthedatawillbewrittentothediskdrives.
Andfinally,mostdiskdriveshavecaches.Somearewrite-throughwhilesomearewrite-back,and
thesameconcernsaboutdatalossexistforwrite-backdrivecachesasfordiskcontrollercaches.
Consumer-gradeIDEandSATAdrivesareparticularlylikelytohavewrite-backcachesthatwillnot
surviveapowerfailure.Manysolid-statedrives(SSD)alsohavevolatilewrite-backcaches.
Thesecachescantypicallybedisabled;however,themethodfordoingthisvariesbyoperatingsystem
anddrivetype:
•
OnLinux,IDEandSATAdrivescanbequeriedusing
hdparm -I
;writecachingisenabledifthere
isa
*
nextto
Write cache
.
hdparm -W 0
canbeusedtoturnoffwritecaching.SCSIdrivescan
bequeriedusingsdparm
1
.Use
sdparm --get=WCE
tocheckwhetherthewritecacheisenabled
and
sdparm --clear=WCE
todisableit.
•
OnFreeBSD,IDEdrivescanbequeriedusing
atacontrol
andwritecachingturnedoffusing
hw.ata.wc=0
in
/boot/loader.conf
; SCSI drives can be queried using
camcontrol
identify
,andthewritecachebothqueriedandchangedusing
sdparm
whenavailable.
•
OnSolaris,thediskwritecacheiscontrolledby
format -e
.(TheSolarisZFSfilesystemissafe
withdiskwrite-cacheenabledbecauseitissuesitsowndiskcacheflushcommands.)
1. http://sg.danny.cz/sg/sdparm.html
638
How to add image to pdf form - insert images into PDF in C#.net, ASP.NET, MVC, Ajax, WinForms, WPF
Sample C# code to add image, picture, logo or digital photo into PDF document page using PDF page editor control
add photo pdf; adding images to pdf forms
How to add image to pdf form - VB.NET PDF insert image library: insert images into PDF in vb.net, ASP.NET, MVC, Ajax, WinForms, WPF
Guide VB.NET Programmers How to Add Images in PDF Document
acrobat insert image in pdf; add image to pdf online
Chapter29.ReliabilityandtheWrite-AheadLog
•
On
Windows,
if
wal_sync_method
is
open_datasync
(the
default),
write
caching
can
be
disabled
by
unchecking
My Computer\Open\
disk
drive
\Properties\Hardware\Properties\Policies\Enable write caching on the
disk
.Alternatively,set
wal_sync_method
to
fsync
or
fsync_writethrough
,whichprevent
writecaching.
•
On OS X, write caching can be prevented by setting
wal_sync_method
to
fsync_writethrough
.
RecentSATAdrives(thosefollowingATAPI-6orlater)offeradrivecacheflushcommand(
FLUSH
CACHE EXT
), whileSCSIdriveshavelongsupportedasimilarcommand
SYNCHRONIZE CACHE
.
ThesecommandsarenotdirectlyaccessibletoPostgreSQL,butsomefilesystems(e.g.,ZFS,ext4)
canusethemtoflushdatatotheplattersonwrite-back-enableddrives.Unfortunately,suchfilesys-
temsbehavesuboptimallywhencombinedwithbattery-backupunit(BBU)diskcontrollers.Insuch
setups,thesynchronizecommandforcesalldatafromthecontrollercachetothedisks,eliminating
muchofthebenefitoftheBBU.Youcanrunthepg_test_fsyncprogramtoseeifyouareaffected.If
youareaffected,theperformancebenefitsoftheBBUcanberegainedbyturningoffwritebarriersin
thefilesystemorreconfiguringthediskcontroller,ifthatisanoption.Ifwritebarriersareturnedoff,
makesurethebatteryremainsfunctional;afaultybatterycanpotentiallyleadtodataloss.Hopefully
filesystemanddiskcontrollerdesignerswilleventuallyaddressthissuboptimalbehavior.
Whentheoperatingsystemsendsawriterequesttothestoragehardware,thereislittleitcandoto
makesurethedatahasarrivedatatrulynon-volatilestoragearea.Rather,itistheadministrator’s
responsibilitytomakecertainthatallstoragecomponentsensureintegrityforbothdataandfile-
systemmetadata.Avoiddiskcontrollersthathavenon-battery-backedwritecaches.Atthedrivelevel,
disablewrite-backcachingifthedrivecannotguaranteethedatawillbewrittenbeforeshutdown.If
youuseSSDs,beawarethatmanyofthesedonothonorcacheflushcommandsbydefault.Youcan
testforreliableI/Osubsystembehaviorusing
diskchecker.pl
2
.
Anotherriskofdatalossisposedbythediskplatterwriteoperationsthemselves.Diskplattersare
dividedintosectors,commonly512byteseach.Everyphysicalreadorwriteoperationprocessesa
wholesector.Whenawriterequestarrivesatthedrive,itmightbeforsomemultipleof512bytes
(PostgreSQLtypicallywrites8192bytes,or16sectors,atatime),andtheprocessofwritingcouldfail
duetopowerlossatanytime,meaningsomeofthe512-bytesectorswerewrittenwhileotherswere
not.Toguardagainstsuchfailures,PostgreSQLperiodicallywritesfullpageimagestopermanent
WALstoragebeforemodifyingtheactualpageondisk.Bydoingthis,duringcrashrecoveryPost-
greSQLcanrestorepartially-writtenpagesfromWAL.Ifyouhavefile-systemsoftwarethatprevents
partialpagewrites(e.g.,ZFS),youcanturnoffthispageimagingbyturningoffthefull_page_writes
parameter.Battery-BackedUnit(BBU)diskcontrollersdonotpreventpartialpagewritesunlessthey
guaranteethatdataiswrittentotheBBUasfull(8kB)pages.
PostgreSQLalsoprotectsagainstsomekindsofdatacorruptiononstoragedevicesthatmayoccur
becauseofhardwareerrorsormediafailureovertime,suchasreading/writinggarbagedata.
•
EachindividualrecordinaWALfileisprotectedbyaCRC-32(32-bit)checkthatallowsustotell
ifrecordcontentsarecorrect.TheCRCvalueissetwhenwewriteeachWALrecordandchecked
duringcrashrecovery,archiverecoveryandreplication.
•
Datapagesarenotcurrentlychecksummedbydefault,thoughfullpageimagesrecordedinWAL
recordswillbeprotected;seeinitdbfordetailsaboutenablingdatapagechecksums.
•
Internal data structures such as
pg_clog
,
pg_subtrans
,
pg_multixact
,
pg_serial
,
pg_notify
,
pg_stat
,
pg_snapshots
arenotdirectlychecksummed,norarepagesprotected
byfullpagewrites.However,wheresuchdatastructuresarepersistent,WALrecordsarewritten
2. http://brad.livejournal.com/2116715.html
639
VB.NET PDF Form Data Read library: extract form data from PDF in
featured PDF software, it should have functions for processing text, image as well can help you have a quick evaluation of our PDF SDK. Add necessary references
add a picture to a pdf; add picture to pdf document
C# PDF Form Data Read Library: extract form data from PDF in C#.
featured PDF software, it should have functions for processing text, image as well can help you have a quick evaluation of our PDF SDK. Add necessary references
add photo to pdf reader; add image to pdf file
Chapter29.ReliabilityandtheWrite-AheadLog
thatallowrecentchangestobeaccuratelyrebuiltatcrashrecoveryandthoseWALrecordsare
protectedasdiscussedabove.
•
Individualstatefilesin
pg_twophase
areprotectedbyCRC-32.
•
TemporarydatafilesusedinlargerSQLqueriesforsorts,materializationsandintermediateresults
arenotcurrentlychecksummed,norwillWALrecordsbewrittenforchangestothosefiles.
PostgreSQLdoesnotprotectagainstcorrectablememoryerrorsanditisassumedyouwilloperate
usingRAMthatusesindustrystandardErrorCorrectingCodes(ECC)orbetterprotection.
29.2. Write-Ahead Logging (WAL)
Write-AheadLogging(WAL)isastandardmethodforensuringdataintegrity.Adetaileddescription
canbefoundinmost(ifnotall)booksabouttransactionprocessing.Briefly,WAL’scentralconcept
isthatchangestodatafiles(wheretablesandindexesreside)mustbewrittenonlyafterthosechanges
havebeenlogged,thatis,afterlogrecordsdescribingthechangeshavebeenflushedtopermanent
storage.Ifwefollowthisprocedure,wedonotneedtoflushdatapagestodiskoneverytransaction
commit,becauseweknowthatintheeventofacrashwewillbeabletorecoverthedatabaseusing
thelog:anychangesthathavenotbeenappliedtothedatapagescanberedonefromthelogrecords.
(Thisisroll-forwardrecovery,alsoknownasREDO.)
Tip:BecauseWALrestoresdatabasefilecontentsafteracrash,journaledfilesystemsarenot
necessaryforreliable storage ofthe data filesorWAL files.Infact, journaling overhead can
reduceperformance,especiallyifjournalingcausesfilesystemdatatobeflushedtodisk.Fortu-
nately,dataflushingduringjournalingcanoftenbedisabledwithafilesystemmountoption,e.g.
data=writeback
onaLinuxext3filesystem.Journaledfilesystemsdoimprovebootspeedafter
acrash.
UsingWALresultsinasignificantlyreducednumberofdiskwrites,becauseonlythelogfileneedsto
beflushedtodisktoguaranteethatatransactioniscommitted,ratherthaneverydatafilechangedby
thetransaction.Thelogfileiswrittensequentially,andsothecostofsyncingthelogismuchlessthan
thecostofflushingthedatapages.Thisisespeciallytrueforservershandlingmanysmalltransactions
touchingdifferentpartsofthedatastore.Furthermore, whentheserverisprocessingmanysmall
concurrenttransactions,one
fsync
ofthelogfilemaysufficetocommitmanytransactions.
WALalsomakesitpossibletosupporton-linebackupandpoint-in-timerecovery,asdescribedin
Section24.3.ByarchivingtheWALdatawecansupportrevertingtoanytimeinstantcoveredby
theavailableWALdata:wesimplyinstallapriorphysicalbackupofthedatabase,andreplaythe
WALlogjustasfarasthedesiredtime.What’smore,thephysicalbackupdoesn’thavetobean
instantaneoussnapshotofthedatabasestate—ifitismadeoversomeperiodoftime,thenreplaying
theWALlogforthatperiodwillfixanyinternalinconsistencies.
29.3. Asynchronous Commit
Asynchronouscommitisanoptionthatallowstransactionstocompletemorequickly,atthecostthat
themostrecenttransactionsmaybelostifthedatabaseshouldcrash.Inmanyapplicationsthisisan
acceptabletrade-off.
640
C# PDF Image Extract Library: Select, copy, paste PDF images in C#
Scan image to PDF, tiff and various image formats. Get image information, such as its location, zonal information Able to edit, add, delete, move, and output PDF
adding images to a pdf document; add jpg to pdf file
VB.NET PDF Password Library: add, remove, edit PDF file password
passwordSetting.IsAnnot = True ' Allow to fill form. passwordSetting document. passwordSetting.IsAssemble = True ' Add password to PDF file. PDFDocument
acrobat add image to pdf; how to add jpg to pdf file
Chapter29.ReliabilityandtheWrite-AheadLog
Asdescribedintheprevioussection,transactioncommitisnormallysynchronous:theserverwaitsfor
thetransaction’sWALrecordstobeflushedtopermanentstoragebeforereturningasuccessindication
totheclient.Theclientisthereforeguaranteedthatatransactionreportedtobecommittedwillbe
preserved,evenintheeventofaservercrashimmediatelyafter.However,forshorttransactionsthis
delayisamajorcomponentofthetotaltransactiontime.Selectingasynchronouscommitmodemeans
thattheserverreturnssuccessassoonasthetransactionislogicallycompleted,beforetheWAL
recordsitgeneratedhaveactuallymadetheirwaytodisk.Thiscanprovideasignificantboostin
throughputforsmalltransactions.
Asynchronouscommitintroducestheriskofdataloss.Thereisashorttimewindowbetweenthe
reportoftransactioncompletiontotheclientandthetimethatthetransactionistrulycommitted
(thatis,itisguaranteednottobelostiftheservercrashes).Thusasynchronouscommitshouldnot
beusediftheclientwilltakeexternalactionsrelyingontheassumptionthatthetransactionwillbe
remembered.Asanexample,abankwouldcertainlynotuseasynchronouscommitforatransaction
recordinganATM’sdispensingofcash.Butinmanyscenarios,suchaseventlogging,thereisno
needforastrongguaranteeofthiskind.
Theriskthatistakenbyusingasynchronouscommitisofdataloss,notdatacorruption.Ifthedatabase
shouldcrash,itwillrecoverbyreplayingWALuptothelastrecordthatwasflushed.Thedatabase
willthereforeberestoredtoaself-consistentstate,butanytransactionsthatwerenotyetflushedto
diskwillnotbereflectedinthatstate.Theneteffectisthereforelossofthelastfewtransactions.
Becausethetransactionsarereplayedincommitorder,noinconsistencycanbeintroduced—for
example,iftransactionBmadechangesrelyingontheeffectsofaprevioustransactionA,itisnot
possibleforA’seffectstobelostwhileB’seffectsarepreserved.
Theusercanselectthecommitmodeofeachtransaction,sothatitispossibletohavebothsyn-
chronousandasynchronouscommittransactionsrunningconcurrently.Thisallowsflexibletrade-offs
betweenperformanceandcertaintyoftransactiondurability.Thecommitmodeiscontrolledbythe
user-settableparametersynchronous_commit,whichcanbechangedinanyofthewaysthatacon-
figurationparametercanbeset.Themodeusedforanyonetransactiondepends onthevalueof
synchronous_commit
whentransactioncommitbegins.
Certainutilitycommands, forinstance
DROP TABLE
,areforcedtocommitsynchronouslyregard-
lessofthesettingof
synchronous_commit
.Thisistoensureconsistencybetweentheserver’sfile
systemandthelogicalstateofthedatabase.Thecommandssupportingtwo-phasecommit,suchas
PREPARE TRANSACTION
,arealsoalwayssynchronous.
Ifthedatabasecrashesduringtheriskwindowbetweenanasynchronouscommitandthewritingof
thetransaction’sWALrecords,thenchangesmadeduringthattransactionwillbelost.Theduration
oftheriskwindowislimitedbecauseabackgroundprocess(the“WALwriter”)flushesunwritten
WALrecordstodiskeverywal_writer_delaymilliseconds.Theactualmaximumdurationoftherisk
windowisthreetimes
wal_writer_delay
becausetheWALwriterisdesignedtofavorwriting
wholepagesatatimeduringbusyperiods.
Caution
Animmediate-modeshutdownisequivalenttoaservercrash,andwilltherefore
causelossofanyunflushedasynchronouscommits.
Asynchronouscommitprovidesbehaviordifferentfromsettingfsync=off.
fsync
isaserver-wide
settingthatwillalterthebehaviorofalltransactions.ItdisablesalllogicwithinPostgreSQLthat
attemptstosynchronizewritestodifferentportionsofthedatabase,andthereforeasystemcrash(that
is,ahardwareoroperatingsystemcrash,notafailureofPostgreSQLitself)couldresultinarbitrarily
badcorruptionofthedatabasestate.Inmanyscenarios,asynchronouscommitprovidesmostofthe
641
VB.NET PDF Image Extract Library: Select, copy, paste PDF images
multiple types of image from PDF file in VB.NET, like XObject Image, XObject Form, Inline Image DLLs for PDF Image Extraction in VB.NET. Add necessary references
add image to pdf acrobat reader; adding image to pdf file
C# PDF Password Library: add, remove, edit PDF file password in C#
passwordSetting.IsAnnot = true; // Allow to fill form. passwordSetting document. passwordSetting.IsAssemble = true; // Add password to PDF file. PDFDocument
how to add an image to a pdf; how to add image to pdf file
Chapter29.ReliabilityandtheWrite-AheadLog
performanceimprovementthatcouldbeobtainedbyturningoff
fsync
,butwithouttheriskofdata
corruption.
commit_delayalsosoundsverysimilartoasynchronouscommit,butitisactuallyasynchronouscom-
mitmethod(infact,
commit_delay
isignoredduringanasynchronouscommit).
commit_delay
causesadelayjustbeforeatransactionflushesWALtodisk,inthehopethatasingleflushexecuted
byonesuchtransactioncanalsoserveothertransactionscommittingataboutthesametime.Theset-
tingcanbethoughtofasawayofincreasingthetimewindowinwhichtransactionscanjoinagroup
abouttoparticipateinasingleflush,toamortizethecostoftheflushamongmultipletransactions.
29.4. WAL Configuration
ThereareseveralWAL-relatedconfigurationparametersthataffectdatabaseperformance.Thissec-
tionexplainstheiruse.ConsultChapter18forgeneralinformationaboutsettingserverconfiguration
parameters.
Checkpointsarepointsinthesequenceoftransactionsatwhichitisguaranteedthattheheapandindex
datafileshavebeenupdatedwithallinformationwrittenbeforethatcheckpoint.Atcheckpointtime,
alldirtydatapagesareflushedtodiskandaspecialcheckpointrecordiswrittentothelogfile.(The
changerecordswerepreviouslyflushedtotheWALfiles.)Intheeventofacrash,thecrashrecovery
procedurelooksatthelatestcheckpointrecordtodeterminethepointinthelog(knownastheredo
record)fromwhichitshouldstarttheREDOoperation.Anychangesmadetodatafilesbeforethat
pointareguaranteedtobealreadyondisk.Hence,afteracheckpoint,logsegmentsprecedingthe
onecontainingtheredorecordarenolongerneededandcanberecycledorremoved.(WhenWAL
archivingisbeingdone,thelogsegmentsmustbearchivedbeforebeingrecycledorremoved.)
ThecheckpointrequirementofflushingalldirtydatapagestodiskcancauseasignificantI/Oload.
Forthisreason,checkpointactivityisthrottledsothatI/Obeginsatcheckpointstartandcompletes
beforethenextcheckpointisduetostart;thisminimizesperformancedegradationduringcheckpoints.
Theserver’scheckpointerprocessautomaticallyperformsacheckpointeverysooften.Acheckpoint
isbeguneverycheckpoint_segmentslogsegments,oreverycheckpoint_timeoutseconds,whichever
comesfirst. Thedefaultsettingsare 3segments and300seconds(5minutes), respectively. Ifno
WAL has been writtensince the previous checkpoint, new checkpoints will be skippedeven if
checkpoint_timeout
haspassed.(IfWALarchivingisbeingusedandyouwanttoputalower
limitonhowoftenfilesarearchivedinordertoboundpotentialdataloss, youshouldadjustthe
archive_timeoutparameterratherthan the checkpointparameters.)Itis alsopossible to force a
checkpointbyusingtheSQLcommand
CHECKPOINT
.
Reducing
checkpoint_segments
and/or
checkpoint_timeout
causes checkpoints to occur
moreoften.Thisallowsfasterafter-crashrecovery,sincelessworkwillneedtoberedone.However,
one must balance this against the increased cost of flushing dirty data pages more often. If
full_page_writesisset(asisthedefault), thereisanotherfactortoconsider.Toensuredatapage
consistency,thefirstmodificationofadatapageaftereachcheckpointresultsinloggingtheentire
pagecontent.Inthatcase,asmallercheckpointintervalincreasesthevolumeofoutputtotheWAL
log,partiallynegatingthegoalofusingasmallerinterval,andinanycasecausingmorediskI/O.
Checkpointsarefairlyexpensive,firstbecausetheyrequirewritingoutallcurrentlydirtybuffers,and
secondbecausetheyresultinextrasubsequentWALtrafficasdiscussedabove.Itisthereforewiseto
setthecheckpointingparametershighenoughsothatcheckpointsdon’thappentoooften.Asasimple
sanitycheckonyourcheckpointingparameters,youcansetthecheckpoint_warningparameter.If
checkpointshappenclosertogetherthan
checkpoint_warning
seconds,amessagewillbeoutput
totheserverlogrecommendingincreasing
checkpoint_segments
.Occasionalappearanceofsuch
amessageisnotcauseforalarm,butifitappearsoftenthenthecheckpointcontrolparametersshould
642
C# Create PDF from images Library to convert Jpeg, png images to
List<Bitmap> images = new List<Bitmap>(); images.Add(new Bitmap(Program.RootPath + "\\" 1.gif")); / Build a PDF document with GIF image.
how to add a photo to a pdf document; how to add a jpeg to a pdf file
C# PDF Sticky Note Library: add, delete, update PDF note in C#.net
C#.NET PDF SDK - Add Sticky Note to PDF Page in C#.NET. Able to add notes to PDF using C# source code in Visual Studio .NET framework.
add picture to pdf file; add image pdf acrobat
Chapter29.ReliabilityandtheWrite-AheadLog
beincreased.Bulkoperationssuchaslarge
COPY
transfersmightcauseanumberofsuchwarningsto
appearifyouhavenotset
checkpoint_segments
highenough.
ToavoidfloodingtheI/Osystemwithaburstofpagewrites,writingdirtybuffersduringacheckpoint
isspreadoveraperiodoftime.Thatperiodiscontrolledbycheckpoint_completion_target,which
is givenas afractionofthecheckpointinterval. TheI/Orateis adjustedsothatthe checkpoint
finisheswhenthegivenfractionof
checkpoint_segments
WALsegmentshavebeenconsumed
since checkpoint start, or the given fraction of
checkpoint_timeout
seconds have elapsed,
whichever is sooner. With the default value of 0.5, PostgreSQL can be expected to complete
each checkpoint in about half the time before the next checkpoint starts. On a system that’s
very close to maximum I/O throughput during normal operation, you might want to increase
checkpoint_completion_target
toreducetheI/Oloadfromcheckpoints.Thedisadvantageof
this isthatprolongingcheckpointsaffectsrecoverytime,becausemoreWALsegmentswillneed
to be kept around for possible use in recovery. Although
checkpoint_completion_target
canbesetashighas1.0,itisbesttokeepitlessthanthat(perhaps0.9atmost)sincecheckpoints
includesomeotheractivitiesbesideswritingdirtybuffers.Asettingof1.0isquitelikelytoresultin
checkpointsnotbeingcompletedontime,whichwouldresultinperformancelossduetounexpected
variationinthenumberofWALsegmentsneeded.
There willalways be atleastoneWAL segmentfile, andwill normallynotbemorethan (2 +
checkpoint_completion_target
)*
checkpoint_segments
+1or
checkpoint_segments
+wal_keep_segments+1files.Eachsegmentfileisnormally16MB(thoughthissizecanbealtered
whenbuildingtheserver).YoucanusethistoestimatespacerequirementsforWAL.Ordinarily,when
oldlogsegmentfilesarenolongerneeded,theyarerecycled(thatis,renamedtobecomefutureseg-
mentsinthenumberedsequence).If,duetoashort-termpeakoflogoutputrate,therearemorethan
3*
checkpoint_segments
+1segmentfiles,theunneededsegmentfileswillbedeletedinsteadof
recycleduntilthesystemgetsbackunderthislimit.
Inarchiverecoveryorstandbymode,theserverperiodicallyperformsrestartpoints,whicharesimilar
tocheckpointsinnormaloperation:theserverforcesallitsstatetodisk,updatesthe
pg_control
filetoindicatethatthealready-processedWALdataneednotbescannedagain,andthenrecycles
anyoldlogsegmentfilesinthe
pg_xlog
directory.Restartpointscan’tbeperformedmorefrequently
thancheckpointsinthemasterbecauserestartpointscanonlybeperformedatcheckpointrecords.
Arestartpointistriggeredwhenacheckpointrecordisreachedifatleast
checkpoint_timeout
secondshavepassedsincethelastrestartpoint.Instandbymode,arestartpointisalsotriggeredifat
least
checkpoint_segments
logsegmentshavebeenreplayedsincethelastrestartpoint.
TherearetwocommonlyusedinternalWALfunctions:
XLogInsert
and
XLogFlush
.
XLogInsert
isusedtoplaceanewrecordintotheWALbuffersinsharedmemory.Ifthereisnospaceforthe
newrecord,
XLogInsert
willhavetowrite(movetokernelcache)afewfilledWALbuffers.This
isundesirablebecause
XLogInsert
isusedoneverydatabaselowlevelmodification(forexample,
rowinsertion)atatimewhenanexclusivelockisheldonaffecteddatapages,sotheoperationneeds
tobeasfastaspossible.Whatisworse,writingWALbuffersmightalsoforcethecreationofanew
logsegment,whichtakesevenmoretime.Normally,WALbuffersshouldbewrittenandflushedby
an
XLogFlush
request,whichismade,forthemostpart,attransactioncommittimetoensurethat
transactionrecordsareflushedtopermanentstorage.Onsystemswithhighlogoutput,
XLogFlush
requestsmightnotoccuroftenenoughtoprevent
XLogInsert
fromhavingtodowrites.Onsuch
systemsoneshouldincreasethenumberofWALbuffersbymodifyingthewal_buffersparameter.
Whenfull_page_writesissetandthesystemisverybusy,setting
wal_buffers
higherwillhelp
smoothresponsetimesduringtheperiodimmediatelyfollowingeachcheckpoint.
Thecommit_delayparameterdefinesforhowmanymicrosecondsagroupcommitleaderprocesswill
sleepafteracquiringalockwithin
XLogFlush
,whilegroupcommitfollowersqueueupbehindthe
leader.ThisdelayallowsotherserverprocessestoaddtheircommitrecordstotheWALbuffersso
thatallofthemwillbeflushedbytheleader’seventualsyncoperation.Nosleepwilloccuriffsync
643
Chapter29.ReliabilityandtheWrite-AheadLog
isnotenabled,oriffewerthancommit_siblingsothersessionsarecurrentlyinactivetransactions;
thisavoidssleepingwhenit’sunlikelythatanyothersessionwillcommitsoon.Notethatonsome
platforms,theresolutionofasleeprequestistenmilliseconds,sothatanynonzero
commit_delay
settingbetween1and10000microsecondswouldhavethesameeffect.Notealsothatonsomeplat-
forms,sleepoperationsmaytakeslightlylongerthanrequestedbytheparameter.
Sincethepurposeof
commit_delay
istoallowthecostofeachflushoperationtobeamortized
acrossconcurrentlycommittingtransactions(potentiallyattheexpenseoftransactionlatency),itis
necessarytoquantifythatcostbeforethesettingcanbechosenintelligently.Thehigherthatcost
is,themoreeffective
commit_delay
isexpectedtobeinincreasingtransactionthroughput,upto
apoint.Thepg_test_fsyncprogramcanbeusedtomeasuretheaveragetimeinmicrosecondsthata
singleWALflushoperationtakes.Avalueofhalfoftheaveragetimetheprogramreportsittakesto
flushafterasingle8kBwriteoperationisoftenthemosteffectivesettingfor
commit_delay
,sothis
valueisrecommendedasthestartingpointtousewhenoptimizingforaparticularworkload.While
tuning
commit_delay
isparticularlyusefulwhentheWALlogisstoredonhigh-latencyrotating
disks,benefitscanbesignificantevenonstoragemediawithveryfastsynctimes,suchassolid-state
drivesorRAIDarrayswithabattery-backedwritecache;butthisshoulddefinitelybetestedagainsta
representativeworkload.Highervaluesof
commit_siblings
shouldbeusedinsuchcases,whereas
smaller
commit_siblings
valuesareoftenhelpfulonhigherlatencymedia.Notethatitisquite
possiblethatasettingof
commit_delay
thatistoohighcanincreasetransactionlatencybysomuch
thattotaltransactionthroughputsuffers.
When
commit_delay
issettozero(thedefault),itisstillpossibleforaformofgroupcommitto
occur,buteachgroupwillconsistonlyofsessionsthatreachthepointwheretheyneedtoflushtheir
commitrecordsduringthewindowinwhichthepreviousflushoperation(ifany)isoccurring.At
higherclientcountsa“gangwayeffect”tendstooccur,sothattheeffectsofgroupcommitbecome
significantevenwhen
commit_delay
iszero,andthusexplicitlysetting
commit_delay
tendsto
helpless.Setting
commit_delay
canonlyhelpwhen(1)therearesomeconcurrentlycommitting
transactions,and(2)throughputislimitedtosomedegreebycommitrate;butwithhighrotational
latencythissettingcanbeeffectiveinincreasingtransactionthroughputwithasfewastwoclients
(thatis,asinglecommittingclientwithonesiblingtransaction).
Thewal_sync_methodparameterdetermineshowPostgreSQLwillaskthekerneltoforceWALup-
datesouttodisk.Alltheoptionsshouldbethesameintermsofreliability,withtheexceptionof
fsync_writethrough
,whichcansometimesforceaflushofthediskcacheevenwhenotherop-
tionsdonotdoso.However,it’squiteplatform-specificwhichonewillbethefastest.Youcantestthe
speedsofdifferentoptionsusingthepg_test_fsyncprogram.Notethatthisparameterisirrelevantif
fsync
hasbeenturnedoff.
Enablingthewal_debugconfigurationparameter(providedthatPostgreSQLhasbeencompiledwith
supportforit)willresultineach
XLogInsert
and
XLogFlush
WALcallbeingloggedtotheserver
log.Thisoptionmightbereplacedbyamoregeneralmechanisminthefuture.
29.5. WAL Internals
WALisautomaticallyenabled;noactionisrequiredfromtheadministratorexceptensuringthatthe
disk-spacerequirementsfortheWALlogsaremet,andthatanynecessarytuningisdone(seeSection
29.4).
WALlogsarestoredinthedirectory
pg_xlog
underthedatadirectory,asasetofsegmentfiles,
normallyeach16MBinsize(butthesizecanbechangedbyalteringthe
--with-wal-segsize
configureoptionwhenbuildingthe server).Each segmentis dividedintopages, normally8 kB
each (this size can be changed via the
--with-wal-blocksize
configure option). The log
644
Chapter29.ReliabilityandtheWrite-AheadLog
recordheadersaredescribedin
access/xlog.h
;therecordcontentisdependentonthetypeof
eventthatis beinglogged.Segmentfiles aregivenever-increasingnumbersasnames, startingat
000000010000000000000000
.Thenumbersdonotwrap,butitwilltakeavery,verylongtimeto
exhausttheavailablestockofnumbers.
Itisadvantageousifthelogislocatedonadifferentdiskfromthemaindatabasefiles.Thiscanbe
achievedbymovingthe
pg_xlog
directorytoanotherlocation(whiletheserverisshutdown,of
course)andcreatingasymboliclinkfromtheoriginallocationinthemaindatadirectorytothenew
location.
TheaimofWAListoensurethatthelogiswrittenbeforedatabaserecordsarealtered,butthiscan
besubvertedbydiskdrivesthatfalselyreportasuccessfulwritetothekernel,wheninfacttheyhave
onlycachedthedataandnotyetstoreditonthedisk.Apowerfailureinsuchasituationmightlead
toirrecoverabledatacorruption.AdministratorsshouldtrytoensurethatdisksholdingPostgreSQL’s
WALlogfilesdonotmakesuchfalsereports.(SeeSection29.1.)
Afteracheckpointhasbeenmadeandthelogflushed,thecheckpoint’spositionissavedinthefile
pg_control
.Therefore,atthestartofrecovery,theserverfirstreads
pg_control
andthenthe
checkpointrecord;thenitperformstheREDOoperationbyscanningforwardfromthelogposition
indicatedinthecheckpointrecord.Becausetheentirecontentofdatapagesissavedinthelogon
thefirstpagemodificationafteracheckpoint(assumingfull_page_writesisnotdisabled),allpages
changedsincethecheckpointwillberestoredtoaconsistentstate.
Todealwiththecasewhere
pg_control
iscorrupt,weshouldsupportthepossibilityofscanning
existinglogsegmentsinreverseorder—newesttooldest—inordertofindthelatestcheckpoint.
Thishasnotbeenimplementedyet.
pg_control
issmallenough(lessthanonediskpage)thatit
isnotsubjecttopartial-writeproblems,andasofthiswritingtherehavebeennoreportsofdatabase
failuresduesolelytotheinabilitytoread
pg_control
itself.Sowhileitistheoreticallyaweakspot,
pg_control
doesnotseemtobeaprobleminpractice.
645
Chapter 30. Regression Tests
TheregressiontestsareacomprehensivesetoftestsfortheSQLimplementationinPostgreSQL.
TheyteststandardSQLoperationsaswellastheextendedcapabilitiesofPostgreSQL.
30.1. Running the Tests
Theregressiontestscanberunagainstanalreadyinstalledandrunningserver,orusingatempo-
raryinstallationwithinthebuildtree.Furthermore,thereisa“parallel”anda“sequential”modefor
runningthetests.Thesequentialmethodrunseachtestscriptalone,whiletheparallelmethodstarts
upmultipleserverprocessestorungroupsoftestsinparallel.Paralleltestingaddsconfidencethat
interprocesscommunicationandlockingareworkingcorrectly.
30.1.1.RunningtheTestsAgainstaTemporaryInstallation
Toruntheparallelregressiontestsafterbuildingbutbeforeinstallation,type:
make check
inthetop-leveldirectory.(Oryoucanchangeto
src/test/regress
andrunthecommandthere.)
Attheendyoushouldseesomethinglike:
=======================
All 115 tests passed.
=======================
orotherwiseanoteaboutwhichtestsfailed.SeeSection30.2belowbeforeassumingthata“failure”
representsaseriousproblem.
Becausethistestmethodrunsatemporaryserver,itwillnotworkifyoudidthebuildastherootuser,
sincetheserverwillnotstartasroot.Recommendedprocedureisnottodothebuildasroot,orelse
toperformtestingaftercompletingtheinstallation.
IfyouhaveconfiguredPostgreSQLtoinstallintoalocationwhereanolderPostgreSQLinstallation
alreadyexists,andyouperform
make check
beforeinstallingthenewversion,youmightfindthatthe
testsfailbecausethenewprogramstrytousethealready-installedsharedlibraries.(Typicalsymptoms
arecomplaintsaboutundefinedsymbols.)Ifyouwishtorunthetestsbeforeoverwritingtheold
installation,you’llneedtobuildwith
configure --disable-rpath
.Itisnotrecommendedthat
youusethisoptionforthefinalinstallation,however.
TheparallelregressionteststartsquiteafewprocessesunderyouruserID.Presently,themaximum
concurrencyistwentyparalleltestscripts, whichmeans fortyprocesses:there’s aserverprocess
andapsqlprocessforeachtestscript.Soifyoursystemenforcesaper-userlimitonthenumberof
processes,makesurethislimitisatleastfiftyorso,elseyoumightgetrandom-seemingfailuresinthe
paralleltest.Ifyouarenotinapositiontoraisethelimit,youcancutdownthedegreeofparallelism
bysettingthe
MAX_CONNECTIONS
parameter.Forexample:
make MAX_CONNECTIONS=10 check
runsnomorethantentestsconcurrently.
646
Chapter30.RegressionTests
30.1.2.RunningtheTestsAgainstanExistingInstallation
Torunthetestsafterinstallation(seeChapter15),initializeadataareaandstarttheserverasexplained
inChapter17,thentype:
make installcheck
orforaparalleltest:
make installcheck-parallel
Thetestswillexpecttocontacttheserveratthelocalhostandthedefaultportnumber,unlessdirected
otherwiseby
PGHOST
and
PGPORT
environmentvariables.Thetestswillberuninadatabasenamed
regression
;anyexistingdatabasebythisnamewillbedropped. Thetestswillalsotransiently
createsomecluster-wideobjects,suchasuseridentitiesnamed
regressuser
N
.
30.1.3.AdditionalTestSuites
The
make check
and
make installcheck
commandsrunonlythe“core”regressiontests,which
testbuilt-infunctionalityofthePostgreSQLserver.Thesourcedistributionalsocontainsadditional
testsuites,mostofthemhavingtodowithadd-onfunctionalitysuchasoptionalprocedurallanguages.
Torunalltestsuitesapplicabletothemodulesthathavebeenselectedtobebuilt,includingthecore
tests,typeoneofthesecommandsatthetopofthebuildtree:
make check-world
make installcheck-world
Thesecommandsrunthetestsusingtemporaryserversoranalready-installedserver,respectively,
justaspreviouslyexplainedfor
make check
and
make installcheck
.Otherconsiderationsare
thesameaspreviouslyexplainedforeachmethod.Notethat
make check-world
buildsaseparate
temporaryinstallationtreeforeachtestedmodule,soitrequiresagreatdealmoretimeanddiskspace
than
make installcheck-world
.
Alternatively,youcanrunindividualtestsuitesbytyping
make check
or
make installcheck
intheappropriatesubdirectoryofthebuildtree.Keepinmindthat
make installcheck
assumes
you’veinstalledtherelevantmodule(s),notonlythecoreserver.
Theadditionalteststhatcanbeinvokedthiswayinclude:
•
Regressiontestsforoptionalprocedurallanguages(otherthanPL/pgSQL,whichistestedbythe
coretests).Thesearelocatedunder
src/pl
.
•
Regressiontestsfor
contrib
modules,locatedunder
contrib
.Notall
contrib
moduleshave
tests.
•
RegressiontestsfortheECPGinterfacelibrary,locatedin
src/interfaces/ecpg/test
.
•
Testsstressingbehaviorofconcurrentsessions,locatedin
src/test/isolation
.
•
Testsofclientprogramsunder
src/bin
.SeealsoSection30.4.
When using
installcheck
mode, these tests will destroy any existing databases named
pl_regression
,
contrib_regression
,
isolationtest
,
regress1
, or
connectdb
, aswell
as
regression
.
647
Documents you may be interested
Documents you may be interested