itextsharp pdf to image c# example: Allow users to save pdf form SDK Library service wpf .net html dnn Automate%20the%20Boring%20Stuff%20with%20Python%20(2015)26-part1615

Web Scraping

237

downloading files from the web with the requests module

The

requests

module lets you easily download ﬁles from the Web without

having to worry about complicated issues such as network errors, connec-

tion problems, and data compression. The

requests

module doesn’t come

with Python, so you’ll have to install it ﬁrst. From the command line, run

pip install requests

. (Appendix A has additional details on how to install

third-party modules.)

The

requests

module was written because Python’s

urllib2

module is

too complicated to use. In fact, take a permanent marker and black out this

entire paragraph. Forget I ever mentioned

urllib2

. If you need to download

things from the Web, just use the

requests

module.

Next, do a simple test to make sure the

requests

module installed itself

correctly. Enter the following into the interactive shell:

>>> import requests

If no error messages show up, then the

requests

module has been suc-

cessfully installed.

Downloading a Web Page with the requests.get() Function

The

requests.get()

function takes a string of a URL to download. By calling

type()

requests.get()

’s return value, you can see that it returns a

Response

object, which contains the response that the web server gave for your request.

I’ll explain the

Response

object in more detail later, but for now, enter the

following into the interactive shell while your computer is connected to

the Internet:

>>> import requests

u >>> res = requests.get('http://www.gutenberg.org/cache/epub/1112/pg1112.txt')

>>> type(res)

v >>> res.status_code == requests.codes.ok

True

>>> len(res.text)

178981

>>> print(res.text[:250])

The Project Gutenberg EBook of Romeo and Juliet, by William Shakespeare

This eBook is for the use of anyone anywhere at no cost and with

almost no restrictions whatsoever. You may copy it, give it away or

re-use it under the terms of the Proje

The URL goes to a text web page for the entire play of Romeo and Juliet,

provided by Project Gutenberg u. You can tell that the request for this web

page succeeded by checking the

status_code

attribute of the

Response

object.

Allow users to save pdf form - C# PDF Field Edit Library: insert, delete, update pdf form field in C#.net, ASP.NET, MVC, Ajax, WPF

Online C# Tutorial to Insert, Delete and Update Fields in PDF Document

create a pdf form that can be filled out; pdf form creator

Allow users to save pdf form - VB.NET PDF Field Edit library: insert, delete, update pdf form field in vb.net, ASP.NET, MVC, Ajax, WPF

How to Insert, Delete and Update Fields in PDF Document with VB.NET Demo Code

pdf save form data; adding image to pdf form

238

Chapter 11

If it is equal to the value of

requests.codes.ok

, then everything went ﬁne v.

(Incidentally, the status code for “OK” in the HTTP protocol is 200. You

may already be familiar with the 404 status code for “Not Found.”)

If the request succeeded, the downloaded web page is stored as a string

in the

Response

object’s

text

variable. This variable holds a large string of

the entire play; the call to

len(res.text)

shows you that it is more than

178,000 characters long. Finally, calling

print(res.text[:250])

displays only

the ﬁrst 250 characters.

Checking for Errors

As you’ve seen, the

Response

object has a

status_code

attribute that can be

checked against

requests.codes.ok

to see whether the download succeeded. A

simpler way to check for success is to call the

raise_for_status()

method on

the

Response

object. This will raise an exception if there was an error down-

loading the ﬁle and will do nothing if the download succeeded. Enter the

following into the interactive shell:

>>> res = requests.get('http://inventwithpython.com/page_that_does_not_exist')

>>> res.raise_for_status()

Traceback (most recent call last):

File "<pyshell#138>", line 1, in <module>

res.raise_for_status()

File "C:\Python34\lib\site-packages\requests\models.py", line 773, in raise_for_status

raise HTTPError(http_error_msg, response=self)

requests.exceptions.HTTPError: 404 Client Error: Not Found

The

raise_for_status()

method is a good way to ensure that a program

halts if a bad download occurs. This is a good thing: You want your program

to stop as soon as some unexpected error happens. If a failed download isn’t

a deal breaker for your program, you can wrap the

raise_for_status()

line

with

try

and

except

statements to handle this error case without crashing.

import requests

res = requests.get('http://inventwithpython.com/page_that_does_not_exist')

try:

res.raise_for_status()

except Exception as exc:

print('There was a problem: %s' % (exc))

This

raise_for_status()

method call causes the program to output the

following:

There was a problem: 404 Client Error: Not Found

Always call

raise_for_status()

after calling

requests.get()

. You want to be

sure that the download has actually worked before your program continues.

C# PDF Sticky Note Library: add, delete, update PDF note in C#.net

Allow users to add comments online in ASPX webpage. Able to change font size in PDF comment box. Able to save and print sticky notes in PDF file.

add date to pdf form; change pdf to fillable form

C# PDF Text Highlight Library: add, delete, update PDF text

Able to save highlighted content to original PDF document. Users will often face the situation that you may is such a functionality which allow users to give

add editable fields to pdf; adding text to a pdf form

Web Scraping

239

Saving downloaded files to the hard drive

From here, you can save the web page to a ﬁle on your hard drive with the

standard

open()

function and

write()

method. There are some slight differ-

ences, though. First, you must open the ﬁle in write binary mode by passing

the string

'wb'

as the second argument to

open()

. Even if the page is in plain-

text (such as the Romeo and Juliet text you downloaded earlier), you need to

write binary data instead of text data in order to maintain the Unicode encod-

ing of the text.

To write the web page to a ﬁle, you can use a

for

loop with the

Response

object’s

iter_content()

method.

>>> import requests

>>> res = requests.get('http://www.gutenberg.org/cache/epub/1112/pg1112.txt')

>>> res.raise_for_status()

>>> playFile = open('RomeoAndJuliet.txt', 'wb')

>>> for chunk in res.iter_content(100000):

playFile.write(chunk)

100000

78981

>>> playFile.close()

The

iter_content()

method returns “chunks” of the content on each

iteration through the loop. Each chunk is of the bytes data type, and you

get to specify how many bytes each chunk will contain. One hundred

thousand bytes is generally a good size, so pass

100000

as the argument to

iter_content()

The ﬁle RomeoAndJuliet.txt will now exist in the current working direc-

tory. Note that while the ﬁlename on the website was pg1112.txt, the ﬁle

on your hard drive has a different ﬁlename. The

requests

module simply

handles downloading the contents of web pages. Once the page is down-

loaded, it is simply data in your program. Even if you were to lose your

Internet connection after downloading the web page, all the page data

would still be on your computer.

uniCoDe e nCoDingS

Unicode encodings are beyond the scope of this book, but you can learn more

about them from these web pages:

• Joel on Software: The Absolute Minimum Every Software Developer

Absolutely, Positively Must Know About Unicode and Character Sets

(No Excuses!): http://www.joelonsoftware.com/articles/Unicode.html

• Pragmatic Unicode: http://nedbatchelder.com/text/unipain.html

C# Word - Document Processing in C#.NET

control (XDoc.Word).//More TODO allow users to create Word provides a method for users to achieve ISection section = doc0.CreateSection(0); //Save the document

add print button to pdf form; adding text field to pdf

VB.NET Image: DMS - Load, Draw & Store Web Document Annotations

NET and JavaScript controls that can allow users to create the provided imaging saving solutions, users are able and copy the sample code to save annotations as

create a pdf form in word; pdf create fillable form

240

Chapter 11

The

write()

method returns the number of bytes written to the ﬁle. In

the previous example, there were 100,000 bytes in the ﬁrst chunk, and the

remaining part of the ﬁle needed only 78,981 bytes.

To review, here’s the complete process for downloading and saving a ﬁle:

1. Call

requests.get()

to download the ﬁle.

2. Call

open()

with

'wb'

to create a new ﬁle in write binary mode.

3. Loop over the

Response

object’s

iter_content()

method.

4. Call

write()

on each iteration to write the content to the ﬁle.

5. Call

close()

to close the ﬁle.

That’s all there is to the

requests

module! The

for

loop and

iter_content()

stuff may seem complicated compared to the

open()

write()

close()

work-

ﬂow you’ve been using to write text ﬁles, but it’s to ensure that the

requests

module doesn’t eat up too much memory even if you download massive

ﬁles. You can learn about the

requests

module’s other features from http://

requests.readthedocs.org/.

html

Before you pick apart web pages, you’ll learn some HTML basics. You’ll also

see how to access your web browser’s powerful developer tools, which will

make scraping information from the Web much easier.

Resources for Learning HTML

Hypertext Markup Language (HTML) is the format that web pages are written

in. This chapter assumes you have some basic experience with HTML, but

if you need a beginner tutorial, I suggest one of the following sites:

• http://htmldog.com/guides/html/beginner/

• http://www.codecademy.com/tracks/web/

• https://developer.mozilla.org/en-US/learn/html/

A Quick Refresher

In case it’s been a while since you’ve looked at any HTML, here’s a quick

overview of the basics. An HTML ﬁle is a plaintext ﬁle with the .html ﬁle

extension. The text in these ﬁles is surrounded by tags, which are words

enclosed in angle brackets. The tags tell the browser how to format the web

page. A starting tag and closing tag can enclose some text to form an element.

The text (or inner HTML) is the content between the starting and closing

tags. For example, the following HTML will display Hello world! in the

browser, with Hello in bold:

<strong>Hello</strong> world!

VB.NET PDF Text Highlight Library: add, delete, update PDF text

text is such a functionality which allow users to give SDK is based on .NET framework 2.0, users are enabled to to the page page.AddPDFAnnot(annot) ' save to a

add image field to pdf form; change font pdf fillable form

VB Imaging - VB ISSN Barcode Generating

generated ISSN barcode to GIF, PNG, JPEG, BMP image formats, our users can even create ISSN barcode on PDF, TIFF, Excel Allow users to save created ISSN

convert word document to editable pdf form; add text field to pdf acrobat

Web Scraping

241

This HTML will look like Figure 11-1 in a browser.

Figure 11-1: Hello world! rendered in

the browser

The opening

tag says that the enclosed text will appear in bold.

The closing

</strong>

tags tells the browser where the end of the bold text is.

There are many different tags in HTML. Some of these tags have extra

properties in the form of attributes within the angle brackets. For example,

the

<a>

tag encloses text that should be a link. The URL that the text links

to is determined by the

href

attribute. Here’s an example:

Al's free <a href="http://inventwithpython.com">Python books</a>.

This HTML will look like Figure 11-2 in a browser.

Figure 11-2: The link rendered in the browser

Some elements have an

attribute that is used to uniquely identify

the element in the page. You will often instruct your programs to seek out an

element by its

attribute, so ﬁguring out an element’s

attribute using the

browser’s developer tools is a common task in writing web scraping programs.

Viewing the Source HTML of a Web Page

You’ll need to look at the HTML source of the web pages that your pro-

grams will work with. To do this, right-click (or

ctrl

-click on OS X) any

web page in your web browser, and select View Source or View page source

to see the HTML text of the page (see Figure 11-3). This is the text your

browser actually receives. The browser knows how to display, or render, the

web page from this HTML.

VB.NET Word: Create Linear and 2D Barcodes to Word Page Within VB.

of VB.NET Barcode Generator Add-on can allow users to draw Users are supposed to use our Word Barcode Drawing desired ISBN barcode on Word and save the created

add submit button to pdf form; change font in pdf form

C# PowerPoint: C# Codes to Combine & Split PowerPoint Documents

Allow users to store the combined PowerPoint file in two ways: form of the folder into a single one and save it to code in C# is offered to help users to work

create a pdf form; adding an image to a pdf form

242

Chapter 11

Figure 11-3: Viewing the source of a web page

I highly recommend viewing the source HTML of some of your favor-

ite sites. It’s ﬁne if you don’t fully understand what you are seeing when

you look at the source. You won’t need HTML mastery to write simple web

scraping programs—after all, you won’t be writing your own websites. You

just need enough knowledge to pick out data from an existing site.

Opening Your Browser’s Developer Tools

In addition to viewing a web page’s source, you can look through a page’s

HTML using your browser’s developer tools. In Chrome and Internet

Explorer for Windows, the developer tools are already installed, and you

can press F12 to make them appear (see Figure 11-4). Pressing F12 again

will make the developer tools disappear. In Chrome, you can also bring

up the developer tools by selecting View4Developer4Developer Tools.

In OS X, pressing

option

-I will open Chrome’s Developer Tools.

VB Imaging - VB Code 93 Generator Tutorial

that are compatible in VB class allow users to adjust Users can freely adjust the coordinates to change DrawBarcode(image, 150F, 150F) image.Save(ImageType.PNG

changing font size in pdf form field; acrobat create pdf form

C# Create PDF from CSV to convert csv files to PDF in C#.net, ASP.

Advanced .NET PDF Component Allows C# Users to Create PDF from CSV File in C# Programming Language in .NET Description: Convert to PDF/TIFF and save it on

add text field pdf; change font size in fillable pdf form

Web Scraping

243

Figure 11-4: The Developer Tools window in the Chrome browser

In Firefox, you can bring up the Web Developer Tools Inspector by

pressing

ctrl

ShiFt

-c on Windows and Linux or by pressing

option

-c

on OS X. The layout is almost identical to Chrome’s developer tools.

In Safari, open the Preferences window, and on the Advanced pane

check the Show Develop menu in the menu bar option. After it has been

enabled, you can bring up the developer tools by pressing

option

-i.

After enabling or installing the developer tools in your browser, you can

right-click any part of the web page and select Inspect Element from the

context menu to bring up the HTML responsible for that part of the page.

This will be helpful when you begin to parse HTML for your web scraping

programs.

Don’t uSe re gulAr ex PreSSionS to PArSe html

Locating a speciﬁc piece of HTML in a string seems like a perfect case for

regular expressions‮ However, I advise you against it‮ There are many differ-

ent ways that HTML can be formatted and still be considered valid HTML, but

trying to capture all these possible variations in a regular expression can be

tedious and error prone‮ A module developed speciﬁcally for parsing HTML,

such as Beautiful Soup, will be less likely to result in bugs‮

You can ﬁnd an extended argument for why you shouldn’t to parse HTML

with regular expressions at http://stackoverﬂow.com/a/1732454/1893164/‮

244

Chapter 11

Using the Developer Tools to Find HTML Elements

Once your program has downloaded a web page using the

requests

module,

you will have the page’s HTML content as a single string value. Now you

need to ﬁgure out which part of the HTML corresponds to the information

on the web page you’re interested in.

This is where the browser’s developer tools can help. Say you want

to write a program to pull weather forecast data from http://weather.gov/.

Before writing any code, do a little research. If you visit the site and search

for the 94105 ZIP code, the site will take you to a page showing the fore-

cast for that area.

What if you’re interested in scraping the temperature information for

that ZIP code? Right-click where it is on the page (or

control

-click on OS X)

and select Inspect Element from the context menu that appears. This will

bring up the Developer Tools window, which shows you the HTML that pro-

duces this particular part of the web page. Figure 11-5 shows the developer

tools open to the HTML of the temperature.

Figure 11-5: Inspecting the element that holds the temperature text with the developer tools

From the developer tools, you can see that the HTML responsible

for the temperature part of the web page is

<p class="myforecast-current

-lrg">57°F</p>

. This is exactly what you were looking for! It seems that

the temperature information is contained inside a

<p>

element with the

myforecast-current-lrg

class. Now that you know what you’re looking for, the

BeautifulSoup

module will help you ﬁnd it in the string.

Web Scraping

245

Parsing html with the BeautifulSoup module

Beautiful Soup is a module for extracting information from an HTML

page (and is much better for this purpose than regular expressions). The

BeautifulSoup

module’s name is

bs4

(for Beautiful Soup, version 4). To install

it, you will need to run

pip install beautifulsoup4

from the command line.

(Check out Appendix A for instructions on installing third-party modules.)

While

beautifulsoup4

is the name used for installation, to import Beautiful

Soup you run

import bs4

For this chapter, the Beautiful Soup examples will parse (that is, analyze

and identify the parts of) an HTML ﬁle on the hard drive. Open a new

ﬁle editor window in IDLE, enter the following, and save it as example.html.

Alternatively, download it from http://nostarch.com/automatestuff/.

<html><head><title>The Website Title</title></head>

<body>

<p>Download my <strong>Python</strong> book from <a href="http://

inventwithpython.com">my website</a>.</p>

<p class="slogan">Learn Python the easy way!</p>

<p>By <span id="author">Al Sweigart</span></p>

</body></html>

As you can see, even a simple HTML ﬁle involves many different tags

and attributes, and matters quickly get confusing with complex websites.

Thankfully, Beautiful Soup makes working with HTML much easier.

Creating a BeautifulSoup Object from HTML

The

bs4.BeautifulSoup()

function needs to be called with a string contain-

ing the HTML it will parse. The

bs4.BeautifulSoup()

function returns is a

BeautifulSoup

object. Enter the following into the interactive shell while

your computer is connected to the Internet:

>>> import requests, bs4

>>> res = requests.get('http://nostarch.com')

>>> res.raise_for_status()

>>> noStarchSoup = bs4.BeautifulSoup(res.text)

>>> type(noStarchSoup)

This code uses

requests.get()

to download the main page from the

No Starch Press website and then passes the

text

attribute of the response

bs4.BeautifulSoup()

. The

BeautifulSoup

object that it returns is stored in a

variable named

noStarchSoup

246

Chapter 11

You can also load an HTML ﬁle from your hard drive by passing a

File

object to

bs4.BeautifulSoup()

. Enter the following into the interactive shell

(make sure the example.html ﬁle is in the working directory):

>>> exampleFile = open('example.html')

>>> exampleSoup = bs4.BeautifulSoup(exampleFile)

>>> type(exampleSoup)

Once you have a

BeautifulSoup

object, you can use its methods to locate

speciﬁc parts of an HTML document.

Finding an Element with the select() Method

You can retrieve a web page element from a

BeautifulSoup

object by calling

the

select()

method and passing a string of a CSS selector for the element you

are looking for. Selectors are like regular expressions: They specify a pattern

to look for, in this case, in HTML pages instead of general text strings.

A full discussion of CSS selector syntax is beyond the scope of this

book (there’s a good selector tutorial in the resources at http://nostarch.com/

automatestuff/), but here’s a short introduction to selectors. Table 11-2 shows

examples of the most common CSS selector patterns.

table 11-2: Examples of CSS Selectors

Selector passed to the select() method

Will match ‮ ‮ ‮

soup.select('div')

All elements named <div>

soup.select('#author')

The element with an id attribute of author

soup.select('.notice')

All elements that use a CSS class attri-

bute named notice

soup.select('div span')

All elements named <span> that are within

an element named <div>

soup.select('div > span')

All elements named <span> that are

directly within an element named <div>,

with no other element in between

soup.select('input[name]')

All elements named <input> that have a

name attribute with any value

soup.select('input[type="button"]')

All elements named <input> that have an

attribute named type with value button

The various selector patterns can be combined to make sophisticated

matches. For example,

soup.select('p #author')

will match any element that

has an

attribute of

author

, as long as it is also inside a

<p>

element.

The

select()

method will return a list of

Tag

objects, which is how

Beautiful Soup represents an HTML element. The list will contain one

Tag

object for every match in the

BeautifulSoup

object’s HTML. Tag values

can be passed to the

str()

function to show the HTML tags they represent.

Documents you may be interested

itextsharp pdf to image c# example : Allow users to save pdf form SDK Library service wpf .net html dnn Automate%20the%20Boring%20Stuff%20with%20Python%20(2015)26-part1615