itextsharp pdf to image c# example: Change tab order in pdf form control software system web page winforms .net console Automate%20the%20Boring%20Stuff%20with%20Python%20(2015)27-part1616

Web Scraping

247

Tag values also have an

attrs

attribute that shows all the HTML attributes

of the tag as a dictionary. Using the example.html ﬁle from earlier, enter the

following into the interactive shell:

>>> import bs4

>>> exampleFile = open('example.html')

>>> exampleSoup = bs4.BeautifulSoup(exampleFile.read())

>>> elems = exampleSoup.select('#author')

>>> type(elems)

>>> len(elems)

>>> type(elems[0])

>>> elems[0].getText()

'Al Sweigart'

>>> str(elems[0])

'Al Sweigart'

>>> elems[0].attrs

{'id': 'author'}

This code will pull the element with

id="author"

out of our example

HTML. We use

select('#author')

to return a list of all the elements with

id="author"

. We store this list of

Tag

objects in the variable

elems

, and

len(elems)

tells us there is one

Tag

object in the list; there was one match.

Calling

getText()

on the element returns the element’s text, or inner

HTML. The text of an element is the content between the opening and

closing tags: in this case,

'Al Sweigart'

Passing the element to

str()

returns a string with the starting and clos-

ing tags and the element’s text. Finally,

attrs

gives us a dictionary with the

element’s attribute,

'id'

, and the value of the

attribute,

'author'

You can also pull all the

elements from the

BeautifulSoup

object.

Enter this into the interactive shell:

>>> pElems = exampleSoup.select('p')

>>> str(pElems[0])

'Download my Python book from <a href="http://

inventwithpython.com">my website</a>.'

>>> pElems[0].getText()

'Download my Python book from my website.'

>>> str(pElems[1])

'Learn Python the easy way!'

>>> pElems[1].getText()

'Learn Python the easy way!'

>>> str(pElems[2])

'By Al Sweigart'

>>> pElems[2].getText()

'By Al Sweigart'

This time,

select()

gives us a list of three matches, which we store in

pElems

. Using

str()

pElems[0]

pElems[1]

, and

pElems[2]

shows you each ele-

ment as a string, and using

getText()

on each element shows you its text.

Change tab order in pdf form - C# PDF Field Edit Library: insert, delete, update pdf form field in C#.net, ASP.NET, MVC, Ajax, WPF

Online C# Tutorial to Insert, Delete and Update Fields in PDF Document

adding form fields to pdf; adding signature to pdf form

Change tab order in pdf form - VB.NET PDF Field Edit library: insert, delete, update pdf form field in vb.net, ASP.NET, MVC, Ajax, WPF

How to Insert, Delete and Update Fields in PDF Document with VB.NET Demo Code

cannot save pdf form; create a form in pdf

248

Chapter 11

Getting Data from an Element’s Attributes

The

get()

method for

Tag

objects makes it simple to access attribute values

from an element. The method is passed a string of an attribute name and

returns that attribute’s value. Using example.html, enter the following into

the interactive shell:

>>> import bs4

>>> soup = bs4.BeautifulSoup(open('example.html'))

>>> spanElem = soup.select('span')[0]

>>> str(spanElem)

'Al Sweigart'

>>> spanElem.get('id')

'author'

>>> spanElem.get('some_nonexistent_addr') == None

True

>>> spanElem.attrs

{'id': 'author'}

Here we use

select()

to ﬁnd any

elements and then store the

ﬁrst matched element in

spanElem

. Passing the attribute name

'id'

get()

returns the attribute’s value,

'author'

Project: “i’m feeling lucky” google Search

Whenever I search a topic on Google, I don’t look at just one search result

at a time. By middle-clicking a search result link (or clicking while hold-

ing

ctrl

), I open the ﬁrst several links in a bunch of new tabs to read later.

I search Google often enough that this workﬂow—opening my browser,

searching for a topic, and middle-clicking several links one by one—is

tedious. It would be nice if I could simply type a search term on the com-

mand line and have my computer automatically open a browser with all

the top search results in new tabs. Let’s write a script to do this.

This is what your program does:

• Gets search keywords from the command line arguments.

• Retrieves the search results page.

• Opens a browser tab for each result.

This means your code will need to do the following:

• Read the command line arguments from

sys.argv

• Fetch the search result page with the

requests

module.

• Find the links to each search result.

• Call the

webbrowser.open()

function to open the web browser.

Open a new ﬁle editor window and save it as lucky.py.

C#: How to Edit XDoc.HTML5 Viewer Toolbar Commands

C# HTML5 Viewer: Change the Order of Command Tabs. This principle also applies equally to changing tabs order. Create a new Tab.

create pdf forms; create a form in pdf from word

C#: How to Add HTML5 Document Viewer Control to Your Web Page

but at the same time you should change the appsetting _viewerTopToolbar.addTab( _tabRedact); //add Tab "Sample Files In order to use certain functions, you may

create a pdf form from excel; best pdf form creator

Web Scraping

249

Step 1: Get the Command Line Arguments and Request the Search Page

Before coding anything, you ﬁrst need to know the URL of the search result

page. By looking at the browser’s address bar after doing a Google search,

you can see that the result page has a URL like https://www.google.com/

search?q=SEARCH_TERM_HERE. The

requests

module can download this

page and then you can use Beautiful Soup to ﬁnd the search result links in

the HTML. Finally, you’ll use the

webbrowser

module to open those links in

browser tabs.

Make your code look like the following:

#! python3

# lucky.py - Opens several Google search results.

import requests, sys, webbrowser, bs4

print('Googling...') # display text while downloading the Google page

res = requests.get('http://google.com/search?q=' + ' '.join(sys.argv[1:]))

res.raise_for_status()

# TODO: Retrieve top search result links.

# TODO: Open a browser tab for each result.

The user will specify the search terms using command line arguments

when they launch the program. These arguments will be stored as strings in

a list in

sys.argv

Step 2: Find All the Results

Now you need to use Beautiful Soup to extract the top search result links

from your downloaded HTML. But how do you ﬁgure out the right selec-

tor for the job? For example, you can’t just search for all

<a>

tags, because

there are lots of links you don’t care about in the HTML. Instead, you must

inspect the search result page with the browser’s developer tools to try to

ﬁnd a selector that will pick out only the links you want.

After doing a Google search for Beautiful Soup, you can open the

browser’s developer tools and inspect some of the link elements on the

page. They look incredibly complicated, something like this:

<a href="/url?sa

=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&

amp;ved=0CCgQFjAA&url=http%3A%2F%2Fwww.crummy.com%2Fsoftware%2FBeautifulSoup

%2F&ei=LHBVU_XDD9KVyAShmYDwCw&usg=AFQjCNHAxwplurFOBqg5cehWQEVKi-TuLQ&a

mp;sig2=sdZu6WVlBlVSDrwhtworMA" onmousedown="return rwt(this,'','','','1','AFQ

jCNHAxwplurFOBqg5cehWQEVKi-TuLQ','sdZu6WVlBlVSDrwhtworMA','0CCgQFjAA','','',ev

ent)" data-href="http://www.crummy.com/software/BeautifulSoup/">Beautiful

Soup: We called him Tortoise because he taught us.</a>

It doesn’t matter that the element looks incredibly complicated. You just

need to ﬁnd the pattern that all the search result links have. But this

<a>

ele-

ment doesn’t have anything that easily distinguishes it from the nonsearch

result

<a>

elements on the page.

VB.NET PowerPoint: Sort and Reorder PowerPoint Slides by Using VB.

and drag to specific position on the Slide tab in normal like "4 2 3 1 5" or change a certain index) Next ' Sort the PowerPoint slides using the correct order.

best way to create pdf forms; change font on pdf form

VB.NET TIFF: VB.NET Sample Codes to Sort TIFF File with .NET

Demonstrated in the code tab below is the Visual Basic (VB will save the edited TIFF doc with updated order as a For example, you may want to change the TIFF in

allow users to save pdf form; android edit pdf forms

250

Chapter 11

Make your code look like the following:

#! python3

# lucky.py - Opens several google search results.

import requests, sys, webbrowser, bs4

--snip--

# Retrieve top search result links.

soup = bs4.BeautifulSoup(res.text)

# Open a browser tab for each result.

linkElems = soup.select('.r a')

If you look up a little from the

<a>

element, though, there is an element

like this:

. Looking through the rest of the HTML source,

it looks like the

class is used only for search result links. You don’t have

to know what the CSS class

is or what it does. You’re just going to use

it as a marker for the

<a>

element you are looking for. You can create a

BeautifulSoup

object from the downloaded page’s HTML text and then use

the selector

'.r a'

to ﬁnd all

<a>

elements that are within an element that

has the

CSS class.

Step 3: Open Web Browsers for Each Result

Finally, we’ll tell the program to open web browser tabs for our results. Add

the following to the end of your program:

#! python3

# lucky.py - Opens several google search results.

import requests, sys, webbrowser, bs4

--snip--

# Open a browser tab for each result.

linkElems = soup.select('.r a')

numOpen = min(5, len(linkElems))

for i in range(numOpen):

webbrowser.open('http://google.com' + linkElems[i].get('href'))

By default, you open the ﬁrst ﬁve search results in new tabs using the

webbrowser

module. However, the user may have searched for something that

turned up fewer than ﬁve results. The

soup.select()

call returns a list of all

the elements that matched your

'.r

selector, so the number of tabs you

want to open is either

or the length of this list (whichever is smaller).

The built-in Python function

min()

returns the smallest of the integer

or ﬂoat arguments it is passed. (There is also a built-in

max()

function that

C# PDF: C# Code to Create Mobile PDF Viewer; C#.NET Mobile PDF

under "General" tab: right-click in "General"tab, select "Choose FileUrl = Server.MapPath("./Demo_Docs/").Replace("\\" Sample.pdf"; this.SessionId

create a fillable pdf form from a word document; pdf form save in reader

C# Image: How to Deploy .NET Imaging SDK in Visual C# Applications

in C#.NET Project. In order to deploy RasterEdge.Imaging.MSWordDocx.dll; RasterEdge.Imaging.PDF.dll; Under application tab, change target Framework to Frameworks

add fillable fields to pdf; adding text fields to pdf

Web Scraping

251

returns the largest argument it is passed.) You can use

min()

to ﬁnd out

whether there are fewer than ﬁve links in the list and store the number of

links to open in a variable named

numOpen

. Then you can run through a

for

loop by calling

range(numOpen)

On each iteration of the loop, you use

webbrowser.open()

to open a new

tab in the web browser. Note that the

href

attribute’s value in the returned

<a>

elements do not have the initial

http://google.com

part, so you have to

concatenate that to the

href

attribute’s string value.

Now you can instantly open the ﬁrst ﬁve Google results for, say, Python

programming tutorials by running

lucky python programming tutorials

on the

command line! (See Appendix B for how to easily run programs on your

operating system.)

Ideas for Similar Programs

The beneﬁt of tabbed browsing is that you can easily open links in new tabs

to peruse later. A program that automatically opens several links at once

can be a nice shortcut to do the following:

• Open all the product pages after searching a shopping site such as

Amazon

• Open all the links to reviews for a single product

• Open the result links to photos after performing a search on a photo

site such as Flickr or Imgur

Project: downloading All xkcd comics

Blogs and other regularly updating websites usually have a front page with

the most recent post as well as a Previous button on the page that takes you

to the previous post. Then that post will also have a Previous button, and so

on, creating a trail from the most recent page to the ﬁrst post on the site.

If you wanted a copy of the site’s content to read when you’re not online,

you could manually navigate over every page and save each one. But this is

pretty boring work, so let’s write a program to do it instead.

XKCD is a popular geek webcomic with a website that ﬁts this structure

(see Figure 11-6). The front page at http://xkcd.com/ has a Prev button that

guides the user back through prior comics. Downloading each comic by

hand would take forever, but you can write a script to do this in a couple of

minutes.

Here’s what your program does:

• Loads the XKCD home page.

• Saves the comic image on that page.

• Follows the Previous Comic link.

• Repeats until it reaches the ﬁrst comic.

VB.NET Image: VB.NET Codes to Read UPC-A Barcode in .NET Barcoding

PDF with VB.NET. In the code tab below we Similar to PDF barcode decoding, RasterEdge will firstly convert area to reduce scanning district in order to improve

create pdf form; change text size pdf form

VB.NET TIFF: VB.NET Sample Codes to Add Watermark in a TIFF Image

We have provided below in the code tab a detailed In order to make use of that code you to provide powerful & profession imaging controls, PDF document, image

chrome save pdf form; pdf form save with reader

252

Chapter 11

Figure 11-6: XKCD, “a webcomic of romance, sarcasm, math, and language”

This means your code will need to do the following:

• Download pages with the

requests

module.

• Find the URL of the comic image for a page using Beautiful Soup.

• Download and save the comic image to the hard drive with

iter_content()

• Find the URL of the Previous Comic link, and repeat.

Open a new ﬁle editor window and save it as downloadXkcd.py.

Step 1: Design the Program

If you open the browser’s developer tools and inspect the elements on the

page, you’ll ﬁnd the following:

• The URL of the comic’s image ﬁle is given by the

href

attribute of an

<img>

element.

• The

<img>

element is inside a

element.

• The Prev button has a

rel

HTML attribute with the value

• The ﬁrst comic’s Prev button links to the http://xkcd.com/# URL, indicat-

ing that there are no more previous pages.

Make your code look like the following:

#! python3

# downloadXkcd.py - Downloads every single XKCD comic.

import requests, os, bs4

url = 'http://xkcd.com' # starting url

os.makedirs('xkcd', exist_ok=True) # store comics in ./xkcd

VB.NET Image: VB.NET Codes to Load Images from File / Stream in .

In order to use the Visual Basic .NET sample codes it is understandable if you want to change the image Visual Basic .NET method listed in the below code tab.

pdf editable fields; adding images to pdf forms

VB.NET Image: VB.NET Codes to Add Antique Effect to Image with .

In the code tab below we have provided you the Visual Basic Please note that you should change corresponding information in the VB codings in order for it

adding form fields to pdf files; add forms to pdf

Web Scraping

253

while not url.endswith('#'):

# TODO: Download the page.

# TODO: Find the URL of the comic image.

# TODO: Download the image.

# TODO: Save the image to ./xkcd.

# TODO: Get the Prev button's url.

print('Done.')

You’ll have a

url

variable that starts with the value

'http://xkcd.com'

and repeatedly update it (in a

for

loop) with the URL of the current page’s

Prev link. At every step in the loop, you’ll download the comic at

url

. You’ll

know to end the loop when

url

ends with

'#'

You will download the image ﬁles to a folder in the current working

directory named xkcd. The call

os.makedirs()

ensures that this folder exists,

and the

exist_ok=True

keyword argument prevents the function from throw-

ing an exception if this folder already exists. The rest of the code is just

comments that outline the rest of your program.

Step 2: Download the Web Page

Let’s implement the code for downloading the page. Make your code look

like the following:

#! python3

# downloadXkcd.py - Downloads every single XKCD comic.

import requests, os, bs4

url = 'http://xkcd.com' # starting url

os.makedirs('xkcd', exist_ok=True) # store comics in ./xkcd

while not url.endswith('#'):

# Download the page.

print('Downloading page %s...' % url)

res = requests.get(url)

res.raise_for_status()

soup = bs4.BeautifulSoup(res.text)

# TODO: Find the URL of the comic image.

# TODO: Download the image.

# TODO: Save the image to ./xkcd.

# TODO: Get the Prev button's url.

print('Done.')

254

Chapter 11

First, print

url

so that the user knows which URL the program is about to

download; then use the

requests

module’s

request.get()

function to download

it. As always, you immediately call the

Response

object’s

raise_for_ status()

method to throw an exception and end the program if something went

wrong with the download. Otherwise, you create a

BeautifulSoup

object from

the text of the downloaded page.

Step 3: Find and Download the Comic Image

Make your code look like the following:

#! python3

# downloadXkcd.py - Downloads every single XKCD comic.

import requests, os, bs4

--snip--

# Find the URL of the comic image.

comicElem = soup.select('#comic img')

if comicElem == []:

print('Could not find comic image.')

else:

comicUrl = comicElem[0].get('src')

# Download the image.

print('Downloading image %s...' % (comicUrl))

res = requests.get(comicUrl)

res.raise_for_status()

# TODO: Save the image to ./xkcd.

# TODO: Get the Prev button's url.

print('Done.')

From inspecting the XKCD home page with your developer tools, you

know that the

<img>

element for the comic image is inside a

<div>

element

with the

attribute set to

comic

, so the selector

'#comic img'

will get you the

correct

<img>

element from the

BeautifulSoup

object.

A few XKCD pages have special content that isn’t a simple image ﬁle.

That’s ﬁne; you’ll just skip those. If your selector doesn’t ﬁnd any elements,

then

soup.select('#comic img')

will return a blank list. When that happens,

the program can just print an error message and move on without down-

loading the image.

Otherwise, the selector will return a list containing one

<img>

element. You

can get the

src

attribute from this

<img>

element and pass it to

requests.get()

to download the comic’s image ﬁle.

Web Scraping

255

Step 4: Save the Image and Find the Previous Comic

Make your code look like the following:

#! python3

# downloadXkcd.py - Downloads every single XKCD comic.

import requests, os, bs4

--snip--

# Save the image to ./xkcd.

imageFile = open(os.path.join('xkcd', os.path.basename(comicUrl)), 'wb')

for chunk in res.iter_content(100000):

imageFile.write(chunk)

imageFile.close()

# Get the Prev button's url.

prevLink = soup.select('a[rel="prev"]')[0]

url = 'http://xkcd.com' + prevLink.get('href')

print('Done.')

At this point, the image ﬁle of the comic is stored in the

res

variable.

You need to write this image data to a ﬁle on the hard drive.

You’ll need a ﬁlename for the local image ﬁle to pass to

open()

The

comicUrl

will have a value like

'http://imgs.xkcd.com/comics/heartbleed

_explanation.png'

—which you might have noticed looks a lot like a ﬁle path.

And in fact, you can call

os.path.basename()

with

comicUrl

, and it will return

just the last part of the URL,

'heartbleed_explanation.png'

. You can use this

as the ﬁlename when saving the image to your hard drive. You join this

name with the name of your

xkcd

folder using

os.path.join()

so that your

program uses backslashes (

) on Windows and forward slashes (

) on OS X

and Linux. Now that you ﬁnally have the ﬁlename, you can call

open()

open a new ﬁle in

'wb'

“write binary” mode.

Remember from earlier in this chapter that to save ﬁles you’ve downloaded

using Requests, you need to loop over the return value of the

iter_content()

method. The code in the

for

loop writes out chunks of the image data (at

most 100,000 bytes each) to the ﬁle and then you close the ﬁle. The image

is now saved to your hard drive.

Afterward, the selector

'a[rel="prev"]'

identiﬁes the

<a>

element with

the

rel

attribute set to

, and you can use this

<a>

element’s

href

attribute

to get the previous comic’s URL, which gets stored in

url

. Then the

while

loop begins the entire download process again for this comic.

The output of this program will look like this:

Downloading page http://xkcd.com...

Downloading image http://imgs.xkcd.com/comics/phone_alarm.png...

Downloading page http://xkcd.com/1358/...

256

Chapter 11

Downloading image http://imgs.xkcd.com/comics/nro.png...

Downloading page http://xkcd.com/1357/...

Downloading image http://imgs.xkcd.com/comics/free_speech.png...

Downloading page http://xkcd.com/1356/...

Downloading image http://imgs.xkcd.com/comics/orbital_mechanics.png...

Downloading page http://xkcd.com/1355/...

Downloading image http://imgs.xkcd.com/comics/airplane_message.png...

Downloading page http://xkcd.com/1354/...

Downloading image http://imgs.xkcd.com/comics/heartbleed_explanation.png...

--snip--

This project is a good example of a program that can automatically

follow links in order to scrape large amounts of data from the Web. You

can learn about Beautiful Soup’s other features from its documentation

at http://www.crummy.com/software/BeautifulSoup/bs4/doc/.

Ideas for Similar Programs

Downloading pages and following links are the basis of many web crawling

programs. Similar programs could also do the following:

• Back up an entire site by following all of its links.

• Copy all the messages off a web forum.

• Duplicate the catalog of items for sale on an online store.

The

requests

and

BeautifulSoup

modules are great as long as you can

ﬁgure out the URL you need to pass to

requests.get()

. However, sometimes

this isn’t so easy to ﬁnd. Or perhaps the website you want your program to

navigate requires you to log in ﬁrst. The

selenium

module will give your pro-

grams the power to perform such sophisticated tasks.

controlling the Browser with the selenium module

The

selenium

module lets Python directly control the browser by program-

matically clicking links and ﬁlling in login information, almost as though

there is a human user interacting with the page. Selenium allows you to

interact with web pages in a much more advanced way than Requests and

Beautiful Soup; but because it launches a web browser, it is a bit slower and

hard to run in the background if, say, you just need to download some ﬁles

from the Web.

Appendix A has more detailed steps on installing third-party modules.

Starting a Selenium-Controlled Browser

For these examples, you’ll need the Firefox web browser. This will be the

browser that you control. If you don’t already have Firefox, you can down-

load it for free from http://getﬁrefox.com/.

Documents you may be interested

itextsharp pdf to image c# example : Change tab order in pdf form control software system web page winforms .net console Automate%20the%20Boring%20Stuff%20with%20Python%20(2015)27-part1616