12.6 Script a Web Application Session

Problem

You want to interact with a website or application that requires dynamic cookies, logins, or multiple requests.

Solution

Use the Invoke-WebRequest cmdlet to download a web page, and access the -SessionVariable and -WebSession parameters. For example, to retrieve the number of active Facebook notifications:

$cred = Get-Credential
$login = Invoke-WebRequest http://www.facebook.com/login.php -SessionVariable fb
$login.Forms[0].Fields.email = $cred.GetNetworkCredential().UserName
$login.Forms[0].Fields.pass = $cred.GetNetworkCredential().Password
$main = Invoke-WebRequest $login.Forms[0].Action
    -WebSession $fb -Body $login -Method Post
$main.ParsedHtml.getElementById("notificationsCountValue").InnerText

Discussion

While many pages on the internet provide their information directly when you access a web page, many others aren’t so simple. For example, the site may be protected by a login page (which then sets cookies), followed by another form (which requires those cookies) that returns a search result.

Automating these scenarios almost always requires a fairly in-depth understanding of the web application in question, as well as how web applications work in general.

Even with that understanding, automating these scenarios usually requires a vast amount of scripting: parsing HTTP headers, sending them in subsequent requests, hand-crafting form POST responses, and more.

As an example of bare scripting of a Facebook login, consider the following example that merely determines the login cookie to be used in further page requests:

$Credential = Get-Credential

## Get initial cookies
$wc = New-Object System.Net.WebClient
$wc.Headers.Add("User-Agent", "User-Agent: Mozilla/4.0 (compatible; MSIE 7.0;)")

$result = $wc.DownloadString("https://www.facebook.com/")
$cookie = $wc.ResponseHeaders["Set-Cookie"]
$cookie = ($cookie.Split(',') -match '^\S+=\S+;' -replace ';.*','') -join '; '

$wc = New-Object System.Net.WebClient
$wc.Headers.Add("User-Agent", "User-Agent: Mozilla/4.0 (compatible; MSIE 7.0;)")
$wc.Headers.Add("Cookie", $cookie)
$postValues = New-Object System.Collections.Specialized.NameValueCollection
$postValues.Add("email", $credential.GetNetworkCredential().Username)
$postValues.Add("pass", $credential.GetNetworkCredential().Password)

## Get the resulting cookie, and convert it into the form to be returned
## in the query string
$result = $wc.UploadValues(
    "https://login.facebook.com/login.php?login_attempt=1", $postValues)
$cookie = $wc.ResponseHeaders["Set-Cookie"]
$cookie = ($cookie.Split(',') -match '^\S+=\S+;' -replace ';.*','') -join '; '
$cookie

This is just for the login. Scripting a full web session using this manual approach can easily take hundreds of lines of script.

If supported in your version of PowerShell, the -SessionVariable and -WebSession parameters of the Invoke-WebRequest cmdlet don’t remove the need to understand how your target web application works. They do, however, remove the drudgery and complexity of dealing with the bare HTTP requests and responses. This improved session support comes primarily through four features:

Automated cookie management

Most web applications store their state in cookies—session IDs and login information being the two most common things to store. When a web application requests that a cookie be stored or deleted, Invoke-WebRequest automatically records this information in the provided session variable. Subsequent requests that use this session variable automatically supply any cookies required by the web application. You can see the cookies in use by looking at the Cookies property of the session variable:

$fb.Cookies.GetCookies("https://www.facebook.com") | Select Name,Value

Automatic redirection support

After you submit a web form (especially a login form), many sites redirect through a series of intermediate pages before you finally land on the destination page. In basic HTTP scripting, this forces you to handle the many HTTP redirect status codes, parse the Location header, and resubmit all the appropriate values. The Invoke-WebRequest cmdlet handles this for you; the result it returns comes from the final page in any redirect sequences. If you wish to override this behavior, use the -MaximumRedirection parameter.

Form detection

Applications that require advanced session scripting tend to take most of their input data from fields in HTML forms, rather than items in the URL itself. Invoke-WebRequest exposes these forms through the Forms property of its result. This collection returns the form ID (useful if there are multiple forms), the form action (URL that should be used to submit the form), and fields defined by the form.

Form submission

In traditional HTTP scripting, submitting a form is a complicated process. You need to gather all the form fields, encode them properly, determine the resulting encoded length, and POST all of this data to the destination URL.
Invoke-WebRequest makes this very simple through the -Body parameter used as input when you select POST as the value of the -Method parameter. The -Body parameter accepts input in one of three formats:

The result of a previous Invoke-WebRequest call, in which case values from the first form are used (if the response contains only one form).
A specific form (as manually selected from the Forms property of a previous Invoke-WebRequest call), in which case values from that form are used.
An IDictionary (hashtable), in which case names and values from that dictionary are used.
An XML node, in which case the XML is encoded directly. This is used primarily for scripting REST APIs, and is unlikely to be used when scripting web application sessions.
A byte array, in which case the bytes are used and encoded directly. This is used primarily for scripting data uploads.

Let’s take a look at how these play a part in the script from the Solution, which detects how many notifications are pending on Facebook. Given how fast web applications change, it’s unlikely that this example will continue to work for long. It does demonstrate the thought process, however.

When you first connect to Facebook, you need to log in. Facebook funnels this through a page called login.php:

$login = Invoke-WebRequest http://www.facebook.com/login.php -SessionVariable fb

If you look at the page that gets returned, there is a single form that includes email and pass fields:

PS > $login.Forms.Fields

Key                Value
---                -----
(...)
return_session     0
legacy_return      1
session_key_only   0
trynum             1
email
pass
persist_box        1
default_persistent 0
(...)

We fill these in:

$cred = Get-Credential
$login.Forms[0].Fields.email = $cred.UserName
$login.Forms[0].Fields.pass = $cred.GetNetworkCredential().Password

And submit the form. We use $fb for the -WebSession parameter, as that is what we used during the original request. We POST to the URL referred to in the Action field of the login form, and use the $login variable as the request body. The $login variable is the response that we got from the first request, where we customized the email and pass form fields. PowerShell recognizes that this was the result of a previous web request, and uses that single form as the POST body:

$mainPage = Invoke-WebRequest $login.Forms[0].Action -WebSession $fb `
    -Body $login -Method Post

If you look at the raw HTML returned by this response (the Content property), you can see that the notification count is contained in a span element with the ID of notificationsCountValue:

(...) <span id="notificationsCountValue">1</span> (...)

To retrieve this element, we use the ParsedHtml property of the response, call the GetElementById method, and return the InnerText property:

$mainPage.ParsedHtml.getElementById("notificationsCountValue").InnerText

Using these techniques, we can unlock a great deal of functionality on the internet previously hidden behind complicated HTTP scripting.

For more information about using the ParsedHtml property to parse and analyze web pages, see Recipe 12.5.

12.6 Script a Web Application Session

Problem

Solution

Discussion

See Also