4.5 Process Time-Consuming Action in Parallel

Problem

You have a set of data or actions that you want to run at the same time.

Solution

Use the -parallel switch of the ForEach-Object cmdlet:

PS > Measure-Command { 1..5 | ForEach-Object { Start-Sleep -Seconds 5 } }

(...)
TotalSeconds      : 25.0247856
(...)

PS > Measure-Command { 1..5 | ForEach-Object -parallel { Start-Sleep -Seconds 5 } }

(...)
TotalSeconds      : 5.1354752
(...)

Discussion

There are times in PowerShell when you can significantly speed up a long-running operation by running parts of it at the same time. Perfect opportunities for this are scenarios where your script spends most of its time waiting on network resources (such as downloading files or web pages) or slow operations (such as restarting a series of slow services).

In these scenarios, you can use the -parallel parameter of ForEach-Object to perform these actions at the same time. Under the covers, PowerShell uses background jobs to run each branch. It caps the number of branches running at the same time to whatever you specify in the -ThrottleLimit parameter, with a default of 5.

Since PowerShell runs these branches as background jobs, you need to use either the $USING syntax to bring outside variables into this background job (PowerShell brings $_ by default) or provide the variables in the -ArgumentList parameter. For example:

PS > $greeting = "World"
PS > 1..5 | ForEach-Object -parallel { "Hello $greeting" }
Hello
Hello
Hello
Hello
Hello

PS > 1..5 | ForEach-Object -parallel { "Hello $USING:greeting" }
Hello World
Hello World
Hello World
Hello World
Hello World

PowerShell runs these background jobs in your main PowerShell process, so you can act on input as live instances:

$processes = 1..10 | ForEach-Object { Start-Process notepad -PassThru }
$processes | ForEach-Object -parallel { $_.Kill() }

If you need the branches of your parallel loop to communicate back to your main shell, the recommended approach is to accomplish this through script block output and then have your main shell process the results. It’s tempting to do this with live objects, but beware that the path is treacherous and difficult. Let’s take a simple example—running a parallel operation to increment a counter.

It might initially seem like you should use:

$counter = 0
1..10 | ForEach-Object -parallel {
    $myCounter = $USING:counter
    $myCounter = $myCounter + 1
}

However, when you type $counter = $counter + 1 in PowerShell, PowerShell updates the $counter variable in the current scope. If you want to change an object from a background job, you need to do so by setting a property on a live object rather than trying to replace the object. Fortunately, PowerShell has a type called [ref] for this kind of scenario:

$counter = [ref] 0
1..10 | ForEach-Object -parallel {
    $myCounter = $USING:counter
    $myCounter.Value = $myCounter.Value + 1
}

Initially, this seems to work:

PS > $counter

Value
-----
   10

Now that we’re proud of ourselves, let’s really do this in parallel:

$counter = [ref] 0
1..10000 | ForEach-Object -throttlelimit 100 -parallel {
    $myCounter = $USING:counter
    $myCounter.Value = $myCounter.Value + 1
}

PS > $counter

Value
-----
 9992

Oops! Because we’ve done this with massive parallelism, $myCounter.Value can change at any time during the parts of the pipeline where PowerShell runs $myCounter.Value = $myCounter.Value + 1. This is called a race condition, and is common to any language that lets code from multiple simultaneous blocks of code run at the same time. To get rid of the weird intermediate states, we have to use the Interlocked Increment class from the .Net Framework:

$counter = [ref] 0
1..10000 | ForEach-Object -throttlelimit 100 -parallel {
    $myCounter = $USING:counter
    $null = [Threading.Interlocked]::Increment($myCounter)
}

Which correctly gives us:

PS > $counter

Value
-----
10000

These problems are gnarly, and bite even professional programmers with regularity. The best practice to handle this class of issue is to avoid the area altogether by not processing or operating on shared state.

See Also

Recipe 4.4, “Repeat Operations with Loops”

Recipe 29.5, “Invoke a Command on a Remote Computer”