The Power of Pipelines (with Filters)

AlaskanPipeline

Upon revisiting the Wrap-Function and Wrap-History functions, some questions on these arose. Let me try to address at least one of those questions.

“Your Wrap-Function definition looks like a function, but is called a filter. What does that mean? Also, how does it actually process a file or bunch of commands and turn them into a function?”

 

That’s two questions. Before delving into answers, let’s first go back and look at another definition of Wrap-Function which we now call Wrap-FunctionClassic. It’s not called Classic because it actually is classy or anything like that but because it’s older. The distinction between the two versions which have similar functionality can be educational. Let’s take a look at the classic version first before dissecting the newer one and actually answering the questions.

function global:Wrap-FunctionClassic { 
    param( $file=”.script.ps1”, 
        $fun=”Wrapped” , $scope=””) 
    $_lines = Get-Content $file 
    for( $i = 0; $i -lt $_lines.count; ++$i ){
        $_lines[$i] = “`t” + $_lines[$i] 
    } 
    Write-Output “function $scope$fun {“ 
    $_lines 
    Write-Output “}” 
}

 

Note that this function could be significantly optimized, yet as it is, it serves as a point of reference as a classic programming approach to PowerShell scripting. It gets the contents of a named file, adds a tab at the beginning of each line, and then outputs the set of lines as a function. Although it reads from a file, it does not save the output to a file yet merely uses the standard output channel for the resultant file-turned-function. In summary, Wrap-FunctionClassic is a function, and it reads from a file, but doesn’t write to a file.

Now, let’s take a look at a variation which is defined as a filter rather than a function. It’s called Wrap-Function. Although this version could be optimized as well (e.g. “Write-Output” doesn’t need to be explicit), look at the simplicity compared with the Wrap-FunctionClassic version.

filter global:Wrap-Function { 
    param( $fun=”Wrapped” ) 
    # include global: or other scope 
    # in name as appropriate 
    BEGIN{ Write-Output “function $fun {“ }
    PROCESS{ “`t” + $_ } 
    END{ Write-Output “}” } 
}

 

Filters are like functions in that they may be invoked with parameters. They are called in the same way. Like awk scripts of UNIX heritage, the body of the filter can have more than one code block. This differentiates filters from functions. In fact, even if the keyword “function” had been used instead of “filter” in the definition, the presence of BEGIN, PROCESS, and/or END code blocks within a function causes it to behave as a filter. In other words, including a code block named BEGIN, PROCESS, or END in a function converts it to a filter.

But just what is the distinction between an ordinary function and a filter? A function runs through its one code block from top to bottom with whatever flow control it contains – once for all input. When a filter is invoked, it’s BEGIN block (if any) is run first. Then the PROCESS block is run once for each input object. Finally, the END block runs. That may sound simple, but it has immense power.

Consider the following.

function x(){ “one”; $_; “two” } 
filter y(){ “one”; $_; “two” }

 

We could call the function x and the filter y by passing an object down the pipeline to either one.

“three” | x 
“three” | y

 

What’s the output of the function x in this example? one, two. That’s it. No three. How about the filter? One, three, two. If that makes sense, try sending more than one object to the filter (e.g. “three”,”four” | y).

filter z(){ 
    BEGIN{ “zero” } 
    PROCESS{ “one”; $_; “two” } 
    END{ “infinity” } 
}

 

 

“three”,”four” | z

The filter z includes separate BEGIN, PROCESS, and END blocks. With the filter y, the body of y was effectively assumed to be the PROCESS block. I’d recommend becoming familiar with the behavior of simple examples like these to help understand how filters process the objects input to them.

Now that we’ve looked at a few basic filters, take another look at the Wrap-Function filter. It’s really quite straight forward once you know how filters work. The BEGIN block emits the beginning of a function declaration. The PROCESS block, which is executed for each object/line of input, emits a tab character followed by the original line. The END block emits the closing of the function definition. That’s it.

Using Wrap-Function to wrap a script or history is fairly straight forward. Let’s first take a look at converting the contents of a script file into a function with the body of that original file and saving that resultant function into another file. Then we’ll revisit the wrapping of history into a function and saving that to a file.

get-content original.ps1 | Wrap-Function | 
out-file result.ps1

 

This pipeline simply uses Get-Content to obtain all of the lines contained in the original.ps1 file. Those lines are then sent to Wrap-Function, and the resultant function is saved by Out-File into the result.ps1 script. Many variations could be made, such as using other cmdlets or functions to get the body of the code to convert to a function.

Although this pipeline is fairly easy to type, it could either be abbreviated by using various aliases and shorthand notations.

cat original.ps1 | Wrap-Function >result.ps1

 

Of course, we could define this sequence as a function for convenient use.

function Convert-ScriptToFunction( $file, $result ){ 
    Get-Content $file | Wrap-Function | Out-File $result 
}

 

And then merely invoke the function when we need to convert a script file into a function-wrapped body stored in another script file.

Convert-ScriptToFunction original.ps1 result.ps1

 

Another use of a pipeline with the Wrap-Function filter has been shown in the Wrap-History examples I’ve posted previously to this blog. The guts of this is essentially as follows.

Get-History -Count 32 `
  | ForEach-Object { $_.CommandLine } `
  | Wrap-Function RecentCommands `
  | Out-File recent.ps1

 

While the possibilities for filters and pipelines are seemingly endless, hopefully these few short examples have illustrated a tidbit of the power of pipelines and filters.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.