Revisiting Remote Administration – The Event Log Collector

Think globally, act locally may be a great mantra for many aspects of computing, but wearing sweaters or a parka in the summer just to cozy up with all your servers in the cooly air conditioned data center hasn’t been considered cool in at least a decade. Remoting is typically far preferred even for most server administrators with just one server room.

Most devices, routers, switches, and of course workstation and server operating systems support one or more forms of remote management. Windows Server 2008 (both releases) support technologies such as Remote Desktop and Web Services Management (WS-Management). Microsoft often refers to their implementation of the industry standard WS-Management as WSMan or Windows Remote Management (WinRM). Microsoft actually started supporting the foundations for such web services back in Windows Server 2003, with IIS 6.0 supporting the Simple Object Access Protocol (SOAP) which is the foundation for a family of Web Services standards.

While most Windows system administrators likely do not care about the details of SOAP itself, I regularly get questions about the features and services which are dependent upon it. Like Remote Procedure Call (RPC) and Windows Management Instrumentation (WMI), peoples’ interests often lay in what can be done with the technology. One particular feature which several students were curious about in a class this week was Event Subscriptions.

Internet standards for collection, storage, and analysis of event log information, derived from the syslog subsystem of UNIX and sendmail heritage, have existed for several years, in the form RFC 3164 and its successor RFC 5424. Syslog’s architecture supports originators, relays, and collectors. Similarly, Microsoft’s Event Collector Service supports originator and collector roles.

Both Windows Server 2008 and 2008 R2 support using the Event Collector service in source initiated and collector initiated subscriptions. An event collector subscription is the configuration of an agreement between two Windows systems for one system to send specific event log entries to be received by the other. Although the configuration of such subscriptions is manual, once configured, these subscriptions are persistent. Yes, if you’re creating event collector subscriptions for hundreds or thousands of computers, automation of these relationships via tools such as Windows PowerShell could be essential. For starters, we shall just look at the basic configuration.

The configuration of the Windows Event Collector (WEC) and WinRM on which it depends could be done with great care for particular details, or there is the quick configuration (QC), a.k.a. quickconfig. Consider the following two commands which could be used on both computers prior to setting up one or more subscriptions.

wecutil qc
winrm qc

What is also needed beyond the quick configuration of the WEC and WinRM is to allow the two (or more) systems to communicate as trusted hosts via WinRM, and then to configure the WEC with one or more subscriptions.

winrm set winrm/config/client @{TrustedHosts=”localhost”}

This command simply configures WinRM to trust itself. While that is useful as a basic test of the functionality on a test machine, in a test network or production system, the names or addresses of the trusted systems would be used instead of merely localhost.

wecutil cs mySubscription.xml

This command creates an event collector subscription (cs = “create subscription”) as defined in the XML file mySubscription.xml. The details of this configuration file are beyond the scope of this article, however we’ll likely delve into such details soon!

Grokking grokking, and well… Grokking DHCP Server Logs

Some people didn’t exactly grok my previous post, thus I would like to repost it again, this time prefaced with a bit of an explanation of what I meant by “grok.”

One of the problems of effective automation and scripting the double-edged sword of (a) achieving a solid enough understanding of what you really want to do, and (b) codifying it’s expression so that your computer also understands the problem. In computer vernacular, the verb grok, or grokking, as coined by Robert A. Heinlein in Stranger in a Strange Land, (see <http://en.wikipedia.org/wiki/Grok/>) means to assimilate a technical concept to the degree to which you become one with the topic at hand. In essence, grokking means achieving a state of fluency and mastery with processing particular concepts or data.

Recently, I wrote a note about how to extract the DHCP activity log table out of a DHCP Server log file on Windows Server using PowerShell. That example used static file names and expected that you were running the commands or script at the proper location, assumed to be within the DHCP folder (typically C:WindowsSystem32DHCP). Now let’s take a look at how to grab one or more of the many log files there. Consider the following PowerShell function:

function Get-DHCPLogFile( [string]$Day = “”, [int]$IPVersion = 4, [string]$LogName = “” ){
$LogFolder = “$env:SystemRootSystem32DHCP”
if( $LogName -eq “” ){
if( $Day -eq “” ){
$Day = [string](get-date).dayOfWeek.toString()[0..2] -replace ” “,””
}
switch( $IPVersion ){
4 { $base = “” }
6 { $base = “V6” }
default { $base = $IPVersion }
}
$LogName = “Dhcp{0}SrvLog-{1}.log” -f $base,$Day
}
# $LogName could be ‘*’ or ‘*.log’ for wildcard, or $Day could be ‘*’
Get-ChildItem $LogFolder$LogName
}

Simply invoking this function with no parameters would determine the current day, such as “Thu” for Thursday, and fetch today’s log. The above function doesn’t perform any log parsing or processing, but merely obtains references to the desired file or files. That’s one part of the equation. Combine with that the extraction of the DHCP activity log from a file described in my previous article, and we can start to actually process the log information.

function Extract-DHCPLogActivity( $file ){
$log = Get-Content $file
$x = ($log | Select-String “ID,Date,Time”).LineNumber-1
$y = $log.count-1
$dhcpLog = ($log[$x..$y] | ConvertFrom-CSV) # or via file for PowerShell v1.0
return $dhcpLog
}

Connecting both pieces, we have:

function Get-DHCPActivity( [string]$Day = “”, [int]$IPVersion = 4, [string]$LogName = “” ){
Get-DHCPLogFile $Day $IPVersion $LogName | %{Extract-DHCPLogActivity $_}
}

If we just want to see all of the activity from the current log file, we could invoke the Get-DHCPActivity function with no parameters. Based on the column headings defined in the CSV activity log part of the DHCP log files, the activity information from each line in the file has the following properties (attributes, fields), shown here with example values.

ID : 30
Date : 07/15/10
Time : 11:25:00
Description : DNS Update Request
IP Address : 10.10.0.65
Host Name : NYC-CL1.WoodgroveBank.com
MAC Address :
User Name :
TransactionID : 0
QResult : 6
ProbationTime :
CorrelationID. :

Now that we have the data in object form in PowerShell, we can use any of several lovely cmdlets for searching, selecting, and sorting that data to build a report, generate some statistics, create a history for a particular network or machine, which hopefully helps meet some DHCP troubleshooting or analysis goal. For example, if we want to find activity for a particular machine and a useful subset of that information, we could use the Where-Object (aliased as ‘?’) and Format-Table (aliased as ‘FT’) as follows:

get-dhcpactivity | ?{ _.’host name’ -match “NYC-CL1” } | FT id,time,’IP Address’,Description -auto

Naturally, if this is the sort of thing you would often do, you could write another little function such as:

function Get-DHCPClientActivity( $machine = $(hostname) ){
get-dhcpactivity | ?{ _.’host name’ -match $machine } | FT id,time,’IP Address’,Description -auto
}

Well, technically, using the hostname as a default for the machine name on a function which fetches files locally from a server wouldn’t be incredibly useful, however the main point is that you can easily customize such DHCP functions to meet your actual needs. If those actual needs include being able to run these tools remotely from a workstation against several servers and look for activity related to a collection of machines or about a certain subnet or scope, these functions could be augmented or extended to handle such needs.

Grokking DHCP Server Logs

Recently, I wrote a note about how to extract the DHCP activity log table out of a DHCP Server log file on Windows Server using PowerShell. That example used static file names and expected that you were running the commands or script at the proper location, assumed to be within the DHCP folder (typically C:WindowsSystem32DHCP). Now let’s take a look at how to grab one or more of the many log files there. Consider the following PowerShell function:

function Get-DHCPLogFile( [string]$Day = “”, [int]$IPVersion = 4, [string]$LogName = “” ){
$LogFolder = “$env:SystemRootSystem32DHCP”
if( $LogName -eq “” ){
if( $Day -eq “” ){
$Day = [string](get-date).dayOfWeek.toString()[0..2] -replace ” “,””
}
switch( $IPVersion ){
4 { $base = “” }
6 { $base = “V6” }
default { $base = $IPVersion }
}
$LogName = “Dhcp{0}SrvLog-{1}.log” -f $base,$Day
}
# $LogName could be ‘*’ or ‘*.log’ for wildcard, or $Day could be ‘*’
Get-ChildItem $LogFolder$LogName
}

Simply invoking this function with no parameters would determine the current day, such as “Thu” for Thursday, and fetch today’s log. The above function doesn’t perform any log parsing or processing, but merely obtains references to the desired file or files. That’s one part of the equation. Combine with that the extraction of the DHCP activity log from a file described in my previous article, and we can start to actually process the log information.

function Extract-DHCPLogActivity( $file ){
$log = Get-Content $file
$x = ($log | Select-String “ID,Date,Time”).LineNumber-1
$y = $log.count-1
$dhcpLog = ($log[$x..$y] | ConvertFrom-CSV) # or via file for PowerShell v1.0
return $dhcpLog
}

Connecting both pieces, we have:

function Get-DHCPActivity( [string]$Day = “”, [int]$IPVersion = 4, [string]$LogName = “” ){
Get-DHCPLogFile $Day $IPVersion $LogName | %{Extract-DHCPLogActivity $_}
}

If we just want to see all of the activity from the current log file, we could invoke the Get-DHCPActivity function with no parameters. Based on the column headings defined in the CSV activity log part of the DHCP log files, the activity information from each line in the file has the following properties (attributes, fields), shown here with example values.

ID : 30
Date : 07/15/10
Time : 11:25:00
Description : DNS Update Request
IP Address : 10.10.0.65
Host Name : NYC-CL1.WoodgroveBank.com
MAC Address :
User Name :
TransactionID : 0
QResult : 6
ProbationTime :
CorrelationID. :

Now that we have the data in object form in PowerShell, we can use any of several lovely cmdlets for searching, selecting, and sorting that data to build a report, generate some statistics, create a history for a particular network or machine, which hopefully helps meet some DHCP troubleshooting or analysis goal. For example, if we want to find activity for a particular machine and a useful subset of that information, we could use the Where-Object (aliased as ‘?’) and Format-Table (aliased as ‘FT’) as follows:

get-dhcpactivity | ?{ _.’host name’ -match “NYC-CL1” } | FT id,time,’IP Address’,Description -auto

Naturally, if this is the sort of thing you would often do, you could write another little function such as:

function Get-DHCPClientActivity( $machine = $(hostname) ){
get-dhcpactivity | ?{ _.’host name’ -match $machine } | FT id,time,’IP Address’,Description -auto
}

Well, technically, using the hostname as a default for the machine name on a function which fetches files locally from a server wouldn’t be incredibly useful, however the main point is that you can easily customize such DHCP functions to meet your actual needs. If those actual needs include being able to run these tools remotely from a workstation against several servers and look for activity related to a collection of machines or about a certain subnet or scope, these functions could be augmented or extended to handle such needs.

Powering Through DHCP Server Logs

Windows Server has many events logs, trace logs, and can support logging to performance, IIS, and other data to databases. In some cases, the activity logs for some services are in plain old fashioned – yes, you guessed it – text files. While there are many free and pricey tools out in the wild for managing various aspects of Windows services, many people still ask me how they can work with certain kinds of logs and actually use the data.

This week in a Windows Network Infrastructure class, we were looking at some log files for the Dynamic Host Configuration Protocol (DHCP), which are text-oriented log files. While there are often .NET classes, WMI classes, and other such application programming interfaces (APIs) for managing Windows subsystem, many systems administrators could probably just learn to read the raw data in the files far more easily than learning some of the APIs for managing a service such as DHCP. Instead, let’s look at a third option – an alternative to both learning the API, and learning the details of the file format.

For some log files, we can simply use PowerShell commands such as:

Import-CSV EasyLogFile.csv

and then quickly manipulate the data to suit our needs.

DHCP logs happen to be a bit more complicated. The files start with a line with a couple of tabs and the title “Microsoft DHCP Service Activity Log.” After a couple of blank lines, there is a table of Event ID values and their meanings. Each line in the body of the table has an numeric event ID followed by a tab character and then a string representing the meaning of the particular event id. Then, after another blank line is a single line which defines some values for the QResult and ProbationTime attributes of the subsequent log entries. Indeed, following yet another blank line is a comma-separated header line which identifies the fields of each event, followed by each of the comma-separated events themselves.

Parsing all of this is relatively easy, and we could take a look at writing a PowerShell script to absorb all of it, however we’ll start simple. Let’s first just target the raw events. While there are probably a bazillion ways to process this data, consider the following technique. How do we find the actual events in the file? Select-String to the rescue! If you heard of the grep command from the UNIX world, you could think of PowerShell’s Select-String cmdlet as a .NET oriented, PowerShell oriented, regular expression matcher that, while not exactly grep, is still pretty powerful. To find the line with the headers for the event table in a DHCP log file, simply use:

Select-String “ID,Date,Time” DhcpSrvLog-Wed.log

Wonderful, so Select-String can show us the line in the file that we already knew the first three words of. How is this useful? Not very. However, knowing that the output of Select-String is an object rather than just text, and extracting the LineNumber property of the search results might help.

$x = (select-string “ID,Date,Time” DhcpSrvLog-Wed.log).LineNumber-1
$log = get-content DhcpSrvLog-Wed.log
$y = $log.count-1
$log[$x..$y]

The above PowerShell code snippet extracts the events from a DHCP activity log as follows. First, we find the line number of the line with the “ID,Date,Time” string, subtract one from it so it can be used as an index later (line numbers are one-based as reported by Select-String, yet integer-indexed arrays are zero-based) and save this value in a variable $x. Next, we get the contents of the whole file and keep it in a variable $log. This isn’t necessary, as there are many other techniques to extract the proper lines from the file, however our chosen technique in the next couple lines is simplified by doing this. Thirdly, we take the number of lines in the log, subtract one, and save the difference in the variable $y. Finally, we reference the activity log table directly with $log[$x..$y] which uses the substring operator .. (dot-dot) to extract the valid lines. This technique could be used when the table is in the middle of a file, and not just the remainder of a file after a certain point like with the DHCP activity logs.

Tying this back in with Import-CSV, unfortunately, the Import-CSV cmdlet does not support bringing in input from a pipeline or even a simple variable. Luckily there are two approaches we can use instead. The method which is PowerShell version 1.0 compatible is to redirect the extracted information ($log[$x..$y] in this example) to a file and use Import-CSV against the file. Of course, a unique pseudo-random file name should be used, however here’s the basic idea with a fixed name:

$log[$x..$y] >temp.csv
$dhcpLog = Import-CSV temp.csv

Another approach is possible with PowerShell version 2.0, which is to use the “new” version 2.0 ConvertFrom-CSV cmdlet, which accepts an -InputObject parameter or pipeline input.

$dhcpLog = ($log[$x..$y] | ConvertFrom-CSV)

Certainly, this sort of extraction and conversion technique can be used with IIS log files, DHCP activity logs, and other files which may or may not contain pure comma-separated-values (CSV). In the next article, we’ll look at another aspect of working with DHCP activity logs.