Powershell – ForEach-Parallel

Background

In PowerShell I often find the need to speed things up when having to do querying lots of similar things – usually getting the same stuff from multiple machines. So I ventured out onto the Interwebs and over time slap together a PowerShell module that does parallel processing. I’m not claiming this to be purely my work – I only added and refined it a bit for my needs.

This example works with ‘older’ versions of PowerShell (think version 4/5 and perhaps 3). The latest versions of PowerShell (afaik version 7) now has something like this built in.

Implementation

Using any sort of parallel coding requires a different approach than normal procedural (Start at the top and run code line by line until you get to the end). Basically it is multi-threading but PowerShell ‘out of the box’ has nothing like it. To implement multi threading you need to use ‘runspaces’ which effectively means you are running multiple instances of PowerShell.

The ForEach-Parallel function takes a scriptblock as input. Optionally you can also specify the number of threads used to do the parallel processing.

#ForEach-Parallel.psm1
function ForEach-Parallel {
     param(
         [Parameter(Mandatory=$true,position=0)]
         [System.Management.Automation.ScriptBlock] $ScriptBlock,
         [Parameter(Mandatory=$true,ValueFromPipeline=$true)]
         [PSObject]$InputObject,
         [Parameter(Mandatory=$false)]
         [int]$MaxThreads=5
     )
     BEGIN {
         $iss = [system.management.automation.runspaces.initialsessionstate]::CreateDefault()
         $pool = [Runspacefactory]::CreateRunspacePool(1, $maxthreads, $iss, $host)
         $pool.open()
         $threads = @()
         $ScriptBlock = $ExecutionContext.InvokeCommand.NewScriptBlock("param($_)r`n" + $Scriptblock.ToString())
     }
     PROCESS {
         $powershell = ::Create().addscript($scriptblock).addargument($InputObject)
         $powershell.runspacepool=$pool
         $threads+= @{
             instance = $powershell
             handle = $powershell.begininvoke()
         }
     }
     END {
         $notdone = $true
         while ($notdone) {
             $notdone = $false
             for ($i=0; $i -lt $threads.count; $i++) {
                 $thread = $threads[$i]
                 if ($thread) {
                     if ($thread.handle.iscompleted) {
                         $thread.instance.endinvoke($thread.handle)
                         $thread.instance.dispose()
                         $threads[$i] = $null
                     }
                     else {
                         $notdone = $true
                     }
                 }
             }
         }
     }
 }
 export-modulemember -function ForEach-Parallel 

To make use of this ‘module’ save it to a file named ForEach-Parallel.psm1 and place it in C:\Users\{your user account}\Documents\WindowsPowerShell\modules\ForEach-Parallel

Example

#Example
$computers = 'computer1','computer2','computer3'
 $computers | ForEach-Parallel -MaxThreads 20 -ScriptBlock {
     $computerToPing = $_
     [int]$ResponseTime = 0
     [string]$ResolvedName = ''
     $StepPing = (Test-Connection -ComputerName $computerToPing -Count 1 -ErrorAction SilentlyContinue)
   if ($StepPing -ne $null){
      $ResponseTime = $StepPing.ResponseTime 
   } 
   New-Object psobject -Property @{     
    Name               = $computerToPing     
    ResponseTime       = $ResponseTime                  
   }
 }

Summary

There are a few things to watch out for – like any multi-threading coding. While the code is running there is no indication how far each ‘thread’ is – you have to wait until all of them are done. Threads are independent so they cannot reference each other or even variables in the calling thread.

So.. happy multi-threading.

SSH on Windows 10

I remember that they (Microsoft) mentioned the bash shell on Windows 10 a while ago and other Linux/Ubuntu integration stuff but at the time it seemed to be just a gimmick.

But since I’m looking into Nagios and some Windows monitoring I now the need to daily connect to my Nagios server making configuration changes. So far I’ve been using good old Putty/WinSCP and it works well. Then I started looking at a way to generate configurations files for some Windows machines and upload them automatically via PowerShell (or it could end up being another tool created in C# eventually). This is how I stumble across the Windows 10 built in SSH functionality – it’s been there for at least a year! (since build 1803).

That means you can simply use the ssh command in a Windows command shell (or PowerShell) and send commands to your Linux server. To start it simple open a command prompt (cmd.exe) and run ssh:

As can be seen it supports all the basics of any ssh client.

So that cover the basics. The real power comes when you start mixing it with PowerShell and proper scripting and/or batch files.

One very interesting (and incredibly useful) example is something Scott Hanselman posted – how to automatically log into a remote linux machine – see this.

My quick Nagios client install guide

Due to changing winds at my day job I’ve been tasked to start looking into Nagios for monitoring a Windows environment. I’ve taken this as a learning opportunity and also to compare monitoring systems (aka SCOM). This blog post is not about comparing Nagios with SCOM or any other monitoring system but rather a summary of the process of installing Nagios client (NSClient++ on Windows and nrpe on Linux). Installing the server portion of Nagios is an affair for another day – as it is a rather hairy story only for the brave…

Installing a Linux client

Since I have a mixed test environment (at home) I also made use of Nagios to monitor them too. I mostly use Ubuntu/Debian based distros (like Ubuntu and Mint) so this guide will only cover them.
1. First make sure no other updates are outstanding:
sudo apt-get update
2. Then download/install the nrpe package (which includes both server and plug-in parts).
sudo apt-get install nagios-nrpe-server nagios-plugins
3. Once installed you need to configure the client so it can act as a monitoring end-point for the Nagios server. The bare minimum you have to specify is to allow the server to connect to the client. To do this you need to edit the nrpe.cfg file. By default it should be installed at /etc/nagios
sudo gedit /etc/nagios/nrpe.cfg
then search for the line that starts with allowed_hosts= and set it to the ip address for the Nagios server.
allowed_hosts=<ip address of nagios server>
4. Lastly you need to restart the agent:
sudo /etc/init.d/nagios-nrpe-server restart
Once all is done you (or the Nagios admin) need to configure the server portion. This will be covered later as it is basically the same for Linux and/or Windows clients.

Installing a Windows client

For Windows monitoring I’m using the NSClient++ agent. It is available for most Windows versions – from Windows XP to Windows 10 as well as Windows server.
1. Download the appropriate MSI version for your version of Windows from
https://nsclient.org/download/
2. Install the MSI. You can do it manually – double-click the MSI or using the command line
3. I tend to choose the the Complete option to choose
Enable common checkplugins
Enable nsclient server (check_nt)
Enable NRPE server (check_nrpe)’ with Safe mode (default)
Enable NSCA client (optional)
4. If for some reason you want to use the command line you can use something like this (which is for a silent install) – this is for the current latest x64 version 0.5.2.35-x64.
msiexec /qn /l* NSCPInstall.log /i NSCP-0.5.2.35-x64.msi /norestart CONF_CAN_CHANGE=1 CONF_NSCLIENT=1 CONF_NRPE=1 CONF_NSCA=1 ADDLOCAL=ALL

Configuring the server

This quick guide is not intended as complete reference for configuring Nagios so I’ll only be giving a summary here. By default Nagios config is found at /usr/local/nagios/etc (for version 5.x afaik).
I tend to put host configs (aka clients) in a separate directory ‘/usr/local/nagios/etc/servers’ – it just makes it easier to manage. For this quick guide I simply show how to add the ‘host’ entry and not how to configure services and any other things.
1. Create a new file for the the client/agent. I use the name ‘hostname.cfg’
2. Edit the file
example config here for a linux host
define host {
use linux-server
host_name <hostname>
alias <hostname>
address <x.x.x.x or dns name>
}
Windows host
define host {
use windows-server
host_name <hostname>
alias <hostname>
address <x.x.x.x or dns name>
}
3. After all config editing has been done I always check that changed config is okay (for typos and so on…) using the following command on the Nagios server
/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg
4. If all checks out you can restart the Nagios server
systemctl restart nagios.service

Summary
For now this is the quick guide. I’ll update it from time to time if I find more things that can be included.

QuickMon 5.1.2

A new version of QuickMon is available.

Summary of changes
Add option to ‘not check’ URL on exit(ok button) when editing config to Ping, Performance counter and file system collectors.

URL: https://github.com/RudolfHenning/QuickMon/releases/tag/v5.1.2

Update: found a bug when saving templates. Fixed now.

QuickMon 5.1.1

Hi QuickMon’ers

Just a heads-up that version 5.1.1. has been released. Also, I forgot to post about version 5.1 which is the first version that made a major change in the .Net dependency. Since .Net 4.5.2 is the oldest version of .Net still supported by Microsoft the whole tool has been recompiled to use .Net 4.5.2. It (the .Net version) does not provide any new functionality as such.

Summary of changes since 5.1

————–
Version 5.1.0
**************
First version using .Net 4.5.2 (which is the oldest supported .Net at this time)
Added Jumplist shortcuts – Shortcuts available on right-click on the Windows Taskbar.

————–
Version 5.1.1
**************
Fix to Web Service collector to display actual value used for testing alert

QuickMon 5.0.10

Just a quick note to mention QuickMon reached version 5.0.10

More advanced Contain string function

Just thought I share this useful function I’ve been using recently to enhance some of my applications to make text/string searching more powerful.

If you want to allow things like ‘and’, ‘or’ and ‘not’ in a text filter this could be useful to you. Additionally I also added keywords for things like ‘matchexactly’, ‘startswith’ and ‘endswith’.

public static class StringCompareUtils
{
  public static bool ContainEx(string source, string findText, bool caseSensitive = false)
  {
    bool found = false;
    string lowerCasefindText = findText.ToLower();
    if (findText.Trim().Length == 0)
    {
      found = true;
    }
    else if (lowerCasefindText.Contains(" and "))
    {
      string[] parts = findText.Split(new string[] { " and ", " AND " }, StringSplitOptions.RemoveEmptyEntries);
      found = true;
      foreach (string part in parts)
      {
        if (!ContainEx(source, part))
        {
          found = false;
          break;
        }
      }
    }
    else if (lowerCasefindText.Contains(" or "))
    {
      string[] parts = findText.Split(new string[] { " or ", " OR " }, StringSplitOptions.RemoveEmptyEntries);
      foreach (string part in parts)
      {
        if (ContainEx(source, part))
        {
          found = true;
          break;
        }
      }
    }
    else
    {
      if (lowerCasefindText.TrimStart().StartsWith("not "))
      {
        found = !source.ContainEx2(findText.TrimStart().Substring(4), caseSensitive);
      }
      else
        found = source.ContainEx2(findText, caseSensitive);
    }
    return found;
  }
  private static bool ContainEx2(this string text, string findText, bool caseSensitive = false)
  {
    string matchText = caseSensitive ? text : text.ToLower();
    string matchFindText = caseSensitive ? findText : findText.ToLower();
    if (matchFindText.ToLower().StartsWith("matchexactly "))
    {
      return text == findText.Substring("matchexactly ".Length);
    }
    else if (findText.ToLower().StartsWith("startswith "))
    {
      return text.StartsWith(findText.Substring("startswith ".Length));
    }
    else if (findText.ToLower().StartsWith("endswith "))
    {
      return text.EndsWith(findText.Substring("endswith ".Length));
    }
    else
    {
      return text.Contains(findText);
    }
  }
}

The default behavior is like the ‘like’ operator (text contains search filter). This allow you to find a string like ‘are’ in ‘You are welcome’. Using this function you can specify a filter like ‘startswith You’ or ‘endswith welcome’.

QuickMon 5.0.4

Another update to Quickmon.

Not going to repeat all the release notes here but basically this release improves on using config variables while editing the config.

Link: QuickMon 5.0.4

Happy monitoring

 

QuickMon 5.0.2

Another small update.

  • There is now an option to have the quick recent monitor pack list sorted by display name or not.
  • Details about which notifiers fired are now visible in the collector detail view.
  • When setting up ‘Success, Warning or Error test conditions for some collectors a context menu provide some shortcut values.

Version 5.0.2

QuickMon 5.0.1

I’ve added a small but useful piece of functionality (probably long overdue) to the File/Directory collector of QuickMon. You can now specify the last X number of lines of a text file to check for the specified text to search for.

This is specifically intended for checking Text log files where lines of text are appended the whole time.

Download from Github