Tag Archives: monitoring

SCOM agent Info script

When you have a case where you start integrating your in house SCOM 2012 R2 environment with VMs that are hosted in Azure and ‘other’ people keep on screwing up your SCOM agents with the OMS version you need a way to track what version is actually installed on a machine – plus whether the original SCOM 2012 R2 config is still in tact – which is  not the case since the upgrade process from SCOM 2012 R2 agent to the OMS version totally screws up the SCOM 2012 R2 config (like in F*ing deletes it !!!)

Therefore I had to create a little Powershell script to help check suspected machines if they have been affected by this issue. From the SCOM console side it just appears the Agent is ‘dead’ even though if you check manually it is still running (but of course the config is gone).

if ($args.count -ne 0) {
    $ComputerName = $args[0]    
}
else {
    $ComputerName = Read-Host -Prompt 'Computer Name'
}

$pingTest = (Test-Connection $ComputerName -Count 1 -TTL 255 -ErrorAction SilentlyContinue) 
if ($pingTest -ne $null -and $pingTest.IPV4Address -ne $null){
    write-host "IP Address : " $pingTest.IPV4Address.IPAddressToString -foregroundcolor "green"
    $serviceDetails = Get-Service -ComputerName $ComputerName | where DisplayName -Like 'Microsoft Monitoring Agent'
    if ($serviceDetails -ne $null ) {
        write-host "Health Service state: " $serviceDetails.Status.ToString() -foregroundcolor "green"

        $Reg = [Microsoft.Win32.RegistryKey]::OpenRemoteBaseKey('LocalMachine', $ComputerName)
        $AgentGroupKeys = $Reg.OpenSubKey("SYSTEM\CurrentControlSet\Services\HealthService\Parameters\Management Groups")
        write-host "Groups: " $AgentGroupKeys.SubKeyCount.ToString() -foregroundcolor "green"
        $AgentGroupKeys.GetSubKeyNames() | % {
            if ($_ -like "AOI*") {
                write-host " OMS" -foregroundcolor "yellow"
            } else {
                write-host " $_" -foregroundcolor "green"
            }            
        }        
        
        $VersionNo = "NOT found!"
        $RegKeyProducts = $Reg.OpenSubKey("SOFTWARE\Classes\Installer\Products")
        $RegKeyProducts.GetSubKeyNames() | % {
            $RegKeyProduct = $Reg.OpenSubKey("SOFTWARE\Classes\Installer\Products\" + $_)
            if ($RegKeyProduct.GetValue('ProductName') -eq "Microsoft Monitoring Agent") {
                $VersionNo = $RegKeyProduct.GetValue('Version').ToString("X")
            }
        }

        if ($VersionNo -ne "NOT found!") {
            write-host "Agent version" $VersionNo -foregroundcolor "green"
        }
        else {
            write-host "Agent version not found!" -foregroundcolor "red"
        }

    }
    else {
        write-host "Health service NOT found!" -foregroundcolor "red"
    }
}
else {
    write-host "$ComputerName is not pingable!" -foregroundcolor "red"
}

This script first checks whether the machine (VM) is pingable, if it is then whether the HealthService service is installed, if so what state it is in and what version it is. It also lists the management groups the agent reports to including OMS.

Have fun kidz…

QuickMon 4

Just a quick post to announce the first (official) release of QuickMon 4.

There might be some minor bugs that must be ironed out but the basic product is done.

Have fun.

QuickMon 4 Beta

Heads up folks! A public Beta of QuickMon 4 has been released. Please go to QuickMon to download and play with it.

People that has not seen the Alpha yet would notice some major changes compared to version 3.x. The biggest change it that a Collector (now called Collector host) can contain multiple agents of multiple types all tested together to raise one single alert if needed. Of course you can still use it the old way with a collector host containing only a single agent type and have child collector hosts depending on it.

Another area of big change is that viewing the details of a collector’s state, stats, history, alert details and other info gathered are all placed inside a single view (window). This makes it a lot easier to see what is happening with alerts. This detail view is now also ‘Remote host’ aware so it will run and gather details from a (QuickMon 4) remote host as if it is running on ‘that’ machine.

Ok, go download it and have fun… and report back if you have issues or suggestions.

QuickMon HTTP Activation catch

I recently created a test Windows 8 (8.1) VM to just play around and took the opportunity to test the QuickMon install as well. Usually I simply test on my own machine where I know it works already. This is of course not a major problem in itself but it means I’m not testing what will happen to the tool if you install it on a fresh/clean machine.

Surprisingly/luckily just about everything worked first time – not that I expected less (ye ye a bit arrogant I know 😉 ) and I was happy that QuickMon ‘out of the box’ is so good. However…. there is one thing that did not work as expected (ok ok it did not work at all) – The Remote host functionality built into the Windows Service.

This is actually not a QuickMon problem per se – it is a requirement of or dependency on the underlying operating system that the WCF functionality must be available. The remote host functionality of QuickMon uses HTTP Activation which is not installed by default on a new Windows 8 (or probably previous versions) installation.

The ‘fix’ is really simple – simply go to ‘Control Panel’ -> ‘Turn Windows features on or off’ -> .Net Framework 4.5 Advanced Services (or 4.0 if present) -> WCF Services -> check HTTP Activation on. Then do a reboot even though Windows doesn’t ask for it after the install – somehow the functionality does not really work right away. It should then be working provided you checked all the other know issues – like firewalls and stuff.

I’ll see if I can build in some check into the service itself to generate an error (Event log or something) to high-light if this feature is not available on the operating system where it needs to run.

One more thing to add to QuickMon… hehe

QuickMon 3.7

It has been a while since I mentioned any updates of my QuickMon monitoring tool. It has now reached version 3.7 with many new and improved features. There are so many that I can’t remember all of them now but here are some highlights:

  • Context menu for Collectors (and Notifiers) improved
  • View statistics of collector polls
  • Copy and Paste of Collectors or whole branches in the tree view
  • General UI has been simplified (several times)
  • Remote Agents (query resources via another machine)
  • An OleDb query collector has been added (with limitations)
  • Restoration scripts (opposite or corrective scripts)
  • Alert suppression based on # of polls
  • Ability to register the Service (and remote host) plus add firewall exception rule from application itself
  • Some minor fixes to existing collectors

QuickMon3.7

Get it from CodePlex here.

QuickMon 3

QuickMon 3 has been released (3.1 is already out)

Hope you like it and please give feedback. To see some screen shots look at this.

QuickMon 3 Beta

A quick heads-up. I’ve released a Beta version of QuickMon3 on CodePlex. I’ll post a more detail listing of all the features once the final version is relased but here is a quick summary:

1. Introduction of ‘Remote agents’, aka The Windows service acts as a remote host so collectors can be run as if they run on the host machine (where the service runs). It makes use of WCF.
2. Consolidation of existing Ping collectors (ICMP, HTTP and sockets)
3. Simplification of UI (as far as possible)
4. ‘Editing’ configuration can be done straight from the main UI (no separate editing window)
5. ‘Service windows’ have been changed a bit (simplified I hope)
6. It is not 100% compatible with qmconfig files of version 2.0 (will attempt to upgrade them but then QuickMon 2 won’t be able to use them anymore.
7. It can run side-by-side with QuickMon 2 – provided it is installed into a separate directory (default anyway)

QuickMon 3 use the .Net 4.0 framework and was developed using VS2013 Express for Desktop. It has been tested on Windows 7 (x64), Windows 8, 2003, 2008 so far – but it should work on all versions compatible with .Net 4.0.

Go grab it here.

MainWindow Example

MainWindow Example

QuickMon 2.13

Just a quick heads-up for any QuickMon users. A new version was added recently with a couple of small but nice improvements.

Some of the changes:

  • UI client now automatically refresh the config of ‘Detail’ windows (if the particular Collector’s config was edited)
  • The WMI collector was basically rewritten to improve it and add a little WMI query builder interface. This help people that are not experts using WMI syntax or know all the classes/properties that are available.

 

Adding performance counters to your application (C#)

Nothing about this is exactly new or ground breaking but few developers (enterprise) these days add proper monitoring abilities to their apps. We (now that I mostly do administration and management of other peoples apps) that has to support these things don’t come equipped with x-ray eyes or ESP pills so knowing how the the internals of ‘your’ works is an issue…

Classically developers tend to use stuff like log file, event log and so on to ‘see’ what is going on inside their apps. However, these things also pick up a lot of noise so they usually get turn off for production use but that still leaves us with the requirements that we want to know ‘what your app is doing all the time’ – when we want to track it. Ok, to make a long story shorter, there is such a thing like performance counters that can help us to know if your app is performing or not. What they are and what they can do for you you can go read up on MSDN, Google etc.

Now, it appears most developers are scared of these things – or just too lazy to implement them. I’ve created a little helper class that makes it at least a little easier to implement – without all the worries about what happens if things blow up etc. It is written is such a way that if itself ‘breaks’ it should not break the rest of the app (don’t contribute to a bigger problem).

Setting it all up.

Creating a ‘Category’ is the first step in setting up a performance counter. Once it is created you don’t have to recreate it again (unless you go and delete it which would be rather stupid). The following routine do just that.

PerformanceCounterCategory InitializeCategory(string category, CounterCreationData[] creationData, string instance = "")
        {
            PerformanceCounterCategory pcc = null;
            if (!PerformanceCounterCategory.Exists(category))
            {
                if (instance.Length > 0)
                    pcc = PerformanceCounterCategory.Create(category, category, PerformanceCounterCategoryType.MultiInstance, new CounterCreationDataCollection(creationData));
                else
                    pcc = PerformanceCounterCategory.Create(category, category, PerformanceCounterCategoryType.SingleInstance, new CounterCreationDataCollection(creationData));
            }
            else
                pcc = new PerformanceCounterCategory(category);
            return pcc;
        }

All this does is create the category for you if it doesn’t exist yet or else just returns the existing one. All you specify is the name and the definitions of the counters you want to use. Optionally you can specify an ‘instance name’ which is handy if you have multiple instances of the same set of ‘things’ you want to ‘monitor’. The default is a single instance (e.g. like a global instance). Keep in mind that a single category can only have counters of one of two types – either single instance or multi-instance counters. This cannot be changed unless the whole category is deleted and recreated again.

The next step is to create run-time ‘instances’ of the counters.

PerformanceCounter InitializePerfCounter(string categoryName, string counterName, string instanceName = "")
        {
            PerformanceCounter counter = null;
            try
            {
                if (instanceName.Length > 0)
                {
                    if (IsCategoryMultiInstance(categoryName))
                        counter = new PerformanceCounter(categoryName, counterName, instanceName, false);
                    else
                        RaiseWarningMessage(string.Format("Performance category '{0}' is not set up for multiple instances! ({1})", categoryName, counterName));
                }
                else
                {
                    if (!IsCategoryMultiInstance(categoryName))
                        counter = new PerformanceCounter(categoryName, counterName, false);
                    else
                        RaiseWarningMessage(string.Format("Performance category '{0}' is not set up for single instance use! ({1})", categoryName, counterName));
                }
                if (counter != null)
                {
                    counter.BeginInit();
                    counter.RawValue = 0;
                    counter.EndInit();
                }
            }
            catch (Exception ex)
            {
                RaiseErrorMessage(string.Format("Error initializing performance counter '{0}'!\r\n{1}", counterName, ex.ToString()));
            }
            return counter;
        }

There are a couple of sub-routines not listed yet. Please just ignore for now – they will be included in the attached sample code.

Next step is just how to use them – setting the values. I tend to only use two types of counters – a ‘rate of’ and absolute values. The ‘rate of’ type of counters are typically used to track the number of events or sum of something per second while absolute values track the actual value of something as it is ‘now’ at that moment of tracking – e.g. CPU % usage.

void SetCounterValue(PerformanceCounter counter, long value)
        {
            try
            {
                if (counter == null)
                {
                    RaiseWarningMessage("Performance counter not set up or installed!");
                }
                else
                {
                    counter.RawValue = value;
                }
            }
            catch (Exception ex)
            {
                RaiseErrorMessage(string.Format("Increment performance counter error! : {0}\r\n{1}", counter.CounterName, ex.ToString()));
            }
        }
void IncrementCounter(PerformanceCounter counter)
        {
            IncrementCounterBy(counter, 1);
        }
void IncrementCounterBy(PerformanceCounter counter, long incBy)
        {
            try
            {
                if (counter == null)
                {
                    RaiseWarningMessage("Performance counter not set up or installed!");
                }
                else
                {
                    counter.IncrementBy(incBy);
                }
            }
            catch (Exception ex)
            {
                RaiseErrorMessage(string.Format("Increment performance counter error! : {0}\r\n{1}", counter.CounterName, ex.ToString()));
            }
        }

Events
In order to provide all this functionality without ‘breaking’ anything else I make use of ‘events’ e.g. RaiseWarningMessage and RaiseErrorMessage. The calling class/code don’t have to make use of the events but of course that would make troubleshooting a bit harder if needed. The code for this is fairly simple:

        public event MessageRaisedDelegate ErrorRaised;
        internal void RaiseErrorMessage(string message)
        {
            if (ErrorRaised != null)
            {
                ErrorRaised(message);
            }
        }
        public event MessageRaisedDelegate WarningRaised;
        internal void RaiseWarningMessage(string message)
        {
            if (WarningRaised != null)
            {
                WarningRaised(message);
            }
        }

Implementation
It is possible to use the utility class ‘as is’ but I would recommend doing the following to provide ‘cleaner’ code. Inheriting from the class ‘PerfCounterUtilBase’ (not shown in above code) and implementing all the specific performance counter functionality ‘you’ need is a better way to so it. The following is just example code to initialize the performance counter category and counters you want to use:

void InitializePerformanceCounters()
        {
            string lastErr = "Create performance counter category error!: {0}";
            try
            {
                CounterCreationData[] creationData = new CounterCreationData[]
                    {
                        new CounterCreationData("Clicks/Sec", "Clicks per second", PerformanceCounterType.RateOfCountsPerSecond32),
                        new CounterCreationData("Total X value/Sec", "Sum of X values per second", PerformanceCounterType.RateOfCountsPerSecond32),
                        new CounterCreationData("Total Y value/Sec", "Sum of Y values per second", PerformanceCounterType.RateOfCountsPerSecond32),
                        new CounterCreationData("Last X value", "Last X value", PerformanceCounterType.NumberOfItems32),
                        new CounterCreationData("Last Y value", "Last Y value", PerformanceCounterType.NumberOfItems32)
                    };
                InitializeCategory(Category, creationData, Instance);

                lastErr = "Initialize performance counter(s) error!: {0}";
                clicksPerSecond = InitializePerfCounter(Category, "Clicks/Sec", Instance);
                xValuePerSecond = InitializePerfCounter(Category, "Total X value/Sec", Instance);
                yValuePerSecond = InitializePerfCounter(Category, "Total Y value/Sec", Instance);
                lastXValue = InitializePerfCounter(Category, "Last X value", Instance);
                lastYValue = InitializePerfCounter(Category, "Last Y value", Instance);
            }
            catch (Exception ex)
            {
                RaiseErrorMessage(string.Format(lastErr, ex.Message));
            }
        }

Then you can simply have specific methods like these to use in your own code:

public void IncClicks()
        {
            IncrementCounter(clicksPerSecond);
        }
        public void IncXValues(long value)
        {
            IncrementCounterBy(xValuePerSecond, value);
        }
...

In ‘your’ code you simply do this then:

pcu = new PerfCounterExampleImplemetation();
pcu.ErrorRaised += new MessageRaisedDelegate(pcu_ErrorRaised);
pcu.WarningRaised += new MessageRaisedDelegate(pcu_WarningRaised);
pcu.Category = txtCategory.Text;
pcu.Instance = txtInstance.Text;
pcu.InitializePerformanceCounters();
...
void pcu_ErrorRaised(string message)
        {
            MessageBox.Show(message, "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
        }

void SomeMethod()
        {
            long val = GetSomeValue();

            if (pcu != null)
            {
                pcu.IncClicks();
                pcu.IncXValues(val);
                ...
            }
        }

Now go have fun…

Example code: ShowPerfCounterUtils

QuickMon 2.8

I recently got a few suggestions about additional functionality that could be useful in my QuickMon tool – extending Collector service window options and supporting TLS in the SMTP notifier. Both of these modifications have been implemented with some modifications or further enhancements.

Also recently I added a new collector to call SOAP Web services which allows you poll/test web service responses and report success/failure based on specified criteria.

In the mean while I’m looking to add some performance counter functionality into the base component of QuickMon so it would be possible to monitor it (monitor the monitor kind of idea…). The intention is that then you can see that QuickMon is actually performing work and you get some idea of how busy it is.

Then, since version 2.7 I have a new icon/logo for the product – two lighting bolts that are shaped to look like a ‘Q’ (sort of). It looks better than the previous dark colored arrow logo.

QMon3

The UI tool statuses icons have also been updated to use the logo plus the previous colored orbs/balls.

Go grab it here.