Watching the watcher (powershell)

It's always frustrating when Orion itself goes down and no one is aware of it.  

This little powershell script will verify that the Orion Website (https) is up and running (returning a code of 200).  It it does not, then 1) validate that the SQL port is accepting connection; 2) validate at SolarWind ports are accepting connections; 3) validate that SolarWinds services are up and running; 4) email everything to someone.

This can be a starting point for a much elaborate script.

add-type @"

    using System.Net;

    using System.Security.Cryptography.X509Certificates;

    public class TrustAllCertsPolicy : ICertificatePolicy {

        public bool CheckValidationResult(

            ServicePoint srvPoint, X509Certificate certificate,

            WebRequest request, int certificateProblem) {

            return true;

        }

    }

"@

$head = @'

<style>

Body  {

        font-family: New Courier;

        font-size: 12px;

        margin-left: 5px;

        margin-top: 5px;  

        margin-right: 5px;

        margin-bottom: 5px; }

table { border: thin solid #000000;}

td {    font-family: Courier;

        font-size: 12px;

        border-top: 1px solid #999999;

        border-right: 1px solid #999999;

        border-bottom: 1px solid #999999;

        border-left: 1px solid #999999;

        padding-top: 2px;

        padding-right: 10px;

        padding-bottom: 2px;

        padding-left: 4px; } 

</style>

'@

$ipaddress = 127.0.0.1

$tcpports =

            @((80,'default port for http binding'),

            (443,'default port for https binding'),

            (1443,'Orion to SQL, NTA Flow Storage and SQ'),

            (1801,'HA - main Orion server and its standby server'),

            (2055,'NTA collector (default)'),

            (4369,'MSMQ WCF binding'),

            (5671,'default port for https binding'),

            (5672,'HA - communication between HA pool members'),

            (17777,'poller to Orion web console'),

            (17778,'SWSI API, agent communications, NetPath'),

            (17779,'Toolset integration over http'),

            (17780,'Toolset integration over https'),

            (17791,'agent communication on Orion server running Windows Server 2008 R2 SP1'),

            (25672,'HA - main Orion Server and standby server'))

# Check for web site

$username = "ashah"

$password = ConvertTo-SecureString 'XXXXX' -AsPlainText -Force

[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

$cred = new-object -typename System.Management.Automation.PSCredential -argumentlist $username, $password

$form = Invoke-WebRequest -Credential $cred -Uri 'https://XXXXX.com/Orion/Login.aspx' -SessionVariable Session | select -ExpandProperty forms

$form.fields['ctl00$BodyContent$Username'] = $cred.UserName

$form.fields['ctl00$BodyContent$Password'] = $cred.GetNetworkCredential().Password

$results = Invoke-WebRequest -Uri 'https://XXXXX.com/Orion/Login.aspx' -WebSession $Session -Method POST -Body $form.Fields

#Write-Output $results

if ($results.StatusCode -ne "200")

{

# check for services

$result = @()

$SWServices = 'SolarWinds TFTP Server','SolarWindsAdministration',

'SolarWindsAgent64','SolarWindsAlertingServiceV2','SolarWindsCortex',

'SolarWindsHighAvailability','SolarWindsRecommendations','SolarWindsSyslogService',

'SolarWindsTrapService','SWBrowserIntegration','SWCollectorService','SWInfoServiceSvc',

'SWInfoServiceSvcV3','SWJMXBridgeSvc','SWJobEngineSvc2',

'SWNTMJobSchedulerSvc','SWSEUMAgentProxySvc','SWSEUMAgentSvc'

foreach($service in $swservices) {

  Try {

    $s = Get-Service -name $service -ErrorAction Stop

    $o = New-Object PSObject -Property @{ service=$service; status=$s.status }

    $result += ,$o

  }

  Catch {

    $o = New-Object PSObject -Property @{ service=$service; status="No Exist"}

    $result += ,$o

  }

}

$Body = "Web SIte Down - Server Service Report$(Get-Date -Format D)"

# check for SQL connection

$port = 1433

$message="SQL Port"

$sqlserver='192.168.25.51'

try {$connection = New-Object System.Net.Sockets.TcpClient($sqlserver, $port) -ErrorAction stop

     if ($connection.Connected)

          {  $Body = $Body + "<br><br>Connection to SQL port - Success"

    } else { $Body = $Body + "<br><br>Connection to SQL port - Failed"}

}

catch {$Body = $Body + "<br><br>Connection to SQL port - Failed"}

# Check for TCP ports

$Body = $Body + "<br><br>TCP Port Status"

foreach ($port in $tcpports) {

   try {$connection = New-Object System.Net.Sockets.TcpClient($ipaddress, $port[0])

       if ($connection.Connected) { $Body = $Body + '<br>Port: ' +$port[0]+ ' TCP - Success: '+$port[1]}

        else {$Body = $Body + '<br>Port: ' +$port[0]+' TCP - Failed: '+$port[1]}

       }

   catch {$Body = $Body + '<br>Port: ' +$port[0]+' TCP - Failed: '+$port[1]}

   }

$Body = $Body + '<br>'

# Setup email

$Report = $result | Sort Service,Status | ConvertTo-Html -Fragment -As Table | Out-String

#test the output

ConvertTo-Html -Head $head -PostContent $Report -Body $Body | out-file "d:\xxx4.htm"

$all=ConvertTo-Html -Head $head -PostContent $Report -Body $Body | Out-String

$EmailFrom = "XXXXX.com"

$EmailTo = "XXXX.com"

$EmailSubject = "Orion Website down"

$SMTPServer = "XXXXX.com"

$username = "XXXXX"

$password = ConvertTo-SecureString 'XXXXXX' -AsPlainText -Force

[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

$cred = new-object -typename System.Management.Automation.PSCredential -argumentlist $username, $password

Send-MailMessage -Port 25 -SmtpServer $SMTPServer -Credential $cred -From $EmailFrom -To $EmailTo -Subject $EmailSubject -encoding UTF8  -Body $all -Bodyashtml;

}

Sample of output:

Web SIte Down - Server Service ReportThursday, August 16, 2018

Connection to SQL port - Success

TCP Port Status
Port: 80 TCP - Failed: default port for http binding
Port: 443 TCP - Success: default port for https binding
Port: 1443 TCP - Failed: Orion to SQL, NTA Flow Storage and SQ
Port: 1801 TCP - Failed: HA - main Orion server and its standby server
Port: 2055 TCP - Failed: NTA collector (default)
Port: 4369 TCP - Failed: MSMQ WCF binding
Port: 5671 TCP - Failed: default port for https binding
Port: 5672 TCP - Failed: HA - communication between HA pool members
Port: 17777 TCP - Failed: poller to Orion web console
Port: 17778 TCP - Failed: SWSI API, agent communications, NetPath
Port: 17779 TCP - Failed: Toolset integration over http
Port: 17780 TCP - Failed: Toolset integration over https
Port: 17791 TCP - Failed: agent communication on Orion server running Windows Server 2008 R2 SP1
Port: 25672 TCP - Failed: HA - main Orion Server and standby server

Thank you

Amit