ERROR TrapService.TrapService - Bad trap packet received from Node with IP 192.168.1.1. Error description : Unknown user and engine. Packet discarded

Why receiving such Events when Node sending the Traps to Orion

SNMP V3 Trap packet will be verified against the SNMP Authentication for the Node in order to translate encrypted trap information

You can easily reproduce the issue by changing the SNMP information .

You will need to make sure the SNMP information sent by the Node within Trap  is matching against the Node in Orion DB

Following needs to be verify . (Edit the Node in Orion ) (Note down the SNMPV3 Username)

Issue # 1

Image may be NSFW.
Clik here to view.

You can also verify the user name from the Nodes table in Orion DB as below.

Image may be NSFW.
Clik here to view.

You will need to Run the Wireshark on Orion server  filter the traffic for Traps sent by the Node and verify the User name should be matching when received the Trap

In below packet capture example where Trap sent by the  Node 192.168.1.1  user ABC which caused the above Event that should correct in the Node .

Image may be NSFW.
Clik here to view.

Following example included where the correct SNMP user is been sent by the node within Trap information

Image may be NSFW.
Clik here to view.

Resolution

So in nut shell you will need to make sure the Node is sending Traps with the same SNMP user account  what is been used in Orion when adding the Node . If it will not match you will have the above Error Event in Orion. Trap packet will be entertained only if sent with correct SNMP User .

Issue # 2.

You will  noticed that the IP address in the error message was actually assigned to an interfaces on the routers and that the traps were being sent from this  interfaces.

Since the SNMPv3 credentials are used by the router and not the interfaces you then  needed to find a way to source all traps from the router's Loopback address themselves.

Resolution

The easiest way to do this was to enter the following line within the SNMP portion of the router configuration.

"snmp-server trap-source Loopbackxxxx" Where xxxx is the loopback # that the router's IP address is assigned to.

I've used custom views to show clients and my Team many different things that NPM monitors, and those folks really appreciate the service.  You can be a rock star in their eyes when you give them something that helps them improve how they and their clients understand their systems and their performance.

Here's how to build a custom View that shows current and recent WAN interface utilization between routers connecting six different data centers.  Remember, your new View doesn't have to be of just interfaces on the same switch, or on the same router.

You can build this kind of report for as many interfaces as you'd like, from any switch or router or device interface that's monitored by your NPM.  This one shows a single WAN interface on six different routers that connect into the same MPLS cloud:

Image may be NSFW.
Clik here to view.

Here's how to do it:

Create the view:

1. From NPM, go to Settings > Manage Views > Add

2. Customize the new view:

• Name it intuitively
• Select Summary (there are MANY types of views.  Read up on them, test some to get a feel for the extreme versatility of NPM)
• Click Submit

3. Edit the View.

• You can change its name if you want.
• I chose not to enable Left Navigation--this will be simple, and won't load a lot of info.
• I added a Custom HTML Resource to Column 1, then cloned it three times.  Next I moved one instance into Column 2, and then cloned that one twice.  I ended up with three Custom HTML Resources in each column.
• I set the width to 500 px for each
• I set No View Limitation (but you can add one in if you need it)
• Click Done or Preview.  Preview lets you see it right away, without assigning it to any Menu Bar.  Plus, you can edit those custom HTML fields.

4. Here's the sweet spot to make you rock in the eyes of your boss & peers because it gives you the power to make a really useful view:

• Open a second browser window to your NPM
• Browse to a router node and open it
• Open the interface that faces the WAN
• Look at the URL bar and note the NetObject= section.   Example:
• http://your_server_name_here/Orion/Interfaces/InterfaceDetails.aspx?NetObject=I:XXXXX&view=InterfaceDetails
• Copy or write or memorize the NetObject number for use in the next step

5. Edit your new View in Preview, or assign it to a Menu Bar and open it

• Click Edit in the top left Custom HTML window
• Give it an intuitive Title
• Fill in the Subtitle if that helps you better identify the interface or feature you're displaying
• Copy and paste this into the html window:

<img src="/Orion/Netperfmon/Chart.aspx?ChartName=MMAvgBps&Title=&SubTitle=&SubTitle2=&Width=640&Height=0&NetObject=I:XXXX&CustomPollerID=&SampleSize=1H&Period=Last 10 Days&FontSize=1&NetObjectPrefix=I&SubsetColor=&RYSubsetColor=&Printable=true&ShowTrend=True&ResourceID=27" />

• Where I've put in the X's, remove them and enter in the NetObject ID number you wrote down from the previous step
• Click Submit

Image may be NSFW.
Clik here to view.

The new page should open, but this time the top left window won't say Custom HTML, it'll show the bandwidth used by that router's WAN interface for the last 10 days:

Image may be NSFW.
Clik here to view.

Now repeat this for every Custom HTML entry you've made, but change the NodeID info to the Node ID that's shown in the Interface Details page URL for new/different WAN routers and interfaces.

OK, you've made a new View.  But your team can't access it easily until you assign your new View to the Menu Bar they use.  If you don't know how to do that . . .

Here's how to define a new View to add to a Menu Bar:

1. Go to NPM > Settings > Customize Menu Bars
2. Edit the Menu Bar you or your team will use to access your new View
3. Find your new View in the Available Items column on the left
4. Drag it to the right column, drop it in where you want it.

If you can't find your new View, you'll need to discover its URL.  Follow this process:

1. Open a second browser window and Edit the View by clicking the Preview open
2. Copy the URL for the new View when you're previewing it
3. Go back to the original browser window from step 3 and click Add
4. Type in the Name of your View, paste in the URL address, give it a good Description
5. I clicked the option to Open in a New Window.  It's your choice.
6. Click OK

Now go back to NPM's Home Page, find the new View in the Menu Bar, and click on it.

     Image may be NSFW.
Clik here to view.

Voila!

Tweaking and tuning:

If you're not afraid of a little keyboard editing of html, you can easily change the 10 days to show the last 7 days or last 23 hours or last 17 minutes or last pretty-much-anything.  Here's how:

Look for "SampleSize" and "Period" in the code you copied & pasted

<img src="/Orion/Netperfmon/Chart.aspx?ChartName=MMAvgBps&Title=&SubTitle=&SubTitle2=&Width=640&Height=0&NetObject=I:XXXX&CustomPollerID=&SampleSize=1H&Period=Last 10 Days&FontSize=1&NetObjectPrefix=I&SubsetColor=&RYSubsetColor=&Printable=true&ShowTrend=True&ResourceID=27" />

Note the SampleSize is 1H, or One Hour.

Node the Period is the Last 10 Days

Edit each window in your view and change the SampleSize and Period to whatever works best for you.  Keep in mind that some combinations of SampleSize and Period don't go well together (e.g.:  SampleSize=1M and Period=Last 10 Months) because NPM needs to have you be reasonable.  If you want a graph showing changes every 1 minute, you have to be polling & collecting the data pretty frequently.  Pick a Period that's reasonable for seeing good granularity of 1 minute points, like Last 20 Minutes.  If you just see points instead of a line in the chart, you've picked a combination of sample size & period that's too granular for what your NPM has polled.  Adjust until you've got something that meets your needs.

Remember to keep the spacing in the html code exactly the same as the example I show above.  If you add a space between words, or remove one, you risk breaking the pretty graph output.

Some examples:

Show the Last 3 Hours, in 10-minute increments uses this code:

<img src="/Orion/Netperfmon/Chart.aspx?ChartName=MMAvgBps&Title=&SubTitle=&SubTitle2=&Width=640&Height=0&NetObject=I:XXXX&CustomPollerID=&SampleSize=10M&Period=Last 3 Hours&FontSize=1&NetObjectPrefix=I&SubsetColor=&RYSubsetColor=&Printable=true&ShowTrend=True&ResourceID=27" />

Note how the graph automatically changed "Last 10 Days" to "Last 3 Hours" in the title.  Sweet!

Image may be NSFW.
Clik here to view.

Suppose you like the 10 minute samples, but want to show the last 24 hours.  Just change "Last 10 Hours" to "Last 24 Hours":

<img src="/Orion/Netperfmon/Chart.aspx?ChartName=MMAvgBps&Title=&SubTitle=&SubTitle2=&Width=640&Height=0&NetObject=I:25091&CustomPollerID=&SampleSize=10M&Period=Last 24 Hours&FontSize=1&NetObjectPrefix=I&SubsetColor=&RYSubsetColor=&Printable=true&ShowTrend=True&ResourceID=27" />

Image may be NSFW.
Clik here to view.

If it looks great, fire off an e-mail to your Team or your Boss and show off the new functionality you've provided to the organization.

If it doesn't look great, spend a little time fixing it up.  Look for typos, fix spelling & punctuation--this is your baby, and folks will judge you by your work.

Swift Packets!

Rick Schroeder

If you find that you are running Windows 2008/2008 R2 and it is taking an inordinate time to start the Solarwinds services after a reboot, then you may want to check the MSMQ service and subsequent settings. In your Windows system log, look for event ID 7044. In that event you will see the following message: "The following service is taking more than 16 minutes to start and may have stopped responding: Message Queuing." If you see this issue on your polling engines, it will cause the polling engine to go red as the following Solarwinds services rely on MSMQ to start:

- Solarwinds Information Service

- Solarwinds Information Service V3

- Solarwinds Collector Data Processor

- Solarwinds Collector Polling Controller

To completely verify the issue, check out your Windows\System32\MSMQ\Storage. The default journal setting is 8GB. If the folder is 8GB, you have verified your problem. Furthermore, if the poller has only 8GB, that will explain why nothing can be accomplished on the server until this is complete, as MSMQ is trying to load all messages into RAM. See the following TechNet article to understand more and correct the problem at the domain level: https://technet.microsoft.com/en-us/library/cc733166(v=ws.10).aspx.

        -DaveB7114  

    Loop1 Systems: SolarWinds Training and Professional Services      

•                 LinkedIN: Loop1 Systems          
•                 Facebook: Loop1 Systems          
•                 Twitter: @Loop1Systems          

Here is an example to use SWQL to build a view to display problematic nodes (servers) with issues from one or more flowing areas:

•    Node Status (column name: CONN) - (1 UP, 2 Down, ignore other status)

•    Node Response Time (column name: M_SECS) - in milliseconds, (> 0 OR When Node is Down, it is -1). If M_SECS> 500: Warning, If  M_SECS> 500: Critical

•    Node CPU Load (column name:  C_LOAD) - in percentage, (Between 0 - 100). If  C_LOAD > 95: Warning, If  C_LOAD > 98: Warning, If  C_LOAD =100: Down

•    Node Memory Usage (column name:  R_Load) - percentage, (Between 0 - 100). If  R_LOAD > 95: Warning, If  R_LOAD > 98: Warning, If  R_LOAD =100: Down

•    Node Highest Volume Usage (column name:  V_PERCENT) - (Between 0 - 100). If  V_PERCENT > 95: Warning, If  V_PERCENT > 98: Warning, If  V_PERCENT =100: Down

•    Node Hardware Components worst  Status (column name: HW_Status) -  (UP, Undefined, Unknown, Warning, Critical, n/a)

•    Node  Application worst Status (column name:  APP_Status)  - (UP, Unmanaged, Unknown, Unreachable, Warning, Critical, Down, n/a)

Image may be NSFW.
Clik here to view.

In order to the worst (highest priority) condition are shown on the top of the list I gave each status different scores, and each column different weights. Then calculate total score as the priority. Here is the calculation:

•    wConn (Connection), scores: Down - 1000, Up - 0; weight 1.00
•    wTime (Response Time), scores:  > 1000ms - 80, >500ms - 10, other - 0; Weight 0.75
•    wCPU (CPU Load), scores: 100% - 600, >98% - 80, >95% - 10, Other - 0; Weight 1.00
•    wRAM (Memory Load), scores: 100% - 600, >98% - 80, >95% - 10, Other 0; Weight 1.00
•    wVol (MAX(Volume Usage)), the highest volume usage of all volumes on a node, scores: 100% - 600, >98% - 80, >95% - 10, Other 0; Weight 0.75
•    wHW  (Hardware Status (worst Value)), the worst HW component status of a node with HW monitor enabled   scores: Critical - 80, Warning - 10, Up - 0, other 1; Weight 0.50
•    wApp (Application Status (worst value), the worst application statues of a node with application monitors assigned.  scores: Down - 600, Critical - 80, Warning - 10, Up - 0, other 1; Weight 0.50

Maximum Total Weighted Score (Exclude wConn):  80*0.75 + 600*1.00 + 600*1.00 + 600*0.75 + 80*0.50 + 600 *0.50  = 2050

Priority = ROUND((t1.wTime*0.75 + t1.wCPU*1.0 + t1.wRAM*1.0 + t1.wVol*0.75 + t1.wHW*0.5 + t1.wApp*0.5)/2.05 + t1.wConn*1.00, 2)

Final Priority value is between 0 and 1000.

You can change the score and weight to meeting your requirement.

Steps:

• Create a view; add “Custom Query” resource.

Image may be NSFW.
Clik here to view.

• In the view, edit Custom Query:
• In the Custom SWQL Query box, add the codes in attached file “thwack-swql-alerts.txt”
• Enable search, and in Search SWQL Query box, add the codes in attached file “thwack-swql-alerts-withSearch.txt”

Done!

Using Search:

•    By Node Name
If you want to just display a node or a group of nodes with similar names, type node name or part of the name in the search box and click search button.
•    By Connection Status
If you want to just display nodes in DOWN status, type “n 1” (white space between n and 1) in the search box and click search button.
•    By CPU or RAM or Volume usage
If you want to just display node with CPU or RAM or Volume usage above certain level, using the following:

     o    “c 80”  (CPU usage above 80%)
     o    “r 80” (Memory usage above 80%)
     o    “v 80” (Volume usage above 80%)
•    By Hardware Status
If you want to just display node with certain hardware status, type “h status” (‘status’ can be one of the following: UP, undefined, Unknown, Warning, Critical, n/a).
•    By Application Status
If you want to just display node with certain application status, type “a status” (‘status‘ can be one of the following: UP, Unmanaged, Unknown, Unreachable, Warning, Critical, Down, n/a).

You can customise the query to meeting your requirements.

Thanks Alex Soul's post https://thwack.solarwinds.com/docs/DOC-174568, which is very helpful!

===========================

Update:  As Alex suggested, I have updated the query and new files are attached. Thanks Alex!

===========================

Update: 11/March/2015

I have added 2 addition columns for Alert Prioritising Dashboard.

One column is AlertTime, another one is Acknowledge (Ack). The Ack column is click-able. Right click it and open a new windows to View or Acknowledge an alert.

Please see the additional document at https://thwack.solarwinds.com/docs/DOC-176727

Image may be NSFW.
Clik here to view.

< BACK TO TO THE MAGIC OF SQL SCRIPTS SERIES LIST

End Result:

Image may be NSFW.
Clik here to view.

How To:

To achieve this just use the below SQL script to create Report in Report Writer and then use this report as a resource on node's page.

You can also download report file directly (attached). Just copy this file into your default folder with reports and report will automatically appear in your list. Then add it as a resource on your node page

Remember: Update SQL script with the correct database name as per your installation

Note: This will work with NPM 11.5+. If you want this to work with earlier versions you will need to update SQL script, the part where user ID is being extracted and change ID 28 to ID 27 (appear two times)

/* ============================================
Author: Alex Soul
Date: 13/05/2015
Description:  * Information about unmnaged nodes.  * Suitable for NPM 11.5. Earleir versions of NPM will require slight adjustemnt as Audit Event IDs are not the same. You will need to change ID from 28 to 27 for it to pick up user  * To be used as an on-page report for "Report from Orion Writer" resource
Version:  Number | Comments | Date | Author  1.1 | Adopt for THWACK | 15/07/2015 | AS
--============================================ */
SELECT      n.NodeID --hidden     ,n.Caption --hidden  ,CASE      WHEN DATEDIFF(YEAR, n.UnManageFrom, getdate()) > 3      THEN CONVERT(NVARCHAR(50), DATEDIFF(YEAR, n.UnManageFrom, getdate())) + ' years ago'      ELSE CASE   WHEN DATEDIFF(MONTH, n.UnManageFrom, getdate()) > 3      THEN CONVERT(NVARCHAR(50), DATEDIFF(MONTH, n.UnManageFrom, getdate())) + ' month ago'      ELSE CASE      WHEN DATEDIFF(DAY, n.UnManageFrom, getdate()) > 3      THEN CONVERT(NVARCHAR(50), DATEDIFF(DAY, n.UnManageFrom, getdate())) + ' days ago'      ELSE CASE      WHEN DATEDIFF(HOUR, n.UnManageFrom, getdate()) > 3      THEN CONVERT(NVARCHAR(50), DATEDIFF(HOUR, n.UnManageFrom, getdate())) + ' hours ago'      ELSE CONVERT(NVARCHAR(50), DATEDIFF(MINUTE, n.UnManageFrom, getdate())) + ' min ago'      END      END      END     END AS 'WHEN'  ,CASE      WHEN DATEDIFF(YEAR, n.UnManageFrom, n.UnManageUntil) > 3      THEN CONVERT(NVARCHAR(50), DATEDIFF(YEAR, n.UnManageFrom, n.UnManageUntil)) + ' years'      ELSE CASE   WHEN DATEDIFF(MONTH, n.UnManageFrom, n.UnManageUntil) > 3      THEN CONVERT(NVARCHAR(50), DATEDIFF(MONTH, n.UnManageFrom, n.UnManageUntil)) + ' month'      ELSE CASE      WHEN DATEDIFF(DAY, n.UnManageFrom, n.UnManageUntil) > 3      THEN CONVERT(NVARCHAR(50), DATEDIFF(DAY, n.UnManageFrom, n.UnManageUntil)) + ' days'      ELSE CASE      WHEN DATEDIFF(HOUR, n.UnManageFrom, n.UnManageUntil) > 3      THEN CONVERT(NVARCHAR(50), DATEDIFF(HOUR, n.UnManageFrom, n.UnManageUntil)) + ' hours'      ELSE CONVERT(NVARCHAR(50), DATEDIFF(MINUTE, n.UnManageFrom, n.UnManageUntil)) + ' min'      END      END      END     END AS 'FOR'  ,CASE      WHEN DATEDIFF(YEAR, getdate(), n.UnManageUntil) > 3      THEN CONVERT(NVARCHAR(50), DATEDIFF(YEAR, getdate(), n.UnManageUntil)) + ' years'      ELSE CASE   WHEN DATEDIFF(MONTH, getdate(), n.UnManageUntil) > 3      THEN CONVERT(NVARCHAR(50), DATEDIFF(MONTH, getdate(), n.UnManageUntil)) + ' month'      ELSE CASE      WHEN DATEDIFF(DAY, getdate(), n.UnManageUntil) > 3      THEN CONVERT(NVARCHAR(50), DATEDIFF(DAY, getdate(), n.UnManageUntil)) + ' days'      ELSE CASE      WHEN DATEDIFF(HOUR, getdate(), n.UnManageUntil) > 3      THEN CONVERT(NVARCHAR(50), DATEDIFF(HOUR, getdate(), n.UnManageUntil)) + ' hours'      ELSE CONVERT(NVARCHAR(50), DATEDIFF(MINUTE, getdate(), n.UnManageUntil)) + ' min'      END      END      END     END AS 'REMAINS'    ,n.UnManageFrom AS 'FROM'  ,n.UnManageUntil AS 'UNTIL'    ,r1.AccountID AS 'BY'
FROM Solarwinds.dbo.Nodes n WITH(NOLOCK)


--below two joints are joining last recorded "Node unmanaged" audit event (Event ID 27) for this node to be able to display who has unmanaged it
LEFT JOIN (
  SELECT NetObjectID, MAX([TimeLoggedUtc]) AS max_datetime  FROM Solarwinds.dbo.AuditingEvents WITH(NOLOCK)  WHERE ActionTypeID = '28'  GROUP BY NetObjectID) MaxDates ON MaxDates.NetObjectID = n.NodeID
LEFT JOIN Solarwinds.dbo.AuditingEvents r1 WITH(NOLOCK) ON (r1.TimeLoggedUtc = MaxDates.max_datetime AND r1.NetObjectID = MaxDates.NetObjectID AND r1.ActionTypeID = '28')


WHERE
  n.NodeID = '${NodeID}' AND  n.UnManaged = '1'

Have fun,

Alex Soul

www.pixace.com

Hi All,

I have set one alert condition from BGP neighbor change between my Cisco routers and Service provider.

Below is the alert conditions

Trigger Alert when all of the following apply

   OID is equal to 1.3.6.1.2.1.15.3.1.2

   Numeric Status is not equal to 6

   Trigger Alert when any of the follow apply

      IP Address is equal to XX.XX.XX.XX

      IP Address is equal to XX.XX.XX.XX

Account to the MIB or the OID ..... if the numerical value is 6 then the bgp session is established.

So I have set the alert like if the value changes from 6...send me a mail alert.

Now the issue is I am not getting the alert when the value changes from 6.

I have verified that the value is changing..but still alert is not triggering.

Any helpy would be very much helpful

Thanks in advance.

Here is the calculation for the Disk / Volume / RAM /below and example.

What object IDs (OIDs) does Orion NPM poll for volume information? What types of volume information does Orion NPM poll?

http://knowledgebase.solarwinds.com/kb/questions/1194/What+object+IDs+%28OIDs%29+does+Orion+NPM+poll+for+volume+information%3F+What+types+of+volume+information+does+Orion+NPM+poll%3F

Image may be NSFW.
Clik here to view.

You can run the MIB Walk for the Node in Question in order to find out what values been returned by the Node for required OID's as below.

Here is MIB walk tool download Link

http://downloads.solarwinds.com/solarwinds/Release/PreRelease/SolarWindsSnmpWalk.zip

******** MIb Walk Result from the Node **********

.1.3.6.1.2.1.25.2.3.1.3.1 = String: "Physical RAM"

.1.3.6.1.2.1.25.2.3.1.3.2 = String: "Virtual Memory"

.1.3.6.1.2.1.25.2.3.1.3.3 = String: "/"

.1.3.6.1.2.1.25.2.3.1.3.4 = String: "/proc"

.1.3.6.1.2.1.25.2.3.1.3.5 = String: "/dev/pts"

.1.3.6.1.2.1.25.2.3.1.3.6 = String: "/proc/bus/usb"

.1.3.6.1.2.1.25.2.3.1.3.7 = String: "/partB"

.1.3.6.1.2.1.25.2.3.1.3.8 = String: "/common"

.1.3.6.1.2.1.25.2.3.1.3.9 = String: "/grub"

.1.3.6.1.2.1.25.2.3.1.3.10 = String: "/dev/shm"

.1.3.6.1.2.1.25.2.3.1.3.11 = String: "/spare"

.1.3.6.1.2.1.25.2.3.1.3.12 = String: "/proc/sys/fs/binfmt_misc"

.1.3.6.1.2.1.25.2.3.1.4.1 = INTEGER: 4096

.1.3.6.1.2.1.25.2.3.1.4.2 = INTEGER: 4096

.1.3.6.1.2.1.25.2.3.1.4.3 = INTEGER: 4096

.1.3.6.1.2.1.25.2.3.1.4.4 = INTEGER: 4096

.1.3.6.1.2.1.25.2.3.1.4.5 = INTEGER: 4096

.1.3.6.1.2.1.25.2.3.1.4.6 = INTEGER: 4096

.1.3.6.1.2.1.25.2.3.1.4.7 = INTEGER: 4096

.1.3.6.1.2.1.25.2.3.1.4.8 = INTEGER: 4096

.1.3.6.1.2.1.25.2.3.1.4.9 = INTEGER: 1024

.1.3.6.1.2.1.25.2.3.1.4.10 = INTEGER: 4096

.1.3.6.1.2.1.25.2.3.1.4.11 = INTEGER: 4096

.1.3.6.1.2.1.25.2.3.1.4.12 = INTEGER: 4096

.1.3.6.1.2.1.25.2.3.1.5.1 = INTEGER: 1021591

.1.3.6.1.2.1.25.2.3.1.5.2 = INTEGER: 512034

.1.3.6.1.2.1.25.2.3.1.5.3 = INTEGER: 4232703

.1.3.6.1.2.1.25.2.3.1.5.4 = INTEGER: 0

.1.3.6.1.2.1.25.2.3.1.5.5 = INTEGER: 0

.1.3.6.1.2.1.25.2.3.1.5.6 = INTEGER: 0

.1.3.6.1.2.1.25.2.3.1.5.7 = INTEGER: 4232699

.1.3.6.1.2.1.25.2.3.1.5.8 = INTEGER: 17502476

.1.3.6.1.2.1.25.2.3.1.5.9 = INTEGER: 252879

.1.3.6.1.2.1.25.2.3.1.5.10 = INTEGER: 510795

.1.3.6.1.2.1.25.2.3.1.5.11 = INTEGER: 8468466

.1.3.6.1.2.1.25.2.3.1.5.12 = INTEGER: 0

.1.3.6.1.2.1.25.2.3.1.6.1 = INTEGER: 447260

.1.3.6.1.2.1.25.2.3.1.6.2 = INTEGER: 52

.1.3.6.1.2.1.25.2.3.1.6.3 = INTEGER: 2837178

.1.3.6.1.2.1.25.2.3.1.6.4 = INTEGER: 0

.1.3.6.1.2.1.25.2.3.1.6.5 = INTEGER: 0

.1.3.6.1.2.1.25.2.3.1.6.6 = INTEGER: 0

.1.3.6.1.2.1.25.2.3.1.6.7 = INTEGER: 2736606

.1.3.6.1.2.1.25.2.3.1.6.8 = INTEGER: 7646375

.1.3.6.1.2.1.25.2.3.1.6.9 = INTEGER: 4303

.1.3.6.1.2.1.25.2.3.1.6.10 = INTEGER: 5422

.1.3.6.1.2.1.25.2.3.1.6.11 = INTEGER: 8256

****************** Physical Disks ******************

hrStorageDescr = .1.3.6.1.2.1.25.2.3.1.3.3 = String: "/"

hrStorageAllocationUnits=   .1.3.6.1.2.1.25.2.3.1.4.3 = INTEGER: 4096

hrStorageSize .1.3.6.1.2.1.25.2.3.1.5.3 = INTEGER: 4232703

hrStorageUsed = .1.3.6.1.2.1.25.2.3.1.6.3 = INTEGER: 2837178

Calculations :

Volume size       4232703 x 4096  17337151488 Bytes   16.146 GB

Volume used       2837178 x 4096  11621081088 Bytes   10.822 GB

space available   17337151488 - 11621081088 = 5716070400 5.323 GB .

Percentage        10.8/16.14 * 100 65%

****************************************************

hrStorageDescr = .1.3.6.1.2.1.25.2.3.1.3.1 = String: "Physical RAM"

hrStorageAllocationUnits= .1.3.6.1.2.1.25.2.3.1.4.1 = INTEGER: 4096

hrStorageSize    .1.3.6.1.2.1.25.2.3.1.5.1 = INTEGER: 1021591

hrStorageUsed = .1.3.6.1.2.1.25.2.3.1.6.1 = INTEGER: 447260

Calculations : Physical RAM

Volume size       1021591 x 4096  4184436736 Bytes       3.897 GB

Volume used       447260 x 4096   1831976960 Bytes       1.706 GB

space available   1831976960 - 4184436736 = 2352459776     2.1 GB

Percentage        1.706/3.897 * 100 43%

*********************************************************

hrStorageDescr = .1.3.6.1.2.1.25.2.3.1.3.2 = String: "Virtual Memory"

hrStorageAllocationUnits= .1.3.6.1.2.1.25.2.3.1.4.2 = INTEGER: 4096

hrStorageSize    .1.3.6.1.2.1.25.2.3.1.5.2 = INTEGER: 512034

hrStorageUsed = .1.3.6.1.2.1.25.2.3.1.6.2 = INTEGER: 52

Calculations : Virtual Memory

Volume size       512034 x 4096  2097291264 Bytes   1.953 GB

Volume used       52 x 4096 212992 Bytes           0.0001 GB

space available   212992 - 2097291264 = 2097078272  1.953 GB .

Percentage 0.0001/1.953 * 100 0.0%

///////////////////////////////////////////////////////////////

Adding Few more explanation here for Cisco Call Manager Showing 99% Memory Utilization in Orion .( just in case someone having this issue )

Image may be NSFW.
Clik here to view.

The reason you are seeing 99% is due to the Cisco MCS returning back those values.

Why its doing this, you have to Open Cisco TAC.

NPM is returning back the values its being supplied with, in this case showing 99%.

Poller type: Used .

1. N.Memory.SNMP.CpqHostPhysicalMemory =

Memory poller for OpenVMS systems which support proprietary CPQ-Host MIB

Poller utilizes GetSubtree request with two OIDs to gather total and used memory:

            cpqHoPhysicalMemorySize = 1.3.6.1.4.1.232.11.2.13.1.

            cpqHoPhysicalMemoryFree = 1.3.6.1.4.1.232.11.2.13.2.

Total memory is computed using this formula:

            memoryTotal = cpqHoPhysicalMemorySize * 1024

Used memory is computed using this formula:

            memoryUsed = memoryTotal - (cpqHoPhysicalMemoryFree * 1024)

MIB Walk shows:

            .1.3.6.1.4.1.232.11.2.13.1.0 = INTEGER: 3990 = cpqHoPhysicalMemorySize

            .1.3.6.1.4.1.232.11.2.13.2.0 = INTEGER: 60 = cpqHoPhysicalMemoryFree

memoryTotal = 3990 * 1024 = 4085760

memoryUsed = 4085760 - (60 * 1024) = 4085760 – 61440 = 4024320

PercentUsed = 4024320/4085760 = 98.496% = 99%

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

For Linux Server for Poller assigned N.Memory.SNMP.NetSnmpReal

Image may be NSFW.
Clik here to view.

If poller assigned

N.Memory.SNMP.NetSnmpReal

/// memTotalReal = 1.3.6.1.4.1.2021.4.5.
/// memAvailReal = 1.3.6.1.4.1.2021.4.6.
/// memCached = 1.3.6.1.4.1.2021.4.15.
/// memBuffer = 1.3.6.1.4.1.2021.4.14.
/// UsedMemory = (totalMem - availMem - cachedMem - bufferMem) * 1024

.1.3.6.1.4.1.2021.4.5.0 = INTEGER: 2073620 (Total )

.1.3.6.1.4.1.2021.4.6.0 = INTEGER: 1869968 (Available )

.1.3.6.1.4.1.2021.4.15.0 = INTEGER: 88008 (cachedMemory )

.1.3.6.1.4.1.2021.4.14.0 = INTEGER: 51868 (BufferMemory )

( 2073620– 1869968– 88008– 51868 ) = 63776  

63776 * 1024 = 65306624 bytes

********************

Vmware ESX Memory Calculation

Image may be NSFW.
Clik here to view.

Image may be NSFW.
Clik here to view.

Supported Poller

Image may be NSFW.
Clik here to view.

The "Memory" Usage and the "Physical Memory" Utilization are calculated differently by Orion NPM.

Memory:

The Memory is the sum of the memory used by the processes listed in the HOST-RESOURCES-MIB:hrSWRunPerfTable (1.3.6.1.2.1.25.5.1).

The OID used is the hrSWRunPerfMem (1.3.6.1.2.1.25.5.1.1.2).

Physical Memory:

The Physical Memory is calculated based on the values populated in the HOST-RESOURCES-MIB:hrStorageTable (1.3.6.1.2.1.25.2.2.3).

The OIDs used are hrStorageSize (1.3.6.1.2.1.25.2.3.1.5), hrStorageUsed (1.3.6.1.2.1.25.2.3.1.6) and hrStorageAllocationUnits (1.3.6.1.2.1.25.2.3.1.4) .

Calculation :-

Used: .1.3.6.1.2.1.25.5.1.1.2

Free:  .1.3.6.1.2.1.25.5.1.1.2

Total. 5933728 + 7325457 = 13259185

% Used : 5933728 % 13259185 = 44.751 %

I've used custom views to show clients and my Team many different things that NPM monitors, and those folks really appreciate the service.  You can be a rock star in their eyes when you give them something that helps them improve how they and their clients understand their systems and their performance.

Here's how to build a custom View that shows current and recent WAN interface utilization between routers connecting six different data centers.  Remember, your new View doesn't have to be of just interfaces on the same switch, or on the same router.

You can build this kind of report for as many interfaces as you'd like, from any switch or router or device interface that's monitored by your NPM.  This one shows a single WAN interface on six different routers that connect into the same MPLS cloud:

Image may be NSFW.
Clik here to view.

Here's how to do it:

Create the view:

1. From NPM, go to Settings > Manage Views > Add

2. Customize the new view:

• Name it intuitively
• Select Summary (there are MANY types of views.  Read up on them, test some to get a feel for the extreme versatility of NPM)
• Click Submit

3. Edit the View.

• You can change its name if you want.
• I chose not to enable Left Navigation--this will be simple, and won't load a lot of info.
• I added a Custom HTML Resource to Column 1, then cloned it three times.  Next I moved one instance into Column 2, and then cloned that one twice.  I ended up with three Custom HTML Resources in each column.
• I set the width to 500 px for each
• I set No View Limitation (but you can add one in if you need it)
• Click Done or Preview.  Preview lets you see it right away, without assigning it to any Menu Bar.  Plus, you can edit those custom HTML fields.

4. Here's the sweet spot to make you rock in the eyes of your boss & peers because it gives you the power to make a really useful view:

• Open a second browser window to your NPM
• Browse to a router node and open it
• Open the interface that faces the WAN
• Look at the URL bar and note the NetObject= section.   Example:
• http://your_server_name_here/Orion/Interfaces/InterfaceDetails.aspx?NetObject=I:XXXXX&view=InterfaceDetails
• Copy or write or memorize the NetObject number for use in the next step

5. Edit your new View in Preview, or assign it to a Menu Bar and open it

• Click Edit in the top left Custom HTML window
• Give it an intuitive Title
• Fill in the Subtitle if that helps you better identify the interface or feature you're displaying
• Copy and paste this into the html window:

<img src="/Orion/Netperfmon/Chart.aspx?ChartName=MMAvgBps&Title=&SubTitle=&SubTitle2=&Width=640&Height=0&NetObject=I:XXXX&CustomPollerID=&SampleSize=1H&Period=Last 10 Days&FontSize=1&NetObjectPrefix=I&SubsetColor=&RYSubsetColor=&Printable=true&ShowTrend=True&ResourceID=27" />

• Where I've put in the X's, remove them and enter in the NetObject ID number you wrote down from the previous step
• Click Submit

Image may be NSFW.
Clik here to view.

The new page should open, but this time the top left window won't say Custom HTML, it'll show the bandwidth used by that router's WAN interface for the last 10 days:

Image may be NSFW.
Clik here to view.

Now repeat this for every Custom HTML entry you've made, but change the NodeID info to the Node ID that's shown in the Interface Details page URL for new/different WAN routers and interfaces.

OK, you've made a new View.  But your team can't access it easily until you assign your new View to the Menu Bar they use.  If you don't know how to do that . . .

Here's how to define a new View to add to a Menu Bar:

1. Go to NPM > Settings > Customize Menu Bars
2. Edit the Menu Bar you or your team will use to access your new View
3. Find your new View in the Available Items column on the left
4. Drag it to the right column, drop it in where you want it.

If you can't find your new View, you'll need to discover its URL.  Follow this process:

1. Open a second browser window and Edit the View by clicking the Preview open
2. Copy the URL for the new View when you're previewing it
3. Go back to the original browser window from step 3 and click Add
4. Type in the Name of your View, paste in the URL address, give it a good Description
5. I clicked the option to Open in a New Window.  It's your choice.
6. Click OK

Now go back to NPM's Home Page, find the new View in the Menu Bar, and click on it.

     Image may be NSFW.
Clik here to view.

Voila!

Tweaking and tuning:

If you're not afraid of a little keyboard editing of html, you can easily change the 10 days to show the last 7 days or last 23 hours or last 17 minutes or last pretty-much-anything.  Here's how:

Look for "SampleSize" and "Period" in the code you copied & pasted

<img src="/Orion/Netperfmon/Chart.aspx?ChartName=MMAvgBps&Title=&SubTitle=&SubTitle2=&Width=640&Height=0&NetObject=I:XXXX&CustomPollerID=&SampleSize=1H&Period=Last 10 Days&FontSize=1&NetObjectPrefix=I&SubsetColor=&RYSubsetColor=&Printable=true&ShowTrend=True&ResourceID=27" />

Note the SampleSize is 1H, or One Hour.

Node the Period is the Last 10 Days

Edit each window in your view and change the SampleSize and Period to whatever works best for you.  Keep in mind that some combinations of SampleSize and Period don't go well together (e.g.:  SampleSize=1M and Period=Last 10 Months) because NPM needs to have you be reasonable.  If you want a graph showing changes every 1 minute, you have to be polling & collecting the data pretty frequently.  Pick a Period that's reasonable for seeing good granularity of 1 minute points, like Last 20 Minutes.  If you just see points instead of a line in the chart, you've picked a combination of sample size & period that's too granular for what your NPM has polled.  Adjust until you've got something that meets your needs.

Remember to keep the spacing in the html code exactly the same as the example I show above.  If you add a space between words, or remove one, you risk breaking the pretty graph output.

Some examples:

Show the Last 3 Hours, in 10-minute increments uses this code:

<img src="/Orion/Netperfmon/Chart.aspx?ChartName=MMAvgBps&Title=&SubTitle=&SubTitle2=&Width=640&Height=0&NetObject=I:XXXX&CustomPollerID=&SampleSize=10M&Period=Last 3 Hours&FontSize=1&NetObjectPrefix=I&SubsetColor=&RYSubsetColor=&Printable=true&ShowTrend=True&ResourceID=27" />

Note how the graph automatically changed "Last 10 Days" to "Last 3 Hours" in the title.  Sweet!

Image may be NSFW.
Clik here to view.

Suppose you like the 10 minute samples, but want to show the last 24 hours.  Just change "Last 10 Hours" to "Last 24 Hours":

<img src="/Orion/Netperfmon/Chart.aspx?ChartName=MMAvgBps&Title=&SubTitle=&SubTitle2=&Width=640&Height=0&NetObject=I:25091&CustomPollerID=&SampleSize=10M&Period=Last 24 Hours&FontSize=1&NetObjectPrefix=I&SubsetColor=&RYSubsetColor=&Printable=true&ShowTrend=True&ResourceID=27" />

Image may be NSFW.
Clik here to view.

If it looks great, fire off an e-mail to your Team or your Boss and show off the new functionality you've provided to the organization.

If it doesn't look great, spend a little time fixing it up.  Look for typos, fix spelling & punctuation--this is your baby, and folks will judge you by your work.

Swift Packets!

Rick Schroeder

It took me a while to get the right combination of AIX SNMP Version 3 settings that will work with Authentication & Privacy enabled for Solar Winds.

Hopefully the steps below will help you get it working too. A list of sources which provided the clues I needed is at the end, its a long one as no single place had everything you will need. Hopefully this doco corrects that.

AIX Configuration

These steps worked ok on "AIX 6 TL08" and "AIX 7 TL 03".

• Install the snmp.crypto fileset to enable encryption (Obtained mine from AIX 6 Expansion Pack DVD 5765-G62 11/2012)

> lslpp -cl snmp.crypto

#Fileset:Level:PTF Id:State:Type:Description:EFIX Locked

/usr/lib/objrepos:snmp.crypto:6.1.2.0::COMMITTED:I:56-bit DES Encrypted SNMPV3 Support:

/etc/objrepos:snmp.crypto:6.1.2.0::COMMITTED:I:56-bit DES Encrypted SNMPV3 Support:

• Backup the /etc/rc.tcpip file as it's about to be modified
• Turn on encryption with the snmp switch command:

> /usr/sbin/snmpv3_ssw -e

This command will create symbolic links as required to enable/disable the encrypted/non-encrypted versions of snmpd & clsnmp

• Confirm and update /etc/rc.tcpip so that the following lines are no longer commented. Comment out dpid2 if it hasn't already been by the above.

---cut---

# Start up the Simple Network Management Protocol (SNMP) daemon

start /usr/sbin/snmpd "$src_running"

# Start up the hostmibd daemon

start /usr/sbin/hostmibd "$src_running"

# Start up the snmpmibd daemon

start /usr/sbin/snmpmibd "$src_running"

# Start up the aixmibd daemon

start /usr/sbin/aixmibd "$src_running"

---cut---

It looks like /usr/sbin/dpid2 functionality has been rolled into one of the above from at least AIX 6 TL08 onwards.

• Backup your existing snmp configuration files

---cut---

/etc/snmpdv3.conf

/etc/clsnmp.conf

/etc/snmpd.boots

/etc/snmpd.peers

---cut---

• Pick one of your servers /etc/snmpd.boots files and make that file uniform across all your servers. Here's an example of its contents:

---cut---

00000002000000000A454172 0000000082

---cut---

The first value is your EngineID, which can be something you made up, or one provided by the vendor. The second is the number of times snmp has been restarted. If you ensure this file is consistent across your AIX servers you can reuse your /etc/snmpdv3.conf file across them all. In turn, you can reuse the credentials when adding the nodes to Solarwinds. The auth/priv keys are married to the EngineID and won't work on another server if the EngineID is different there.

• Generate a new auth key with your local EngineID.

---cut---

pwtokey -e -u auth <auth password> $(cat /etc/snmpd.boots | cut -f2 -d' ')

Display of 16 byte HMAC-MD5 privKey:

5xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0

Display of 16 byte HMAC-MD5 localized privKey:

axxxxxxxxxxxxxxxxxxxxxxxxxxxxxx5

---cut---

Make a note of the non-localized key value. E.g 5xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0 of the two above. Also ensure you make a note of the passwords of course. You _must_ use the passwords when adding the node to Solarwinds. Attempts to use the key instead met with failure, likely due to the need for the EngineID to be paired up with the key somehow (context field didn't help).

• Generate a new priv key with your local EngineID. Again you're only interested in the non-localized key value. I'll use axxxxxxxxxxxxxxxxxxxxxxxxxxxxxx7 below for this.

---cut---

pwtokey -e -u priv <priv password> $(cat /etc/snmpd.boots | cut -f2 -d' ')

---cut---

• Clear your command history if you're worried about maintaining the privacy of these keys (good habit but a touch paranoid!)

---cut---

> ~/.sh_history

---cut---

• Update the /etc/clsnmp.conf file so it has an entry for local snmp testing. I'm using swro aka Solar-Winds-Read-Only. I might dabble with read-write later and want them segregated. Plug in your freshly generated auth/priv keys.

---cut---

/etc/clsnmp.conf

#winSnmpName  targetAgent  admin  secName  password  context secLevel authProto  authKey                           privProto  privKey

#----------------------------------------------------------------------------------------------------------------------------------------------------------

swro          127.0.0.1    snmpv3 swro     -         -       AuthPriv HMAC-MD5   5xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0  DES axxxxxxxxxxxxxxxxxxxxxxxxxxxxxx7

---cut---

You can replace the first "swro" with the local server name if you like. In fact multiple duplicate lines with each server/IP (and consistant snmpd.boots & snmpdv3.conf files) will allow you to kick off clsnmp commands from this server to any other that has been updated with this process. Handy if you want to setup scripting to pull specific MIB/OID values etc from all servers etc.

• Update your snmpd.peers file to ensure it has the details required for the snmpd process to access other components (e.g hostmibd/snmpmibd etc) for specific MIB/OID resources.

---cut---

/etc/snmpd.peers

###############################################################################

#

"gated"     1.3.6.1.4.1.2.3.1.2.1.2     "gated_password"

"dpid2"     1.3.6.1.4.1.2.3.1.2.2.1.1.2 "dpid_password"

"muxatmd" 1.3.6.1.4.1.2.3.1.2.3.1.1 "muxatmd_password"

#

# Enables cpu & volume information visibility to snmpd

"xmtopas"       1.3.6.1.4.1.2.3.1.2.1.3 "xmtopas_pw"

#

## EOF

---cut---

The default AIX set of "passwords" is being used above (and in the following snmpdv3.conf) which should get you sorted. Sing out if you spot any issues with this approach as it depends on locking out non-local access to snmpd via the snmpdv3.conf file except for auth/priv key holders.

For example, I didn't have an entry here for xmtopas. Once I put that in place the SolarWinds discovered resources list suddenly included "Volume Utilization" values aka filesystem and logical volume info. Once selected they appear in the "Asset Inventory" tab under logical volumes.

• Update your /etc/snmpdv3.conf file with the one below. Swap out the auth/priv keys with the ones you generated above.

---cut---

##

## Solar Winds Specific Entries

##

#

USM_USER swro 00000002000000000A454172 HMAC-MD5 5xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0 DES axxxxxxxxxxxxxxxxxxxxxxxxxxxxxx7 N -

#

VACM_GROUP swrogrp - swro readonly

#

VACM_VIEW swroview internet                    - included -

VACM_VIEW swroview 1.3.6.1.4.1.2               - included -

VACM_VIEW swroview 1.3.6.1.4.1.2.2             - included -

VACM_VIEW swroview 1.3.6.1.4.1.2.3             - included -

VACM_VIEW swroview 1.3.6.1.4.1.2.5             - included -

VACM_VIEW swroview 1.3.6.1.4.1.2.6             - included -

VACM_VIEW swroview directory                   - included -

VACM_VIEW swroview mgmt                        - included -

VACM_VIEW swroview mib-2                       - included -

VACM_VIEW swroview system                      - included -

VACM_VIEW swroview aix                         - included -

VACM_VIEW swroview 1.3.6.1.4                   - included -

VACM_VIEW swroview 1.3.6.1.6                   - included -

VACM_VIEW swroview 1.3.6.1.6.3.1.1.5           - included -

VACM_VIEW swroview 1.3.6.1.4.1.2021            - included -

VACM_VIEW swroview 1.3.6.1.4.1.2.3.1.2.2.2.1.4 - included -

#

# Include snmpv3 managed MIBs with this view

VACM_VIEW swroview snmpModules                 - included -

# Include aixmibd managed MIBS with this view

VACM_VIEW swroview 1.3.6.1.4.1.2.6.191         - included -

#

VACM_ACCESS swrogrp - - AuthPriv - swroview - sworoview -

##

## AIX Internal SNMP Agent Specific Entries

##

#

# Allow localhost(only) SNMPv1 general access

COMMUNITY public public noAuthNoPriv 127.0.0.1 255.255.255.255  -

VACM_GROUP group1 SNMPv1  public  -

VACM_ACCESS  group1 - - noAuthNoPriv SNMPv1   defaultView - defaultView -

#

VACM_VIEW defaultView internet                    - included -

# Exclude snmpv3 related MIBs from the default view

VACM_VIEW defaultView snmpModules                 - excluded -

VACM_VIEW defaultView 1.3.6.1.6.3.1.1.4           - included -

VACM_VIEW defaultView 1.3.6.1.6.3.1.1.5           - included -

# Exclude aixmibd managed MIBS from this view

VACM_VIEW defaultView 1.3.6.1.4.1.2.6.191         - excluded -

#

# Access to data from gated/muxatmd/xmservd/dpid

smux   1.3.6.1.4.1.2.3.1.2.1.2   gated_password  # gated

smux   1.3.6.1.4.1.2.3.1.2.3.1.1   muxatmd_password #muxatmd

smux   1.3.6.1.4.1.2.3.1.2.1.3   xmservd_pw   #xmservd

smux   1.3.6.1.4.1.2.3.1.2.2.1.1.2   dpid_password   #dpid

#

#

# These entries appear to be for IBM director at a guess

#  They allow it to participate with the above

#VACM_GROUP director_group SNMPv2c public -

#VACM_ACCESS director_group - - noAuthNoPriv SNMPv2c defaultView - defaultView -

#

# Trap definitions

NOTIFY notify1 traptag trap -

TARGET_ADDRESS Target1 UDP 127.0.0.1 traptag trapparms1 - - -

TARGET_PARAMETERS trapparms1 SNMPv1  SNMPv1  public  noAuthNoPriv -

##

## Global Defaults

##

#

# Set no access unless explicitly allowed by previous entries

DEFAULT_SECURITY no-access - -

#

# Set log location, maximum size, log level

logging         file=/usr/tmp/snmpdv3.log       enabled

#logging         size=100000                     level=0

logging         size=100000                     level=2

#

## EOF

---cut---

Still a work in progress locking down the AIX Internal SNMP agents and of course getting the right set of MIB included in the Solar Winds view. At least now I've something working I can fine tune and will be looking at other related posts here.

• Stop all snmp related services

---cut---

stopsrc -s snmpmibd;stopsrc -s aixmibd;stopsrc -s snmpd;stopsrc -s hostmibd;stopsrc -s dpid2

---cut---

• Start all snmp related servers (excluding the now redundant dpid2)

---cut---

startsrc -s snmpmibd;startsrc -s aixmibd;startsrc -s snmpd;startsrc -s hostmibd

---cut---

• Test things out locally by using the walk option on clsnmp (yup, IBM included a cleverly disguised snmpwalk command). I'm using the "internet" MIB in this example, lots of output!

---cut---

clsnmp -h swro walk internet

---cut---

If you get an error here, odds are you've a mismatched EngineID and auth/priv keys. Check out your /usr/tmp/snmpdv3.log for more details. As mentioned earlier the "swro" here is a reference to the matching line in /etc/clsnmp.conf. A server name could be used here (if defined there) instead and would result in a remote test.

• Once the dust settles, turn down the log level in snmpdv3.conf (level=0) to avoid excessive logging for daily operation.

Solar Winds Configuration

It should now be possible to add this node in Solarwinds.

Key Values

• SNMP Version: SNMPv3
• SNMPv3 Username: swro
• SNMPv3 Authentication Method: MD5
• SNMPv3 Authentication Password: Use the password from pwtokey above (don't use the key, it doesn't work)
• SNMPv3 Privacy Method: DES56
• SNMPv3 Privacy Password: Use the password from pwtokey above (don't use the key, it doesn't work)

Hit the test button to see if all is ok. If it isn't make sure the services have been started and check the snmpdv3.log for more clues (especially the "did solar winds get here at all" clue).

As I've standardized the /etc/snmpd.boot and /etc/snmpdv3.conf files across the estate I can re-use these credentials and have saved them as "aix-swro".

Image may be NSFW.
Clik here to view.

Update 09/11/2015 - Post AIX upgrade (TL08/SP02 -> TL09/SP04) - Repair links to encrypted binaries

After the upgrade the links to the alternate, encrypted binaries for SNMP were reset to the default non-encrypted ones. This wasn't immediately apparent on Solarwinds until you try to "List Resources" for the node, which fails.

Solarwinds error: "<node> is currently down, unreachable, or provided credentials are not valid"

• Login as root on the target server
• Attempt an snmp walk to confirm this is the same problem

> clsnmp -h swro walk internet 1>/dev/null

Error reading file /etc/clsnmp.conf(Line 46):    Invalid securityLevel

clsnmp: 1473-406 Error converting destinationName swro to Entity.

• Check the current snmp binaries being linked to

> ls -la /usr/sbin/snmpd /usr/sbin/clsnmp

lrwxrwxrwx 1 root system 9 Oct 14 12:02 /usr/sbin/snmpd -> snmpdv3ne

lrwxrwxrwx 1 root system 9 Oct 14 12:02 /usr/sbin/clsnmp -> clsnmpne

• Change the links to the encrypted snmpd binary with the command:

> snmpv3_ssw -e

In /etc/rc.tcpip file, comment out the line that contains: dpid2

In /etc/rc.tcpip file, remove the comment from the line that contains: snmpmibd

Stop daemon: snmpdMake the symbolic link from /usr/sbin/snmpd to /usr/sbin/snmpdv3e

Make the symbolic link from /usr/sbin/clsnmp to /usr/sbin/clsnmpe

Start daemon: snmpd

• Check the linked binary again

> ls -la /usr/sbin/snmpd /usr/sbin/clsnmpe

lrwxrwxrwx 1 root system 18 Nov 09 14:49 /usr/sbin/snmpd -> /usr/sbin/snmpdv3e

lrwxrwxrwx 1 root system 18 Nov 09 14:49 /usr/sbin/snmpd -> /usr/sbin/clsnmpe

• On the solarwinds console, retry "list resources" on the node to confirm all is ok again

Sources

- IBM Doco

-- IBM SNMPv3 Documentation [http://www-01.ibm.com/support/knowledgecenter/api/content/ssw_aix_71/com.ibm.aix.networkcomm/snmpv3_intro.htm]

-- Creating users in SNMPv3 in AIX [http://www-01.ibm.com/support/knowledgecenter/api/content/ssw_aix_71/com.ibm.aix.networkcomm/HT_commadmn_create_snmpv3_user.htm#create_snmpv3_user]

-- AIX 7.1 snmpdv3.conf file setup [http://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.files/snmpdv3.conf.htm?lang=en]

-- IBM SNMP key generation [http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.commadmn/doc/commadmndita/snmpv3_genkeys.htm]

-- Using the clsnmp command [http://www-01.ibm.com/support/knowledgecenter/ssw_aix_53/com.ibm.aix.cmds/doc/aixcmds1/clsnmp.htm?cp=ssw_aix_53%2F1-2-0-2-122&lang=en]

--- Useful to test an snmp configuration as this can interrogate your snmp server to confirm its working ok.

- Solar Winds Doco

-- Solarwinds SNMPV3 Implementation Guide [http://www.solarwinds.com/support/Orion/docs/Implementing_SNMPv3r1.pdf]

-- Managing SNMP credentials guidelines [http://www.solarwinds.com/netperfmon/solarwinds/wwhelp/wwhimpl/js/html/wwhelp.htm]

- Solar Winds Forums

-- Example SNMPv3 Configuration on AIX from Solarwinds forum [https://thwack.solarwinds.com/message/249258]

-- Help with SNMPv3.1 on AIX [http://thwack.solarwinds.com/thread/42695]

-- Using SNMPv3.1 on AIX [http://thwack.solarwinds.com/thread/36507]

-- Configuring 3rd party Net-SNMP [http://thwack.solarwinds.com/thread/19323]

-- AIX Specific mods [http://thwack.solarwinds.com/community/application-and-server_tht/server-and-application-monitor/content?filterID=content~category[application-monitor-templates]&filterID=content~objecttype~objecttype[document]&query=unix]

- Make AIX a Solarwinds client on snmpv3 [http://odme.blogspot.com.au/2012/09/make-aix-solarwinds-client-on-snmpv3.html]

- Configuring AIX snmpd for MIB subagent access [http://odme.blogspot.com.au/2012/09/snmpdv3-wont-talk-to-mib-subagents.html]

-- The aixmbid, snmpmibd and similar subsystems depend on snmp to talk to each other! Here's the settings they use to do so.

-- Very detailed setup of the AIX snmpd for MIB subagent config including extra mib defaultviews [http://forums.cacti.net/viewtopic.php?t=19040]

- Nagios setup for snmp [http://nagios.frank4dd.com/howto/aix-snmp-setup.htm]

-- Examples on using snmpwalk

-- Example snmpdv3.conf for Nagios with highlights for nagios specifics

- [http://lparbox.com/how-to/powerha-cluster/21]

- [http://www.mcpressonline.com/system-administration/techtip-monitoring-aix-with-snmp.html]

- Using snmpwalk [http://www.net-snmp.org/tutorial/tutorial-5/commands/snmpv3.html]

Here are some SQL queries as doing weekly Solarwinds Maintenance and creating custom alerts that we as an MSP can't live without. I will most likely update this often but here are some I store in my evernote for now but I will update this frequently when i start looking through everything I use.

This first one isn't a SQL query but good for everyone use FoE to know:

Checking Ghost or Unknown Interfaces

selectInterfaceName,Nodes.CaptionfromInterfaces

I ran into an issue where every hyperlink in Solarwinds was working except license details. The issue turned out to be that in the recent past, the customer had renamed the Solarwinds server, and the old hostname never got removed from the allengines table. After removing the old entry in allengines, the license details screen was immediately accessible.

        -DaveB7114  

    Loop1 Systems: SolarWinds Training and Professional Services      

•                 LinkedIN: Loop1 Systems          
•                 Facebook: Loop1 Systems          
•                 Twitter: @Loop1Systems