MonAMI

Gidon Moont (of the 3D Real-Time Monitor fame) has come up with another monitoring tool. Using the data collected from all the WLCG Resource Brokers, graphs are generated that plot the number of jobs each VO has running and queued at your site. ... [More] More information is available at the Real Time Monitoring page (the "GridLoad Graphs" section).What's particularly nice is he's included support for Google Gadgets. Gadgets, if you've not come across them, are a small bit of web content wrapped up so they're easy to handle. You can add Gadgets to iGoogle, to your desktop or even within to your webpages.MonAMI includes a framework that (amongst other things) extends Ganglia's default web front-end to include support for Gadgets (e.g. Glasgow's Torque monitoring).So, with Google Gadgets, you can see your local batch system monitoring side-by-side with a per-VO view of what the Resource Brokers think your site is up to. [Less]

Gidon Moont (of the 3D Real-Time Monitor fame) has come up with another monitoring tool. Using the data collected from all the WLCG Resource Brokers, graphs are generated that plot the number of jobs each VO has running and queued at your site. ... [More] More information is available at the Real Time Monitoring page (the "GridLoad Graphs" section).What's particularly nice is he's included support for Google Gadgets. Gadgets, if you've not come across them, are a small bit of web content wrapped up so they're easy to handle. You can add Gadgets to iGoogle, to your desktop or even within to your webpages.MonAMI includes a framework that (amongst other things) extends Ganglia's default web front-end to include support for Gadgets (e.g. Glasgow's Torque monitoring).So, with Google Gadgets, you can see your local batch system monitoring side-by-side with a per-VO view of what the Resource Brokers think your site is up to. [Less]

Gidon Moont (of the 3D Real-Time Monitor fame) has come up with another monitoring tool. Using the data collected from all the WLCG Resource Brokers, graphs are generated that plot the number of jobs each VO has running and queued at your site. ... [More] More information is available at the Real Time Monitoring page (the "GridLoad Graphs" section).What's particularly nice is he's included support for Google Gadgets. Gadgets, if you've not come across them, are a small bit of web content wrapped up so they're easy to handle. You can add Gadgets to iGoogle, to your desktop or even within to your webpages.MonAMI includes a framework that (amongst other things) extends Ganglia's default web front-end to include support for Gadgets (e.g. Glasgow's Torque monitoring).So, with Google Gadgets, you can see your local batch system monitoring side-by-side with a per-VO view of what the Resource Brokers think your site is up to. [Less]

Yesterday, I helped Phil install MonAMI on the Durham CE and update his Ganglia web front-end so it now has the nice graphs.However, we hit a snag: a few of the metrics "disappear" every so often. This is most likely happening because gmond is ... [More] loosing the UDP (multicast) metric update messages. After "a while" (the DMAX value), gmond assumes that the metric is no longer being monitored and purges it. The purged metrics no longer have their data written to the RRD file by gmetad, leaving a gap in the graph.When we encountered this with Glasgow it was caused by incoming UDP packets overflowing gmond's network buffer. The ganglia MonAMI plugin has a work-around: every 200 packets it will pause "a while" (100 ms by default). Looks like this isn't enough for Durham.The long term solution is for someone to fix gmond: it should be multithreaded (to stop gmetad downloads from blocking metric updates) or for it to accept data using a reliable transport (e.g. TCP). [Less]

Yesterday, I helped Phil install MonAMI on the Durham CE and update his Ganglia web front-end so it now has the nice graphs.However, we hit a snag: a few of the metrics "disappear" every so often. This is most likely happening because gmond is ... [More] loosing the UDP (multicast) metric update messages. After "a while" (the DMAX value), gmond assumes that the metric is no longer being monitored and purges it. The purged metrics no longer have their data written to the RRD file by gmetad, leaving a gap in the graph.When we encountered this with Glasgow it was caused by incoming UDP packets overflowing gmond's network buffer. The ganglia MonAMI plugin has a work-around: every 200 packets it will pause "a while" (100 ms by default). Looks like this isn't enough for Durham.The long term solution is for someone to fix gmond: it should be multithreaded (to stop gmetad downloads from blocking metric updates) or for it to accept data using a reliable transport (e.g. TCP). [Less]

Yesterday, I helped Phil install MonAMI on the Durham CE and update his Ganglia web front-end so it now has the nice graphs.However, we hit a snag: a few of the metrics "disappear" every so often. This is most likely happening because gmond is ... [More] loosing the UDP (multicast) metric update messages. After "a while" (the DMAX value), gmond assumes that the metric is no longer being monitored and purges it. The purged metrics no longer have their data written to the RRD file by gmetad, leaving a gap in the graph.When we encountered this with Glasgow it was caused by incoming UDP packets overflowing gmond's network buffer. The ganglia MonAMI plugin has a work-around: every 200 packets it will pause "a while" (100 ms by default). Looks like this isn't enough for Durham.The long term solution is for someone to fix gmond: it should be multithreaded (to stop gmetad downloads from blocking metric updates) or for it to accept data using a reliable transport (e.g. TCP). [Less]

Yesterday, I helped Phil install MonAMI on the Durham CE and update his Ganglia web front-end so it now has the nice graphs.However, we hit a snag: a few of the metrics "disappear" every so often. This is most likely happening because gmond is ... [More] loosing the UDP (multicast) metric update messages. After "a while" (the DMAX value), gmond assumes that the metric is no longer being monitored and purges it. The purged metrics no longer have their data written to the RRD file by gmetad, leaving a gap in the graph.When we encountered this with Glasgow it was caused by incoming UDP packets overflowing gmond's network buffer. The ganglia MonAMI plugin has a work-around: every 200 packets it will pause "a while" (100 ms by default). Looks like this isn't enough for Durham.The long term solution is for someone to fix gmond: it should be multithreaded (to stop gmetad downloads from blocking metric updates) or for it to accept data using a reliable transport (e.g. TCP). [Less]

How do you best configure Ganglia to work with MonAMI? What's a good Nagios configuration? MonAMI is designed to fit in with existing monitoring tools; but, sometimes those external tools need to be tweaked to get the best out of the available ... [More] data.External is a collection of scripts, configuration hints, and similar "useful stuff". It's material not for MonAMI, but rather for the programs MonAMI communicates with (hence "external").The current focus has been on getting decent graphs within Ganglia. External has a framework for building RRDTool graphs, pie charts, and frames of related information. It also includes a fair number of examples showing how to use the framework. The torque and maui frames are excellent examples: see the output from UKI-SCOTGRID-Glasgow.For now, external is available as a CVS module (browse, instructions).Enjoy! [Less]

How do you best configure Ganglia to work with MonAMI? What's a good Nagios configuration? MonAMI is designed to fit in with existing monitoring tools; but, sometimes those external tools need to be tweaked to get the best out of the available ... [More] data.External is a collection of scripts, configuration hints, and similar "useful stuff". It's material not for MonAMI, but rather for the programs MonAMI communicates with (hence "external").The current focus has been on getting decent graphs within Ganglia. External has a framework for building RRDTool graphs, pie charts, and frames of related information. It also includes a fair number of examples showing how to use the framework. The torque and maui frames are excellent examples: see the output from UKI-SCOTGRID-Glasgow.For now, external is available as a CVS module (browse, instructions).Enjoy! [Less]

How do you best configure Ganglia to work with MonAMI? What's a good Nagios configuration? MonAMI is designed to fit in with existing monitoring tools; but, sometimes those external tools need to be tweaked to get the best out of the available ... [More] data.External is a collection of scripts, configuration hints, and similar "useful stuff". It's material not for MonAMI, but rather for the programs MonAMI communicates with (hence "external").The current focus has been on getting decent graphs within Ganglia. External has a framework for building RRDTool graphs, pie charts, and frames of related information. It also includes a fair number of examples showing how to use the framework. The torque and maui frames are excellent examples: see the output from UKI-SCOTGRID-Glasgow.For now, external is available as a CVS module (browse, instructions).Enjoy! [Less]