Depending on the application, sometimes you want to have some socket operations going (such as loading a website) and have stdin being read. There are plenty of examples for this in python which usually boil down to making stdin behave like a socket and mixing it into the list of sockets select() cares about.
A while ago I asked an email list could I have pysnmp use a different socket map so I could add my own sockets in (UDP, TCP and a zmq to name a few) and the Ilya the author of pysnmp explained how pysnmp can use a foreign socket map. (more…)
While the website might of been idle there has been a great deal of work behind the scenes (unless you’re looking at github) updating and improving RoseNMS, the python based Network Management System. The code has had a great many changes including:
Simplifying the admin back to TG standard using the Twitter Bootstrap option
Making several GUI changes to make it easier to switch screens or get more information
Update to the pysnmp module to the latest, which is a much better module
Changing from paste to cliff for the command lines and bringing it all into one binary
I’m hoping to get to release one in the next few months. The back-end is largely complete with some work required on the daemon and also more webGUI work to make sure its easy to get the right information at the right time.
Hmm, bring up a python interpreter and type the last line in with the imports. Works fine.
It’s the first line that is the problem, I typoed it and made it utf-7 not utf-8. I suppose it means that it is case-insenstive. Still, it wasn’t too clear to me at least, what was going on.
Even though the backend of Rosenberg NMS uses rrdtool RRD files, the front end graphs are created using jqPlot. The idea is to have a set of templates for the different types of graphs and just apply them to the various data sets. It makes things a lot simpler for new graphs because you just select which one you want; unless you want something a lot different which would involve a new graph template.
In any case, anyone that looks enough at the standard rrdtool graphs will know they are a series of steps. While it depends on the RRA, usually they are 5 minute steps, so a graph showing an increasing rate might show 5 minutes of 2 Mbps and then the next 5 minutes of 11 Mbps. jqPlot graphs as I’ve currently got them draws a line between two data points, so there would be a sloping line starting at 2.5 minutes (half of the first 5 minute interval) and 2Mbps sloping up to 11 Mbps at 7.5 minutes.
At first I thought this was wrong and spent some time attempting to “fix” the graph by making it look like a rrdtool graph more. Someway through that process I stopped and wondered, what IS the right way? The answer like a lot of other things, is “it depends”.
For a graph showing a rate, such as the output bits per second on an interface, the way this is done is at regular intervals a counter is measured. So if at time 0 the counter is 140 and time 60 the counter is 200 and finally at time 120 the counter is at 800 there has been an average rate of 1 [(200-140)/60] and 100 [(800-200)/60)]. rrdtool would show a horizontal bar at 1 bps and then another horizontal bar at 100 at the next time interval. jqPlot would show a sloping line going from 30,1 up to 90,100.
Two graphs looking very different from the same data, what gives? Each graph is right, or rather is showing an estimation of different things, hence their differences.
rrdtool is showing the average rate for that time period. It is in some ways accurate because leaving aside missed wrap-arounds and resets that many bits did pass through that interface for the time specified. However often the graphs are interpreted incorrectly as the real rate and not an average. We can be reasonably sure for a rate that it would not be 1 for exactly a minute and then immediately jump to 100 for another minute. This isn’t rrdtool’s fault, it is just how the graphs can be interpreted.
jqplot will show more “realistic” graphs, with a curve sloping up. However this too makes assumptions that the rate increase is linear which often it is not. It just gives the illusion that the graph knows more about the data than it really does.
In the end, both graphs are at the same time accurate and misleading. It’s important when looking at any graph in general (not just these two types) that you understand its limits and assumptions. To give one example of the problems that can be missed, traffic interfaces may have microbursts (large amount of traffic in short amount of time) which, due to the averaging that goes on in graphing are invisible to graphs and give an incorrect account of what is going on.
The code is still pre-alpha but it is now available to try out. This means you can try out this new network management system. I have done this to hopefully shake out some of the bugs before the first release. The program has minimal documentation but does work, kinda.
You might of looked at Ralph Bean’s tutorial on graphs and thought, that’s nice but I’d like something different. The great thing about ToscaWidgets using the jqPlot library is that pretty much anything you can do in jqPlot, you can do in ToscaWidgets and by extension in TurboGears. You want different graph types? jqPlot has heaps!
My little application needed a Pie Chart to display the overall status of attributes. There are 6 states an attribute can be in: Up, Alert, Down, Testing, Admin Down and Unknown. The Pie Chart would show them all with some sort of colour representing the status. For this example I’ve used hard-coded data but you would normally extract it from the database using a TurboGears model and possibly some bucket sorting code.
I’ve divided my code into widget specific, which is found in myapp/widgets/attribute.py and the controller code found unsurprisingly at myapp/controllers/attribute.py Also note that some versions of ToscaWidgets have a bug which means any jqPlot widget won’t work, version 2.0.4 has the fix for issue 80 that explains briefly the bug.
The widget code looks like this:
from tw2.jqplugins.jqplot import JQPlotWidget
from tw2.jqplugins.jqplot.base import pieRenderer_js
import tw2.core as twc
class AttributeStatusPie(JQPlotWidget):
"""
Pie Chart of the Attributes' Status """
id = 'attribute-status-pie'
resources = JQPlotWidget.resources + [
pieRenderer_js,
]
options = {
'seriesColors': [ "#468847", "#F89406", "#B94A48", "#999999", "#3887AD", "#222222"],
'seriesDefaults' : {
'renderer': twc.js_symbol('$.jqplot.PieRenderer'),
},
'legend': {
'show': True,
'location': 'e',
},
}
Some important things to note are:
resources are the way of pulling in the javascript includes that actually do the work, generally if you have something like a renderer using js_symbol further on, it needs to be listed in the resources too.
seriesColors is how you make a specific data item a specific colour, or perhaps change the range of colours. It’s not required if you use the default set, which is defined in the jqPlot options.
The renderer tells jqPlot what sort of graph we want, the line above says we want pie
Next the controller needs to be defined:
from myapp.widgets.attribute import AttributeStatusPie
@expose('myapp.templates.widget')
def statuspie(self):
data = [[
['Up', 20], ['Alert', 7], ['Down', 12], ['Admin Down', 3], ['Testing', 1], ['Unknown', 4],
]]
pie = AttributeStatusPie(data=data)
return dict(w=pie)
And that is about it, we now have a controller path attributes/statuspie which shows us the pie chart.
My template is basically a bare template with a ${w.display | n} in it to just show the widget for testing.
RRDTool is a neat utility for collecting and graphing statistics such as server loads or network traffic. There are two main modules for interfacing with RRDTool files within python; rrdpython and pyrrd.
rrdpython is the basic bindings of the rrdtool library within python. The API is very familiar for people who program in C or use the command line tools which for me is both so it works well. However if you were expecting a “pythonic” API you will be disappointed. As there is the direct binding, you have to have either a pre-compiled module or compile it yourself with librrd-dev package installed. Depending on your setup this could be trivial or a real pain.
Well, the problem is its not pure python. The hooks are there for it (in backend.py) for it to be implemented in python but it falls back to using the external method, and external method is a bunch of Popen calls. pyrrd also does not
support the full set of rrdtool commands.
My intention is to keep using rrdpython despite the compile hassle and possibly even use a fancier graph setup such as High Charts though there is a problem with their license for me.
My Turbogears project has recently reached an important milestone; one of the back-end processes now runs pretty much continuously and the plugins it use (or at least the ones I can see) are also working. That means I can turn to the front-end which displays the data the back-end collected.
For some of the data I am using a ToscaWidgets (or TW2) widget called a jqGridWidget which is a very nice jquery device that separates the presentation and data using a json query. I’ve mentioned previously about how to get a jqGridWidget working but left the pre-filtering out until now. This meant that my grid showed all the items in the database, not just the ones I wanted to see, but it was a start.
Now this widget displays things called Attributes which are basically children of another model called Hosts. Basically, Attributes are things you want to check or track about a Host. My widget used to show all Attributes but often on a Host screen, I want to see its hosts only. So, this is how I got my widget to behave; I’m not sure this is THE CORRECT way of doing it, but it does work.
First, in the Hosts controller you need to create the widget and pass along the host_id that the controller has obtained. I was not able to use the sub-classing trick you see in some TW2 documentation but actually make a widget instance.
class HostsController(BaseController):
# other stuff here
class por2(portlets.Portlet):
title = 'Host Attributes'
widget = AttributeGrid()
widget.host_id = host_id
Next, the prepare() method in the Widget needs to get hold of the host_id and put it into the postData list. I needed to do it before calling super() because the options become one long string in the sub-class.
class AttributeGrid(jqgrid.jqGridWidget):
def prepare(self, **kw):
if self.host_id is not None:
self.options['postData'] = {'hostid': self.host_id}
super(AttributeGrid, self).prepare()
This means the jqgrid when it asks for its JSON data will include the hostid parameter as well. We need that method to “see” the host ID so we can filter the database access.
Finally in the JSON method for the Attribute we pick up and filter on the hostid.
Turbogears 2 uses Toscawidgets 2 for a series of very clever widgets and base objects that you can use in your projects. One I have been playing with is the jqGridWidget which uses jquery to display a grid. The idea with jquery is creating a grid or other object and then using javascript to call back to the webserver to obtain the data which usually comes from a database using the very excellent SQLAlchemy framework.
I found it a bit fiddly to get going and it is unforgiving, you get one thing wrong and you end up with an empty grid but when it works, it works very well.
from tw2.jqplugins.jqgrid import jqGridWidget
from tw2.jqplugins.jqgrid.base import word_wrap_css
class ItemsWidget(jqGridWidget):
def prepare(self):
self.resources.append(word_wrap_css)
super(ItemsWidget, self).prepare()
options = {
'pager': 'item-list-pager2',
'url': '/items/jqgrid',
'datatype': 'json',
'colNames': ['Date', 'Type', 'Area', 'Description'],
The code above is a snippet of the widget I use to make the graph. This is the visual side of the graph and shows just the graph with nothing in it. You need the datatype and url lines in the options which tell the code the url to get the data. I have an items controller and within it there is a jqgrid method. Note the number of columns I have, the data sent by the jqgrid method has to match this.
Deep inside the items controller is a jqgrid method. The idea is to return some rows of data back to the grid in the exact way the grid expects it otherwise you get, yep more empty grids.
The four items in the value for the cell in the dictionary must equal the number defined for the colNames. I found it was useful to have firebug looking at my gets and another widget based upon SQLAjqGridWidget which is a lot simpler to setup but much less flexible too.
I have had an opportunity to rework some code to query NTP servers directly in python rather than running ntpq -pn and scraping the result. The over-the-wire protocol format is reasonably straight forward and my little module is now passing all of its nosetests which is wonderful.
While its only a single table, ntpq actually does multiple requests to get its results. NTP control messages are all mode 6 which tells the client and server its not time sort of things, but more about queries. A control packet has a sequence number which the server responds with so you can line up your requests with responses. For large responses, the more, offset and count fields work together to knit up a large block of data. The first packet of the response will have the offset field of 0. The count is the size of the data payload in bytes. If the response is too big, the more bit will be set to 1 to say there is more coming. The second packet will have an offset that is the same value as the previous packets count value, so you can stitch them together.
The first query is a READSTAT which uses opcode 1. This returns a list of 4 byte responses with the association ID taking the first two bytes and the peer status taking the second. If you look at the output of ntpq you will notice a character in the first column which is the peer selection. The important one for us is asterix, which equates to 110 in the lower 3 bits of the first status byte. This is the system peer which is the main one. There may be others, up to 2, (usually symbol +) which are compared to the system peer. ntpq doesn’t really care about the different peers and just displays the status but I want to know what peer the server is synchronised to.
Next we need more details about the system peer (or all peers in ntpq). To do this a READVAR or opcode 2 is sent to the server. The association ID is set to the association or peer we are interested in. The response for my setup is sent over two packets which need to be reassembled and it is all in plain ascii which is a comma-separated list with a key=value pair. I use the following python line to split it up.
self.assoc_data.update(dict([tuple(ad.strip().split("=")) for ad in data[header_len:].split(",") if ad.find("=") > -1]))
I’m sure people that know python better than I will say I am doing it wrong (it looks a bit klunky to me too) but it does work. From that I can query the various values that are returned by the server about its peers, including the source of its clock.
