While the website might of been idle there has been a great deal of work behind the scenes (unless you’re looking at github) updating and improving RoseNMS, the python based Network Management System. The code has had a great many changes including:
Simplifying the admin back to TG standard using the Twitter Bootstrap option
Making several GUI changes to make it easier to switch screens or get more information
Update to the pysnmp module to the latest, which is a much better module
Changing from paste to cliff for the command lines and bringing it all into one binary
I’m hoping to get to release one in the next few months. The back-end is largely complete with some work required on the daemon and also more webGUI work to make sure its easy to get the right information at the right time.
I wanted to use the jqGrid for my admin pages as I liked that look and it matches some of the other screens outside the admin controller. The admin controller, or rather the CrudRestController out of tgext.crud, has a way of exporting results in json format. So surely its a matter of changing the TableBase to use jqGrid in the admin controller and we’re done?
Well, no.
First you need to adjust the jsonReader options so that it lines up to the field names that the controller sends and this was one of the first (or last snags). The json output looks like:
Now this is a little odd because of the top-level dictionary that is being used here. Most of the examples have everything that is inside the value_list being returned. In fact adjusting the controller to return only those items in the value_list values works.
To look inside this structure we need to adjust the jsonReader options. jqGrid documentation uses the format “toptier>subtier” for the XML reader so that was the intial attempt. It was also an intial fail, it doesn’t work and you get your very familiar empty grid.
The trick is close to this format, but slightly different for json. You change the options to “toptier.subtier”. In other words change the greater than symbol to a full stop for json access.
The jqGridWidget now has the following options (amongst others):
There might be a way of saying all entries sit under value_list inside jqGrid, but I couldn’t find it. Those options given above do give a working jqGrid on the admin screens.
Even though the backend of Rosenberg NMS uses rrdtool RRD files, the front end graphs are created using jqPlot. The idea is to have a set of templates for the different types of graphs and just apply them to the various data sets. It makes things a lot simpler for new graphs because you just select which one you want; unless you want something a lot different which would involve a new graph template.
In any case, anyone that looks enough at the standard rrdtool graphs will know they are a series of steps. While it depends on the RRA, usually they are 5 minute steps, so a graph showing an increasing rate might show 5 minutes of 2 Mbps and then the next 5 minutes of 11 Mbps. jqPlot graphs as I’ve currently got them draws a line between two data points, so there would be a sloping line starting at 2.5 minutes (half of the first 5 minute interval) and 2Mbps sloping up to 11 Mbps at 7.5 minutes.
At first I thought this was wrong and spent some time attempting to “fix” the graph by making it look like a rrdtool graph more. Someway through that process I stopped and wondered, what IS the right way? The answer like a lot of other things, is “it depends”.
For a graph showing a rate, such as the output bits per second on an interface, the way this is done is at regular intervals a counter is measured. So if at time 0 the counter is 140 and time 60 the counter is 200 and finally at time 120 the counter is at 800 there has been an average rate of 1 [(200-140)/60] and 100 [(800-200)/60)]. rrdtool would show a horizontal bar at 1 bps and then another horizontal bar at 100 at the next time interval. jqPlot would show a sloping line going from 30,1 up to 90,100.
Two graphs looking very different from the same data, what gives? Each graph is right, or rather is showing an estimation of different things, hence their differences.
rrdtool is showing the average rate for that time period. It is in some ways accurate because leaving aside missed wrap-arounds and resets that many bits did pass through that interface for the time specified. However often the graphs are interpreted incorrectly as the real rate and not an average. We can be reasonably sure for a rate that it would not be 1 for exactly a minute and then immediately jump to 100 for another minute. This isn’t rrdtool’s fault, it is just how the graphs can be interpreted.
jqplot will show more “realistic” graphs, with a curve sloping up. However this too makes assumptions that the rate increase is linear which often it is not. It just gives the illusion that the graph knows more about the data than it really does.
In the end, both graphs are at the same time accurate and misleading. It’s important when looking at any graph in general (not just these two types) that you understand its limits and assumptions. To give one example of the problems that can be missed, traffic interfaces may have microbursts (large amount of traffic in short amount of time) which, due to the averaging that goes on in graphing are invisible to graphs and give an incorrect account of what is going on.
I’ve been working on the Rosenberg NMS graphs slowly migrating them from using rrdtool graph and using jqplot. While there have been many false-starts and re-works, I now have a working set of graphs, two of which are shown on the page.
The graphs look a lot slicker and I have also simplified the admin screens. I found I kept having to type the same thing in over and over for the rrdgraphs and have narrowed down the type of graphs to approximately 5. They’re certainly not bulletproof and need more testing but they’re a good start.
The graphs are based upon the ToscaWidgets2 series of widgets that then provide a nice “handle” for the jqPlot code. My graphs even have some hand-coded Javascript to give nice units on the Y axis.
You might of looked at Ralph Bean’s tutorial on graphs and thought, that’s nice but I’d like something different. The great thing about ToscaWidgets using the jqPlot library is that pretty much anything you can do in jqPlot, you can do in ToscaWidgets and by extension in TurboGears. You want different graph types? jqPlot has heaps!
My little application needed a Pie Chart to display the overall status of attributes. There are 6 states an attribute can be in: Up, Alert, Down, Testing, Admin Down and Unknown. The Pie Chart would show them all with some sort of colour representing the status. For this example I’ve used hard-coded data but you would normally extract it from the database using a TurboGears model and possibly some bucket sorting code.
I’ve divided my code into widget specific, which is found in myapp/widgets/attribute.py and the controller code found unsurprisingly at myapp/controllers/attribute.py Also note that some versions of ToscaWidgets have a bug which means any jqPlot widget won’t work, version 2.0.4 has the fix for issue 80 that explains briefly the bug.
The widget code looks like this:
from tw2.jqplugins.jqplot import JQPlotWidget
from tw2.jqplugins.jqplot.base import pieRenderer_js
import tw2.core as twc
class AttributeStatusPie(JQPlotWidget):
"""
Pie Chart of the Attributes' Status """
id = 'attribute-status-pie'
resources = JQPlotWidget.resources + [
pieRenderer_js,
]
options = {
'seriesColors': [ "#468847", "#F89406", "#B94A48", "#999999", "#3887AD", "#222222"],
'seriesDefaults' : {
'renderer': twc.js_symbol('$.jqplot.PieRenderer'),
},
'legend': {
'show': True,
'location': 'e',
},
}
Some important things to note are:
resources are the way of pulling in the javascript includes that actually do the work, generally if you have something like a renderer using js_symbol further on, it needs to be listed in the resources too.
seriesColors is how you make a specific data item a specific colour, or perhaps change the range of colours. It’s not required if you use the default set, which is defined in the jqPlot options.
The renderer tells jqPlot what sort of graph we want, the line above says we want pie
Next the controller needs to be defined:
from myapp.widgets.attribute import AttributeStatusPie
@expose('myapp.templates.widget')
def statuspie(self):
data = [[
['Up', 20], ['Alert', 7], ['Down', 12], ['Admin Down', 3], ['Testing', 1], ['Unknown', 4],
]]
pie = AttributeStatusPie(data=data)
return dict(w=pie)
And that is about it, we now have a controller path attributes/statuspie which shows us the pie chart.
My template is basically a bare template with a ${w.display | n} in it to just show the widget for testing.
I’ve been trying to tidy up my Mako templates within my Turbogears 2 project. As part of that I was looking at some of them that are quickstarted including one which is the About page.
What was curious was there was all this CSS work all done already for you, including icons. Digging further I found out that one of the many projects Turbogears takes in, is Bootstrap. It’s website describes Bootstrap as “Sleek, intuitive, and powerful front-end framework for faster and easier web development.” but what it means to me is a bunch of guys who understand HTML and CSS way better than me have made it easy for me to build a decent web application.
While the framework won’t suit everyone’s tastes, it makes a lot of the formatting decisions so much easier. The thing is, in all the Turbogears documentation I have read I’ve not heard it mentioned. Not sure why, its pretty awesome (not SQLAlchemy awesome but not many things are).
My Turbogears project has recently reached an important milestone; one of the back-end processes now runs pretty much continuously and the plugins it use (or at least the ones I can see) are also working. That means I can turn to the front-end which displays the data the back-end collected.
For some of the data I am using a ToscaWidgets (or TW2) widget called a jqGridWidget which is a very nice jquery device that separates the presentation and data using a json query. I’ve mentioned previously about how to get a jqGridWidget working but left the pre-filtering out until now. This meant that my grid showed all the items in the database, not just the ones I wanted to see, but it was a start.
Now this widget displays things called Attributes which are basically children of another model called Hosts. Basically, Attributes are things you want to check or track about a Host. My widget used to show all Attributes but often on a Host screen, I want to see its hosts only. So, this is how I got my widget to behave; I’m not sure this is THE CORRECT way of doing it, but it does work.
First, in the Hosts controller you need to create the widget and pass along the host_id that the controller has obtained. I was not able to use the sub-classing trick you see in some TW2 documentation but actually make a widget instance.
class HostsController(BaseController):
# other stuff here
class por2(portlets.Portlet):
title = 'Host Attributes'
widget = AttributeGrid()
widget.host_id = host_id
Next, the prepare() method in the Widget needs to get hold of the host_id and put it into the postData list. I needed to do it before calling super() because the options become one long string in the sub-class.
class AttributeGrid(jqgrid.jqGridWidget):
def prepare(self, **kw):
if self.host_id is not None:
self.options['postData'] = {'hostid': self.host_id}
super(AttributeGrid, self).prepare()
This means the jqgrid when it asks for its JSON data will include the hostid parameter as well. We need that method to “see” the host ID so we can filter the database access.
Finally in the JSON method for the Attribute we pick up and filter on the hostid.
Turbogears 2 uses Toscawidgets 2 for a series of very clever widgets and base objects that you can use in your projects. One I have been playing with is the jqGridWidget which uses jquery to display a grid. The idea with jquery is creating a grid or other object and then using javascript to call back to the webserver to obtain the data which usually comes from a database using the very excellent SQLAlchemy framework.
I found it a bit fiddly to get going and it is unforgiving, you get one thing wrong and you end up with an empty grid but when it works, it works very well.
from tw2.jqplugins.jqgrid import jqGridWidget
from tw2.jqplugins.jqgrid.base import word_wrap_css
class ItemsWidget(jqGridWidget):
def prepare(self):
self.resources.append(word_wrap_css)
super(ItemsWidget, self).prepare()
options = {
'pager': 'item-list-pager2',
'url': '/items/jqgrid',
'datatype': 'json',
'colNames': ['Date', 'Type', 'Area', 'Description'],
The code above is a snippet of the widget I use to make the graph. This is the visual side of the graph and shows just the graph with nothing in it. You need the datatype and url lines in the options which tell the code the url to get the data. I have an items controller and within it there is a jqgrid method. Note the number of columns I have, the data sent by the jqgrid method has to match this.
Deep inside the items controller is a jqgrid method. The idea is to return some rows of data back to the grid in the exact way the grid expects it otherwise you get, yep more empty grids.
The four items in the value for the cell in the dictionary must equal the number defined for the colNames. I found it was useful to have firebug looking at my gets and another widget based upon SQLAjqGridWidget which is a lot simpler to setup but much less flexible too.
When the programming language Python came out while it seemed like an interesting idea, I didn’t really see the point. I already know Perl, PHP and C (plus a few others) why learn another language? So until recently, I didn’t
Recent frustrations with PHP and how “loose” it is with things like variables made me have another look at Python and what it could do for me. As just learning a language by itself it pretty boring I decided to write some small programs in the language and used the framework Turbogears to do it.
First of all, learning the basics of python from perl or PHP is dead simple. The idea of what a language looks like is not too different between these three so it was a snap to write simple stuff in python. So, what does yet-another language give me?
Variables must be defined before being used, but assigning them defines them too. This is a nice compromise between PHP where you can check undefined variables and C where it all must be defined first, or else.
try and except. For code that might fail, use try and then catch the error in except. We’ve all been guilty of using the @ symbol in PHP to supress stupid error messages, this does it correctly.
No PHP extract() or compact(). Actually just no extract function is good enough. That function is pure evil.
Sqlalchemy – A proper “you look after the database and ill play with objects” database extraction layer thingamy. I don’t want to play with databases just let me write code and thats what sqlalchemy does well.
Testing – Turbogears uses nosetests which does functional and unit testing. I do get some annoying artifacts based upon testing but generally its all good.
Python has help() everywhere and pydoc on the command line. Not sure what variable foo can do? Just type “help(foo)” or even stick it in your script.
paster! You can run a development site off the command-line using a simple sqlite database. Don’t like it, dump the DB and reinstall it with one command.
It’s not one-way though, there have been some challenges. Quite likely these are more my deficiencies than the languages.
Error messages take up a whole screen and often make it difficult to isolate. It’s not quite 10-screens-full ala Eiffel but sometimes the error messages tell you something is going wrong, but not what. Mess up your object versus database table references and you get something rather obscure.
When is an integer not an integer? The answer is, sometimes. Sometimes they work, sometimes they break. It’s probably more to do with what pysnmp is doing with them.
Documentation – some is great, some is downright misleading. It comes down to the writer of the documentation for modules. At least it doesn’t have ‘well get round to it’ like some functions in PHP on their website. Documents that say you can directly iterate dictionaries were another dead-end I wandered down. If you are stuck here, add a .items() at the end of it.
Deployment. It’s not just a matter of taring the files up and sticking them into /var/www/sitename and you’re done. There are probably good reasons why, but its just annoying.
Despite the minor problems, python really does give you much much more than PHP and some more than Perl. For me and what I am intending on writing it is worth learning yet another language.
Well JFFNMS 0.9.0 was a, well, its a good example of what can go wrong without adequate testing. Having it written in PHP makes it difficult to test, because you can have entire globs of code that are completely wrong but are not activated (because the “if” statement proves false for example) and you won’t get an error.
It also had some database migration problems. This is the most difficult and annoying part of releasing versions of JFFNMS. There are so many things to check, like:
What has changed between this version of the database?
Does the development database have all the new fields?
Will importing an old database and using the diffs give me the same result as importing the new database
Does the structure field match the object which matches the database table
I’ve been looking at sqlalchemy which is part of turbogears. It’s a pretty impressive setup and lets you get on with writing your application and not mucking around with the low-level stuff. It’s a bit of a steep learning curve learning python AND sqlalchemy AND turbogears but I’ve got some rudimentary code running ok and its certainly worth it (python erroring on un-assigned varables but not forcing you to define them is a great compromise). The best thing is that you can develop on a sqlite database but deploy using mysql or postgresql with a minor change.
Python and turbogears both emphasise automatic testing. Ideally the testing should cover all the code you write. The authors even suggest you write the test first then implement the feature. After chasing down several bugs, some of which I introduced fixing other bugs, automatic testing would make my life a lot easier and perhaps I wouldn’t dread the release cycle so often.
