Turbogears – Small Dropbear

September 17, 2013Craig Blog 2 Comments

jqGrid in TurboGears2 Admin Screens

I wanted to use the jqGrid for my admin pages as I liked that look and it matches some of the other screens outside the admin controller. The admin controller, or rather the CrudRestController out of tgext.crud, has a way of exporting results in json format. So surely its a matter of changing the TableBase to use jqGrid in the admin controller and we’re done?

Well, no.

First you need to adjust the jsonReader options so that it lines up to the field names that the controller sends and this was one of the first (or last snags). The json output looks like:

{
  "value_list": {
    "total": 20,
    "items_per_page": 7,
    "page": 1,
    "entries": [(lots of entries)...]
  }
}

Now this is a little odd because of the top-level dictionary that is being used here. Most of the examples have everything that is inside the value_list being returned. In fact adjusting the controller to return only those items in the value_list values works.

To look inside this structure we need to adjust the jsonReader options. jqGrid documentation uses the format “toptier>subtier” for the XML reader so that was the intial attempt. It was also an intial fail, it doesn’t work and you get your very familiar empty grid.

The trick is close to this format, but slightly different for json. You change the options to “toptier.subtier”. In other words change the greater than symbol to a full stop for json access.

The jqGridWidget now has the following options (amongst others):

options = {
  'datatype': 'json',
  'jsonReader': {
    'repeatitems': False,
    'root': 'value_list.entries',
    'total': 'value_list.total',
    'page': 'value_list.page',
  }
}

There might be a way of saying all entries sit under value_list inside jqGrid, but I couldn’t find it. Those options given above do give a working jqGrid on the admin screens.

July 18, 2013Craig Blog 1 Comment

Step or Sloping Graphs

Even though the backend of Rosenberg NMS uses rrdtool RRD files, the front end graphs are created using jqPlot. The idea is to have a set of templates for the different types of graphs and just apply them to the various data sets. It makes things a lot simpler for new graphs because you just select which one you want; unless you want something a lot different which would involve a new graph template.

In any case, anyone that looks enough at the standard rrdtool graphs will know they are a series of steps. While it depends on the RRA, usually they are 5 minute steps, so a graph showing an increasing rate might show 5 minutes of 2 Mbps and then the next 5 minutes of 11 Mbps. jqPlot graphs as I’ve currently got them draws a line between two data points, so there would be a sloping line starting at 2.5 minutes (half of the first 5 minute interval) and 2Mbps sloping up to 11 Mbps at 7.5 minutes.

At first I thought this was wrong and spent some time attempting to “fix” the graph by making it look like a rrdtool graph more. Someway through that process I stopped and wondered, what IS the right way? The answer like a lot of other things, is “it depends”.

For a graph showing a rate, such as the output bits per second on an interface, the way this is done is at regular intervals a counter is measured. So if at time 0 the counter is 140 and time 60 the counter is 200 and finally at time 120 the counter is at 800 there has been an average rate of 1 [(200-140)/60] and 100 [(800-200)/60)]. rrdtool would show a horizontal bar at 1 bps and then another horizontal bar at 100 at the next time interval. jqPlot would show a sloping line going from 30,1 up to 90,100.

Two graphs looking very different from the same data, what gives? Each graph is right, or rather is showing an estimation of different things, hence their differences.

rrdtool is showing the average rate for that time period. It is in some ways accurate because leaving aside missed wrap-arounds and resets that many bits did pass through that interface for the time specified. However often the graphs are interpreted incorrectly as the real rate and not an average. We can be reasonably sure for a rate that it would not be 1 for exactly a minute and then immediately jump to 100 for another minute. This isn’t rrdtool’s fault, it is just how the graphs can be interpreted.

jqplot will show more “realistic” graphs, with a curve sloping up. However this too makes assumptions that the rate increase is linear which often it is not. It just gives the illusion that the graph knows more about the data than it really does.

In the end, both graphs are at the same time accurate and misleading. It’s important when looking at any graph in general (not just these two types) that you understand its limits and assumptions. To give one example of the problems that can be missed, traffic interfaces may have microbursts (large amount of traffic in short amount of time) which, due to the averaging that goes on in graphing are invisible to graphs and give an incorrect account of what is going on.

June 20, 2013Craig Blog 1 Comment

jqplot in Turbogears

I’ve been working on the Rosenberg NMS graphs slowly migrating them from using rrdtool graph and using jqplot. While there have been many false-starts and re-works, I now have a working set of graphs, two of which are shown on the page.

The graphs look a lot slicker and I have also simplified the admin screens. I found I kept having to type the same thing in over and over for the rrdgraphs and have narrowed down the type of graphs to approximately 5. They’re certainly not bulletproof and need more testing but they’re a good start.

The graphs are based upon the ToscaWidgets2 series of widgets that then provide a nice “handle” for the jqPlot code. My graphs even have some hand-coded Javascript to give nice units on the Y axis.

TurboGears Just Keeps Getting Better (programming.oreilly.com)

May 15, 2013Craig Blog 3 Comments

itools is back

My last post I said that I had to remove my internet query tools due to some bugs that were a concern. Some of the code was hard to maintain and probably had holes and I had noticed that it looped at times.

I’m happy to say that I have restored some of those tools now, still located at http://enc.com.au/itools

This code is completely re-written in Python using the TurboGears toolkit which means it is a lot cleaner in how it works and how it looks. Some of the lookup tables use a database rather than an array for ease of updating and querying. The downside is the backends will take time. It currently only does nslookup queries and whois only works for IPv4 addresses. The domain name queries will be a while off as these are the most complicated to handle. To give you an idea, all IPv4 and IPv6 address information comes from 5 sources with two formats while domain names come from over 200 sources with about 40 formats. This means the information from Regional Internet Registries will be done first.

April 13, 2013Craig Blog 3 Comments

Pie Charts in TurboGears

You might of looked at Ralph Bean’s tutorial on graphs and thought, that’s nice but I’d like something different. The great thing about ToscaWidgets using the jqPlot library is that pretty much anything you can do in jqPlot, you can do in ToscaWidgets and by extension in TurboGears. You want different graph types? jqPlot has heaps!

My little application needed a Pie Chart to display the overall status of attributes. There are 6 states an attribute can be in: Up, Alert, Down, Testing, Admin Down and Unknown. The Pie Chart would show them all with some sort of colour representing the status. For this example I’ve used hard-coded data but you would normally extract it from the database using a TurboGears model and possibly some bucket sorting code.

I’ve divided my code into widget specific, which is found in myapp/widgets/attribute.py and the controller code found unsurprisingly at myapp/controllers/attribute.py Also note that some versions of ToscaWidgets have a bug which means any jqPlot widget won’t work, version 2.0.4 has the fix for issue 80 that explains briefly the bug.

The widget code looks like this:

from tw2.jqplugins.jqplot import JQPlotWidget
from tw2.jqplugins.jqplot.base import pieRenderer_js
import tw2.core as twc

class AttributeStatusPie(JQPlotWidget):
    """
    Pie Chart of the Attributes' Status """
    id = 'attribute-status-pie'
    resources = JQPlotWidget.resources + [
            pieRenderer_js,
            ]

    options = {
            'seriesColors': [ "#468847", "#F89406", "#B94A48", "#999999", "#3887AD", "#222222"],
            'seriesDefaults' : {
                'renderer': twc.js_symbol('$.jqplot.PieRenderer'),
                },
            'legend': {
                'show': True,
                'location': 'e',
                },
            }

Some important things to note are:

resources are the way of pulling in the javascript includes that actually do the work, generally if you have something like a renderer using js_symbol further on, it needs to be listed in the resources too.
seriesColors is how you make a specific data item a specific colour, or perhaps change the range of colours. It’s not required if you use the default set, which is defined in the jqPlot options.
The renderer tells jqPlot what sort of graph we want, the line above says we want pie

Next the controller needs to be defined:

from myapp.widgets.attribute import AttributeStatusPie

@expose('myapp.templates.widget')
    def statuspie(self):
        data = [[
                ['Up', 20], ['Alert', 7], ['Down', 12], ['Admin Down', 3], ['Testing', 1], ['Unknown', 4],
                ]]
        pie = AttributeStatusPie(data=data)
        return dict(w=pie)

And that is about it, we now have a controller path attributes/statuspie which shows us the pie chart.
My template is basically a bare template with a ${w.display | n} in it to just show the widget for testing.

Related articles