NTP control messages

I have had an opportunity to rework some code to query NTP servers directly in python rather than running ntpq -pn and scraping the result.  The over-the-wire protocol format is reasonably straight forward and my little module is now passing all of its nosetests which is wonderful.

While its only a single table, ntpq actually does multiple requests to get its results.  NTP control messages are all mode 6 which tells the client and server its not time sort of things, but more about queries. A control packet has a sequence number which the server responds with so you can line up your requests with responses.  For large responses, the more, offset and count fields work together to knit up a large block of data.  The first packet of the response will have the offset field of 0. The count is the size of the data payload in bytes.  If the response is too big, the more bit will be set to 1 to say there is more coming.  The second packet will have an offset that is the same value as the previous packets count value, so you can stitch them together.

The first query is a READSTAT which uses opcode 1.  This returns a list of 4 byte responses with the association ID taking the first two bytes and the peer status taking the second.  If you look at the output of ntpq you will notice a character in the first column which is the peer selection.  The important one for us is asterix, which equates to 110 in the lower 3 bits of the first status byte.  This is the system peer which is the main one. There may be others, up to 2, (usually symbol +) which are compared to the system peer.  ntpq doesn’t really care about the different peers and just displays the status but I want to know what peer the server is synchronised to.

Next we need more details about the system peer (or all peers in ntpq).  To do this a READVAR or opcode 2 is sent to the server. The association ID is set to the association or peer we are interested in.  The response for my setup is sent over two packets which need to be reassembled and it is all in plain ascii which is a comma-separated list with a key=value pair.  I use the following python line to split it up.

self.assoc_data.update(dict([tuple(ad.strip().split("=")) for ad in data[header_len:].split(",") if ad.find("=") > -1]))

I’m sure people that know python better than I will say I am doing it wrong (it looks a bit klunky to me too) but it does work. From that I can query the various values that are returned by the server about its peers, including the source of its clock.

Enhanced by Zemanta

First Look at Python

When the programming language Python came out while it seemed like an interesting idea, I didn’t really see the point. I already know Perl, PHP and C (plus a few others) why learn another language? So until recently, I didn’t

Recent frustrations with PHP and how “loose” it is with things like variables made me have another look at Python and what it could do for me.  As just learning a language by itself it pretty boring I decided to write some small programs in the language and used the framework Turbogears to do it.

First of all, learning the basics of python from perl or PHP is dead simple. The idea of what a language looks like is not too different between these three so it was a snap to write simple stuff in python.  So, what does yet-another language give me?

  • Variables must be defined before being used, but assigning them defines them too.  This is a nice compromise between PHP where you can check undefined variables and C where it all must be defined first, or else.
  • try and except. For code that might fail, use try and then catch the error in except.  We’ve all been guilty of using the @ symbol in PHP to supress stupid error messages, this does it correctly.
  • No PHP extract() or compact(). Actually just no extract function is good enough. That function is pure evil.
  • Sqlalchemy – A proper “you look after the database and ill play with objects” database extraction layer thingamy.  I don’t want to play with databases just let me write code and thats what sqlalchemy does well.
  • Testing – Turbogears uses nosetests which does functional and unit testing. I do get some annoying artifacts based upon testing but generally its all good.
  • Python has help() everywhere and pydoc on the command line. Not sure what variable foo can do? Just type “help(foo)” or even stick it in your script.
  • paster! You can run a development site off the command-line using a simple sqlite database. Don’t like it, dump the DB and reinstall it with one command.

It’s not one-way though, there have been some challenges.  Quite likely these are more my deficiencies than the languages.

  • Error messages take up a whole screen and often make it difficult to isolate. It’s not quite 10-screens-full ala Eiffel but sometimes the error messages tell you something is going wrong, but not what. Mess up your object versus database table references and you get something rather obscure.
  • When is an integer not an integer? The answer is, sometimes. Sometimes they work, sometimes they break. It’s probably more to do with what pysnmp is doing with them.
  • Documentation – some is great, some is downright misleading. It comes down to the writer of the documentation for modules. At least it doesn’t have ‘well get round to it’ like some functions in PHP on their website. Documents that say you can directly iterate dictionaries were another dead-end I wandered down. If you are stuck here, add a .items() at the end of it.
  • Deployment. It’s not just a matter of taring the files up and sticking them into /var/www/sitename and you’re done.  There are probably good reasons why, but its just annoying.

Despite the minor problems, python really does give you much much more than PHP and some more than Perl.  For me and what I am intending on writing it is worth learning yet another language.

Enhanced by Zemanta

What happens without software testing


New TurboGears logo
Image via Wikipedia


Well JFFNMS 0.9.0 was a, well, its a good example of what can go wrong without adequate testing.  Having it written in PHP makes it difficult to test, because you can have entire globs of code that are completely wrong but are not activated (because the “if” statement proves false for example) and you won’t get an error.

It also had some database migration problems.  This is the most difficult and annoying part of releasing versions of JFFNMS.  There are so many things to check, like:

  • What has changed between this version of the database?
  • Does the development database have all the new fields?
  • Will importing an old database and using the diffs give me the same result as importing the new database
  • Does the structure field match the object which matches the database table

I’ve been looking at sqlalchemy which is part of turbogears.  It’s a pretty impressive setup and lets you get on with writing your application and not mucking around with the low-level stuff.  It’s a bit of a steep learning curve learning python AND sqlalchemy AND turbogears but I’ve got some rudimentary code running ok and its certainly worth it (python erroring on un-assigned varables but not forcing you to define them is a great compromise).  The best thing is that you can develop on a sqlite database but deploy using mysql or postgresql with a minor change.

Python and turbogears both emphasise automatic testing. Ideally the testing should cover all the code you write.  The authors even suggest you write the test first then implement the feature.  After chasing down several bugs, some of which I introduced fixing other bugs, automatic testing would make my life a lot easier and perhaps I wouldn’t dread the release cycle so often.


Enhanced by Zemanta