Tag: Programming

  • Fixing iCalendar feeds

    Fixing iCalendar feeds

    The local government here has all the schools use an iCalendar feed for things like when school terms start and stop and other school events occur. The department’s website also has events like public holidays. The issue is that all of them don’t make it an all-day event but one that happens at midnight, or one past midnight.

    The events synchronise fine, though Google’s calendar is known for synchronising when it feels like it, not at any particular time you would like it to.

    Screenshot of Android Calendar showing a tiny bar at midnight which is the event.

    Even though a public holiday is all day, they are sent as appointments for midnight.

    That means on my phone all the events are these tiny bars that appear right up the top of the screen and are easily missed, especially when the focus of the calendar is during the day.

    On the phone, you can see the tiny purple bar at midnight. This is how the events appear. It’s not the calendar’s fault, as far as it knows the school events are happening at midnight.

    You can also see Lunar New Year and Australia Day appear in the all-day part of the calendar and don’t scroll away. That’s where these events should be.

    Why are all the events appearing at midnight? The reason is the feed is incorrectly set up and has the time. The events are sent in an iCalendar format and a typical event looks like this:

    BEGIN:VEVENT
    DTSTART;TZID=Australia/Sydney:20230206T000000
    DTEND;TZID=Australia/Sydney:20230206T000000
    SUMMARY:School Term starts
    END:VEVENT

    The event starting and stopping date and time are the DTSTART and DTEND lines. Both of them have the date of 2023/02/06 or 6th February 2023 and a time of 00:00:00 or midnight. So the calendar is doing the right thing, we need to fix the feed!

    The Fix

    I wrote a quick and dirty PHP script to download the feed from the real site, change the DTSTART and DTEND lines to all-day events and leave the rest of it alone.

    <?php
    $site = $_GET['s'];
    if ($site == 'site1') {
        $REMOTE_URL='https://site1.example.net/ical_feed';
    } elseif ($site == 'site2') {
        $REMOTE_URL='https://site2.example.net/ical_feed';
    } else {
        http_response_code(400);
        die();
    }
    
    $fp = fopen($REMOTE_URL, "r");
    if (!$fp) {
        die("fopen");
    }
    header('Content-Type: text/calendar');
    while (( $line = fgets($fp, 1024)) !== false) {
        $line = preg_replace(
            '/^(DTSTART|DTEND);[^:]+:([0-9]{8})T000[01]00/',
            '${1};VALUE=DATE:${2}',
            $line);
        echo $line;
    }
    ?>

    It’s pretty quick and nasty but gets the job done. So what is it doing?

    • Lines 2-10: Check the given variable s and match it to either “site1” or “site2” to obtain the URL. If you only had one site to fix you could just set the REMOTE_URL variable.
    • Lines 12-15: A typical fopen() and nasty error handling.
    • Line 16: set the content type to a calendar.
    • Line 17: A while loop to read the contents of the remote site line by line.
    • Line 18-21: This is where the “magic” happens, preg_replace is a Perl regular expression replacement. The PCRE is:
      • Finding lines starting with DTSTART or DTEND and store it in capture 1
      • Skip everything that isn’t a colon. This is the timezone information. I wasn’t sure if it was needed and how to combine it so I took it out. All the all-day events I saw don’t have a time zone.
      • Find 8 numerics (this is for YYYYMMDD) and store it in capture 2.
      • Scan the Time part, a literal “T” then HHMMSS. Some sites use midnight some use one minute past, so it covers both.
      • Replace the line with either DTSTART or DTEND (capture 1), set the value type to DATE as the default is date/time and print the date (capture 2).
    • Line 22: Print either the modified or original line.

    You need to save the script on your web server somewhere, possibly with an alias command.

    The whole point of this is to change the type from a date/time to a date-only event and only print the date part of it for the start and end of it. The resulting iCalendar event looks like this:

    BEGIN:VEVENT
    DTSTART;VALUE=DATE:20230206
    DTEND;VALUE=DATE:20230206
    SUMMARY:School Term starts
    END:VEVENT

    The calendar then shows it properly as an all-day event. I would check the script works before doing the next step. You can use things like curl or wget to download it. If you use a normal browser, it will probably just download the translated file.

    If you’re not seeing the right thing then it’s probably the PCRE failing. You can check it online with a regex checker such as https://regex101.com. The site has saved my PCRE and match so you got something to start with.

    Calendar settings

    The last thing to do is to change the URL in your calendar settings. Each calendar system has a different way of doing it. For Google Calendar they provide instructions and you want to follow the section titled “Use a link to add a public Calendar”.

    The URL here is not the actual site’s URL (which you would have put into the REMOTE_URL variable before) but the URL of your script plus the ‘?s=site1″ part. So if you put your script aliased to /myical.php and the site ID was site1 and your website is www.example.com the URL would be “https://www.example.com/myical.php?s=site1”.

    You should then see the events appear as all-day events on your calendar.

  • Forking processes and Gtk2

    I made a change recently on the gjay program. gjay is a gtk program that basically analyzes your music and makes playlists. There is a gui frontend and a analyzer back-end and they communicate through a pipe.

    One really useful debugging option gtk has is to make warnings fatal, so when gtk finds one it crashes at that point and you can use gdb to trap it. The flag is –g-fatal-warnings.  I have been updating gjay and initially it didn’t have this option, so I needed to add the gtk options, which is a simple one-liner.

    (more…)
  • PHP uniqid() not always a unique ID

    For quite some time modern versions of JFFNMS have had a problem. In large installations hosts would randomly appear as down with the reachability interface going red. All other interface types worked, just this one.

    Reachability interfaces are odd, because they call fping or fping6 do to the work. The reason is because to run a ping program you need to have root access to a socket and to do that is far too difficult and scary in PHP which is what JFFNMS is written in.

    To capture the output of fping, the program is executed and the output captured to a temporary file. For my tiny setup this worked fine, for a lot of small setups this was also fine. For larger setups, it was not fine at all. Random failed interfaces and, most bizzarely of all, even though a file disappearing. The program checked for a file to exist and then ran stat in a loop to see if data was there. The file exist check worked but the stat said file not found.

    At first I thought it was some odd load related problem, perhaps the filesystem not being happy and having a file there but not really there. That was, until someone said “Are these numbers supposed to be the same?”

    The numbers he was referring to was the filename id of the temporary file. They were most DEFINITELY not supposed to be the same. They were supposed to be unique. Why were they always unique for me and not for large setups?

    The problem is with the uniqid() function. It is basically a hex representation of the time.  Large setups often have large numbers of child processes for polling devices. As the number of poller children increases, the chance that two child processes start the reachability poll at the same time and have the same uniqid increases. It’s why the problem happened, but not all the time.

    The stat error was another symptom of this bug, what would happen was:

    • Child 1 starts the poll, temp filename abc123
    • Child 2 starts the poll in the same microsecond, temp filename is also abc123
    • Child 1 and 2 wait poller starts, sees that the temp file exists and goes into a loop of stat and wait until there is a result
    • Child 1 finishes, grabs the details, deletes the temporary file
    • Child 2 loops, tries to run stat but finds no file

    Who finishes first is entirely dependent on how quickly the fping returns and that is dependent on how quicky the remote host responds to pings, so its kind of random.

    A minor patch to use tempnam() instead of uniqid() and adding the interface ID in the mix for good measure (no two children will poll the same interface, the parent’s scheduler makes sure of that.) The initial responses is that it is looking good.

     

  • killing a process in TCL

    Suppose you had spawned a process in TCL and knew its PID and wanted to kill it? Sounds simple enough thing to do, right? This problem has plagued me for many months because some things you can assume on a normal system do not hold true on strange environments, such as build deaemons.

    Seems simple enough, I started off with:

    exec kill $pid
    

    Except.. not every environment has the kill binary, and with that piece of code exec has to be a binary and not a shell builtin. The funny thing is that /bin/kill is in the procps package, which is the package having the buildd problems.

    So next idea was to use command -v to check for the existence of kill and skip those tests that needed kill if not found. Good idea except, so I found out later, it also finds built-ins. That means we are back to problem .

    There is a kill command in tcl, but it requires tclx. That seems excessive for just one little simple command. How can I run a shell out of TCL that runs the kill builtin? On the command line, something like below would do it.

    /bin/sh -c 'kill 1234'
    

    I was closer, but then hit TCL quoting hell. No matter what I (initially) did I’d either get the shell to complain or my variable to not be evaluated. In the end, I had to write it to a separate variable for the command line then apply that to the exec. Not perfect but at least it works now.

    The resulting code (found in testsuite/config/unix.exp) looks like:

    proc kill_process pid {
        set cmdline "kill $pid"
        if { [catch { exec /bin/sh -c $cmdline } msg]} {
            warning "Could not kill process: $msgn"
        }
    }
    

    Perhaps there is a more elegant way, I’m certainly no star TCL programmer, but of all the combinations I saw this was the only that worked.

  • Sneak peek of top graphs

    Jim has been busy as part of the procps-ng team that looks after top. Basically all the changes you find in top from around 2.7 or so are by him. Not satisfied enough with fixing top, making it faster and showing more fields, he has given us CPU and memory graphs.

    He also thinks I don’t have enough colours (or as he would put it colors) on my top output so I’ve posted what the new top looks like for me so you can see the graphs and he can see my colours.

    top, with colours
    top, with colours

     

    I think it is both colourful and useful addition. The colours have been available for a while now and the graphs will appear in the next upstream release of procps-ng.

     

  • A python utf gotcha

    This one had me stumped for a while:

    # -*- coding: utf-7 -*-
    import datetime
    from sqlalchemy import ForeignKey, Column
    from sqlalchemy.types import Integer, Unicode, Boolean, DateTime
    
    default_due_date = datetime.datetime.now() + datetime.timedelta(days=30)
    

    Syntax error found on last line.

    Hmm, bring up a python interpreter and type the last line in with the imports. Works fine.

    It’s the first line that is the problem, I typoed it and made it utf-7 not utf-8. I suppose it means that it is case-insenstive. Still, it wasn’t too clear to me at least, what was going on.

    Enhanced by Zemanta
  • jqGridWidget in Turbogears

     

     

    Turbogears 2 uses Toscawidgets 2 for a series of very clever widgets and base objects that you can use in your projects.  One I have been playing with is the jqGridWidget which uses jquery to display a grid. The idea with jquery is creating a grid or other object and then using javascript to call back to the webserver to obtain the data which usually comes from a database using the very excellent SQLAlchemy framework.

    I found it a bit fiddly to get going and it is unforgiving, you get one thing wrong and you end up with an empty grid but when it works, it works very well.

    from tw2.jqplugins.jqgrid import jqGridWidget
    from tw2.jqplugins.jqgrid.base import word_wrap_css
    
    class ItemsWidget(jqGridWidget):
        def prepare(self):
            self.resources.append(word_wrap_css)
            super(ItemsWidget, self).prepare()
        options = {
                'pager': 'item-list-pager2',
                'url': '/items/jqgrid',
                'datatype': 'json',
                'colNames': ['Date', 'Type', 'Area', 'Description'],

    The code above is a snippet of the widget I use to make the graph. This is the visual side of the graph and shows just the  graph with nothing in it.  You need the datatype and url lines in the options which tell the code the url to get the data.  I have an items controller and within it there is a jqgrid method.  Note the number of columns I have, the data sent by the jqgrid method has to match this.

    Deep inside the items controller is a jqgrid method. The idea is to return some rows of data back to the grid in the exact way the grid expects it otherwise you get, yep more empty grids.

       @expose('json')
        def jqgrid(self, page=1, rows=30, sidx=1, soid='asc', _search='false',
                searchOper=u'', searchField=u'', searchString=u'', **kw):
    
            qry = DBSession.query(Item)
            qry = qry.filter()
            qry = qry.order_by()
            result_count = qry.count()
            rows = int(rows)
    
            offset = (int(page)-1) * rows
            qry = qry.offset(offset).limit(rows)
    
            records = [{'id': rw.id,
               'cell': [ rw.created.strftime('%d %b %H:%M:%S'), rw.item_type.display_name), rw.display_name, rw.text()]} for rw in qry]
            total = int(math.ceil(result_count / float(rows)))
            return dict(page=int(page), total=total, records=result_count, rows=records)

    The four items in the value for the cell in the dictionary must equal the number defined for the colNames.  I found it was useful to have firebug looking at my gets and another widget based upon SQLAjqGridWidget which is a lot simpler to setup but much less flexible too.

     

    Enhanced by Zemanta
  • JFFNMS 0.9.3

    JFFNMS version 0.9.3 has been released today.  This is a vast improvement over the 0.9.x releases and anyone using that train is strongly recommended to upgrade.So what changed? What didn’t change!  A nice summary would be fixing a lot of things that were broken or needed some tweaking. A really, really big thanks to Marek for all the testing and bug reports and also patient “just run this and tell me what it says” tests he did too.  If something wasn’t right before and works now, it is quite likely it is working because Marek told me how it broke.

    A brief overview of what has changed:

    • TFTP transfers work again
    • A lot of the wierd polling effects due to caching fixed
    • Lots of the selects in sub-tables now work
    • The PHP string-to-float brokeness in SLAs worked-around
    • Even more SNMP library cruft removed or escaped
    • HostMIB apps match properly
    • Interface autodiscovery delete and update fields back working

    You can download the file off sourceforge at

    https://sourceforge.net/projects/jffnms/files/JFFNMS%20Releases/

    Enhanced by Zemanta
  • JFFNMS 0.9.2 Released

    JFFNMS Interfaces and Events

    JFFNMS version 0.9.2 was released today both as an upstream tar.gz file and a new Debian package.  This version fixes some bugs including making sure it works with PHP5.4.

    The biggest change in PHP 5.4 is that you can no longer call by reference.  Previously you could call a function like myfunc(&blah); which would send a pointer to blah and not the item itself. Now the function definition needs to define what it wants rather than change it each time.

     

    Enhanced by Zemanta
  • psmisc 22.16 Released

    psmisc version 22.16 was released today.  It is a bugfix release that bascially fixes a problem around strings in C.  Process name lengths are only supposed to be 16 characters long, so a 17 bye buffer is ok; until you have processes with brackets which means the string is 18 characters.

    The next wrinkle is that at times the brackets are stripped out so matches fail because the lengths don’t quite line up. You’ll see this with the Debian 22.15-2 version of psmisc where killall won’t find long-named processes.

    So, 22.16 fixes all that.

    Test Processes

    It really shows that psmisc needs a set of tests like procps has already. The difficulty with both is that its not simple in the DejaGNU framework to make test processes. These are not the programs within the package but other processes that the programs can work on.  There really needs to be an equivalent to touch for processes just for this sort of thing.  Creating processes is rather simple, but ensuring they go away is the tricky part, or they die with certain signals.

    Enhanced by Zemanta