Tag: PHP

  • Fixing iCalendar feeds

    Fixing iCalendar feeds

    The local government here has all the schools use an iCalendar feed for things like when school terms start and stop and other school events occur. The department’s website also has events like public holidays. The issue is that all of them don’t make it an all-day event but one that happens at midnight, or one past midnight.

    The events synchronise fine, though Google’s calendar is known for synchronising when it feels like it, not at any particular time you would like it to.

    Screenshot of Android Calendar showing a tiny bar at midnight which is the event.

    Even though a public holiday is all day, they are sent as appointments for midnight.

    That means on my phone all the events are these tiny bars that appear right up the top of the screen and are easily missed, especially when the focus of the calendar is during the day.

    On the phone, you can see the tiny purple bar at midnight. This is how the events appear. It’s not the calendar’s fault, as far as it knows the school events are happening at midnight.

    You can also see Lunar New Year and Australia Day appear in the all-day part of the calendar and don’t scroll away. That’s where these events should be.

    Why are all the events appearing at midnight? The reason is the feed is incorrectly set up and has the time. The events are sent in an iCalendar format and a typical event looks like this:

    BEGIN:VEVENT
    DTSTART;TZID=Australia/Sydney:20230206T000000
    DTEND;TZID=Australia/Sydney:20230206T000000
    SUMMARY:School Term starts
    END:VEVENT

    The event starting and stopping date and time are the DTSTART and DTEND lines. Both of them have the date of 2023/02/06 or 6th February 2023 and a time of 00:00:00 or midnight. So the calendar is doing the right thing, we need to fix the feed!

    The Fix

    I wrote a quick and dirty PHP script to download the feed from the real site, change the DTSTART and DTEND lines to all-day events and leave the rest of it alone.

    <?php
    $site = $_GET['s'];
    if ($site == 'site1') {
        $REMOTE_URL='https://site1.example.net/ical_feed';
    } elseif ($site == 'site2') {
        $REMOTE_URL='https://site2.example.net/ical_feed';
    } else {
        http_response_code(400);
        die();
    }
    
    $fp = fopen($REMOTE_URL, "r");
    if (!$fp) {
        die("fopen");
    }
    header('Content-Type: text/calendar');
    while (( $line = fgets($fp, 1024)) !== false) {
        $line = preg_replace(
            '/^(DTSTART|DTEND);[^:]+:([0-9]{8})T000[01]00/',
            '${1};VALUE=DATE:${2}',
            $line);
        echo $line;
    }
    ?>

    It’s pretty quick and nasty but gets the job done. So what is it doing?

    • Lines 2-10: Check the given variable s and match it to either “site1” or “site2” to obtain the URL. If you only had one site to fix you could just set the REMOTE_URL variable.
    • Lines 12-15: A typical fopen() and nasty error handling.
    • Line 16: set the content type to a calendar.
    • Line 17: A while loop to read the contents of the remote site line by line.
    • Line 18-21: This is where the “magic” happens, preg_replace is a Perl regular expression replacement. The PCRE is:
      • Finding lines starting with DTSTART or DTEND and store it in capture 1
      • Skip everything that isn’t a colon. This is the timezone information. I wasn’t sure if it was needed and how to combine it so I took it out. All the all-day events I saw don’t have a time zone.
      • Find 8 numerics (this is for YYYYMMDD) and store it in capture 2.
      • Scan the Time part, a literal “T” then HHMMSS. Some sites use midnight some use one minute past, so it covers both.
      • Replace the line with either DTSTART or DTEND (capture 1), set the value type to DATE as the default is date/time and print the date (capture 2).
    • Line 22: Print either the modified or original line.

    You need to save the script on your web server somewhere, possibly with an alias command.

    The whole point of this is to change the type from a date/time to a date-only event and only print the date part of it for the start and end of it. The resulting iCalendar event looks like this:

    BEGIN:VEVENT
    DTSTART;VALUE=DATE:20230206
    DTEND;VALUE=DATE:20230206
    SUMMARY:School Term starts
    END:VEVENT

    The calendar then shows it properly as an all-day event. I would check the script works before doing the next step. You can use things like curl or wget to download it. If you use a normal browser, it will probably just download the translated file.

    If you’re not seeing the right thing then it’s probably the PCRE failing. You can check it online with a regex checker such as https://regex101.com. The site has saved my PCRE and match so you got something to start with.

    Calendar settings

    The last thing to do is to change the URL in your calendar settings. Each calendar system has a different way of doing it. For Google Calendar they provide instructions and you want to follow the section titled “Use a link to add a public Calendar”.

    The URL here is not the actual site’s URL (which you would have put into the REMOTE_URL variable before) but the URL of your script plus the ‘?s=site1″ part. So if you put your script aliased to /myical.php and the site ID was site1 and your website is www.example.com the URL would be “https://www.example.com/myical.php?s=site1”.

    You should then see the events appear as all-day events on your calendar.

  • PHP uniqid() not always a unique ID

    For quite some time modern versions of JFFNMS have had a problem. In large installations hosts would randomly appear as down with the reachability interface going red. All other interface types worked, just this one.

    Reachability interfaces are odd, because they call fping or fping6 do to the work. The reason is because to run a ping program you need to have root access to a socket and to do that is far too difficult and scary in PHP which is what JFFNMS is written in.

    To capture the output of fping, the program is executed and the output captured to a temporary file. For my tiny setup this worked fine, for a lot of small setups this was also fine. For larger setups, it was not fine at all. Random failed interfaces and, most bizzarely of all, even though a file disappearing. The program checked for a file to exist and then ran stat in a loop to see if data was there. The file exist check worked but the stat said file not found.

    At first I thought it was some odd load related problem, perhaps the filesystem not being happy and having a file there but not really there. That was, until someone said “Are these numbers supposed to be the same?”

    The numbers he was referring to was the filename id of the temporary file. They were most DEFINITELY not supposed to be the same. They were supposed to be unique. Why were they always unique for me and not for large setups?

    The problem is with the uniqid() function. It is basically a hex representation of the time.  Large setups often have large numbers of child processes for polling devices. As the number of poller children increases, the chance that two child processes start the reachability poll at the same time and have the same uniqid increases. It’s why the problem happened, but not all the time.

    The stat error was another symptom of this bug, what would happen was:

    • Child 1 starts the poll, temp filename abc123
    • Child 2 starts the poll in the same microsecond, temp filename is also abc123
    • Child 1 and 2 wait poller starts, sees that the temp file exists and goes into a loop of stat and wait until there is a result
    • Child 1 finishes, grabs the details, deletes the temporary file
    • Child 2 loops, tries to run stat but finds no file

    Who finishes first is entirely dependent on how quickly the fping returns and that is dependent on how quicky the remote host responds to pings, so its kind of random.

    A minor patch to use tempnam() instead of uniqid() and adding the interface ID in the mix for good measure (no two children will poll the same interface, the parent’s scheduler makes sure of that.) The initial responses is that it is looking good.

     

  • jffnms 0.9.4

    JFFNMS version 0.9.4 was released today, this version fixes some bugs that have recently appeared in previous versions.

    Alarmed Interfaces and Events
    Alarmed Interfaces and Events

    The triggers rules editor had a problem where some of the rules clicked off the triggers would not appear or could not be edited correctly.

    Most of the Admin screens have the ability to sort the rows. This, unfortunately, didn’t sort but the functionality has been restored.

    Most users are probably unaware of this, but the database schema is first created for MySQL and is then converted for PostgreSQL. The conversi0n process is far from ideal and hasn’t worked until this release. More testing is required for PostgreSQL support but it should be a lot better.

     

    Enhanced by Zemanta
  • Careful with apache upgrades

    You might (or not if you don’t visit) notice all my websites were down.  A rushed apt-get dist-upgrade and I found two problems:

    1. PHP5 got removed, which is bad if you run a wordpress site that uses PHP to run
    2. The apache configuration has changed.

    Yes, the NEWS entries did warn me, if I read them fully. Yes, I didn’t read them enough.

    Apache now ignores configuration files that don’t end in .conf To give a completely non-theoretical example, if you have your virtual hosts in files such as /etc/apache2/sites-enabled/enc.com.au then this will not be recognised and your sites will show the default “It works” page.

    Stuff that doesn’t fall in the usual places where website stuff should go, which for my setup is a lot of things, will also be denied as the developers have tightened up the rules around what is permitted.  Pretty simple to fix with a few <Directory blah> clauses.

    This isn’t a criticism of the Debian apache developers. They do an awesome job of keeping the package workable, flexible but secure which isn’t easy.  Now it’s all back working, I actually agree with the changes they have made. It is just that the latest changes are, well, tricky so be forewarned.

     

    Enhanced by Zemanta
  • JFFNMS 0.9.3

    JFFNMS version 0.9.3 has been released today.  This is a vast improvement over the 0.9.x releases and anyone using that train is strongly recommended to upgrade.So what changed? What didn’t change!  A nice summary would be fixing a lot of things that were broken or needed some tweaking. A really, really big thanks to Marek for all the testing and bug reports and also patient “just run this and tell me what it says” tests he did too.  If something wasn’t right before and works now, it is quite likely it is working because Marek told me how it broke.

    A brief overview of what has changed:

    • TFTP transfers work again
    • A lot of the wierd polling effects due to caching fixed
    • Lots of the selects in sub-tables now work
    • The PHP string-to-float brokeness in SLAs worked-around
    • Even more SNMP library cruft removed or escaped
    • HostMIB apps match properly
    • Interface autodiscovery delete and update fields back working

    You can download the file off sourceforge at

    https://sourceforge.net/projects/jffnms/files/JFFNMS%20Releases/

    Enhanced by Zemanta
  • PHP floats and locales

    I recently had a bug report in JFFNMS that the SLA checks were failing with bizarre calculations.  Things like 300% disk drive utilization and the like.  Briefly, JFFNMS is written in PHP and checks values that come out of rrdtool and makes various comparisons like have you used more than 80% of your disk or have there been too many errors.

    The logs showed strange input variables coming in, all were integers below 10.  I don’t know of many 1 or 3 kB sized disk drives. What was going on?  I ran a rrdtool fetch command on the relevant file and got output of something like 1,780000e+07 which for an 18GB drive seemed ok. Notice the comma, in this locale that’s a decimal point… hmm.

    In lib/api.rrdtool.inc.php there is this line around the rrdtool_fetch area:

    $value[] = ($line1[$i]=="nan")?0:(float)$line1[$i];

    A quick check and I was finding that my 1,7…e+07 was coming back as 1.  We had a float conversion problem.  Or more specifically, php has a float conversion problem.  I built a small check script like the following:

    setlocale(LC_NUMERIC,'pl_PL.UTF-8');
    $linfo = localeconv();
    $pi='3,14';
    print "Decimal is "$linfo[decimal_point]". Pi is $pi and ".(float)($pi)."n";
    print "Half is ".(1/2)."n";

    Which gave the output of:

    Decimal is “,”. Pi is 3,14 and 3

    Half is 0,5

    So… PHP is saying that decimal point is a comma and it uses it BUT if a string comes in with a comma, its not a decimal point. Really?? Are they serious here?  I tried various combinations and could not make it parse correctly.

    The fix was made easier for me because I know rrdtool fetch only outputs values in scientific notation. That means if there is a string with a comma, then it must be a decimal point as it could never be used for a thousands mark.  By using str_replace to replace any comma with a period the code worked again and didn’t even need the locale to be set correctly, or that the locale for the shell where rrdtool is run is the same as the locale in php.

    Enhanced by Zemanta
  • JFFNMS 0.9.2 Released

    JFFNMS Interfaces and Events

    JFFNMS version 0.9.2 was released today both as an upstream tar.gz file and a new Debian package.  This version fixes some bugs including making sure it works with PHP5.4.

    The biggest change in PHP 5.4 is that you can no longer call by reference.  Previously you could call a function like myfunc(&blah); which would send a pointer to blah and not the item itself. Now the function definition needs to define what it wants rather than change it each time.

     

    Enhanced by Zemanta
  • What happens without software testing

     

    New TurboGears logo
    Image via Wikipedia

     

    Well JFFNMS 0.9.0 was a, well, its a good example of what can go wrong without adequate testing.  Having it written in PHP makes it difficult to test, because you can have entire globs of code that are completely wrong but are not activated (because the “if” statement proves false for example) and you won’t get an error.

    It also had some database migration problems.  This is the most difficult and annoying part of releasing versions of JFFNMS.  There are so many things to check, like:

    • What has changed between this version of the database?
    • Does the development database have all the new fields?
    • Will importing an old database and using the diffs give me the same result as importing the new database
    • Does the structure field match the object which matches the database table

    I’ve been looking at sqlalchemy which is part of turbogears.  It’s a pretty impressive setup and lets you get on with writing your application and not mucking around with the low-level stuff.  It’s a bit of a steep learning curve learning python AND sqlalchemy AND turbogears but I’ve got some rudimentary code running ok and its certainly worth it (python erroring on un-assigned varables but not forcing you to define them is a great compromise).  The best thing is that you can develop on a sqlite database but deploy using mysql or postgresql with a minor change.

    Python and turbogears both emphasise automatic testing. Ideally the testing should cover all the code you write.  The authors even suggest you write the test first then implement the feature.  After chasing down several bugs, some of which I introduced fixing other bugs, automatic testing would make my life a lot easier and perhaps I wouldn’t dread the release cycle so often.

     

    Enhanced by Zemanta