Changing Grafana Legends

I’m not sure if I just can search Google properly, or this really is just not written down much, but I have had problems with Grafana Legends (I would call them the series labels). The issue is that Grafana queries Prometheus for a time series and you want to display multiple lines, but the time-series labels you get are just not quite right.

A simple example is you might be using the black-box exporter to monitor an external TCP port and you would just like the port number separate to display. The default output would look like this:

probe_duration_seconds{instance="example.net:5222",job="blackbox",module="xmpp_banner"} = 0.01
probe_duration_seconds{instance="example.net:5269",job="blackbox",module="xmpp_banner"} = 0.01

I can graph the number of seconds that it takes to probe the 5222 and 5269 TCP ports, but my graph legend is going to have the hostname, making it cluttered. I just want the legend to be the port numbers on Grafana.

The answer is to use a Prometheus function called label_replace that takes an existing label, applies a regular expression, then puts the result into another label. That’s right, regular expressions, and if you get them wrong then the label just doesn’t appear.

Perl REGEX Problems courtesy of XKCD

The label_replace documentation is a bit terse, and in my opinion, the order of parameters is messed up, but after a few goes I had what I needed:

label_replace(probe_duration_seconds{module="xmpp_banner"}, "port", "$1", "instance", ".*:(.*)")

probe_duration_seconds{instance="example.net:5222",job="blackbox",module="xmpp_banner",port="5222"}	0.001
probe_duration_seconds{instance="example.net:5269",job="blackbox",module="xmpp_banner",port="5269"}	0.002

The response now has a new label (or field if you like) called port. So what is this function to our data coming from probe_duration_seconds? The function format is:

label_replace(value, dst_label, replacement, src_label, regex)

So the function does the following:

  1. Evaluate value, which is generally some sort of query such as probe_duration_seconds
  2. Find the required source label src_label, in this example is instance, in this case the values are example.net:5222 and example.net:5269
  3. Apply regular expression regex, for us its “.*:(.*)” That says skip everying before “:” then capture/store everything past “:”. The brackets mean copy what is after the colon and put it in match #1
  4. Make a new label specified in dst_label, for us this is port
  5. Whatever is in replacement goes into dst_label. For this example it is “$1” which means match #1 in our regular expression in the label called port.

In short, the function captures everything after the colon in the instance label and puts that into a new label called port. It does this for each value that is returned into the first parameter.

This means I can use the {{port}} in my Grafana graph Legend and it will show 5222 or 5269 respectively. I have made the Legend “TCP {{port}}” to give the below result, but I could have used {{port}} in Grafana Legend and made the result “TCP $1” in the label_replace function to get the same result.

Grafana console showing the use of the label_replace function

Percent CPU for processes

The ps program gives a snapshot of the processes running on your Unix-like system. On most Linux installations, this will be the ps program from the procps project.

While you can get a lot of information from the tool, a lot of the fields need further explanation or can give “wrong” or confusing information; or putting it another way, they provide the right information that looks wrong.

One of these confusing fields is the %CPU or pcpu field. You can see this as the third field with the ps aux command. You only really need the u option to see it, but ps aux is a pretty common invokation.

More than 100%?

This post was inspired by procps issue 186 where the submitter said that the sum of the processes cannot be more than the number of CPUs times 100%. If you have 1 CPU then the sum of %CPU for all processes should be 100% or less, have 16 CPUs then 1600% is your maximum number.

Some people reason for the oddity of over 100% CPU as some rounding thing gone wrong and at first I did think that; except I know we get a lot of reports about comparing the top header CPU load vs process load not lining up and its because “they’re different”.

The trick here, is ps is reporting a percentage of what? Or, perhaps to give a better clue, a percentage of when?

PCPU Calculations

So to get to the bottom of this, let’s look at the relevant code. In ps/output.c we have a function pr_pcpu that prints the percent CPU. The relevant lines are:

  total_time = pp->utime + pp->stime;
  if(include_dead_children) total_time += (pp->cutime + pp->cstime);
  seconds = cook_etime(pp);
  if(seconds) pcpu = (total_time * 1000ULL / Hertz) / seconds;

OK, ignoring the include _dead_time line (you get this from the S option and means you include the time this process waited for its children processes) and the scaling (process times are in Jiffies, we have the CPU as 0 to 999 for reasons) you can reduce this down to.

%CPU = ( Tutime + Tstime ) / Tetime

So we find the amount of time the CPU(s) have been busy either in userland or the system, add them together, then divide the sum by the total time. The utime and stime increment like a car’s odometer. So if a process uses one Jiffy of CPU time in userland, that counter goes to 1. If it does it again a few seconds later, then that counter goes to 2.

To give an example, if a process has run for ten seconds and within those ten seconds the CPU has been busy in userland for that process, then we get 10/10 = 100% which makes sense.

Not all Start times are the same

Let’s take another example, a process still consumes ten seconds CPU time but been running for twenty seconds, the answer is 10/20 or 50%. With our single CPU example system both of these cannot be running at the same time otherwise we have 150% CPU utilisation which is not possible.

However, let’s adjust this slightly. We have assumed uniform utilisation. But take the following scenario:

  • At time T: Process P1 starts and uses 100% CPU
  • At time T+10 seconds: Process P1 stops using CPU but still runs, perhaps waiting for I/O or sleeping.
  • Also at time T+10 seconds: Process P2 starts and uses 100% CPU
  • At time T+20 we run the ps command and look at the %CPU column

The output for ps -o times,etimes,pcpu,comm would look something like:

    TIME ELAPSED %CPU COMMAND
      10      20   50 P1
      10      10  100 P2

What we will see is P1 has 10/20 or 50% CPU and P2 has 10/10 or 100% CPU. Add those up, and you have 150% CPU, magic!

The key here is the ELAPSED column. P1 has given you the CPU utilisation across 20 seconds of system time and P2 the CPU utilisation across only 10 seconds. You directly add them together you get the wrong answer.

What’s the point of %CPU?

Probably the %CPU column gives results that a lot of people are not expecting, so what’s the point of it? Don’t use it to see why the CPU is running hot; you can see above those two processes were working the CPU hard at different times. What it is useful for is to see how “busy” a process is, but be warned its an average. It’s helpful for something that starts busy but if the process idles or hardly uses CPU for a week then goes bananas you won’t see it.

The top program, because a lot of its statistics are deltas from the last refresh, is a much better program for this sort of information about what is happening right now.

25 Years of Free Software

When did I start writing Free Software, now called Open Source? That’s a tricky question. Does the time start with the first file edited, the first time it compiles or perhaps even some proto-program you use to work out a concept for the real program formed later on.

So using the date you start writing, especially in a era before decent version control systems, is problematic. That is why I use the date of the first release of the first package as the start date. For me, that was Monday 24th July 1995.

Read more 25 Years of Free Software

Sending data in a signal

The well-known kill system call has been around for decades and is used to send a signal to another process. The most common use is to terminate or kill another process by sending the KILL or TERM signal but it can be used for a form of IPC, usually around giving the other process a “kick” to do something.

One thing that isn’t as well known is besides sending a signal to a process, you can send some data to it. This can either be an integer or a pointer and uses similar semantics to the known kill and signal handler. I came across this when there was a merge request for procps. The main changes are using sigqueue instead of kill in the sender and using a signal action not a signal handler in the receiver.

Read more Sending data in a signal

procps-ng 3.3.16

procps-ng version 3.3.16 was released today. Besides some documentation and library updates, there were a few incremental changes.

Zombie Hunting with pgrep

Ever wanted to find zombies? Perhaps processes with other states? pgrep has a shiny new runstate flag to help you which will match process against the runstate. I’m curious to see the use-cases for this flag; it certainly will get used (e.g. find my zombies) but as some processes bounce in and out of states (think Run to Sleep and back) it might add some confusion.

Snice plays nice with PIDs

Speaking of ancient corpses, snice was not matching against PIDs. The best use for snice is to not use it (as the man page says) but some people do and some people noticed it never matched against PIDs.
The issue was reading the process state up to 128 bytes, but process state lines are always longer than 128 bytes so a bounds check failed and it skipped that PID (and every other PID too).

Top Enhancements

Top got a bunch of love again in this release. If you ever wanted your processes to be shown in fuchsia? Perhaps goldenrod? With some earlier versions of top, you could by directly editing the toprc file but now everyone can have more than the standard 8 colours!

If you use the other filters parameter for some fancy process filtering in top, it now will save that configuration.

Collapsed children (process names are weird) get some help. If you are in tree view, you can collapse or fold the children processes under the parent. Their CPU is also added to the parent so there are no “missing” CPU ticks.

For people who use the One True Editor (which is, of course, VIM) you can use the vim navigation keys to move through the process list.

Where to find it?

You’ll find the latest version of procps either at our git repository or download a tarball.