Percent CPU for processes

The ps program gives a snapshot of the processes running on your Unix-like system. On most Linux installations, this will be the ps program from the procps project.

While you can get a lot of information from the tool, a lot of the fields need further explanation or can give “wrong” or confusing information; or putting it another way, they provide the right information that looks wrong.

One of these confusing fields is the %CPU or pcpu field. You can see this as the third field with the ps aux command. You only really need the u option to see it, but ps aux is a pretty common invokation.

More than 100%?

This post was inspired by procps issue 186 where the submitter said that the sum of the processes cannot be more than the number of CPUs times 100%. If you have 1 CPU then the sum of %CPU for all processes should be 100% or less, have 16 CPUs then 1600% is your maximum number.

Some people reason for the oddity of over 100% CPU as some rounding thing gone wrong and at first I did think that; except I know we get a lot of reports about comparing the top header CPU load vs process load not lining up and its because “they’re different”.

The trick here, is ps is reporting a percentage of what? Or, perhaps to give a better clue, a percentage of when?

PCPU Calculations

So to get to the bottom of this, let’s look at the relevant code. In ps/output.c we have a function pr_pcpu that prints the percent CPU. The relevant lines are:

  total_time = pp->utime + pp->stime;
  if(include_dead_children) total_time += (pp->cutime + pp->cstime);
  seconds = cook_etime(pp);
  if(seconds) pcpu = (total_time * 1000ULL / Hertz) / seconds;

OK, ignoring the include _dead_time line (you get this from the S option and means you include the time this process waited for its children processes) and the scaling (process times are in Jiffies, we have the CPU as 0 to 999 for reasons) you can reduce this down to.

%CPU = ( Tutime + Tstime ) / Tetime

So we find the amount of time the CPU(s) have been busy either in userland or the system, add them together, then divide the sum by the total time. The utime and stime increment like a car’s odometer. So if a process uses one Jiffy of CPU time in userland, that counter goes to 1. If it does it again a few seconds later, then that counter goes to 2.

To give an example, if a process has run for ten seconds and within those ten seconds the CPU has been busy in userland for that process, then we get 10/10 = 100% which makes sense.

Not all Start times are the same

Let’s take another example, a process still consumes ten seconds CPU time but been running for twenty seconds, the answer is 10/20 or 50%. With our single CPU example system both of these cannot be running at the same time otherwise we have 150% CPU utilisation which is not possible.

However, let’s adjust this slightly. We have assumed uniform utilisation. But take the following scenario:

  • At time T: Process P1 starts and uses 100% CPU
  • At time T+10 seconds: Process P1 stops using CPU but still runs, perhaps waiting for I/O or sleeping.
  • Also at time T+10 seconds: Process P2 starts and uses 100% CPU
  • At time T+20 we run the ps command and look at the %CPU column

The output for ps -o times,etimes,pcpu,comm would look something like:

    TIME ELAPSED %CPU COMMAND
      10      20   50 P1
      10      10  100 P2

What we will see is P1 has 10/20 or 50% CPU and P2 has 10/10 or 100% CPU. Add those up, and you have 150% CPU, magic!

The key here is the ELAPSED column. P1 has given you the CPU utilisation across 20 seconds of system time and P2 the CPU utilisation across only 10 seconds. You directly add them together you get the wrong answer.

What’s the point of %CPU?

Probably the %CPU column gives results that a lot of people are not expecting, so what’s the point of it? Don’t use it to see why the CPU is running hot; you can see above those two processes were working the CPU hard at different times. What it is useful for is to see how “busy” a process is, but be warned its an average. It’s helpful for something that starts busy but if the process idles or hardly uses CPU for a week then goes bananas you won’t see it.

The top program, because a lot of its statistics are deltas from the last refresh, is a much better program for this sort of information about what is happening right now.

Sending data in a signal

The well-known kill system call has been around for decades and is used to send a signal to another process. The most common use is to terminate or kill another process by sending the KILL or TERM signal but it can be used for a form of IPC, usually around giving the other process a “kick” to do something.

One thing that isn’t as well known is besides sending a signal to a process, you can send some data to it. This can either be an integer or a pointer and uses similar semantics to the known kill and signal handler. I came across this when there was a merge request for procps. The main changes are using sigqueue instead of kill in the sender and using a signal action not a signal handler in the receiver.

Read more Sending data in a signal

procps-ng 3.3.16

procps-ng version 3.3.16 was released today. Besides some documentation and library updates, there were a few incremental changes.

Zombie Hunting with pgrep

Ever wanted to find zombies? Perhaps processes with other states? pgrep has a shiny new runstate flag to help you which will match process against the runstate. I’m curious to see the use-cases for this flag; it certainly will get used (e.g. find my zombies) but as some processes bounce in and out of states (think Run to Sleep and back) it might add some confusion.

Snice plays nice with PIDs

Speaking of ancient corpses, snice was not matching against PIDs. The best use for snice is to not use it (as the man page says) but some people do and some people noticed it never matched against PIDs.
The issue was reading the process state up to 128 bytes, but process state lines are always longer than 128 bytes so a bounds check failed and it skipped that PID (and every other PID too).

Top Enhancements

Top got a bunch of love again in this release. If you ever wanted your processes to be shown in fuchsia? Perhaps goldenrod? With some earlier versions of top, you could by directly editing the toprc file but now everyone can have more than the standard 8 colours!

If you use the other filters parameter for some fancy process filtering in top, it now will save that configuration.

Collapsed children (process names are weird) get some help. If you are in tree view, you can collapse or fold the children processes under the parent. Their CPU is also added to the parent so there are no “missing” CPU ticks.

For people who use the One True Editor (which is, of course, VIM) you can use the vim navigation keys to move through the process list.

Where to find it?

You’ll find the latest version of procps either at our git repository or download a tarball.

WordPress 5.2.4

Hot on the heels of WordPress version 5.2.3 which fixed a bunch of stuff we have WordPress 5.2.4 with fixes for six security issues.

There is a certain trick to matching up what the WordPress Blog thinks has been fixed and the changsets between the old version and the new. The curious thing is there were 6 changsets backported to older versions of WordPress, so you might think “six issues, six changesets, what’s the problem?”. The problem is that two of them fix the same thing (or one sort-of fixed it and the second really did) and another I couldn’t link to any vulnerability, BUT it was to do with directory traversal issue.

The hardest part of maintaining the Debian WordPress packages is the backporting. Trying to link the changes to the bugs is next to impossible so I generally import all the ones they have in the specific major version and hope for the best. This isn’t ideal, but information about what the actual bugs are and how they are fixed is not forthcoming.

WordPress 5.1.1

The Debian packages for WordPress version 5.1.1 are being updated as I write this. This is a security fix for WordPress that stops comments causing a cross-site scripting bug. It’s an important one to update.

The backports should happen soon so even if you are using Debian stable you’ll be covered.