CPU Threading Efficiency: Tactical AIX Monitoring of the Runqueue Value
AIX:vmstat –IWwt 1 is my favorite AIX monitoring syntax. It's a lightweight command for casual 24-7 monitoring of your important LPARs―the keyword being "lightweight."
In my previous article, "Recognizing the Efficiency Benefits of CPU Threading," I offered an outline of instructions for tactical AIX monitoring that begins with:
- Understand the runqueue thread count relative to the total count of logical CPUs.
- Monitor the value of AIX:vmstat -IWwt 1:kthr:r until familiar.
With this article, I'll elaborate on point 1b, specifically AIX:vmstat –IWwt 1, because it's my favorite AIX monitoring syntax. This is an extremely lightweight command for casual 24-7 monitoring of your important LPARs―the keyword being "lightweight." Of course, the better known AIX:nmon does the same type of monitoring, but it's far more CPU-intensive.
The command syntax of AIX:vmstat -IWwt 1 offers six groups of statistics per second, namely:
- The count of a variety of kernel threads (kthr:r/:b/:p/:w)
- Active virtual memory and free memory in 4k pages (memory:avm/:fre)
- Default JFS/JFS2 reads/writes and paging space ins/outs in 4k pages (page:fi/:fo/:pi/:po)
- The count of device interrupts, system calls and context switches (faults:in/:sy:/:cs)
- CPU usage percentages, processor consumed and entitlement used (cpu:us/:sy/:id/:wa/:pc/:ec)
- The time of each interval (time:hr/:mi/:se).
One great advantage with this syntax is that it provides an up-scrolling history of past seconds as the next second appears with each new line. While most other AIX interval commands (including mpstat, iostat and lvmstat) display in this fashion, this capability contrasts sharply with nmon output, which doesn't show any interval history. The up-scrolling history is perfect for following the dynamic rise and fall of the numbers of an on-going workload.
For the remainder of this article, I'll focus on a single statistical value: AIX:vmstat –IWwt 1:kthr:r.
The AIX:vmstat -IWwt 1:kthr:r value is the average count of running plus runnable threads over each one-second interval, (aka the runqueue). Threads are groups of instructions that execute on the logical CPUs of a virtual CPU while this virtual CPU is running on a CPUcore.
Now, you might be thinking: “But I thought processes run on CPUs? Then what are logical CPUs and virtual CPUs running on a CPUcore?” At this point, it's critical to consider the structure specific to POWER CPU virtualization: process:threads are mapped to CPUcore:vCPU:logicalCPUs when executing. I've taught customer classes weekly for nearly five years, and most attendees don't understand the significance of this. Nor do they see how these threads comprise the runqueue that is the AIX:vmstat -IWwt 1:kthr:r value itself.
Historically, a process is a stream of instructions manipulating data in process memory (aka private RAM). This process’s single stream of instructions is a discrete task working with data―i.e., one process:one task:one dataset or [1:1:1].
Next, envision a process with two discrete streams of instructions sharing access to the same data in process memory. In other words, imagine two tasks of the same process manipulating the same dataset in RAM―i.e., one process:two tasks:one dataset or [1:2:1]. This is more efficient than two processes with two copies of the same data in two different regions of process memory―i.e., 2*[1:1:1].
Now envision a process with eight discrete streams of instructions sharing access to the same data in process memory. In other words, imagine eight tasks of the same process manipulating the same dataset in RAM―i.e., one process:eight tasks:one dataset or [1:8:1]. Again, this is more efficient than eight processes with eight copies of the same data in eight different regions of process memory―i.e., 8*[1:1:1].
Given all of this, each discrete stream of instructions (or tasks) of the same process is a thread. Processes are comprised of one or more threads sharing the same data in process memory―i.e., multithreaded processes or [1:X:1].
Notice the PID (Process ID) column. PIDs are always even-numbered in AIX. Also take note of the TID (Thread ID) column. TIDs are always odd-numbered in AIX. A TID number uniquely identifies each thread of every process listed by PID. (Exception: /etc/init always has a PID=1 and a PPID=0.)
Logical CPUs and Virtual CPUs
Let's review: Threads are groups of instructions that execute on the logical CPUs of a virtual CPU while this virtual CPU is running on a CPUcore. The AIX:vmstat -IWwt 1:kthr:r value is the average count of running plus runnable threads (TIDs) over each one-second interval, and the runqueue is comprised of executing threads (TIDs) of processes (PIDs).
Now we can get into logical and virtual CPUs. Logical CPUs ever and only map to their virtual CPUs, and virtual CPUs are manifested by their logical CPUs. Virtual CPUs do not execute threads (TIDs); it is the logical CPUs of virtual CPUs that execute threads (TIDs). As such, virtual CPUs are structures comprised of logical CPUs, and logical CPUs execute threads (TIDs) on the runqueue.
All IBM POWER Systems* LPARs are configured with one or more virtual CPUs. Yes, this includes dedicated CPU LPARs; each dedicated CPU is a virtual CPU that the hypervisor treats in a dedicated fashion. In virtually all POWER Systems installations, the sum of virtual CPUs of all LPARs is routinely greater than the count of active CPUcores. With such in varying degrees, the active CPUcores are shared by the virtual CPUs of all LPARs; the hypervisor manages this CPUcore sharing between LPARs.
Within the hypervisor, each virtual CPU can be thought of as merely an individual data structure on a scheduling array. (This is perhaps why they're called virtual CPUs.) Virtual CPUs have no productive existence until/unless they are running on a CPUcore. Virtual CPUs without a CPUcore can only wait for the hypervisor to assign them to a CPUcore. When a virtual CPU is assigned and running on a CPUcore, the virtual CPU manifests as one or more available logical CPUs that can each execute a thread (TID).
So the virtual CPU manifests as one or more available logical CPUs? Indeed it does. This feature is called simultaneous multithreading (SMT), which I'll discuss later in this series. Until then, please accept that virtual CPUs can rapidly and variably manifest as:
- A single logical CPU (ST/SMT-1) to execute a single thread (TID)
- Two logical CPUs (SMT-2) to execute two threads (2 TIDs)
- Four logical CPUs (SMT-4)―as of POWER7 to execute four threads (4 TIDs)
- Eight logical CPUs―as of POWER8 to execute eight threads (8 TIDs)
The above count of active logical CPUs per virtual CPU, (aka the dynamic SMT-mode) is governed by the demand of scheduled threads on the runqueue―i.e., the AIX:vmstat -IWwt 1:kthr:r value, and the maximum SMT-mode setting of the LPAR.
The second-by-second changing value of AIX:vmstat -IWwt 1:kthr:r is the runqueue thread count that is meaningful when compared to the total count of logical CPUs. The total count of logical CPUs is offered on the first line of AIX:vmstat -IWwt 1 output with lcpu=XX (not unlike what's seen with lcpu=32 in Figure 3 below (click to view larger):
At time=13:29:31, the value of kthr:r=22 of lcpu=32.
At time=13:29:32, the value of kthr:r=28 of lcpu=32.
As a matter of POWER/AIX monitoring practice, the runqueue value is typically ignored. This is unfortunate, because we're missing entire dimensions of meaning―especially as it regards the thorough exploitation of POWER8’s capabilities for best value. Instead, today’s POWER/AIX monitoring utilities (and thus the blameless readership) are fixated on the values of AIX:vmstat -IWwt 1:cpu:pc and AIX:vmstat -IWwt 1:cpu:ec. While this is not incorrect, it is incomplete. What's bAIX:vmstat -IWwteing ignored is how the runqueue reports the instantaneous preference for single-threaded performance or multi-threaded efficiency, as well as the traits of under-threading, optimal-threading and over-threading.
In the next installment in this series, I'll explain how CPU idle (AIX:vmstat:cpu:id) and CPU wait (AIX:vmstat.cpu.wa) are calculated, and show you how to monitor these values.
Earl Jew is a certified expert (Level Two) IT Specialist and senior IT management consultant, IBM Power Systems and IBM Systems Storage.
See more by Earl Jew