Leviathan User Usage

Refresh
Reading the columns
CPU

A live sample, not a lifetime average. The collector reads /proc twice a second apart and reports the difference, so this is what the user is running now — unlike ps -eo pcpu, where a job that pinned a core for six hours and then went idle still reports a huge number.

100% is one hardware thread fully busy. It is not a share of the machine: a leviathan reports 32 threads, so it tops out at 3200%. The figure under each percentage is the same number divided by 100, and it covers the whole fleet. Expanding a user's row breaks it down by leviathan: how many threads they are using on each machine they are logged into, next to what they have resident there. A host at 0.0 is one they are logged into with nothing running.

Memory

PSS (proportional set size): a page shared between several of a user's processes is divided among them. So a forked R job counts its shared memory once across all its workers rather than once per worker, which is the figure you want when deciding who is actually filling the box.

Load (1 / 5 / 15m)

The Unix load average: how many processes are using a CPU or waiting for one, averaged over the last 1, 5 and 15 minutes. On Linux it also counts processes blocked on uninterruptible I/O.

It is a count, not a percentage — compare it against the host's thread count. On a leviathan (32 threads):

~4Nearly idle
~32Fully busy, nothing queueing — ideal
~64Twice as much runnable work as there are threads; everything runs at roughly half speed
~256Badly oversubscribed — someone likely launched a job with the wrong core count

The three numbers show direction. 120 / 40 / 15 means load spiked in the last minute; 15 / 40 / 120 means a big job just finished and the box is draining. Equal numbers mean steady state.

High load with low CPU is an I/O problem, not a CPU one. Because uninterruptible I/O wait counts toward load, a stuck NFS mount to the NAS can drive load into the hundreds while the CPU column sits near zero. That combination is the most useful thing this column tells you.

Snapshot age

Nothing here is live-streamed. Each host writes its own snapshot from cron once a minute; this page fetches all 11 of them in parallel when you load it and stores nothing. So an age of a few seconds up to a minute is normal, and the CPU numbers can be up to a minute behind what is happening right now.

Past 10 minutes a host is flagged stale — its collector has most likely stopped. A host that could not be reached at all is marked down and is left out of every total on this page.

Hosts reporting
11 / 11
Active users
31
1379 processes
CPU in use
182.6 / 352 threads
Memory in use
183.8 GB / 1.6 TB
Usage by user CPU 100% = one thread. Memory is PSS.
User CPU Memory Procs Hosts
agw2 8819.4%
88.2 threads
49.7 GB
160 leviathan8, leviathan10, leviathan11, leviathan9, leviathan5, leviathan7
Host CPU Threads Memory Procs Top command
leviathan8 2766.0% 27.7 16.8 GB 42 python
leviathan10 2758.3% 27.6 15.7 GB 42 python
leviathan11 1737.3% 17.4 16.1 GB 40 python
leviathan9 1557.8% 15.6 1.1 GB 14 python
leviathan5 0.0% 0.0 21.8 MB 12 bash
leviathan7 0.0% 0.0 33.2 MB 10 bash
seunghwn 5337.9%
53.4 threads
55.8 GB
64 leviathan1, leviathan2, leviathan11, leviathan6
Host CPU Threads Memory Procs Top command
leviathan1 1461.8% 14.6 14.3 GB 16 R
leviathan2 1372.5% 13.7 7.9 GB 16 R
leviathan11 1260.9% 12.6 21.0 GB 16 R
leviathan6 1242.7% 12.4 12.7 GB 16 R
plerdput 3398.2%
34.0 threads
16.1 GB
150 leviathan6, leviathan4, leviathan9, leviathan5, leviathan8, leviathan1, leviathan3
Host CPU Threads Memory Procs Top command
leviathan6 1329.1% 13.3 3.3 GB 33 R
leviathan4 1045.1% 10.5 5.2 GB 34 R
leviathan9 976.5% 9.8 4.1 GB 30 R
leviathan5 46.5% 0.5 2.5 GB 27 R
leviathan8 1.0% 0.0 521.0 MB 1 rsession
leviathan1 0.0% 0.0 579.5 MB 13 rsession
leviathan3 0.0% 0.0 19.3 MB 12 xdg-permission-
scolando 500.0%
5.0 threads
3.8 GB
50 leviathan3, leviathan4
Host CPU Threads Memory Procs Top command
leviathan3 500.0% 5.0 3.8 GB 38 R
leviathan4 0.0% 0.0 37.6 MB 12 code-a5b5009513
austinfe 198.0%
2.0 threads
2.2 GB
3 leviathan1, leviathan3, leviathan2
Host CPU Threads Memory Procs Top command
leviathan1 99.0% 1.0 1.2 GB 1 rsession
leviathan3 99.0% 1.0 475.0 MB 1 rsession
leviathan2 0.0% 0.0 559.7 MB 1 rsession
pallotey 2.7%
0.0 threads
45.8 GB
123 leviathan11, leviathan2, leviathan3, leviathan4, leviathan5, leviathan6, leviathan7, leviathan8, leviathan9, leviathan10
Host CPU Threads Memory Procs Top command
leviathan11 2.7% 0.0 307.1 MB 17 R
leviathan2 0.0% 0.0 45.1 GB 11 cpptools
leviathan3 0.0% 0.0 62.7 MB 16 code-1b6a188127
leviathan4 0.0% 0.0 27.8 MB 11 code-1b6a188127
leviathan5 0.0% 0.0 55.8 MB 15 code-1b6a188127
leviathan6 0.0% 0.0 32.1 MB 11 code-1b6a188127
leviathan7 0.0% 0.0 52.3 MB 13 code-f6cfa2ea24
leviathan8 0.0% 0.0 51.9 MB 9 code-f6cfa2ea24
leviathan9 0.0% 0.0 43.8 MB 9 code-f6cfa2ea24
leviathan10 0.0% 0.0 38.5 MB 11 code-f6cfa2ea24
arosenga 2.0%
0.0 threads
6.9 GB
67 leviathan5, leviathan7, leviathan4
Host CPU Threads Memory Procs Top command
leviathan5 1.0% 0.0 2.8 GB 30 node
leviathan7 1.0% 0.0 4.1 GB 24 node
leviathan4 0.0% 0.0 17.4 MB 13 bash
lpreimes 1.0%
0.0 threads
746.7 MB
90 leviathan9, leviathan3, leviathan4, leviathan5, leviathan6, leviathan7
Host CPU Threads Memory Procs Top command
leviathan9 1.0% 0.0 464.2 MB 22 2.1.258
leviathan3 0.0% 0.0 69.0 MB 14 code-110a328ea5
leviathan4 0.0% 0.0 69.8 MB 16 code-110a328ea5
leviathan5 0.0% 0.0 69.2 MB 16 code-a5b5009513
leviathan6 0.0% 0.0 46.7 MB 13 code-a5b5009513
leviathan7 0.0% 0.0 27.9 MB 9 code-f6cfa2ea24
ajshen 1.0%
0.0 threads
571.4 MB
18 leviathan3
gthivier 0.0%
0.0 threads
399.9 MB
24 leviathan7, leviathan8
Host CPU Threads Memory Procs Top command
leviathan7 0.0% 0.0 382.3 MB 15 bash
leviathan8 0.0% 0.0 17.5 MB 9 bash
mwieckso 0.0%
0.0 threads
322.6 MB
122 leviathan7, leviathan8, leviathan9, leviathan10, leviathan11
Host CPU Threads Memory Procs Top command
leviathan7 0.0% 0.0 33.3 MB 24 bash
leviathan8 0.0% 0.0 88.8 MB 25 bash
leviathan9 0.0% 0.0 91.1 MB 27 bash
leviathan10 0.0% 0.0 30.4 MB 24 bash
leviathan11 0.0% 0.0 79.0 MB 22 bash
haolee 0.0%
0.0 threads
203.3 MB
11 leviathan11
jinoojwcho 0.0%
0.0 threads
166.0 MB
83 leviathan1, leviathan4, leviathan5, leviathan6, leviathan7, leviathan8, leviathan9
Host CPU Threads Memory Procs Top command
leviathan1 0.0% 0.0 25.1 MB 12 xdg-permission-
leviathan4 0.0% 0.0 20.2 MB 12 bash
leviathan5 0.0% 0.0 22.3 MB 13 bash
leviathan6 0.0% 0.0 34.1 MB 16 bash
leviathan7 0.0% 0.0 22.4 MB 11 sleep
leviathan8 0.0% 0.0 18.5 MB 9 bash
leviathan9 0.0% 0.0 23.3 MB 10 bash
zlai2 0.0%
0.0 threads
150.3 MB
37 leviathan1
junheeo 0.0%
0.0 threads
129.4 MB
65 leviathan1, leviathan3, leviathan6, leviathan7, leviathan9, leviathan11
Host CPU Threads Memory Procs Top command
leviathan1 0.0% 0.0 20.7 MB 12 bash
leviathan3 0.0% 0.0 16.9 MB 12 bash
leviathan6 0.0% 0.0 21.3 MB 12 bash
leviathan7 0.0% 0.0 16.4 MB 9 bash
leviathan9 0.0% 0.0 19.5 MB 9 bash
leviathan11 0.0% 0.0 34.4 MB 11 vim
diegomar 0.0%
0.0 threads
124.0 MB
34 leviathan2, leviathan5
Host CPU Threads Memory Procs Top command
leviathan2 0.0% 0.0 38.1 MB 13 bash
leviathan5 0.0% 0.0 85.9 MB 21 code-df53daabb1
ozh 0.0%
0.0 threads
92.4 MB
24 leviathan1, leviathan3
Host CPU Threads Memory Procs Top command
leviathan1 0.0% 0.0 56.0 MB 13 code-110a328ea5
leviathan3 0.0% 0.0 36.4 MB 11 code-110a328ea5
woonyouc 0.0%
0.0 threads
91.8 MB
45 leviathan7, leviathan8, leviathan9, leviathan10, leviathan11
Host CPU Threads Memory Procs Top command
leviathan7 0.0% 0.0 16.8 MB 9 bash
leviathan8 0.0% 0.0 18.4 MB 9 bash
leviathan9 0.0% 0.0 20.3 MB 9 bash
leviathan10 0.0% 0.0 16.3 MB 9 bash
leviathan11 0.0% 0.0 20.0 MB 9 bash
efranke 0.0%
0.0 threads
81.9 MB
31 leviathan3, leviathan7, leviathan8
Host CPU Threads Memory Procs Top command
leviathan3 0.0% 0.0 41.7 MB 13 code-125df4672b
leviathan7 0.0% 0.0 19.4 MB 9 bash
leviathan8 0.0% 0.0 20.9 MB 9 bash
ftownes 0.0%
0.0 threads
67.7 MB
22 leviathan1, leviathan3
Host CPU Threads Memory Procs Top command
leviathan1 0.0% 0.0 32.7 MB 11 code-110a328ea5
leviathan3 0.0% 0.0 34.9 MB 11 code-110a328ea5
dhuertao 0.0%
0.0 threads
53.7 MB
33 leviathan4, leviathan6, leviathan8
Host CPU Threads Memory Procs Top command
leviathan4 0.0% 0.0 17.7 MB 12 bash
leviathan6 0.0% 0.0 18.7 MB 12 bash
leviathan8 0.0% 0.0 17.3 MB 9 bash
nynke 0.0%
0.0 threads
48.6 MB
15 leviathan5
axin 0.0%
0.0 threads
42.7 MB
21 leviathan3, leviathan8
Host CPU Threads Memory Procs Top command
leviathan3 0.0% 0.0 21.5 MB 12 bash
leviathan8 0.0% 0.0 21.2 MB 9 bash
mohanx 0.0%
0.0 threads
38.0 MB
11 leviathan1
minghao3 0.0%
0.0 threads
37.7 MB
11 leviathan1
yingzhu4 0.0%
0.0 threads
36.7 MB
11 leviathan1
ltesta 0.0%
0.0 threads
34.8 MB
11 leviathan6
jacobg2 0.0%
0.0 threads
31.1 MB
11 leviathan4
tsaidi 0.0%
0.0 threads
27.4 MB
8 leviathan10
davidwec 0.0%
0.0 threads
20.9 MB
12 leviathan1
gaihara 0.0%
0.0 threads
20.6 MB
12 leviathan1
Hosts
Host CPU Memory used Users Snapshot age
leviathan1
32 threads
1560.8% of 3200%
25.6 GB / 123.4 GB
16.7 / 16.52 / 16.52 13 50s ago
leviathan2
32 threads
1372.5% of 3200%
64.7 GB / 123.4 GB
16.94 / 16.87 / 15.44 4 50s ago
leviathan3
32 threads
600.0% of 3200%
30.4 GB / 123.4 GB
10.22 / 10.6 / 10.7 11 51s ago
leviathan4
32 threads
1045.1% of 3200%
10.4 GB / 123.4 GB
5.07 / 6.95 / 7.82 8 51s ago
leviathan5
32 threads
47.5% of 3200%
9.7 GB / 123.4 GB
14.29 / 13.7 / 13.14 8 51s ago
leviathan6
32 threads
2571.8% of 3200%
27.5 GB / 123.4 GB
26.93 / 25.91 / 25.48 8 50s ago
leviathan7
32 threads
1.0% of 3200%
10.5 GB / 186.3 GB
0.13 / 0.03 / 0.01 10 51s ago
leviathan8
32 threads
2767.0% of 3200%
29.8 GB / 186.3 GB
28.05 / 28.03 / 28.05 10 50s ago
leviathan9
32 threads
2535.3% of 3200%
23.5 GB / 186.3 GB
22.38 / 23.52 / 24.06 8 51s ago
leviathan10
32 threads
2758.3% of 3200%
23.1 GB / 186.3 GB
28.18 / 28.14 / 28.05 5 50s ago
leviathan11
32 threads
3000.9% of 3200%
52.6 GB / 186.3 GB
43.28 / 43.51 / 43.7 7 49s ago