Performance Troubleshooting

Useful commands for investigating & improving system performance problems.

Linux

atop

atop - AT Computing’s System & Process Monitor

This utility is far more feature rich than plain top.

top

Press '1' to see all CPUs.
Press 'c' to view full command lines.
Press 'H' show threads
Press 'r' renice a process (e.g. -19 to boost it's priority in the scheduler)

vmstat

vmstat 5

Watch for high numbers in these columns

procs/b - blocked processes, anything more than 0 is bad (why?)
cpu/wa - wait indicates system is I/O bound
system/cs - context switches, system is CPU bound... to many processes. Consider tickless or kernel divider options?

iostat

iostat comes with the sysstat package. the await and %util numbers from iostat -x are probably the best indicators as to how a disk is doing

iostat -dx 3

Another interesting command that lets you see I/O to all devices at once & over time is

watch iostat -p ALL

FreeBSD

FreeBSD also has top, ps, vmstat and iostat. However there can be subtle to significant differences in the output and usage.

iostat -h 5
tty ad0 cpu
tin tout KB/t tps MB/s us ni sy in id
0 7 22.01 3 0.07 1 0 1 0 99
0 18 27.20 4 0.11 0 0 1 1 98
0 56 64.00 0 0.02 2 0 1 0 97
0 6 26.74 3 0.07 1 0 1 1 97
0 6 0.00 0 0.00 1 0 1 1 98