All our customers are on Linux, They have multiple flavors of Linux actually – Ubuntu, CentOS, RedHat Linux, Oracle Linux, SUSE Linux etc. Though we are an full-service everything MySQL shop, Our consulting, support and managed services are never restricted to only MySQL Ops. , We are experts in Linux, \u00a0DevOps and Site Reliability Engineering (SRE). We have proven methods to deliver Linux performance audit \/ health check \/ diagnostics and recommendations. What are the tools we use for monitoring Linux ops. ? This post is about those tools we use regularly in MinervaDB for monitoring Linux operations:<\/p>\n

How long Linux server is up and running ?\u00a0<\/strong><\/p>\n

[root@localhost ~]# uptime \r\n 12:32:56 up 1800 min,  83 users,  load average: 88.01, 88.52, 88.64\r\n[root@localhost ~]#<\/pre>\n
Print all the processes running as root<\/strong><\/p>\n
[root@localhost ~]# ps -U root -u root \r\n  PID TTY          TIME CMD\r\n    1 ?        00:00:00 systemd\r\n    2 ?        00:00:00 kthreadd\r\n    3 ?        00:00:00 ksoftirqd\/0\r\n    4 ?        00:00:00 kworker\/0:0\r\n    5 ?        00:00:00 kworker\/0:0H\r\n    6 ?        00:00:00 kworker\/u2:0\r\n    7 ?        00:00:00 migration\/0\r\n    8 ?        00:00:00 rcu_bh\r\n    9 ?        00:00:00 rcu_sched\r\n   10 ?        00:00:00 watchdog\/0\r\n   12 ?        00:00:00 kdevtmpfs\r\n   13 ?        00:00:00 netns\r\n   14 ?        00:00:00 khungtaskd\r\n   15 ?        00:00:00 writeback\r\n   16 ?        00:00:00 kintegrityd\r\n   17 ?        00:00:00 bioset\r\n   18 ?        00:00:00 kblockd\r\n<\/pre>\n
 <\/p>\n
[root@localhost ~]# ps -A\r\n\r\n[root@localhost ~]# ps -A\r\n  PID TTY          TIME CMD\r\n    1 ?        00:00:00 systemd\r\n    2 ?        00:00:00 kthreadd\r\n    3 ?        00:00:00 ksoftirqd\/0\r\n    4 ?        00:00:00 kworker\/0:0\r\n\r\n[root@localhost ~]# ps -e \r\n  PID TTY          TIME CMD\r\n    1 ?        00:00:00 systemd\r\n    2 ?        00:00:00 kthreadd\r\n    3 ?        00:00:00 ksoftirqd\/0\r\n    4 ?        00:00:00 kworker\/0:0\r\n    5 ?        00:00:00 kworker\/0:0H\r\n<\/pre>\n
Print all the processes owned by MySQL<\/p>\n
[root@localhost ~]# ps -fG mysql \r\nUID        PID  PPID  C STIME TTY          TIME CMD\r\nmysql     1165     1  0 10:54 ?        00:00:00 \/usr\/sbin\/mysqld --daemonize --pid-file=\/var\/run\/mysqld\/mysqld.pid\r\n[root@localhost ~]# \r\n<\/pre>\n
Top\u00a0<\/strong><\/p>\n
Displays all actively running processes in real-time, The matrices addressed in Linux “top” command are CPU usage, Memory usage, Swap Memory, Cache Size, Buffer Size, Process PID, User, Commands<\/strong>etc.<\/p>\n
[root@localhost ~]# top\r\n\r\ntop - 11:37:31 up 43 min,  3 users,  load average: 0.50, 0.15, 0.08\r\nTasks:  90 total,   1 running,  89 sleeping,   0 stopped,   0 zombie\r\n%Cpu(s):100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st\r\nKiB Mem :  1016156 total,   541664 free,   307024 used,   167468 buff\/cache\r\nKiB Swap:  2097148 total,  2097148 free,        0 used.   553344 avail Mem \r\n\r\n  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                            \r\n 1412 root      20   0   31068   1720   1324 S 99.7  0.2   0:43.38 sysbench                                           \r\n    1 root      20   0  125572   4096   2496 S  0.0  0.4   0:00.76 systemd                                            \r\n    2 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kthreadd                                           \r\n    3 root      20   0       0      0      0 S  0.0  0.0   0:00.04 ksoftirqd\/0                                        \r\n    5 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kworker\/0:0H                                       \r\n    6 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kworker\/u2:0                                       \r\n    7 root      rt   0       0      0      0 S  0.0  0.0   0:00.00 migration\/0                                        \r\n    8 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcu_bh                                             \r\n    9 root      20   0       0      0      0 S  0.0  0.0   0:00.15 rcu_sched                                          \r\n<\/pre>\n
Press “c” option in running top command to know absolute path of running process<\/p>\n
top - 11:43:20 up 49 min,  3 users,  load average: 1.27, 0.88, 0.43\r\nTasks:  90 total,   1 running,  88 sleeping,   1 stopped,   0 zombie\r\n%Cpu(s): 99.2 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.8 si,  0.0 st\r\nKiB Mem :  1016156 total,   540768 free,   307752 used,   167636 buff\/cache\r\nKiB Swap:  2097148 total,  2097148 free,        0 used.   552484 avail Mem \r\n\r\n  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                            \r\n 1412 root      20   0   31068   1720   1324 S 99.7  0.2   6:31.61 sysbench --test=cpu --cpu-max-prime=300000000 run  \r\n 1415 root      20   0       0      0      0 S  0.3  0.0   0:00.14 [kworker\/0:3]                                      \r\n    1 root      20   0  125572   4096   2496 S  0.0  0.4   0:00.77 \/usr\/lib\/systemd\/systemd --switched-root --system +\r\n    2 root      20   0       0      0      0 S  0.0  0.0   0:00.00 [kthreadd]                                         \r\n    3 root      20   0       0      0      0 S  0.0  0.0   0:00.04 [ksoftirqd\/0]                                      \r\n    5 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 [kworker\/0:0H]                                     \r\n    6 root      20   0       0      0      0 S  0.0  0.0   0:00.00 [kworker\/u2:0]                                     \r\n<\/pre>\n
Monitor CPU statistics using IOSTAT<\/p>\n
[root@localhost ~]# iostat -c\r\nLinux 3.10.0-693.21.1.el7.x86_64 (localhost.localdomain) \t04\/21\/2018 \t_x86_64_\t(1 CPU)\r\n\r\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\r\n           4.73    0.00    0.28    0.15    0.00   94.84\r\n<\/pre>\n
Top five CPU consuming processes<\/p>\n
[root@localhost ~]# watch \"ps aux | sort -nrk 3,3 | head -n 5\"\r\n\r\nEvery 2.0s: ps aux | sort -nrk 3,3 | head -n 5                                                          Sat Apr 21 15:44:49 2018\r\n\r\nroot\t  2658 99.8  0.2  39160  2816 pts\/0    Sl+  15:38   6:41 sysbench --test=cpu --cpu-max-prime=30000000 --num-threads=120\r\nmysql     1165  1.8 24.4 1152160 248112 ?      Sl   10:54   5:20 \/usr\/sbin\/mysqld --daemonize --pid-file=\/var\/run\/mysqld\/mysqld.\r\nUSER\t   PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND\r\nroot\t   932  0.0  0.0 113372   940 ?        S    10:54   0:00 \/sbin\/dhclient -d -q -sf \/usr\/libexec\/nm-dhcp-helper -pf \/var\/r\r\nroot        92  0.0  0.0      0     0 ?        S    10:54   0:00 [kauditd]\r\n<\/pre>\n
Monitor % of CPU and memory consumption by Linux processes<\/p>\n
[root@localhost ~]# ps aux k-pcpu | head -6\r\nUSER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND\r\nroot      2658 99.8  0.2  39160  2816 pts\/0    Sl+  15:38  10:05 sysbench --test=cpu --cpu-max-prime=30000000 --num-threads=120 run\r\nmysql     1165  1.8 24.4 1152160 248112 ?      Sl   10:54   5:20 \/usr\/sbin\/mysqld --daemonize --pid-file=\/var\/run\/mysqld\/mysqld.pid\r\nroot         1  0.0  0.2 125572  2148 ?        Ss   10:54   0:00 \/usr\/lib\/systemd\/systemd --switched-root --system --deserialize 21\r\nroot         2  0.0  0.0      0     0 ?        S    10:54   0:00 [kthreadd]\r\nroot         3  0.0  0.0      0     0 ?        S    10:54   0:00 [ksoftirqd\/0]\r\n<\/pre>\n
Monitor active and inactive memory<\/strong><\/p>\n
[root@localhost ~]# vmstat -a \r\nprocs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----\r\n r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa st\r\n 2  0      0 540052 114584 294208    0    0    52    11  131   76  7  0 93  0  0\r\n[root@localhost ~]# \r\n<\/pre>\n
Monitoring \u00a0memory usage with timestamp<\/p>\n
[root@localhost ~]# vmstat -t 1 50\r\nprocs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- -----timestamp-----\r\n r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st                 IST\r\n 2  0      0 540508   2108 165580    0    0    22     5  146   65  9  0 91  0  0 2018-04-21 13:01:38\r\n 2  0      0 540508   2108 165580    0    0     0     0  300   76 100  0  0  0  0 2018-04-21 13:01:39\r\n 2  0      0 540508   2108 165580    0    0     0     0  283   76 100  0  0  0  0 2018-04-21 13:01:40\r\n 1  0      0 540508   2108 165580    0    0     0     0  246   73 100  0  0  0  0 2018-04-21 13:01:41\r\n 1  0      0 540508   2108 165580    0    0     0     0  280   84 96  4  0  0  0 2018-04-21 13:01:42\r\n 2  0      0 540508   2108 165576    0    0     0    24  337   93 100  0  0  0  0 2018-04-21 13:01:43\r\n 1  0      0 540508   2108 165576    0    0     0     0  283   79 100  0  0  0  0 2018-04-21 13:01:44\r\n<\/pre>\n
Monitoring top five memory consuming processes<\/p>\n
[root@localhost ~]# ps -eo pid,comm,%cpu,%mem --sort=-%mem | head -n 5\r\n  PID COMMAND         %CPU %MEM\r\n 1165 mysqld           1.8 24.4\r\n 2405 dhclient         0.0  1.5\r\n  774 NetworkManager   0.0  0.4\r\n 1137 tuned            0.0  0.2\r\n<\/pre>\n
Monitoring disk I\/O statistics<\/strong><\/p>\n
[root@localhost ~]# iostat -d \r\nLinux 3.10.0-693.21.1.el7.x86_64 (localhost.localdomain) \t04\/21\/2018 \t_x86_64_\t(1 CPU)\r\n\r\nDevice:            tps    kB_read\/s    kB_wrtn\/s    kB_read    kB_wrtn\r\nsda              27.68      1191.80      3866.67   13191808   42799263\r\ndm-0             22.33      1185.06      3838.77   13117158   42490503\r\ndm-1              8.38         5.99        27.71      66324     306712\r\n<\/pre>\n
More detailed disk I\/O monitoring at process level using \u00a0‘iotop’\u00a0<\/strong><\/p>\n
To monitor disk I\/O more detailed at process level (very much real-time) we use ‘iotop’<\/p>\n
Total DISK READ :     269.52 M\/s | Total DISK WRITE :      50.85 M\/s\r\nActual DISK READ:     269.52 M\/s | Actual DISK WRITE:      51.96 M\/s\r\n  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                  \r\n 1821 be\/4 root     1564.46 K\/s  204.70 K\/s  0.00 % 50.39 % sysbench fileio --thr~le-test-mode=rndrw run\r\n 1893 be\/4 root      720.09 K\/s  190.07 K\/s  0.00 % 42.68 % sysbench fileio --thr~le-test-mode=rndrw run\r\n 2080 be\/4 root      789.54 K\/s  190.07 K\/s  0.00 % 42.23 % sysbench fileio --thr~le-test-mode=rndrw run\r\n 1867 be\/4 root     1286.66 K\/s  190.07 K\/s  0.00 % 41.35 % sysbench fileio --thr~le-test-mode=rndrw run\r\n 1870 be\/4 root      321.66 K\/s  160.83 K\/s  0.00 % 41.16 % sysbench fileio --thr~le-test-mode=rndrw run\r\n 1833 be\/4 root     1257.41 K\/s  204.70 K\/s  0.00 % 40.66 % sysbench fileio --thr~le-test-mode=rndrw run\r\n 1934 be\/4 root      789.54 K\/s  190.07 K\/s  0.00 % 40.15 % sysbench fileio --thr~le-test-mode=rndrw run\r\n 2035 be\/4 root     1286.66 K\/s  204.70 K\/s  0.00 % 39.73 % sysbench fileio --thr~le-test-mode=rndrw run\r\n 1975 be\/4 root      745.68 K\/s  190.07 K\/s  0.00 % 39.49 % sysbench fileio --thr~le-test-mode=rndrw run\r\n 1851 be\/4 root     1579.08 K\/s  190.07 K\/s  0.00 % 39.36 % sysbench fileio --thr~le-test-mode=rndrw run\r\n 1836 be\/4 root      877.27 K\/s  190.07 K\/s  0.00 % 39.35 % sysbench fileio --thr~le-test-mode=rndrw run\r\n 2001 be\/4 root      160.83 K\/s  131.59 K\/s  0.00 % 39.34 % sysbench fileio --thr~le-test-mode=rndrw run\r\n 1879 be\/4 root     1842.26 K\/s  190.07 K\/s  0.00 % 39.22 % sysbench fileio --thr~le-test-mode=rndrw run\r\n 1872 be\/4 root      263.18 K\/s  204.70 K\/s  0.00 % 38.48 % sysbench fileio --thr~le-test-mode=rndrw run\r\n 1953 be\/4 root        2.81 M\/s  146.21 K\/s  0.00 % 38.35 % sysbench fileio --thr~le-test-mode=rndrw run\r\n 1941 be\/4 root      292.42 K\/s  277.80 K\/s  0.00 % 38.31 % sysbench fileio --thr~le-test-mode=rndrw run\r\n 1913 be\/4 root     1345.14 K\/s  204.70 K\/s  0.00 % 38.02 % sysbench fileio --thr~le-test-mode=rndrw run\r\n 2017 be\/4 root      628.71 K\/s  160.83 K\/s  0.00 % 37.93 % sysbench fileio --thr~le-test-mode=rndrw run\r\n 2040 be\/4 root      745.68 K\/s  190.07 K\/s  0.00 % 37.61 % sysbench fileio --thr~le-test-mode=rndrw run\r\n 1942 be\/4 root      555.60 K\/s  160.83 K\/s  0.00 % 37.49 % sysbench fileio --thr~le-test-mode=rndrw run\r\n 1980 be\/4 root      233.94 K\/s  219.32 K\/s  0.00 % 37.49 % sysbench fileio --thr~le-test-mode=rndrw run<\/pre>\n
This is how usually we begin a MySQL performance benchmarking and audit project, We first understand the load in the Linux and from there our matrices will be completely MySQL biased, We deliver very detailed performance optimization recommendation, which will \/ can be used by our customers for performance optimization and capacity planning \/ sizing.<\/p>\n","protected":false},"excerpt":{"rendered":"
All our customers are on Linux, They have multiple flavors of Linux actually – Ubuntu, CentOS, RedHat Linux, Oracle Linux, SUSE Linux etc. Though we are an full-service everything MySQL shop, Our consulting, support and […]<\/a><\/p>\n<\/div>","protected":false},"author":10,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2,15,56],"tags":[5,124],"yoast_head":"\n