Fixing performance-related points within the IT world is all the time a problem, and if you do not have the appropriate instruments, it could be irritating.
For those who work as a assist engineer in a manufacturing surroundings, you’ll most definitely encounter performance-related points within the Linux surroundings. Let’s undergo among the mostly used Linux command line instruments to diagnose efficiency associated points.
Comment: A number of the instructions under will not be put in by default, so you may want to put in them manually.
lsof
lsof stands for “checklist of open recordsdata”that can assist you discover all open recordsdata and processes together with whoever opened them. The lsof utility could be helpful in some eventualities.
To provide an outline: all recordsdata opened by a given PID.
# lsof –p PID
Rely the variety of recordsdata and processes
[root@localhost ~]# lsof -p 4271 | wc -l 34 [root@localhost ~]#
Examine the presently open log file
# lsof –p | grep log
Discover out the port quantity utilized by the method
lsof -i -P |grep $PID[root@localhost ~]# lsof -i -P |grep 4271 nginx 4271 root 6u IPv4 51306 0t0 TCP *:80 (LISTEN) nginx 4271 root 7u IPv4 51307 0t0 TCP *:443 (LISTEN) [root@localhost ~]#
See extra examples of lsof instructions.
pidstat
pidstat can be utilized to observe duties managed by the Linux kernel. This command lets you simply troubleshoot I/O associated points.
Listing I/O statistics of all PIDs
# pidstat –d
To maneuver I/O statistics for a given PID
# pidstat –p 4271 –d
If you’re performing real-time troubleshooting for a selected course of, you possibly can verify the I/O at intervals. The instance under checks each 5 seconds.
[root@localhost ~]# pidstat -p 4362 -d 5 Linux 3.10.0-327.13.1.el7.x86_64 (localhost.localdomain) 08/13/2016 _x86_64_ (2 CPU) 07:01:30 PM UID PID kB_rd/s kB_wr/s kB_ccwr/s Command 07:01:35 PM 0 4362 0.00 0.00 0.00 nginx 07:01:40 PM 0 4362 0.00 0.00 0.00 nginx 07:01:45 PM 0 4362 0.00 0.00 0.00 nginx 07:01:50 PM 0 4362 0.00 0.00 0.00 nginx
high
Most likely probably the most used instructions on Linux could be high notch. The highest command can be utilized to show system abstract info and present utilization.
Simply by working the highest command, you possibly can see the CPU utilization, course of particulars, some duties, reminiscence utilization, some zombie processes, and so on.
high - 11:48:43 up 13 days, 17:25, 1 person, load common: 0.00, 0.00, 0.00
Duties: 90 whole, 2 working, 88 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.3 us, 0.0 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hello, 0.0 si, 0.0 st
MiB Mem : 1829.7 whole, 388.1 free, 220.3 used, 1221.4 buff/cache
MiB Swap: 0.0 whole, 0.0 free, 0.0 used. 1369.4 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 186820 13400 9092 S 0.0 0.7 0:10.87 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.13 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp
6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/0:0H
8 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_percpu_wq
9 root 20 0 0 0 0 S 0.0 0.0 0:07.35 ksoftirqd/0
10 root 20 0 0 0 0 R 0.0 0.0 0:07.30 rcu_sched
11 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
12 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/0
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
16 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 netns
17 root 20 0 0 0 0 S 0.0 0.0 0:00.68 kauditd
18 root 20 0 0 0 0 S 0.0 0.0 0:00.25 khungtaskd
19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 oom_reaper
20 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 writeback
21 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kcompactd0
22 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd
23 root 39 19 0 0 0 S 0.0 0.0 0:05.63 khugepaged
24 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 crypto
25 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kintegrityd
26 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kblockdTo show course of particulars for a particular person
# high –u username
To complete the method, you possibly can run the highest and press okay. You’ll be prompted to enter the PID you wish to disable.
high - 11:49:39 up 13 days, 17:26, 1 person, load common: 0.00, 0.00, 0.00
Duties: 91 whole, 1 working, 90 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.3 us, 0.0 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hello, 0.0 si, 0.0 st
MiB Mem : 1829.7 whole, 386.9 free, 221.4 used, 1221.4 buff/cache
MiB Swap: 0.0 whole, 0.0 free, 0.0 used. 1368.3 avail Mem
PID to sign/kill [default pid = 21261]
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5834 nginx 20 0 148712 7428 4800 S 0.0 0.4 0:02.37 nginxPS
PS stands for course of standing and generally used command to get a snapshot of the working course of. Very helpful to search out out whether or not a course of is working or not and whether it is working it is going to print PID.
To seek out out the PID and course of particulars with a phrase
[root@lab ~]# ps -ef|grep nginx
root 5833 1 0 May24 ? 00:00:00 nginx: grasp course of /usr/sbin/nginx
nginx 5834 5833 0 May24 ? 00:00:02 nginx: employee course of
root 21267 18864 0 11:50 pts/0 00:00:00 grep --color=auto nginx
[root@lab ~]#tcpdump
Community troubleshooting is all the time a problem, and one of many important instructions to make use of is tcpdump.
You should utilize tcpdump to seize the community packets on a community interface.
To seize the packets on a selected community interface
[root@lab ~]# tcpdump -i eth0 -w /tmp/seize
tcpdump: listening on eth0, link-type EN10MB (Ethernet), seize dimension 262144 bytes
^C9 packets captured
16 packets acquired by filter
0 packets dropped by kernel
[root@lab ~]#As you possibly can see above, the site visitors circulate has been accommodated eth0 couple.
To seize community site visitors between supply and vacation spot IP
# tcpdump src $IP and dst host $IP
Seize community site visitors for vacation spot port 443
# tcpdump dst port 443 tcpdump: knowledge hyperlink kind PKTAP tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on pktap, link-type PKTAP (Packet Faucet), seize dimension 262144 bytes 12:02:30.833845 IP 192.168.1.2.49950 > ec2-107-22-185-206.compute-1.amazonaws.com.https: Flags [.], ack 421458229, win 4096, size 0 12:02:32.076893 IP 192.168.1.2.49953 > 104.25.133.107.https: Flags [S], seq 21510813, win 65535, choices [mss 1460,nop,wscale 5,nop,nop,TS val 353259990 ecr 0,sackOK,eol], size 0 12:02:32.090389 IP 192.168.1.2.49953 > 104.25.133.107.https: Flags [.], ack 790725431, win 8192, size 0 12:02:32.090630 IP 192.168.1.2.49953 > 104.25.133.107.https: Flags [P.], seq 0:517, ack 1, win 8192, size 517 12:02:32.109903 IP 192.168.1.2.49953 > 104.25.133.107.https: Flags [.], ack 147, win 8187, size 0
Learn the captured file
# tcpdump –r filenameFor instance: to learn a file captured above
# tcpdump –r /tmp/take a look atStudy extra about tcpdump to seize and analyze community site visitors.
iostat
iostat stands for input-output statistics and infrequently used to diagnose a efficiency challenge with storage gadgets. You’ll be able to monitor the CPU, system and community file system utilization report with iostat.
View disk I/O statistics
[root@localhost ~]# iostat -d Linux 3.10.0-327.13.1.el7.x86_64 (localhost.localdomain) 08/13/2016 _x86_64_ (2 CPU) System: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 1.82 55.81 12.63 687405 155546 [root@localhost ~]#
View CPU statistics
[root@localhost ~]# iostat -c Linux 3.10.0-327.13.1.el7.x86_64 (localhost.localdomain) 08/13/2016 _x86_64_ (2 CPU) avg-cpu: %person %good %system %iowait %steal %idle 0.59 0.02 0.33 0.54 0.00 98.52 [root@localhost ~]#
ldd
ldd stands for checklist of dynamic dependencies to checklist the shared libraries that the library wants. The ldd command could be helpful to diagnose utility startup drawback.
If a program does not begin as a result of dependencies aren’t obtainable, you should utilize ldd to search out out which shared libraries it is on the lookout for.
[root@localhost sbin]# ldd httpd linux-vdso.so.1 => (0x00007ffe7ebb2000) libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fa4d451e000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fa4d42f9000) libaprutil-1.so.0 => /lib64/libaprutil-1.so.0 (0x00007fa4d40cf000) libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fa4d3e98000) libexpat.so.1 => /lib64/libexpat.so.1 (0x00007fa4d3c6e000) libdb-5.3.so => /lib64/libdb-5.3.so (0x00007fa4d38af000) libapr-1.so.0 => /lib64/libapr-1.so.0 (0x00007fa4d3680000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa4d3464000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fa4d325f000) libc.so.6 => /lib64/libc.so.6 (0x00007fa4d2e9e000) liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fa4d2c79000) /lib64/ld-linux-x86-64.so.2 (0x00007fa4d4a10000) libuuid.so.1 => /lib64/libuuid.so.1 (0x00007fa4d2a73000) libfreebl3.so => /lib64/libfreebl3.so (0x00007fa4d2870000) [root@localhost sbin]#
netstat
netstat (Community statistics) is a well-liked command to print community connections and interface statistics and to unravel numerous network-related issues.
To indicate statistics of all protocols
# netstat –s
You should utilize grep to search out out if there are any errors
[root@localhost sbin]# netstat -s | grep error 0 packet obtain errors 0 obtain buffer errors 0 ship buffer errors [root@localhost sbin]#
To show the kernel routing desk
[root@localhost sbin]# netstat -r Kernel IP routing desk Vacation spot Gateway Genmask Flags MSS Window irtt Iface default gateway 0.0.0.0 UG 0 0 0 eno16777736 172.16.179.0 0.0.0.0 255.255.255.0 U 0 0 0 eno16777736 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 [root@localhost sbin]#
Uncover extra examples of netstat instructions.
free
In case your Linux server is working out of reminiscence or simply wish to know the way a lot reminiscence is offered in obtainable reminiscence, the free command will enable you.
[root@localhost sbin]# free -g whole used free shared buff/cache obtainable Mem: 5 0 3 0 1 4 Swap: 5 0 5 [root@localhost sbin]#
-g means to show the small print in GB. As you possibly can see, the whole obtainable reminiscence is 5 GB and three GB is free.
sar
sar (System exercise report) might be helpful to gather a part of a report together with CPU, reminiscence and system load.
By merely executing sar command reveals you the system utilization for the entire day.
By default, the utilization report is saved inside 10 minutes. For those who want one thing shorter in actual time, you should utilize it as under.
Present CPU report 3 occasions each 3 seconds
[root@localhost sbin]# sar 3 2 Linux 3.10.0-327.13.1.el7.x86_64 (localhost.localdomain) 08/13/2016 _x86_64_ (2 CPU) 11:14:02 PM CPU %person %good %system %iowait %steal %idle 11:14:05 PM all 1.83 0.00 0.50 0.17 0.00 97.51 11:14:08 PM all 1.50 0.00 0.17 0.00 0.00 98.33 Common: all 1.67 0.00 0.33 0.08 0.00 97.92 [root@localhost sbin]#
View reminiscence utilization report
# sar –r
View community report
# sar –n ALL
ipcs
ipcs (InterProcess communication system) supplies a report on the semaphore, shared reminiscence, and message queue.
To show the message queue
# ipcs –q
To enumerate the semaphores
# ipcs –s
To show the shared reminiscence
# ipcs –m
To show the present utilization standing of IPC
[root@localhost sbin]# ipcs -u ------ Messages Standing -------- allotted queues = 0 used headers = 0 used area = 0 bytes ------ Shared Reminiscence Standing -------- segments allotted 5 pages allotted 2784 pages resident 359 pages swapped 0 Swap efficiency: 0 makes an attempt 0 successes ------ Semaphore Standing -------- used arrays = 0 allotted semaphores = 0 [root@localhost sbin]#
iopen
ioping is an exterior command that you may set up from right here. It may be very helpful to observe the disk I/O latency in actual time.
Conclusion
I hope the above instructions assist in the completely different conditions in your system administrator job. The instructions talked about above are good to make use of on-demand. Nevertheless, if it’s good to continuously monitor Linux servers, think about using server monitoring software program.
And for extra info on Linux efficiency, take a look at this Udemy course.