Friday, August 30, 2013

Resource Management : Socket Statistics

“ss” command is another very useful command available in Linux. The “ss” command is used to show socket statistics. Statistics for PACKET sockets, TCP sockets, UDP sockets, DCCP sockets, RAW sockets, Unix domain sockets can be seen using the ss command. This command is much similar to netstat command in linux. The command displays the list of all socket and port status and related information's.

Basic Example

Dev:vx11aaa:jbs002-~ $ ss -s
Total: 186 (kernel 218)
TCP: 73 (estab 20, closed 13, orphaned 0, synrecv 0, timewait 1/0), ports 54

Transport Total IP IPv6
*              218    -   -
RAW          0      0   0
UDP         17      9   8
TCP         60     52   8
INET        77     61 16
FRAG       0       0    0

From the above output ,we can see the established , closed ,orphaned and many more details divided by the protocol family.

Display All Open Network Ports
Dev:vx11aaa:jbs002-~ $ ss -l
Recv-Q Send-Q Local Address:Port                               Peer Address:Port
0           0                  :::48578                                      :::*

See the Process Using the Sockets
Dev:vx11aaa:jbs002-~ $ ss -pl
Recv-Q  Send-Q   Local Address:Port            Peer Address:Port
0           0             :::48578                          :::*  users:(("java",32506,12))

In the above out put we can see both the Open Ports as well as the process that is using that (in this case a Java process

Find out who is responsible for opening socket / port
Dev:vx11aaa:jbs002-~ $ ss -lp | grep 10012( )
0        0                :::10012               :::*     users:(("java",32506,67))

connected host's
Dev:vx11aaa:jbs002-~ $ ss -r
State  Recv-Q  Send-Q Local Address:Port    Peer Address:Port
ESTAB   0           0        vx1379:8629              eth0.vx1d81.uprr.com:11580
ESTAB   0           0        vx1379:24035            eth0.vx1d81.uprr.com:4708


Socket Memory usage
Dev:vx11aaa:jbs002-~ $ ss -m
State   Recv-Q Send-Q Local Address:Port          Peer Address:Port
ESTAB  0          0          192.229.125.12:8629     173.229.152.83:11580 mem:(r0,w0,f0,t0)
ESTAB  0          0          192.229.125.12:24035   173.229.152.83:4708 mem:(r0,w0,f0,t0)

r represents the read (inbound) buffer
w represents the write (outbound) buffer
f represents the "forward allocated memory" (memory available to the socket)
t represents the transmit queue (stuff waiting to be sent or waiting on an ACK)

ss -t -a dumps all TCP sockets
ss -u -a dumps all UDP sockets
ss -w -a dumps all RAW sockets
ss -x -a dumps all UNIX sockets

Filtering
as we said earlier , ss is almost equal to netstat command but it provides a lot more than netstat like the filtering capabilities.

“Ss” allows to filter socket states, using keywords state and exclude, followed by some state identifier.

State identifier are standard TCP state names

Find all the Established Ports
Dev:vx11aaa:jbs002-~ $ ss -o state established ( )
Recv-Q Send-Q Local Address:Port                               Address:Port
0           0         192.229.125.12:8629                           173.229.152.83:11580

Where FILTER-NAME can be any from the below list,
established
syn-sent
syn-recv
fin-wait-1
fin-wait-2
time-wait
closed
close-wait
last-ack
listen
closing
all : All of the above states
connected : All the states except for listen and closed
synchronized : All the connected states except for syn-sent
bucket : Show states, which are maintained as mini sockets, i.e. time-wait and syn-recv.
big : Opposite to bucket state.

For example , to dump all tcp sockets except TIME_WAIT:
Dev:vx11aaa:jbs002-~ $ ss exclude TIME_WAIT

Mapping of PID to Port
Dev:vx11aaa:jbs002-~ $ ss -p -l '( sport = :10012 )'
Recv-Q  Send-Q                Local Address:Port                 Peer Address:Port
0           0                         :::10012                  :::* users:(("java",32506,67))

How to list all unique ip address currently connected to a specific port?
ss -o state established '( dport = :10012 )'|awk -F"[\t :]+" 'NR!=1{ ip[$5]+=1 } END{ for (i in ip){n++};print n }'


User ID Information's
Dev:vx11aaa:jbs002-~ $ ss -e | head
State    Recv-Q    Send-Q         Local Address:Port      Peer Address:Port
ESTAB   0            0                  192.229.125.12:8629    173.229.152.83:11580 ino:24265172 sk:f28c4080ffff8800
ESTAB   0           0                   192.229.125.12:24035  173.229.152.83:4708 ino:22405856 sk:9e954080ffff8800
ESTAB   0           0                   192.229.125.12:43730  167.132.85.252:ncube-lm uid:7281 ino:24335602 sk:9e955340ffff8800
ESTAB  0            0                  192.229.125.12:43712  167.132.85.252:ncube-lm uid:7281 ino:24334169 sk:62d6e080ffff8800


More To Come On Advanced Socket Management With ss. Stay Tuned
Read More

Very use full Commands ( For Trouble Shooting )

Some of these Commands may be throw an error when I Use '-' ( hyphen ) ,so for commands using hyphen be a little cautious

Biggest 10 Files
du -sh * | sort -n | tail
du -x -a . | sort -n -r | head -n 10

List All the Process By memory usage
ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10
ps -eo pcpu,pid,user,args | sort -r -k1 | less

find how many files an application is using
lsof +c 0 | cut -d' ' -f1 | sort | uniq –c

Highest CPU Usage
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10 | awk "{ print $2 }"

Total Memory Usage
sar -q 1 | tail -1 | awk '{ print "" $3}' | sed 's/%//g'

CPU Threshold
top -b -n 1 | awk -F'[:,]' '/^Cpu/{sub("\\..*","",$2); print $2}'

Current User With Session Count
who | awk '{ User [$1]++; } END { for (i in User) printf "%-9s %s\n", i, User [i] }'

Memory Details
free -t -m | grep "Total" | awk '{ print "Total Memory space : "$2 " MB";
print "Used Memory Space : "$3" MB";
print "Free Memory : "$4" MB";
}'

Swap memory Details
free -t -m | grep "Swap" | awk '{ print "Total Swap space : "$2 " MB";
print "Used Swap Space : "$3" MB";
print "Free Swap : "$4" MB";
}'

Process Using memory
ps aux | awk '{if ($5 != 0 ) print $2,$5,$6,$11}' | sort -k2n

Largest File Or Directory
du -sk /var/log/* | sort -r -n | head -10

Processes Count Used By Users
ps hax -o user | sort | uniq -c

Who Started this Process
ps -o comm= -p $(ps -o ppid= -p PID)

How Much Ram Is Being Used
ps -o rss -C java | tail -n +2 | (sed 's/^/x+=/'; echo x) | bc

When Was the Process Started
ps -o lstart PID

Environment Variables belong to a Process
ps ewwo command PID | tr ' ' '\n' | grep \=

List Threads by Pid along with Thread Start Time
ps -o pid,lwp,lstart --pid PID -L

CPU usage for EACH cores
ps ax -L -o pid,tid,psr,pcpu,args | sort -nr -k4| head -15 | cut -c 1-90

Memory Percentage Usage of a Process
ps -o comm,%mem,args PID

Total CPU Usage Percentage
ps aux | awk {'sum+=$3;print sum'} | tail -n 1

Find Class Files inSide the Jar Location
find . -iname '*.jar' -printf "unzip -c %p | grep -q 'sample Text' && echo %p\n" | sh

Find the Class File in jars
find . -name "*.jar" | while read line; do unzip -l $line; done | grep

Search The File From Multiple Jar Files
find . -name "*.jar" | xargs -tn1 jar tvf | grep --color "log4j.xml"

List of Files That Are Open For Writing to Disk
lsof | grep -e "[[:digit:]]\+w"

See which Process is Holding the File
lsof -r1 /common/jboss.log

List the files accessed by a program
strace -f -o foo.trace su user -c 'mycommand'?

get size of data that haven't been written to disk yet
The term for that is "dirty" data (data that has been changed, but not yet flushed to permanent storage).
cat /proc/meminfo | grep Dirty
Dirty : 188 kB

Owner of the File
/sbin/fuser admin.lok
admin.lok: 5912
Pmap Output In Kilo Bytes
pmap -x PID | awk '/\[ anon \]/ {total += $3} END {print total}'

determine which application is utilizing a certain port?
lsof -w -n -i tcp:80 ( or any Othe Port)
Start Number of active, and recently torn down TCP sessions
netstat -ant | egrep -i '(ESTABLISHED|WAIT|CLOSING)' | wc -l

Number of sessions waiting for ACK (SYN Flood)
netstat -ant | egrep -i '(SYN)' | wc -l
Find Network Connections
Out Going Going Connections
localhost:root002-~ $ lsof | grep omdx1971
java 1142 dwls977 924u IPv6 141719124 0t0 TCP localMachine :27710->eth0.remoteMachine.nova.com:54426 (ESTABLISHED)

java 1142 dwls977 925u IPv6 141713891 0t0 TCP localMachine :27710->eth0.remoteMachine.nova.com:54141 (ESTABLISHED)

Incoming Connections
Once you got the Out Going Connections , we can get the Port and try it on the local machine to find the connection information

remoteMachine:root002-~ $ netstat | grep 54426
tcp 0 0 localMachine :54426 eth0.remoteMachine.nova.co:27710 ESTABLISHED
All Outgoing Connections
netstat -an | grep -i tcp | grep ESTABLISHED | less
talk to Other over the Network
Dev:vx1379:djbs002-~ $ lsof -p 29118 -a -i ( Process Talk)
show the number of connections active to a port and also the number of connections from that ip in order
netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
List listening TCP sockets
netstat -ant | egrep -i '(LISTEN)'
List Per User Process
ps -e -o uid,vsz | awk '{ usage[$1] += $2 } END { for (uid in usage) { print uid, ":", usage[uid] } }'
Working Directory of a Running process
lsof -bw -d cwd -a -c java
Extract files from war/Ear
jar xf abc.war log4j-test.xml WEB-INF/
All Connections from a Specific process
lsof -p PID -a -i
Scan a port On a Remote Machine
nc -v -w 1 -z
Show The Java Process
top -b -n 1|grep java|wc -l
Smallest To biggest
ls -s | sort -nr | more
Find Files That Exceed a Specified Size Limit
find . -size +400 -print
Grep All Files Of A Certain File Type For A Specific Pattern
find . -type f -name '*.cs' -print0 | xargs -0 grep --color=always -n PATTERN

Zero Length File Sizes
find . -type f -size 0k -exec rm {} \; | awk '{ print $8 }'
Find and Kill a Process
ps ux | grep | grep –v grep | awk ‘{print $2}’ | xargs –r kill -9
Files Opened By a Process
lsof +f | grep PID
Find The Command Line Of the Process Using Specific Port
cat /proc/$(lsof -ti:)/cmdline
All Open Tomcat Threads
ps -ALcf | grep org.apache.catalina.startup.Bootstrap | wc -l

Find The MAC Address
[root@vx111a test]# cat /sys/class/net/eth0/address

List all the Installed Perl packages
perl -MFile::Find=find -MFile::Spec::Functions -Tlwe 'find { wanted => sub { print canonpath $_ if /\.pm\z/ }, no_chdir => 1 }, @INC'
Find Whether a Port is Open On a Remote Machine
nmap -p
Status Of the HTTP
curl -o /dev/null --silent --head --write-out '%{http_code}\n'
200
-o /dev/null throws away the usual output
-silent throws away the progress meter
-head makes a HEAD HTTP request, instead of GET
-write-out '%{http_code}\n' prints the required status code
Find Whether The Process is 32Bit or 64Bit
file -L /proc/PID/exe

The Output would be some thing like this
/proc/6462/exe: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), not stripped
Convert Of Bytes to Mbs
units --terse "3415014314 bytes" "MB"

SSh and Execute a Command
ssh root@ -q 'echo $MYDIR’
Find out what is listening on a series of ports
/sbin/fuser -n tcp {7000..8000}
View details of network activity between
lsof -i :7000-8000
List all files opened by a particular command
lsof -c java
List Threads By PID along with Thread Start Time
ps -o lwp,lstart --pid PID

Count Threads OF a Process
ps uH -p PID | wc –l

Print a stack trace of a running process
Pstack PID
Return Number of kernel Threads Owned by a process
ps -o thcount –p PID
Show Ports that belong to this PID
netstat --all --program | grep PID

Drop all the Connections available now for a Port
iptables -I INPUT -p tcp -dport 80 -j DROP

Find and Grep a File
find $PWD -type f -exec egrep PAVAN {} \;

Find the Time took Process To Complete
/usr/bin/time -v

Memory Information ( Much Better than Free Command)
vmstat -s -S M
List all the Jars loaded by a Process
lsof -p PID | grep jar

All process Running as a Specific User
pgrep -l -f -x -u root
Find and Remove Files Matching a pattern
find $PWD -type f -name "*Jul-*.ESS-A1.log" -exec rm -f {} \;

Find and Zip files matching a Pattern
find $PWD –type f –name “ess_admin.controllermessages.log.” | xargs tar zcvf one.tar

How to list all unique ip address currently connected to a specific port
ss -o state established '( dport = :10012 )'|awk -F"[\t :]+" 'NR!=1{ ip[$5]+=1 } END{ for (i in ip){n++};print n }'

Find and Gzip
find $PWD –type f –name “*.log” | xargs tar zcvf one.tar

Find and Remove
find –type f –name “*.log” | xargs rm
find –type f –name “*.log” -exec rm -f {} \;

Find Files That are Updated in The Last 60 Minutes
find $PWD -mmin -60

Find all files Older than 2 days and gzip them
find $PWD -type f -mtime +2 | xargs gzip

How many Open Tomcat Threads
ps -ALcf | grep org.apache.catalina.startup.Bootstrap | wc -l

Differences between 2 files in remote hosts
diff <(ssh alice cat /etc/apt/sources.list) <(ssh bob cat /etc/apt/sources.list)

Monitor the active thread count of a process (jvm) on linux
ps uH -p 10343 | wc -l

How do I find out which service is listening on a specific port?
lsof -Pnl +M -i4 ( IP 4 )
lsof -Pnl +M -i6 ( IP 6 )
netstat -npl

Display CPU,Memory Usages of the Users
ps axo user,pcpu,pmem,rss --no-heading | awk '{pCPU[$1]+=$2; pMEM[$1]+=$3; sRSS[$1]+=$4} END {for (user in pCPU) if (pCPU[user]>0 || sRSS[user]>10240) printf "%s:@%.1f%% of total CPU,@%.1f%% of total MEM@(%.2f GiB used)\n", user, pCPU[user], pMEM[user], sRSS[user]/1024/1024}' | column -ts@ | sort -rnk2
Kill a Process on the Port
kill -9 `lsof -t -i :port_number`
How Many Established Connection
lsof | grep -c "(ESTABLISHED)"
How to enlarge existing file to specific size
dd if=/dev/zero bs=1 seek=new_filesize count=0 of=your_file
For example this:
dd if=/dev/zero bs=1G seek=1000 count=0 of=test
will enlarge file test to 1000G
Tail the Last Bytes of Files
tail -c 400 jboss.log > jboss.log11
Deleted Files along with File Descriptor(FD)
lsof | awk '(/deleted/) {print "FD :-",$4,"| File Name:-",$9}'
Check the Files For Changes From the last 1 Hour
awk -vDate=`date -d'now-1 hours' +[%d/%b/%Y:%H:%M:%S` ' { if ($4 > Date) print Date FS $4}'
Threads in a Process
ps -eLo pid,ppid,tid,pcpu,comm | grep PID
Grep Multiple Words
grep -w 'warning\|error\|critical' /var/log/messages
Find Out What Partition a File Belongs To
We can use the df command to find out what partition a file belongs,

[root@vx111a perl]# df -T file1
Filesystem Type 1K-blocks Used Available Use% Mounted on
/dev/sda7 ext3 49594196 5499736 41534504 12% /soa
Find Out Port
/sbin/fuser 10011/tcp
10011/tcp: 32506

lsof -i tcp:10012
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 32506 jbs002 67u IPv6 20552857 0t0 TCP *:10012 (LISTEN)
Thread Ids of a process in Linux using Proc
cat /proc/PID/task

How To Find The Number Of Open Files for a Process Name and process pid sorted by number of open files.?
lsof | perl -lane '$x{"$F[0]:$F[1]"}++;END { print "$x{$_}\t$_" for sort {$x{$a}<=>$x{$b}} keys %x}'
To show connections to a specific host
lsof -i@192.168.1.5

Show connections based on the host and the port using @host:port 

Grep All Files Of A Certain File Type For A Specific Pattern
find . -type f -name '*.*' -print0 | xargs -0 grep --color=always -n GCMonitor

Top 20 Process With High File Descriptors
for x in `ps -eF| awk '{ print $2 }'`;do echo `ls /proc/$x/fd 2> /dev/null | wc -l` $x `cat /proc/$x/cmdline 2> /dev/null`;done | sort -n -r | head -n 20

Ping a URL to Find the HTTP Status
Dev:Hunter@root-~ $ printf "GET / HTTP/1.0\r\nvx111a: www\r\n\r\n" | nc vx111a 10011 | head
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=291DBACCB3596F4FBC38ABFBDE99AA7C.jasB2; Path=/
ETag: W/"7857-1235874240000"
Last-Modified: Sun, 01 Mar 2009 02:24:00 GMT
Content-Type: text/html
Content-Length: 7857
Date: Thu, 10 Oct 2013 02:33:39 GMT
Connection: close

Dev:Hunter@root-~ $ nc -zw2 vx111a 10011 || echo http service is down
Connection to vx111a 10011 port [tcp/*] succeeded!

Dev:Hunter@root-~ $ curl -sL -w "%{http_code}\\n" "http://vx111a:10011/wls_monitor/" -o /dev/null
200

Redirect and Print
Dev:vx111a:jbs002-~ $ jmap -heap 922 2>/dev/null | grep MaxPermSize | awk '{print $3}'
268435456

The 2>/dev/null at the end of the find command tells your shell to redirect the error messages (FD #2) to /dev/null, so you don't have to see them on screen. Use /dev/null to to send any unwanted output from program/command. All data written on a /dev/null special file is discarded by the system. To redirect standard error to /dev/null and store file list to output.txt, type:

Find Who is Using Port
Dev:vx111a:djbs002-~ $ /sbin/fuser 10011/tcp
10011/tcp: 32506

Dev:vx111a:djbs002-~ $ lsof -i tcp:10012
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 32506 djbs002 67u IPv6 20552857 0t0 TCP *:10012 (LISTEN)

Search For Out Of memory
find . -type f -exec grep -l java.lang.OutOfMemoryError {} \;
./MANDY-B1_gc.log
./heap.log
./MANDY -B1.log.yyyyMMdd_HHmmss

Conenct to the Linux Machine From windows Machine to Download Files
pscp -i C:\Ateam_uberkey.ppk jbs002@vx111a:/tmp/top.log C:\Users\Jag\Desktop\temp\top.log

Note : pscp is available with putty Software which is free to download


Obtain the Number of Thread States in a Thread Dump
dev:vx1abc:he002:nc-Dumps $ awk '/State: / { print }' < td.log.073716.013908855 | sort | uniq -c
   10    java.lang.Thread.State: RUNNABLE
    8    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    2    java.lang.Thread.State: TIMED_WAITING (sleeping)
    2    java.lang.Thread.State: WAITING (on object monitor)
   24    java.lang.Thread.State: WAITING (parking)

Obtain the Count of Threads based on Thread State
dev:vx1cea:djhe002:nc-stackDetils $ awk '/State: TIMED_WAITING/ { getline; print }' < td.2013-10-17-03.log | sort | uniq -c
   50         at java.lang.Object.wait(Native Method)
   15         at java.lang.Thread.sleep(Native Method)
    5         at sun.misc.Unsafe.park(Native Method)

Grep Multiple Strings 
grep 'RUNNABLE\|ListenPort\|port\|clusters\|broadcastchannel\|multicastaddress\|multicastport\|server_cluster'  td.log.073652.957462762

Number of Connection Hits in between
Dev:vx1322:jbs002-JAS-A2 $ egrep "2013-08-09 16:47:12|2013-08-09 16:50:00" jboss.log  | wc -l
3

How many Requests Happened Per Minutes
egrep "17/Aug/2013:01|17/Aug/2013:02" PHYRES-H1_access.log | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":"$3}' | sort -nk1 -nk2 | uniq -c | sed 's/[ ]*//'

Query packages
dev:vx1abc:he002:nc-~ $ rpm -q --queryformat '\n%{NAME} %{VERSION} %{RELEASE} %{ARCH}\n' glibc
glibc 2.12 1.80.el6_3.6 x86_64
glibc 2.12 1.80.el6_3.6 i686

Processor and Memory Usage Per User
ps axo user,pcpu,pmem,rss --no-heading | awk '{pCPU[$1]+=$2; pMEM[$1]+=$3; sRSS[$1]+=$4} END {for (user in pCPU) if (pCPU[user]>0 || sRSS[user]>10240) printf "%s:@%.1f%% of total CPU,@%.1f%% of total MEM@(%.2f GiB used)\n", user, pCPU[user], pMEM[user], sRSS[user]/1024/1024}' | column -ts@ | sort -rnk2

Read More

Tuesday, August 20, 2013

Resource Management : Vmstat

Vmstat is a command available in linux which displays information about the memory, swap and CPU utilization of the system in real time.

A Basic executions of vmstat gives,

LocalMach:root002-~ $ vmstat
procs       -----------memory----------          ---swap-- -----io---- --system-- -----cpu------
r b           swpd   free           buff          cache        si    so        bi     bo         in  cs       us sy id wa st
1 0                 0   36570164 1584768 12768668 0     0           0      6            0   1         1 0  99 0    0

The data provided above was an average usage of system resources since last reboot. In the above output, there is
0 - nothing is swapped from disk
36570164 – amount of free system memory
1584768 - amount of system data that has yet to be flushed to disk.
12768668 - amount of data that has been read from the disk

LocalMach:root002-~ $ vmstat -a
procs -----------memory----------               ---swap-- -----io---- --system-- -----cpu------
r b      swpd free           inact       active        si so         bi bo          in cs          us sy id wa st
1 0           0 36442616 3303552 25479784    0  0         0   6           0  0             1 0 99 0   0

In the above output we asked vmstat to display information about the number of active and inactive pages. The amount of inactive pages indicates how much of the memory could be swapped to disk and how much is currently being used. In this case, we can see that 33.3552 of memory is active, and only 25479784 is considered inactive

Often vmstat is used with an interval and a count like,

vmstat [interval] [count]

The above output provides you a lot of information like

procs Section
  • r field: Total number of runnable process. The More the load on system more number of process are waiting to get CPU
  • b field: Total number of blocked process or uninterruptable sleeping process. These processes are most likely waiting for a I/O operations or could be any other task. (should be close to Zero)
The value for these fields should be either ‘0’ or close to ‘0’.If there is a high value ,then the system don’t have enough resources like CPU, memory or I/O.

Memory section (in KB)
  • Swpd field: Used swap space
  • Free field: Available free RAM
  • Buff field: RAM used for buffers. This is used to store file metadata such as i-node and data from Raw block devices.
  • Cache field: RAM used for file system cache .This is used for the file data itself.
Swap Section (in KB/Sec)
  • Si field: Amount of memory swapped from disk per second
  • So field: Amount of memory swapped to disk per second

The data under these fields show the amount of swapping going on. If it is more than we need to either reduce the memory demand or increase the Physical Ram.

For Web servers or Data base Servers its very bad, since a trip to the disk may often cause a Performance impact for transactions going.

I/O Section ( in Blocks/Sec)
  • Bi field: Blocks received from disk
  • Bo field: Blocks sent to disk.

This shows us the flow of data to and from the disks. if there are large number of process are blocking and high I/O , then there is an Issue with the I/O. bi and bo represent the data transfers between virtual memory and block devices. It depends on your workload

System Section ( Per Second)
  • In field: Number of interrupts per second.
  • Cs field: Number of context switches per second.

CPU Section
  • Us field: Time spend running user code. (non-kernel code , application specific code comes under this)
  • Sy field: Time spent running kernel code such as root process.
  • Id field: Idle time.
  • Wa field: Time spent waiting for the IO

The information is very important in identifying the information about the CPU. We have one more command ‘top’ which gives the same information. The ‘top’ command gives information about single CPU ,But vmstat gives information about all the CPU.

We can also use vmstat like ,

LocalMach:root002-~ $ vmstat -a -n 10 3
procs -----------memory----------               ---swap-- -----io---- --system-- -----cpu------
r b           swpd free           inact     active       si so         bi bo        in cs        us sy id wa st
1 0               0 36551776 3229768 25445280    0 0           0 6          0 1            1 0 99 0 0
0 0               0 36551972 3229800 25445272    0 0           0 88    277 2602        1 0 99 0 0
0 0               0 36551848 3229856 25445340    0 0           0 100 1265 2545       0 0 100 0 0

The vmstat example above means we want it to delay 10 seconds before the next update and count the update 3 times. The '-n' option is used so that the header does not displayed in every update.

In Order to get the memory Details we can use,

LocalMach:root002-~ $ vmstat -s
65970668 total memory
29418976 used memory
25445424 active memory
3230004 inactive memory
36551692 free memory
1584872 buffer memory
12786880 swap cache
4194296 total swap
0 used swap
4194296 free swap
39488615 non-nice user cpu ticks
178225 nice user cpu ticks
3318192 system cpu ticks
2963059383 idle cpu ticks
41192 IO-wait cpu ticks
94896 IRQ cpu ticks
803239 softirq cpu ticks
0 stolen cpu ticks
10891932 pages paged in
193698804 pages paged out
0 pages swapped in
0 pages swapped out
2664606290 interrupts
723606365 CPU context switches
1346657309 boot time
24413327 forks

In the default operation, vmstat displays memory statistics in kilobytes. vmstat considers a single kilobyte equal to 1024 bytes. vmstat can also display reports with memory sizes reported in megabytes. vmstat reports with the argument "-S m" will consider a single megabyte equal to 1000 kilobytes as follows:

vmstat -S k 1 10
vmstat -S m 1 10

vmstat can provide information about how the Linux kernel allocates its memory. The Linux kernel has a series of "slabs" to hold its dynamic data structures. vmstat displays each of the slabs (Cache), shows how many of the elements are being used (Num), shows how many are allocated (Total), shows the size of each element (Size), and shows the amount of memory in pages (Pages) that the total slab is using.

LocalMach:root002-~ $ vmstat -S k
procs -----------memory----------   ---swap-- -----io---- --system-- -----cpu------
r b      swpd free         buff     cache    si so     bi bo        in  cs       us sy id wa st
0 0           0 37423276 1622970 1...    0    0      0 6          0  1          1 0  99  0  0

The same information can be obtained in MB using, vmstat –S M.

LocalMach:root002-~ $ vmstat -f
24447726 forks

Resource Usage

localMach:root002-sa $ vmstat -m
Cache Num Total Size Pages
rpc_buffers 8 8 2048 2
rpc_tasks 8 10 384 10
rpc_inode_cache 6 10 768 5
ip_fib_alias 10 59 64 59
ip_fib_hash 10 59 64 59
iser_descriptors 0 0 128 30
ib_mad 0 0 448 8
fib6_nodes 5 59 64 59
ip6_dst_cache 4 24 320 12
ndisc_cache 1 15 256 15
RAWv6 7 8 960 4
UDPv6 6 8 896 4
tw_sock_TCPv6 0 0 192 20
request_sock_TCPv6 0 0 192 20

Disk Statistics

localMach:root002-sa $ vmstat -d
disk- ------------reads------------ ------------writes----------- -----IO------
total merged sectors ms total merged sectors ms cur sec
ram0 0 0 0 0 0 0 0 0 0 0
ram1 0 0 0 0 0 0 0 0 0 0
ram2 0 0 0 0 0 0 0 0 0 0
ram3 0 0 0 0 0 0 0 0 0 0
ram4 0 0 0 0 0 0 0 0 0 0
ram5 0 0 0 0 0 0 0 0 0 0
ram6 0 0 0 0 0 0 0 0 0 0
ram7 0 0 0 0 0 0 0 0 0 0
ram8 0 0 0 0 0 0 0 0 0 0
ram9 0 0 0 0 0 0 0 0 0 0
ram10 0 0 0 0 0 0 0 0 0 0
ram11 0 0 0 0 0 0 0 0 0 0
ram12 0 0 0 0 0 0 0 0 0 0
ram13 0 0 0 0 0 0 0 0 0 0
ram14 0 0 0 0 0 0 0 0 0 0


Disk Table
(! 1018)-> vmstat -D
31 disks
2 partitions
1139802 total reads
214276 merged reads
36688696 read sectors
8146064 milli reading
2168133 writes
742101 merged writes
23021306 written sectors
169686644 milli writing
0 inprogress IO
13502 milli spent IO

Happy learning , More To Come :-)
Read More