|
AIX性能監(jiān)控系列學(xué)習(xí)-IO系統(tǒng)瓶頸分析 2011-05-16 16:07:58
------------------- 輸入輸出(I/O)瓶頸
-------------------
1. iostat
iostat命令用于統(tǒng)計(jì)CPU的使用情況及tty設(shè)備,、硬盤和CD-ROM的I/0量,。
運(yùn)行命令：
iostat 5 10
將顯示10次統(tǒng)計(jì)結(jié)果,，下面為其中一次的數(shù)據(jù)：
tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 102.2 0.2 0.7 99.0 0.1
Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk2 0.0 0.0 0.0 0 0
hdisk3 0.0 0.0 0.0 0 0
hdisk0 0.4 33.8 2.6 0 172
hdisk1 0.0 2.4 0.6 0 12
cd0 0.0 0.0 0.0 0 0
其中
%iowait 等待本地 I/O 時(shí)CPU 空閑時(shí)間的百分比
%idle 未等待本地 I/O 時(shí)CPU 空閑時(shí)間的百分比
當(dāng)沒(méi)有需使用CPU的進(jìn)程但至少有一個(gè)進(jìn)程在等待I/O 時(shí),， CPU時(shí)間屬性標(biāo)為iowait,。
如果iowait 時(shí)間的百分比很高，sample示該磁盤輸入輸出(I/O)是導(dǎo)致系統(tǒng)運(yùn)行速度緩慢的主要原因,。
%tm_act 硬盤繁忙的百分比
注: tm_act的值很高,，表明硬盤存在I/O瓶頸。
當(dāng)%tm_act(硬盤繁忙時(shí)間)很高時(shí),，可能會(huì)感覺(jué)到系統(tǒng)運(yùn)行速度在減慢,。有些系統(tǒng)上某個(gè)硬盤的%tm_act值為60%或更高時(shí)，系統(tǒng)性能就會(huì)受到影響,。
兩點(diǎn)建議：
1. 觀察繁忙與空閑的硬盤,，將數(shù)據(jù)從忙的盤移至相對(duì)空閑的的盤上，這會(huì)在一定程度上減輕由此引起的I/O 瓶頸,。
2. 依照上面"內(nèi)存瓶頸"中介紹的方法檢查內(nèi)存換頁(yè)的頻繁程度,。大量的換頁(yè)操作會(huì)增加I/O負(fù)載。
2. filemon
filemon 命令用于查看哪些文件/邏輯卷/硬盤處于繁忙狀態(tài),。在系統(tǒng)I/O 處于繁忙狀態(tài)時(shí)運(yùn)行下面的命令：
#filemon -u -O all -o /tmp/fmon.out; sleep 30; trcstop
30秒鐘后會(huì)生成記錄跟蹤信息的文件 /tmp/fmon.out,。從該文件中可以找出：
1. 最為繁忙的虛擬內(nèi)存段(segment)、邏輯卷和物理卷,；
2. 對(duì)頁(yè)空間(paging space)的讀寫(xiě)次數(shù),，確認(rèn)硬盤I/O 是由于應(yīng)用程序操作還是頻繁的換頁(yè)操作；
3. 最為活躍的文件或邏輯卷,，如果它們存放在繁忙的物理卷上,，可以考慮將數(shù)據(jù)移至相對(duì)空閑的硬盤上，
這樣有助于提高系統(tǒng)性能,。最為繁忙的段的報(bào)sample里列示出最繁忙的文件所對(duì)應(yīng)的文件系統(tǒng)和節(jié)點(diǎn),。
文件系統(tǒng)的安裝點(diǎn)(mount point)及文件的i節(jié)點(diǎn)(inode)可與命令ncheck一起使用，來(lái)找出相對(duì)應(yīng)的文件,。
這個(gè)報(bào)sample可用來(lái)判斷該I/O操作是針對(duì)文件系統(tǒng),、JFS Log 還是系統(tǒng)頁(yè)空間的。
通過(guò)檢查字段"reads"和"read sequences"的值,，可以判斷該操作是順序讀取還是隨機(jī)讀取,。
當(dāng) "read sequences"接近"reads"時(shí)，對(duì)該文件的訪問(wèn)則以隨機(jī)讀取方式居多,。
附filemon 的輸出示例:
P650A:/#filemon -u -O all -o /tmp/fmon.out; sleep 30; trcstop
Mon May 16 15:51:49 2011
System: AIX 5.3 Node: P650A Machine: 00C3EE9E4C00
Run trcstop command to signal end of trace.
P650A:/#[filemon: Reporting started]
[filemon: Reporting completed]
[filemon: 30.013 secs in measured interval]
P650A:/#more /tmp/fmon.out
Mon May 16 15:51:49 2011
System: AIX 5.3 Node: P650A Machine: 00C3EE9E4C00
Cpu utilization: 53.9%
Cpu allocation: 97.1%
9185662 events were lost. Reported data may have inconsistencies or errors.
Most Active Files
------------------------------------------------------------------------
#MBs #opns #rds #wrs file volume:inode
------------------------------------------------------------------------
168.4 0 43111 0 pid=1511880_fd=0
0.8 3 204 0 unix /dev/hd2:37196
0.7 0 47 0 pid=405822_fd=9
0.4 0 25 0 pid=364696_fd=15
0.2 0 158 0 pid=7827780_fd=5
0.1 0 210 0 pid=1_fd=0
0.1 0 7 0 pid=381286_fd=13
0.1 11 22 0 ksh.cat /dev/hd2:66100
0.1 0 5 0 pid=364696_fd=16
0.1 0 41 1 pid=7852356_fd=13
0.1 0 4 0 pid=294986_fd=15
0.1 0 4 0 pid=405822_fd=10
0.1 0 15 0 pid=7885144_fd=0
0.0 0 20 3 pid=7876884_fd=13
0.0 6 7 0 vfs /dev/hd4:930
0.0 0 1 0 pid=294986_fd=16
0.0 3 3 0 ps.cat /dev/hd2:66341
0.0 1 2 0 cmdtrace.cat /dev/hd2:65708
0.0 0 2 0 pid=0_fd=0
0.0 3 1 1 dsmscoutd.lst
Most Active Segments
------------------------------------------------------------------------
#MBs #rpgs #wpgs segid segtype volume:inode
------------------------------------------------------------------------
0.1 0 28 170536 client
0.1 0 28 90588 client
0.1 0 15 30562 client
0.0 0 10 b056a client
0.0 2 0 1705b6 working
0.0 1 0 e05af working
0.0 1 0 d05ac working
0.0 0 1 c280d client
Most Active Logical Volumes
------------------------------------------------------------------------
util #rblk #wblk KB/s volume description
------------------------------------------------------------------------
0.79 0 648 10.8 /dev/oradatalv /oradata
0.51 0 88 1.5 /dev/hd8 jfs2log
0.00 32 0 0.5 /dev/hd6 paging
0.00 0 8 0.1 /dev/hd1 /home
Most Active Physical Volumes
------------------------------------------------------------------------
util #rblk #wblk KB/s volume description
------------------------------------------------------------------------
0.79 24 608 10.5 /dev/hdisk0 N/A
0.17 8 352 6.0 /dev/hdisk1 N/A
------------------------------------------------------------------------
Detailed File Stats
------------------------------------------------------------------------
FILE: pid=1511880_fd=0
opens: 0
total bytes xfrd: 176582656
reads: 43111 (0 errs)
read sizes (bytes): avg 4096.0 min 4096 max 4096 sdev 0.0
read times (msec): avg 0.009 min 0.003 max 36.139 sdev 0.355
FILE: /unix volume: /dev/hd2 inode: 37196
opens: 3
total bytes xfrd: 835584
reads: 204 (0 errs)
read sizes (bytes): avg 4096.0 min 4096 max 4096 sdev 0.0
read times (msec): avg 0.006 min 0.003 max 0.008 sdev 0.001
lseeks: 375
FILE: pid=405822_fd=9
opens: 0
total bytes xfrd: 770048
reads: 47 (0 errs)
read sizes (bytes): avg 16384.0 min 16384 max 16384 sdev 0.0
read times (msec): avg 0.010 min 0.010 max 0.010 sdev 0.000
FILE: pid=364696_fd=15
opens: 0
total bytes xfrd: 409600
reads: 25 (0 errs)
read sizes (bytes): avg 16384.0 min 16384 max 16384 sdev 0.0
read times (msec): avg 0.012 min 0.012 max 0.012 sdev 0.000
FILE: pid=7827780_fd=5
opens: 0
total bytes xfrd: 161792
reads: 158 (158 errs)
read sizes (bytes): avg 1024.0 min 1024 max 1024 sdev 0.0
read times (msec): avg 0.003 min 0.002 max 0.006 sdev 0.001
FILE: pid=1_fd=0
opens: 0
total bytes xfrd: 136080
reads: 210 (0 errs)
read sizes (bytes): avg 648.0 min 648 max 648 sdev 0.0
read times (msec): avg 0.003 min 0.002 max 0.009 sdev 0.001
lseeks: 418
FILE: pid=381286_fd=13
opens: 0
total bytes xfrd: 114688
reads: 7 (0 errs)
read sizes (bytes): avg 16384.0 min 16384 max 16384 sdev 0.0
read times (msec): avg 0.013 min 0.013 max 0.013 sdev 0.000
FILE: /usr/lib/nls/msg/en_US/ksh.cat volume: /dev/hd2 inode: 66100
opens: 11
total bytes xfrd: 90112
reads: 22 (0 errs)
read sizes (bytes): avg 4096.0 min 4096 max 4096 sdev 0.0
read times (msec): avg 0.008 min 0.004 max 0.024 sdev 0.006
lseeks: 70
FILE: pid=364696_fd=16
opens: 0
total bytes xfrd: 81920
reads: 5 (0 errs)
read sizes (bytes): avg 16384.0 min 16384 max 16384 sdev 0.0
read times (msec): avg 0.011 min 0.011 max 0.011 sdev 0.000
FILE: pid=7852356_fd=13
opens: 0
total bytes xfrd: 71272
reads: 41 (0 errs)
read sizes (bytes): avg 1734.6 min 221 max 2064 sdev 541.6
read times (msec): avg 0.042 min 0.003 max 0.271 sdev 0.065
writes: 1 (0 errs)
write sizes (bytes): avg 152.0 min 152 max 152 sdev 0.0
write times (msec): avg 0.017 min 0.017 max 0.017 sdev 0.000
FILE: pid=294986_fd=15
opens: 0
total bytes xfrd: 65536
reads: 4 (0 errs)
read sizes (bytes): avg 16384.0 min 16384 max 16384 sdev 0.0
read times (msec): avg 0.009 min 0.009 max 0.009 sdev 0.000
FILE: pid=405822_fd=10
opens: 0
total bytes xfrd: 65536
reads: 4 (0 errs)
read sizes (bytes): avg 16384.0 min 16384 max 16384 sdev 0.0
read times (msec): avg 0.007 min 0.007 max 0.007 sdev 0.000
FILE: pid=7885144_fd=0
opens: 0
total bytes xfrd: 61440
reads: 15 (0 errs)
read sizes (bytes): avg 4096.0 min 4096 max 4096 sdev 0.0
read times (msec): avg 0.004 min 0.003 max 0.006 sdev 0.001
FILE: pid=7876884_fd=13
opens: 0
total bytes xfrd: 35230
reads: 20 (0 errs)
read sizes (bytes): avg 1738.7 min 551 max 2064 sdev 555.2
read times (msec): avg 1157.846 min 0.004 max 19000.089 sdev 4258.232
writes: 3 (0 errs)
write sizes (bytes): avg 152.0 min 152 max 152 sdev 0.0
write times (msec): avg 0.017 min 0.014 max 0.022 sdev 0.004
FILE: /etc/vfs volume: /dev/hd4 inode: 930
opens: 6
total bytes xfrd: 28672
reads: 7 (0 errs)
read sizes (bytes): avg 4096.0 min 4096 max 4096 sdev 0.0
read times (msec): avg 0.005 min 0.002 max 0.008 sdev 0.002
FILE: pid=294986_fd=16
opens: 0
total bytes xfrd: 16384
reads: 1 (0 errs)
read sizes (bytes): avg 16384.0 min 16384 max 16384 sdev 0.0
read times (msec): avg 0.007 min 0.007 max 0.007 sdev 0.000
FILE: /usr/lib/nls/msg/en_US/ps.cat volume: /dev/hd2 inode: 66341
opens: 3
total bytes xfrd: 12288
reads: 3 (0 errs)
read sizes (bytes): avg 4096.0 min 4096 max 4096 sdev 0.0
read times (msec): avg 0.010 min 0.009 max 0.012 sdev 0.001
lseeks: 12
FILE: /usr/lib/nls/msg/en_US/cmdtrace.cat volume: /dev/hd2 inode: 65708
opens: 1
total bytes xfrd: 8192
reads: 2 (0 errs)
read sizes (bytes): avg 4096.0 min 4096 max 4096 sdev 0.0
read times (msec): avg 0.007 min 0.006 max 0.007 sdev 0.000
lseeks: 8
FILE: pid=0_fd=0
opens: 0
total bytes xfrd: 8192
reads: 2 (0 errs)
read sizes (bytes): avg 4096.0 min 4096 max 4096 sdev 0.0
read times (msec): avg 86.036 min 79.036 max 93.037 sdev 7.000
lseeks: 10
FILE: /tmp/dsmscoutd.lst
opens: 3
total bytes xfrd: 7757
reads: 1 (0 errs)
read sizes (bytes): avg 4096.0 min 4096 max 4096 sdev 0.0
read times (msec): avg 0.004 min 0.004 max 0.004 sdev 0.000
writes: 1 (0 errs)
write sizes (bytes): avg 3661.0 min 3661 max 3661 sdev 0.0
write times (msec): avg 0.010 min 0.010 max 0.010 sdev 0.000
------------------------------------------------------------------------
Detailed VM Segment Stats (4096 byte pages)
------------------------------------------------------------------------
SEGMENT: 170536 segtype: client
segment flags: clnt
writes: 28 (0 errs)
write times (msec): avg 512.575 min 4.273 max 3025.921 sdev 1124.006
write sequences: 7
write seq. lengths: avg 4.0 min 4 max 4 sdev 0.0
SEGMENT: 90588 segtype: client
segment flags: clnt
writes: 28 (0 errs)
write times (msec): avg 511.387 min 3.031 max 3024.705 sdev 1123.995
write sequences: 7
write seq. lengths: avg 4.0 min 4 max 4 sdev 0.0
SEGMENT: 30562 segtype: client
segment flags: clnt
writes: 15 (0 errs)
write times (msec): avg 8.398 min 8.385 max 8.408 sdev 0.007
write sequences: 1
write seq. lengths: avg 15.0 min 15 max 15 sdev 0.0
SEGMENT: b056a segtype: client
segment flags: clnt
writes: 10 (0 errs)
write times (msec): avg 7.167 min 3.844 max 8.103 sdev 1.259
write sequences: 3
write seq. lengths: avg 3.3 min 1 max 5 sdev 1.7
SEGMENT: 1705b6 segtype: working
segment flags: work
reads: 2 (0 errs)
read times (msec): avg 5.557 min 5.021 max 6.094 sdev 0.536
read sequences: 1
read seq. lengths: avg 2.0 min 2 max 2 sdev 0.0
SEGMENT: e05af segtype: working
segment flags: work
reads: 1 (0 errs)
read times (msec): avg 6.231 min 6.231 max 6.231 sdev 0.000
read sequences: 1
read seq. lengths: avg 1.0 min 1 max 1 sdev 0.0
SEGMENT: d05ac segtype: working
segment flags: work
reads: 1 (0 errs)
read times (msec): avg 6.172 min 6.172 max 6.172 sdev 0.000
read sequences: 1
read seq. lengths: avg 1.0 min 1 max 1 sdev 0.0
SEGMENT: c280d segtype: client
segment flags: clnt
writes: 1 (0 errs)
write times (msec): avg 8.514 min 8.514 max 8.514 sdev 0.000
write sequences: 1
write seq. lengths: avg 1.0 min 1 max 1 sdev 0.0
------------------------------------------------------------------------
Detailed Logical Volume Stats (512 byte blocks)
------------------------------------------------------------------------
VOLUME: /dev/oradatalv description: /oradata
writes: 18 (0 errs)
write sizes (blks): avg 36.0 min 8 max 120 sdev 21.2
write times (msec): avg 8.575 min 3.017 max 14.165 sdev 3.649
write sequences: 18
write seq. lengths: avg 36.0 min 8 max 120 sdev 21.2
seeks: 18 (100.0%)
seek dist (blks): init 63840,
avg 83666530.4 min 23712 max 178591480 sdev 88716020.7
time to next req(msec): avg 1523.929 min 0.002 max 11804.280 sdev 2894.606
throughput: 10.8 KB/sec
utilization: 0.79
VOLUME: /dev/hd8 description: jfs2log
writes: 11 (0 errs)
write sizes (blks): avg 8.0 min 8 max 8 sdev 0.0
write times (msec): avg 1014.631 min 2.864 max 10091.925 sdev 3025.765
write sequences: 9
write seq. lengths: avg 9.8 min 8 max 24 sdev 5.0
seeks: 9 (81.8%)
seek dist (blks): init 494280,
avg 8.0 min 8 max 8 sdev 0.0
time to next req(msec): avg 2265.352 min 5.804 max 10045.696 sdev 3589.596
throughput: 1.5 KB/sec
utilization: 0.51
VOLUME: /dev/hd6 description: paging
reads: 4 (0 errs)
read sizes (blks): avg 8.0 min 8 max 8 sdev 0.0
read times (msec): avg 5.875 min 5.017 max 6.228 sdev 0.498
read sequences: 4
read seq. lengths: avg 8.0 min 8 max 8 sdev 0.0
seeks: 4 (100.0%)
seek dist (blks): init 2389536,
avg 5036200.0 min 1713808 max 11610392 sdev 4648745.1
time to next req(msec): avg 1762.830 min 4.373 max 7010.239 sdev 3029.610
throughput: 0.5 KB/sec
utilization: 0.00
VOLUME: /dev/hd1 description: /home
writes: 1 (0 errs)
write sizes (blks): avg 8.0 min 8 max 8 sdev 0.0
write times (msec): avg 8.500 min 8.500 max 8.500 sdev 0.000
write sequences: 1
write seq. lengths: avg 8.0 min 8 max 8 sdev 0.0
seeks: 1 (100.0%)
seek dist (blks): init 957512
time to next req(msec): avg 24910.354 min 24910.354 max 24910.354 sdev 0.000
throughput: 0.1 KB/sec
utilization: 0.00
------------------------------------------------------------------------
Detailed Physical Volume Stats (512 byte blocks)
------------------------------------------------------------------------
VOLUME: /dev/hdisk0 description: N/A
reads: 3 (0 errs)
read sizes (blks): avg 8.0 min 8 max 8 sdev 0.0
read times (msec): avg 5.757 min 5.000 max 6.213 sdev 0.539
read sequences: 3
read seq. lengths: avg 8.0 min 8 max 8 sdev 0.0
writes: 34 (0 errs)
write sizes (blks): avg 17.9 min 8 max 32 sdev 11.8
write times (msec): avg 9.867 min 1.498 max 49.412 sdev 9.138
write sequences: 33
write seq. lengths: avg 18.4 min 8 max 32 sdev 11.7
seeks: 36 (97.3%)
seek dist (blks): init 117212960,
avg 10718459.2 min 8 max 52186200 sdev 18350570.0
seek dist (%tot blks):init 40.87664,
avg 3.73794 min 0.00000 max 18.19932 sdev 6.39955
time to next req(msec): avg 945.888 min 0.003 max 10045.698 sdev 2098.774
throughput: 10.5 KB/sec
utilization: 0.79
VOLUME: /dev/hdisk1 description: N/A
reads: 1 (0 errs)
read sizes (blks): avg 8.0 min 8 max 8 sdev 0.0
read times (msec): avg 6.149 min 6.149 max 6.149 sdev 0.000
read sequences: 1
read seq. lengths: avg 8.0 min 8 max 8 sdev 0.0
writes: 23 (0 errs)
write sizes (blks): avg 15.3 min 8 max 120 sdev 23.7
write times (msec): avg 7.579 min 1.715 max 46.186 sdev 9.665
write sequences: 23
write seq. lengths: avg 15.3 min 8 max 120 sdev 23.7
seeks: 24 (100.0%)
seek dist (blks): init 23402504,
avg 36368105.7 min 8 max 108692392 sdev 48254032.5
seek dist (%tot blks):init 8.16135,
avg 12.68295 min 0.00000 max 37.90520 sdev 16.82803
time to next req(msec): avg 1465.817 min 5.801 max 9765.621 sdev 2781.132
throughput: 6.0 KB/sec
utilization: 0.17
P650A:/#
|
|
|
