2.05: Превышение LA за 5 мин LA1: 1.92, LA5: 2.05, LA15: 1.40, Limits: LA1 < , LA5 < 2.0, LA15 < ------------------------------------------------------ /usr/local/alertmon/store_log.sh: last pid: 60665; load averages: 1.92, 2.05, 1.40 up 22+09:06:56 04:20:01 120 processes: 3 running, 114 sleeping, 3 zombie Mem: 363M Active, 1234M Inact, 298M Wired, 95M Cache, 169M Buf, 12M Free Swap: 2048M Total, 276K Used, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 59292 root 1 -4 0 102M 100M CPU0 0 1:37 9.13% rsync 60640 root 1 -8 0 4680K 4060K piperd 1 0:00 6.00% perl5.8.8 60643 opennet 1 96 0 2832K 2164K RUN 1 0:00 1.00% perl5.8.8 76336 web 1 20 0 4676K 3828K lockf 1 0:29 0.05% httpd 1398 postgres 1 96 0 30620K 27096K select 0 3:21 0.00% postgres 1041 root 1 96 0 2972K 2276K select 1 3:17 0.00% httpd 1096 bind 1 96 0 7688K 6876K select 0 2:49 0.00% named 1033 postgres 1 96 0 30460K 2424K select 0 1:35 0.00% postgres 782 root 1 96 0 6344K 4160K select 0 1:02 0.00% snmpd 59291 root 1 96 0 98084K 95556K select 0 1:01 0.00% rsync 974 pop 1 4 0 3936K 3268K kqread 0 0:56 0.00% memcached 956 root 1 96 0 1520K 984K select 1 0:41 0.00% master 1091 nginx 1 4 0 12416K 1568K kqread 1 0:37 0.00% nginx 76418 web 1 20 0 4852K 4120K lockf 0 0:31 0.00% httpd 1400 postgres 1 96 0 4852K 2400K select 0 0:31 0.00% postgres 76508 web 1 4 0 4676K 3904K sbwait 1 0:31 0.00% httpd 76404 web 1 97 0 4752K 4000K select 0 0:31 0.00% httpd 76403 web 1 97 0 4820K 4112K select 1 0:31 0.00% httpd ====================================== net.inet.tcp.syncache ====================================== vmstat ITEM SIZE LIMIT USED FREE REQUESTS FAILURES UMA Kegs: 140, 0, 71, 1, 71, 0 UMA Zones: 480, 0, 71, 1, 71, 0 UMA Slabs: 64, 0, 3307, 1531, 969518, 0 UMA RCntSlabs: 104, 0, 255, 374, 638953, 0 UMA Hash: 128, 0, 3, 27, 7, 0 16 Bucket: 76, 0, 24, 26, 79, 0 32 Bucket: 140, 0, 32, 52, 124, 0 64 Bucket: 268, 0, 66, 46, 304, 92 128 Bucket: 524, 0, 2425, 256, 236476, 278077 VM OBJECT: 132, 0, 46881, 46905, 668992894, 0 MAP: 192, 0, 7, 13, 7, 0 KMAP ENTRY: 68, 83944, 724, 3644, 6775728, 0 MAP ENTRY: 68, 0, 3926, 3690, 1503941454, 0 PV ENTRY: 24, 2154990, 155057, 115948, 23476748893, 0 DP fakepg: 72, 0, 0, 159, 187, 0 mt_zone: 1024, 0, 202, 130, 202, 0 16: 16, 0, 3118, 739, 79919535, 0 32: 32, 0, 1715, 1562, 20630559, 0 64: 64, 0, 4365, 1181, 259983263, 0 128: 128, 0, 2654, 1876, 587963349, 0 256: 256, 0, 1214, 1531, 61446699, 0 512: 512, 0, 974, 602, 2732100, 0 1024: 1024, 0, 87, 293, 13781486, 0 2048: 2048, 0, 305, 303, 5034069, 0 4096: 4096, 0, 251, 164, 25067109, 0 Files: 72, 0, 390, 670, 923918536, 0 TURNSTILE: 52, 0, 601, 107, 601, 0 PROC: 536, 0, 169, 405, 15629231, 0 THREAD: 384, 0, 574, 26, 574, 0 KSEGRP: 88, 0, 574, 66, 574, 0 UPCALL: 44, 0, 0, 0, 0, 0 SLEEPQUEUE: 32, 0, 601, 190, 601, 0 VMSPACE: 296, 0, 118, 402, 15629202, 0 mbuf_packet: 256, 0, 358, 95, 703239656, 0 mbuf: 256, 0, 5, 292, 1170882895, 0 mbuf_cluster: 2048, 101376, 453, 57, 30497664, 0 mbuf_jumbo_pagesize: 4096, 0, 0, 0, 0, 0 mbuf_jumbo_9k: 9216, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 0, 0, 0, 0, 0 ACL UMA zone: 388, 0, 0, 0, 0, 0 g_bio: 132, 0, 9, 252, 282058430, 0 ata_request: 204, 0, 3, 206, 78817377, 0 ata_composite: 196, 0, 0, 0, 0, 0 VNODE: 272, 0, 45911, 51235, 564281840, 0 VNODEPOLL: 76, 0, 0, 100, 2, 0 S VFS Cache: 68, 0, 48010, 53910, 621770424, 0 L VFS Cache: 291, 0, 778, 873, 3126471, 0 NAMEI: 1024, 0, 0, 332, 5173084501, 0 DIRHASH: 1024, 0, 7801, 155, 6311308, 0 NFSMOUNT: 480, 0, 0, 0, 0, 0 NFSNODE: 464, 0, 0, 0, 0, 0 PIPE: 408, 0, 50, 220, 24490793, 0 KNOTE: 68, 0, 7, 329, 4267636, 0 socket: 356, 36003, 211, 636, 60494087, 0 unpcb: 144, 36018, 76, 383, 36595881, 0 ipq: 32, 3277, 0, 452, 1074264, 0 udpcb: 180, 36014, 7, 147, 1188051, 0 inpcb: 180, 36014, 583, 2189, 22710028, 0 tcpcb: 464, 36000, 128, 688, 22710028, 0 tcptw: 48, 41028, 455, 1729, 14831191, 0 syncache: 100, 15366, 3, 777, 22182323, 0 hostcache: 76, 15400, 1792, 3508, 962209, 0 tcpreass: 20, 6422, 0, 507, 133846, 0 sackhole: 20, 0, 0, 507, 6270695, 0 ripcb: 180, 36014, 0, 66, 126, 0 rtentry: 132, 0, 15, 72, 15, 0 IPFW dynamic rule: 108, 0, 292, 320, 28605470, 0 SWAPMETA: 276, 121576, 25, 199, 1129, 0 Mountpoints: 664, 0, 16, 8, 18, 0 FFS inode: 132, 0, 45851, 50835, 564218577, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 45851, 50239, 564218577, 0 md1: 512, 0, 621, 131, 2418, 0 interrupt total rate irq14: ata0 58 0 irq16: em0 uhci3 579472163 299 irq19: uhci1+ 75152946 38 cpu0: timer 3867169091 1999 cpu1: timer 3867168851 1999 Total 8388963109 4338 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad4 ad5 in sy cs us sy id 0 9 0 558256 108080 2030 1 0 0 1494 121 0 0 1550 1684 1158 18 6 76 ====================================== iostat ====================================== netstat 360/390/750 mbufs in use (current/cache/total) 356/154/510/101376 mbuf clusters in use (current/cache/total/max) 356/95 mbuf+clusters out of packet secondary zone in use (current/cache) 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/0 9k jumbo clusters in use (current/cache/total/max) 0/0/0/0 16k jumbo clusters in use (current/cache/total/max) 802K/405K/1207K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/225/8704 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 374697 requests for I/O initiated by sendfile 18505 calls to protocol drain routines Current listen queue sizes (qlen/incqlen/maxqlen) Proto Listen Local Address tcp4 0/0/32 *.21 tcp4 0/0/80 *.5432 tcp6 0/0/80 *.5432 tcp4 0/0/128 127.0.0.1.953 tcp4 0/0/3 77.234.201.246.53 tcp4 0/0/1024 77.234.201.244.80 tcp4 0/0/511 127.0.0.1.80 tcp4 0/13/511 77.234.201.242.80 tcp4 0/0/511 77.234.201.242.8100 tcp4 0/0/511 77.234.201.242.8101 tcp4 0/0/511 77.234.201.242.8102 tcp4 0/0/511 77.234.201.242.8103 tcp4 0/0/511 77.234.201.242.8104 tcp4 0/0/128 *.22 tcp6 0/0/128 *.22 tcp4 0/0/1024 127.0.0.1.11211 tcp4 0/0/128 *.199 tcp: 501142182 packets sent 374195379 data packets (2741096152 bytes) 12779915 data packets (4289214028 bytes) retransmitted 209193 data packets unnecessarily retransmitted 3996 resends initiated by MTU discovery 98391771 ack-only packets (0 delayed) 0 URG only packets 55226 window probe packets 165264 window update packets 15555304 control packets 403419259 packets received 289097946 acks (for 4288395358 bytes) 33524672 duplicate acks 1284783 acks for unsent data 69856481 packets (3752878894 bytes) received in-sequence 1412340 completely duplicate packets (152039463 bytes) 0 old duplicate packets 12199 packets with some dup. data (4493165 bytes duped) 133488 out-of-order packets (135550710 bytes) 1676 packets (131558 bytes) of data after window 61 window probes 4613784 window update packets 269460 packets received after close 11887 discarded for bad checksums 59 discarded for bad header offset fields 0 discarded because packet too short 612482 connection requests 22063671 connection accepts 745289 bad connection attempts 0 listen queue overflows 550 ignored RSTs in the windows 22621042 connections established (including accepts) 22709445 connections closed (including 2126726 drops) 9655588 connections updated cached RTT on close 9827878 connections updated cached RTT variance on close 6171442 connections updated cached ssthresh on close 1014 embryonic connections dropped 102862895 segments updated rtt (of 110180192 attempts) 7978044 retransmit timeouts 167158 connections dropped by rexmit timeout 65614 persist timeouts 46 connections dropped by persist timeout 27679 keepalive timeouts 24957 keepalive probes sent 2721 connections dropped by keepalive 22570698 correct ACK header predictions 42873286 correct data packet header predictions 22182303 syncache entries added 471416 retransmitted 626296 dupsyn 20 dropped 22063671 completed 0 bucket overflow 0 cache overflow 63497 reset 54453 stale 0 aborted 0 badack 680 unreach 0 zone failures 22182323 cookies sent 0 cookies received 2741163 SACK recovery episodes 3824937 segment rexmits in SACK recovery episodes 1071004962 byte rexmits in SACK recovery episodes 19044206 SACK options (SACK blocks) received 109749 SACK options (SACK blocks) sent 0 SACK scoreboard overflow ====================================== sysctl net.inet.ip.fw.*dyn* ====================================== counters (connection/stablished/open sockets/open files) 673 4227 52116 27 162 2133 ------------------------------------------------------ /usr/local/alertmon/store_log.sh: last pid: 60683; load averages: 1.92, 2.05, 1.40 up 22+09:06:56 04:20:01 122 processes: 4 running, 116 sleeping, 2 zombie Mem: 365M Active, 1237M Inact, 298M Wired, 95M Cache, 169M Buf, 6728K Free Swap: 2048M Total, 276K Used, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 59292 root 1 -4 0 102M 100M getblk 1 1:37 9.13% rsync 60640 root 1 -8 0 4696K 4080K piperd 1 0:00 6.00% perl5.8.8 60643 opennet 1 -8 0 2832K 2164K biord 0 0:00 1.00% perl5.8.8 76336 web 1 20 0 4676K 3828K lockf 0 0:29 0.05% httpd 1398 postgres 1 96 0 30620K 27096K select 0 3:21 0.00% postgres 1041 root 1 96 0 2972K 2276K select 1 3:17 0.00% httpd 1096 bind 1 96 0 7688K 6876K select 0 2:49 0.00% named 1033 postgres 1 96 0 30460K 2424K select 0 1:35 0.00% postgres 782 root 1 96 0 6344K 4160K select 0 1:02 0.00% snmpd 59291 root 1 96 0 98084K 95556K select 0 1:01 0.00% rsync 974 pop 1 4 0 3936K 3268K kqread 0 0:56 0.00% memcached 956 root 1 96 0 1520K 984K select 1 0:41 0.00% master 1091 nginx 1 4 0 12416K 1568K kqread 1 0:37 0.00% nginx 76418 web 1 20 0 4852K 4120K lockf 0 0:31 0.00% httpd 1400 postgres 1 96 0 4852K 2400K select 0 0:31 0.00% postgres 76508 web 1 4 0 4676K 3904K sbwait 1 0:31 0.00% httpd 76404 web 1 97 0 4752K 4000K select 0 0:31 0.00% httpd 76403 web 1 97 0 4820K 4112K select 1 0:31 0.00% httpd ====================================== net.inet.tcp.syncache ====================================== vmstat ITEM SIZE LIMIT USED FREE REQUESTS FAILURES UMA Kegs: 140, 0, 71, 1, 71, 0 UMA Zones: 480, 0, 71, 1, 71, 0 UMA Slabs: 64, 0, 3307, 1531, 969521, 0 UMA RCntSlabs: 104, 0, 255, 374, 638953, 0 UMA Hash: 128, 0, 3, 27, 7, 0 16 Bucket: 76, 0, 24, 26, 79, 0 32 Bucket: 140, 0, 32, 52, 124, 0 64 Bucket: 268, 0, 66, 46, 304, 92 128 Bucket: 524, 0, 2425, 256, 236476, 278077 VM OBJECT: 132, 0, 46935, 46851, 668993405, 0 MAP: 192, 0, 7, 13, 7, 0 KMAP ENTRY: 68, 83944, 724, 3644, 6775734, 0 MAP ENTRY: 68, 0, 4023, 3593, 1503942997, 0 PV ENTRY: 24, 2154990, 156835, 114170, 23476763777, 0 DP fakepg: 72, 0, 0, 159, 187, 0 mt_zone: 1024, 0, 202, 130, 202, 0 16: 16, 0, 3118, 739, 79919591, 0 32: 32, 0, 1695, 1582, 20630720, 0 64: 64, 0, 4365, 1181, 259983599, 0 128: 128, 0, 2656, 1874, 587963693, 0 256: 256, 0, 1223, 1522, 61446743, 0 512: 512, 0, 974, 602, 2732106, 0 1024: 1024, 0, 87, 293, 13781717, 0 2048: 2048, 0, 305, 303, 5034069, 0 4096: 4096, 0, 253, 162, 25067133, 0 Files: 72, 0, 409, 651, 923918935, 0 TURNSTILE: 52, 0, 601, 107, 601, 0 PROC: 536, 0, 171, 403, 15629249, 0 THREAD: 384, 0, 574, 26, 574, 0 KSEGRP: 88, 0, 574, 66, 574, 0 UPCALL: 44, 0, 0, 0, 0, 0 SLEEPQUEUE: 32, 0, 601, 190, 601, 0 VMSPACE: 296, 0, 121, 399, 15629220, 0 mbuf_packet: 256, 0, 355, 78, 703239683, 0 mbuf: 256, 0, 4, 313, 1170882927, 0 mbuf_cluster: 2048, 101376, 433, 77, 30497664, 0 mbuf_jumbo_pagesize: 4096, 0, 0, 0, 0, 0 mbuf_jumbo_9k: 9216, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 0, 0, 0, 0, 0 ACL UMA zone: 388, 0, 0, 0, 0, 0 g_bio: 132, 0, 9, 252, 282058589, 0 ata_request: 204, 0, 3, 206, 78817430, 0 ata_composite: 196, 0, 0, 0, 0, 0 VNODE: 272, 0, 45911, 51235, 564281847, 0 VNODEPOLL: 76, 0, 0, 100, 2, 0 S VFS Cache: 68, 0, 48010, 53910, 621770434, 0 L VFS Cache: 291, 0, 778, 873, 3126471, 0 NAMEI: 1024, 0, 0, 332, 5173085881, 0 DIRHASH: 1024, 0, 7801, 155, 6311308, 0 NFSMOUNT: 480, 0, 0, 0, 0, 0 NFSNODE: 464, 0, 0, 0, 0, 0 PIPE: 408, 0, 59, 211, 24490808, 0 KNOTE: 68, 0, 7, 329, 4267636, 0 socket: 356, 36003, 213, 634, 60494093, 0 unpcb: 144, 36018, 76, 383, 36595883, 0 ipq: 32, 3277, 0, 452, 1074264, 0 udpcb: 180, 36014, 7, 147, 1188051, 0 inpcb: 180, 36014, 587, 2185, 22710032, 0 tcpcb: 464, 36000, 130, 686, 22710032, 0 tcptw: 48, 41028, 457, 1727, 14831193, 0 syncache: 100, 15366, 1, 779, 22182325, 0 hostcache: 76, 15400, 1792, 3508, 962209, 0 tcpreass: 20, 6422, 0, 507, 133846, 0 sackhole: 20, 0, 0, 507, 6270695, 0 ripcb: 180, 36014, 0, 66, 126, 0 rtentry: 132, 0, 15, 72, 15, 0 IPFW dynamic rule: 108, 0, 124, 488, 28605472, 0 SWAPMETA: 276, 121576, 25, 199, 1129, 0 Mountpoints: 664, 0, 16, 8, 18, 0 FFS inode: 132, 0, 45851, 50835, 564218584, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 45851, 50239, 564218584, 0 md1: 512, 0, 621, 131, 2418, 0 interrupt total rate irq14: ata0 58 0 irq16: em0 uhci3 579472231 299 irq19: uhci1+ 75153022 38 cpu0: timer 3867169647 1999 cpu1: timer 3867169408 1999 Total 8388964366 4338 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad4 ad5 in sy cs us sy id 2 11 0 564372 98788 2030 1 0 0 1494 121 0 0 1550 1684 1158 18 6 76 ====================================== iostat ====================================== netstat 390/360/750 mbufs in use (current/cache/total) 384/140/524/101376 mbuf clusters in use (current/cache/total/max) 384/140 mbuf+clusters out of packet secondary zone in use (current/cache) 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/0 9k jumbo clusters in use (current/cache/total/max) 0/0/0/0 16k jumbo clusters in use (current/cache/total/max) 865K/370K/1235K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/225/8704 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 374697 requests for I/O initiated by sendfile 18505 calls to protocol drain routines Current listen queue sizes (qlen/incqlen/maxqlen) Proto Listen Local Address tcp4 0/0/32 *.21 tcp4 0/0/80 *.5432 tcp6 0/0/80 *.5432 tcp4 0/0/128 127.0.0.1.953 tcp4 0/0/3 77.234.201.246.53 tcp4 0/0/1024 77.234.201.244.80 tcp4 0/0/511 127.0.0.1.80 tcp4 0/12/511 77.234.201.242.80 tcp4 0/0/511 77.234.201.242.8100 tcp4 0/0/511 77.234.201.242.8101 tcp4 0/0/511 77.234.201.242.8102 tcp4 0/0/511 77.234.201.242.8103 tcp4 0/0/511 77.234.201.242.8104 tcp4 0/0/128 *.22 tcp6 0/0/128 *.22 tcp4 0/0/1024 127.0.0.1.11211 tcp4 0/0/128 *.199 tcp: 501142243 packets sent 374195421 data packets (2741154668 bytes) 12779915 data packets (4289214028 bytes) retransmitted 209193 data packets unnecessarily retransmitted 3996 resends initiated by MTU discovery 98391784 ack-only packets (0 delayed) 0 URG only packets 55226 window probe packets 165264 window update packets 15555310 control packets 403419309 packets received 289097984 acks (for 4288436002 bytes) 33524674 duplicate acks 1284783 acks for unsent data 69856490 packets (3752880293 bytes) received in-sequence 1412340 completely duplicate packets (152039463 bytes) 0 old duplicate packets 12199 packets with some dup. data (4493165 bytes duped) 133488 out-of-order packets (135550710 bytes) 1676 packets (131558 bytes) of data after window 61 window probes 4613785 window update packets 269460 packets received after close 11887 discarded for bad checksums 59 discarded for bad header offset fields 0 discarded because packet too short 612483 connection requests 22063676 connection accepts 745289 bad connection attempts 0 listen queue overflows 550 ignored RSTs in the windows 22621048 connections established (including accepts) 22709453 connections closed (including 2126726 drops) 9655588 connections updated cached RTT on close 9827878 connections updated cached RTT variance on close 6171442 connections updated cached ssthresh on close 1014 embryonic connections dropped 102862906 segments updated rtt (of 110180207 attempts) 7978044 retransmit timeouts 167158 connections dropped by rexmit timeout 65614 persist timeouts 46 connections dropped by persist timeout 27679 keepalive timeouts 24957 keepalive probes sent 2721 connections dropped by keepalive 22570698 correct ACK header predictions 42873292 correct data packet header predictions 22182306 syncache entries added 471416 retransmitted 626296 dupsyn 20 dropped 22063676 completed 0 bucket overflow 0 cache overflow 63497 reset 54453 stale 0 aborted 0 badack 680 unreach 0 zone failures 22182326 cookies sent 0 cookies received 2741163 SACK recovery episodes 3824937 segment rexmits in SACK recovery episodes 1071004962 byte rexmits in SACK recovery episodes 19044206 SACK options (SACK blocks) received 109749 SACK options (SACK blocks) sent 0 SACK scoreboard overflow ====================================== sysctl net.inet.ip.fw.*dyn* ====================================== counters (connection/stablished/open sockets/open files) 668 4198 51737 24 144 1896