Out Of Memory (OOM)
Out Of Memory
OOM Killer๊ฐ ํน์ ํ๋ก์ธ์ค๋ฅผ ์ข ๋ฃ ์์ผ์, ๋ฉ๋ชจ๋ฆฌ ๊ณต๊ฐ์ ํ๋ณด
OOM Killer ์ปค๋ ๋ฉ์์ง
## /var/log/syslog
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659753] </TASK>
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659754] Mem-Info:
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659756] active_anon:216 inactive_anon:200408 isolated_anon:0
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659756] active_file:18 inactive_file:12 isolated_file:0
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659756] unevictable:6418 dirty:0 writeback:0
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659756] slab_reclaimable:5808 slab_unreclaimable:10210
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659756] mapped:2598 shmem:767 pagetables:1272 bounce:0
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659756] kernel_misc_reclaimable:0
Jan 23 03:56:17 ip-172-31-1-
360kB mlocked:0kB bounce:0kB free_pcp:12kB local_pcp:12kB free_cma:0kB
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659770] lowmem_reserve[]: 0 939 939 939 939
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659774] Node 0 DMA32 free:44284kB min:44332kB low:55412kB high:66492kB reserved_highatomic:0KB active_anon:864kB inactive_anon:791060kB active_file:72kB inactive_file:48kB unevictable:25672kB writepending:0kB present:1032192kB managed:961676kB mlocked:18556kB bounce:0kB free_pcp:256kB local_pcp:256kB free_cma:0kB
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659779] lowmem_reserve[]: 0 0 0 0 0
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659783] Node 0 DMA: 4*4kB (ME) 10*8kB (UME) 10*16kB (UME) 6*32kB (UM) 5*64kB (UME) 3*128kB (UE) 1*256kB (E) 2*512kB (ME) 2*1024kB (ME) 0*2048kB 0*4096kB = 4480kB
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659797] Node 0 DMA32: 981*4kB (UME) 595*8kB (UME) 445*16kB (UME) 280*32kB (UE) 141*64kB (UE) 34*128kB (UME) 18*256kB (UE) 1*512kB (M) 1*1024kB (M) 0*2048kB 0*4096kB = 44284kB
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659811] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659813] 4637 total pagecache pages
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659814] 0 pages in swap cache
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659815] Swap cache stats: add 0, delete 0, find 0/0
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659816] Free swap = 0kB
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659817] Total swap = 0kB
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659818] 262045 pages RAM
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659819] 0 pages HighMem/MovableOnly
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659820] 17786 pages reserved
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659820] 0 pages hwpoisoned
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659821] Tasks state (memory values in pages):
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659821] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659824] [ 162] 0 162 15549 680 139264 0 -250 systemd-journal
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659828] [ 197] 0 197 2486 902 61440 0 -1000 systemd-udevd
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659831] [ 290] 0 290 70052 4501 90112 0 -1000 multipathd
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659833] [ 324] 102 324 22667 604 73728 0 0 systemd-timesyn
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659835] [ 385] 100 385 6820 816 73728 0 0 systemd-network
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659837] [ 387] 101 387 6172 1398 86016 0 0 systemd-resolve
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659840] [ 423] 0 423 60265 348 98304 0 0 accounts-daemon
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659842] [ 424] 0 424 637 182 45056 0 0 acpid
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659844] [ 432]
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659848] [ 442] 0 442 7463 2786 90112 0 0 networkd-dispat
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659850] [ 444] 0 444 60742 617 110592 0 0 polkitd
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659852] [ 445] 104 445 56125 843 77824 0 0 rsyslogd
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659854] [ 448] 0 448 310325 1817 188416 0 0 amazon-ssm-agen
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659857] [ 453] 0 453 4455 624 73728 0 0 systemd-logind
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659859] [ 456] 0 456 98866 482 131072 0 0 udisksd
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659861] [ 462] 0 462 654 351 45056 0 0 sh
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659863] [ 463] 0 463 951 520 45056 0 0 atd
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659865] [ 465] 0 465 321211 3291 286720 0 -999 containerd
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659867] [ 497] 0 497 79699 558 131072 0 0 ModemManager
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659885] [ 631] 0 631 27026 2636 102400 0 0 unattended-upgr
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659887] [ 704] 0 704 329607 5773 372736 0 -500 dockerd
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659889] [ 767] 0 767 624 145 49152 0 0 bpfilter_umh
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659818] 262045 pages RAM
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659819] 0 pages HighMem/MovableOnly
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659820] 17786 pages reserved
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659820] 0 pages hwpoisoned
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659821] Tasks state (memory values in pages):
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659821] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659824] [ 162] 0 162 15549 680 139264 0 -250 systemd-journal
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659828] [ 197] 0 197 2486 902 61440 0 -1000 systemd-udevd
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659831] [ 290] 0 290 70052 4501 90112 0 -1000 multipathd
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659833] [ 324] 102 324 22667 604 73728 0 0 systemd-timesyn
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659835] [ 385] 100 385 6820 816 73728 0 0 systemd-network
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659837] [ 387] 101 387 6172 1398 86016 0 0 systemd-resolve
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659840] [ 423] 0 423 60265 348 98304 0 0 accounts-daemon
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659842] [ 424] 0 424 637 182 45056 0 0 acpid
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659844] [ 432] 0 432 2137 593 53248 0 0 cron
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659846] [ 434] 103 434 1943 801 57344 0 -900 dbus-daemon
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659848] [ 442] 0 442 7463 2786 90112 0 0 networkd-dispat
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659850] [ 444] 0 444 60742 617 110592 0 0 polkitd
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659852] [ 445] 104 445 56125 843 77824 0 0 rsyslogd
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659854] [ 448] 0 448 310325 1817 188416 0 0 amazon-ssm-agen
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659857] [ 453] 0 453 4455 624 73728 0 0 systemd-logind
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659859] [ 456] 0 456 98866 482 131072 0 0 udisksd
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659861] [ 462] 0 462 654 351 45056 0 0 sh
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659863] [ 463] 0 463 951 520 45056 0 0 atd
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659865] [ 465] 0 465 321211 3291 286720 0 -999 containerd
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659867] [ 497] 0 497 79699 558 131072 0 0 ModemManager
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659885] [ 631] 0 631 27026 2636 102400 0 0 unattended-upgr
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659887] [ 704] 0 704 329607 5773 372736 0 -500 dockerd
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659889] [ 767] 0 767 624 145 49152 0 0 bpfilter_umh
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659891] [ 1026] 0 1026 1840 436 57344 0 0 agetty
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659893] [ 1028] 0 1028 1459 394 49152 0 0 agetty
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659895] [ 480120] 0 480120 3048 938 65536 0 -1000 sshd
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659898] [ 482528] 0 482528 311484 2953 258048 0 -900 snapd
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659915] [1123430] 0 1123430 3453 1025 69632 0 0 sshd
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659917] [1123440] 1000 1123440 4734 1081 77824 0 0 systemd
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659919] [1123441] 1000 1123441 42649 1149 94208 0 0 (sd-pam)
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659921] [1123544] 1000 1123544 3487 383 69632 0 0 sshd
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659923] [1123545] 1000 1123545 2539 927 53248 0 0 bash
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659939] [1123600] 1000 1123600 3236 1635 61440 0 0 bash
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659941] [1123887] 1000 1123887 1786 528 53248 0 0 tmux: client
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659943] [1123889] 1000 1123889 2089 783 53248 0 0 tmux: server
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659945] [1123890] 1000 1123890 3209 1641 69632 0 0 bash
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659962] [1123944] 1000 1123944 3209 1646 65536 0 0 bash
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659964] [1124079] 0 1124079 2793 702 57344 0 0 sudo
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659966] [1124080] 0 1124080 1818 351 53248 0 0 tail
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659968] [1124094] 1000 1124094 20997 1191 77824 0 0 stress-ng
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659970] [1124095] 1000 1124095 20997 323 77824 0 0 stress-ng-bighe
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659972] [1124096] 1000 1124096 20997 321 77824 0 0 stress-ng-bighe
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659974] [1124097] 1000 1124097 20997 262 77824 0 0 stress-ng-bighe
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659976] [1124107] 1000 1124107 110182 89318 798720 0 1000 stress-ng-bighe
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659978] [1124108] 0 1124108 1809 21 45056 0 0 sleep
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659980] [1124109] 1000 1124109 74310 53513 512000 0 1000 stress-ng-bighe
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659982] [1124110] 1000 1124110 47302 26497 299008 0 1000 stress-ng-bighe
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659984] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-49943.scope,task=stress-ng-bighe,pid=1124107,uid=1000
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659996] Out of memory: Killed process 1124107 (stress-ng-bighe) total-vm:440728kB, anon-rss:356888kB, file-rss:380kB, shmem-rss:4kB, UID:1000 pgtables:780kB oom_score_adj:1000
๋ฉ๋ชจ๋ฆฌ ํ์ ์ ๋ณด
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659984] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-49943.scope,task=stress-ng-bighe,pid=1124107,uid=1000
Jan 23 03:56:17 ip-172-31-1-240 kernel: [2854486.659996] Out of memory: Killed process 1124107 (stress-ng-bighe) total-vm:440728kB, anon-rss:356888kB, file-rss:380kB, shmem-rss:4kB, UID:1000 pgtables:780kB oom_score_adj:1000OOM Killer์ ํ๋ก์ธ์ค ์ข
๋ฃ ๊ธฐ์ค (v5.19)
๊ฐ์ฅ ๋์ oom score๋ฅผ ๊ฐ์ง ํ๋ก์ธ์ค๊ฐ ์ข ๋ฃ๋จ
## /proc/PID/oom_score
cat /proc/448/oom_score
> 671score ๋ฒ์ : 0 ~ 1000
score๋ฅผ ๋์ด๋ ๊ฒฝ์ฐ?
๋ฉ๋ชจ๋ฆฌ๋ฅผ ๋ง์ด ์ฌ์ฉํ๋ ๊ฒฝ์ฐ
oom_score_adj ๊ฐ ๋์ ๊ฒฝ์ฐ
๋ฉ๋ชจ๋ฆฌ๋ฅผ ๋ง์ด ์ฌ์ฉํ๋๋ผ๋ ์ข ๋ฃ๋ ํ๋ฅ ์ ๋ฎ์ถ๋ ค๋ฉด
oom_score_adj๋ฅผ ๋ฎ๊ฒ ์ค์ ํ๋ค.
ํน์ ํ๋ก์ธ์ค๊ฐ OOM Killer์ ์ํด ์ข ๋ฃ๋์ง ์๊ฒ ํ๋ ค๋ฉด?
oom_score_adj๋ฅผ -1000์ผ๋ก ์ค์
echo -1000 > /proc/448/oom_score_adj
cat /proc/448/oom_score
> 0Last updated