Once again I end up with my clients database swapping. Why? After quick investigation, could see that HugePages were not used on the last restart of the database.<\/p>\n\n\n
\noracle@myvm1:.\/trace\/ [oracle19] grep -B1 -A4 PAGESIZE alert*.log\n2020-04-14T04:36:34.601494+02:00\n PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s)\n2020-04-14T04:36:34.601550+02:00\n 4K Configured 10 10 NONE\n2020-04-14T04:36:34.601642+02:00\n 2048K 247816 8193 8193 NONE\n--\n2020-10-13T22:59:28.856763+02:00\n PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s)\n2020-10-13T22:59:28.856818+02:00\n 4K Configured 10 4186122 NONE\n2020-10-13T22:59:28.856925+02:00\n 2048K 202479 8193 17 NONE\n<\/pre><\/div>\n\n\nWhy was that? I did use a normal start command:<\/p>\n\n\n
\noracle@myvm1:.\/trace\/ [oracle19] srvctl start database -db mydb\n<\/pre><\/div>\n\n\nLet’s put the context. This is a Oracle Restart server, with separation between oracle and grid users.<\/p>\n\n\n\n\n\n\n\n
The database in the past used HugePages, so they are configured (I skip the check here, but I did it).<\/p>\n\n\n\n
I’ve looked around and could see that ulimits were correctly set for both users:<\/p>\n\n\n
\noracle@myvm1:.\/trace\/ [oracle19] grep 'memlock' \/etc\/security\/limits.d\/99-grid-oracle-limits.conf\noracle soft memlock 1342177280\noracle hard memlock 1342177280\ngrid soft memlock 1342177280\ngrid hard memlock 1342177280\n<\/pre><\/div>\n\n\nI see the file was changed after the previous DB restart. The server was not restarted in-between.<\/p>\n\n\n\n
But on the alertlog I see that the ulimit seen by the database changed compared the the last restart:<\/p>\n\n\n
\noracle@myvm1:.\/trace\/ [oracle19] grep -h -B1 "Per proc" alert*.log\n2020-04-14T04:36:34.301642+02:00\n Per process system memlock (soft) limit = 1280G\n--\n2020-10-13T22:59:28.756763+02:00\n Per process system memlock (soft) limit = 64K\n<\/pre><\/div>\n\n\nI decided to stop the database and starting it again but with sqlplus. And HugesPages were used!<\/p>\n\n\n\n
So, where could be the problem?<\/p>\n\n\n\n
The srvctl uses de oraagent.bin to start databases. It is so important to check what the limits used by this process:<\/p>\n\n\n
\ngrid@myvm1:~\/ [grid19] grep memory \/proc\/$(pgrep oraagent.bin)\/limits\nMax locked memory 65536 65536 bytes\n<\/pre><\/div>\n\n\nHere is the problem, the oraagent.bin was started before the change of the limits file. And new limits are only valid for new sessions.<\/p>\n\n\n\n
When we connect to a new session and start a DB with sqlplus, it will get this session limits. But if we use srvctl, it will get the limits that were in place when oraagent.bin started.<\/p>\n\n\n\n
We are on a production server, restart the whole clusterware stack would be quite annoying. What is nice, is that we can simply “kill” the oraagent.bin process and it will respawn. Even better if we do it with the ohasd.bin.<\/p>\n\n\n
\ngrid@myvm1:~\/ [grid19] ps -ef | grep ohasd.bin\ngrid 2854 1 0 Sep02 ? 03:23:19 \/u00\/app\/grid\/product\/19\/bin\/ohasd.bin reboot\n\ngrid@myvm1:~\/ [grid19] kill -9 2854\n\ngrid@myvm1:~\/ [grid19] ps -ef | grep ohasd.bin\ngrid 185684 1 17 14:22 ? 00:00:01 \/u00\/app\/grid\/product\/19\/bin\/ohasd.bin restart\n\ngrid@myvm1:~\/ [grid19] psg pmon\ngrid 6458 1 0 Sep02 ? 00:03:38 asm_pmon_+ASM\noracle 7069 1 0 Oct13 ? 00:05:18 ora_pmon_mydb\n<\/pre><\/div>\n\n\nAnd now the new limits are correctly set for the oraagent.bin:<\/p>\n\n\n
\ngrid@myvm1:~\/ [grid19] grep memory \/proc\/$(pgrep oraagent.bin)\/limits\nMax locked memory 1374389534720 1374389534720 bytes\n<\/pre><\/div>\n\n\nAnd further restarts with srvctl are using HugePages.<\/p>\n\n\n\n
Lesson: check the if the ulimits are correctly applied to oragent.bin. <\/p>\n\n\n\n
If you have a file with a list of servers separated by spaces or lines, then you can use a similar loop to go through all of them:<\/p>\n\n\n
\nfor server in $(cat server_list.txt | grep -v '^#'); do \n echo "=== $server"; \n ssh -o ConnectTimeout=1 -o ConnectionAttempts=1 -oBatchMode=yes $server \\\n "echo '99-grid - ' \\$(grep 'grid soft memlock' \/etc\/security\/limits.d\/99-grid-oracle-limits.conf); echo 'oraagent - ' \\$(grep memory \/proc\/\\$(pgrep oraagent.bin)\/limits)"; \ndone;\n<\/pre><\/div>\n\n\n<\/p>\n","protected":false},"excerpt":{"rendered":"
Once again I end up with my clients database swapping. Why? After quick investigation, could see that HugePages were not used on the last restart of the database. Why was that? I did use a normal start command: Let’s put the context. This is a Oracle Restart server, with separation between oracle and grid users.<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[],"class_list":{"0":"post-406","1":"post","2":"type-post","3":"status-publish","4":"format-standard","6":"category-oracle","7":"czr-hentry"},"_links":{"self":[{"href":"https:\/\/anjo.pt\/keyword-oracle\/wp-json\/wp\/v2\/posts\/406","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/anjo.pt\/keyword-oracle\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/anjo.pt\/keyword-oracle\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/anjo.pt\/keyword-oracle\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/anjo.pt\/keyword-oracle\/wp-json\/wp\/v2\/comments?post=406"}],"version-history":[{"count":3,"href":"https:\/\/anjo.pt\/keyword-oracle\/wp-json\/wp\/v2\/posts\/406\/revisions"}],"predecessor-version":[{"id":452,"href":"https:\/\/anjo.pt\/keyword-oracle\/wp-json\/wp\/v2\/posts\/406\/revisions\/452"}],"wp:attachment":[{"href":"https:\/\/anjo.pt\/keyword-oracle\/wp-json\/wp\/v2\/media?parent=406"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/anjo.pt\/keyword-oracle\/wp-json\/wp\/v2\/categories?post=406"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/anjo.pt\/keyword-oracle\/wp-json\/wp\/v2\/tags?post=406"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}