{"id":542,"date":"2023-03-09T17:00:24","date_gmt":"2023-03-09T16:00:24","guid":{"rendered":"https:\/\/anjo.pt\/keyword-oracle\/?p=542"},"modified":"2023-03-09T17:00:25","modified_gmt":"2023-03-09T16:00:25","slug":"root-mailbox-huge-on-exadata-after-upgrade-to-ol7","status":"publish","type":"post","link":"https:\/\/anjo.pt\/keyword-oracle\/2023\/03\/09\/root-mailbox-huge-on-exadata-after-upgrade-to-ol7\/","title":{"rendered":"Root mailbox huge on Exadata after upgrade to OL7"},"content":{"rendered":"\n<p>Finally I got to work a bit more with an Exadata. And finding its unknown &#8220;features&#8221;. <\/p>\n\n\n\n<p>Our monitoring just detected that &#8220;\/ diskspace 6% free (1.9\/23.5 GB) (DiskWarning)&#8221;. Looking for the guilty folder or file, I end up here:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># for d in \/*; do egrep \" ${d} \" \/proc\/mounts > \/dev\/null || du -sh ${d}; done\n...\n15G     \/var\r\n\n# cd \/var\/\n# du -hs * | sort -h\n...\n12G     spool\r\n\n# cd spool\/\r\n# du -hs * | sort -h\r\n...\n12G     mail\r\n\n# cd mail\/\r\n# du -hs * | sort -h\n...\n12G     root\r<\/code><\/pre>\n\n\n\n<p>The huge file was \/var\/spool\/mail\/root &#8211; the mailbox of the root user.<\/p>\n\n\n\n<p>Trying to open it, just created another small problem:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># mail\r\n\/tmp: No space left on device\r<\/code><\/pre>\n\n\n\n<p>(parenthesis here: on this Exadata VM we have DBFS running. The check done by the CRS for the DBFS &#8211; official mount-dbfs.sh script -, also writes to \/tmp. If \/tmp is full when the DBFS check runs, it fails, and the clusterware moves DBFS to another node. Goldengate, which was using DBFS, just crashed)<\/p>\n\n\n\n<p>The root mailbox is just a text file, so using other tools I saw that is full of emails that come from &#8220;0logwatch&#8221;. This matches something in cron.daily:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># ls -l \/etc\/cron.daily\/\r\r\n-rwx------ 1 root root  408 Apr 15  2022 0logwatch\r<\/code><\/pre>\n\n\n\n<p>Looking on Metalink, quickly we end up on this note:<\/p>\n\n\n\n<p><a href=\"https:\/\/support.oracle.com\/epmos\/faces\/DocContentDisplay?id=2564364.1\" target=\"_blank\" rel=\"noreferrer noopener\">Output of Daily Cronjob 0logwatch Sent to Mail instead of Saving to File on Oracle Linux 7 (Doc ID 2564364.1)<\/a><\/p>\n\n\n\n<p>The problem comes from an Exadata update, which did not care about syntax change of the \/etc\/logwatch\/conf\/logwatch.conf file with OL7. What is a pity, is that later Exadata patches did not seem to care of fixing that. <\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Finally I got to work a bit more with an Exadata. And finding its unknown &#8220;features&#8221;. Our monitoring just detected that &#8220;\/ diskspace 6% free (1.9\/23.5 GB) (DiskWarning)&#8221;. Looking for the guilty folder or file, I end up here: The huge file was \/var\/spool\/mail\/root &#8211; the mailbox of the root user. Trying to open it, [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[67,69],"tags":[],"class_list":{"0":"post-542","1":"post","2":"type-post","3":"status-publish","4":"format-standard","6":"category-engineer-systems","7":"category-exadata","8":"czr-hentry"},"_links":{"self":[{"href":"https:\/\/anjo.pt\/keyword-oracle\/wp-json\/wp\/v2\/posts\/542","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/anjo.pt\/keyword-oracle\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/anjo.pt\/keyword-oracle\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/anjo.pt\/keyword-oracle\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/anjo.pt\/keyword-oracle\/wp-json\/wp\/v2\/comments?post=542"}],"version-history":[{"count":1,"href":"https:\/\/anjo.pt\/keyword-oracle\/wp-json\/wp\/v2\/posts\/542\/revisions"}],"predecessor-version":[{"id":544,"href":"https:\/\/anjo.pt\/keyword-oracle\/wp-json\/wp\/v2\/posts\/542\/revisions\/544"}],"wp:attachment":[{"href":"https:\/\/anjo.pt\/keyword-oracle\/wp-json\/wp\/v2\/media?parent=542"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/anjo.pt\/keyword-oracle\/wp-json\/wp\/v2\/categories?post=542"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/anjo.pt\/keyword-oracle\/wp-json\/wp\/v2\/tags?post=542"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}