Finally I got to work a bit more with an Exadata. And finding its unknown “features”.
Our monitoring just detected that “/ diskspace 6% free (1.9/23.5 GB) (DiskWarning)”. Looking for the guilty folder or file, I end up here:
# for d in /*; do egrep " ${d} " /proc/mounts > /dev/null || du -sh ${d}; done
...
15G /var
# cd /var/
# du -hs * | sort -h
...
12G spool
# cd spool/
# du -hs * | sort -h
...
12G mail
# cd mail/
# du -hs * | sort -h
...
12G root
The huge file was /var/spool/mail/root – the mailbox of the root user.
Trying to open it, just created another small problem:
# mail
/tmp: No space left on device
(parenthesis here: on this Exadata VM we have DBFS running. The check done by the CRS for the DBFS – official mount-dbfs.sh script -, also writes to /tmp. If /tmp is full when the DBFS check runs, it fails, and the clusterware moves DBFS to another node. Goldengate, which was using DBFS, just crashed)
The root mailbox is just a text file, so using other tools I saw that is full of emails that come from “0logwatch”. This matches something in cron.daily:
# ls -l /etc/cron.daily/
-rwx------ 1 root root 408 Apr 15 2022 0logwatch
Looking on Metalink, quickly we end up on this note:
Output of Daily Cronjob 0logwatch Sent to Mail instead of Saving to File on Oracle Linux 7 (Doc ID 2564364.1)
The problem comes from an Exadata update, which did not care about syntax change of the /etc/logwatch/conf/logwatch.conf file with OL7. What is a pity, is that later Exadata patches did not seem to care of fixing that.