Engineer Systems


Using AI to confirm a wrongly cabled Exadata switch – or how to fix verify_roce_cables.py script for Python3.

One of the preparation steps when installing an Exadata X10M is to verify that the cabling of the RoCE switches is correctly done. The next step is to upgrade the Cisco switches with the latest firmware. During my intervention for Tradeware at the customer, the first didn’t work as the provided script is not compatible with Python3 and the latter complained about wrong cabling.

Here I show how studied the wrong cabling of the X10M switches and how I use Claude.ai (ChatGPT and other AI tools probably also work) to quickly fix the Python script provided by Oracle.

(more…)

Oracle postpones release of 23ai on-premises to 2H CY2024

Oracle just updated the Release Schedule of Current Database Releases (Doc ID 742060.1) and changed the release date of database version 23ai on-premises to next half-year. Lets see how many months and bug fixing that means. 🙂

Update on 20.06.2024 – “Added new release dates for Oracle Autonomous Database – Dedicated Exadata Infrastructure, Autonomous Database on Exadata Cloud@Customer, ODA, Exadata and Linux-x86 64”


Root mailbox huge on Exadata after upgrade to OL7

Finally I got to work a bit more with an Exadata. And finding its unknown “features”.

Our monitoring just detected that “/ diskspace 6% free (1.9/23.5 GB) (DiskWarning)”. Looking for the guilty folder or file, I end up here:

# for d in /*; do egrep " ${d} " /proc/mounts > /dev/null || du -sh ${d}; done
...
15G     /var

# cd /var/
# du -hs * | sort -h
...
12G     spool

# cd spool/
# du -hs * | sort -h
...
12G     mail

# cd mail/
# du -hs * | sort -h
...
12G     root

The huge file was /var/spool/mail/root – the mailbox of the root user.

Trying to open it, just created another small problem:

# mail
/tmp: No space left on device

(parenthesis here: on this Exadata VM we have DBFS running. The check done by the CRS for the DBFS – official mount-dbfs.sh script -, also writes to /tmp. If /tmp is full when the DBFS check runs, it fails, and the clusterware moves DBFS to another node. Goldengate, which was using DBFS, just crashed)

The root mailbox is just a text file, so using other tools I saw that is full of emails that come from “0logwatch”. This matches something in cron.daily:

# ls -l /etc/cron.daily/

-rwx------ 1 root root  408 Apr 15  2022 0logwatch

Looking on Metalink, quickly we end up on this note:

Output of Daily Cronjob 0logwatch Sent to Mail instead of Saving to File on Oracle Linux 7 (Doc ID 2564364.1)

The problem comes from an Exadata update, which did not care about syntax change of the /etc/logwatch/conf/logwatch.conf file with OL7. What is a pity, is that later Exadata patches did not seem to care of fixing that.


Solution for ODA 19.13 list-availablepatches error

After patching an ODA to 19.13, often we have the error

[root@oda-01 DB1]# odacli list-availablepatches
DCS-10001:Internal error encountered: For input string: "".

The workaround provided by Oracle on the 19.13 “Known Issues” is incomplete and might not solve the problem in many cases.

The complete solution is to change all patchmetadata.xml files where this “targetVersion” is missing. Here is the sed command that creates a backup (extension .bck) and adds the missing bit:

sed -i.bck -E 's/name="DB" repotag="(1[0-9]\.[0-9]{1,2}\.[0-9]\.[0-9]\.[0-9]{6})"><\/component>/name="DB" repotag="\1" targetVersion="\1"><\/component>/g' /opt/oracle/oak/pkgrepos/System/*/patchmetadata.xml

Now, odacli list-availablepatches is back to normal:

[root@oda-01 DB1]# odacli list-availablepatches
-------------------- ------------------------- -------------------------
ODA Release Version  Supported DB Versions     Available DB Versions
-------------------- ------------------------- -------------------------

19.6.0.0.0           19.6.0.0.200114           Clone not available
                     18.9.0.0.200114           Clone not available
                     12.2.0.1.200114           Clone not available
                     12.1.0.2.200114           Clone not available
                     11.2.0.4.200114           Clone not available

19.13.0.0.0          21.4.0.0.211019           Clone not available
                     19.13.0.0.211019          19.13.0.0.211019
                     12.2.0.1.211019           Clone not available
                     12.1.0.2.211019           Clone not available

19.10.0.0.0          19.10.0.0.210119          Clone not available
                     18.13.0.0.210119          Clone not available
                     12.2.0.1.210119           Clone not available
                     12.1.0.2.210119           Clone not available
                     11.2.0.4.210119           Clone not available

18.8.0.0.0           18.8.0.0.191015           Clone not available
                     18.8.0.0.191015           Clone not available
                     12.2.0.1.191015           Clone not available
                     12.1.0.2.191015           Clone not available
                     11.2.0.4.191015           Clone not available

18.5.0.0.0           11.2.0.4.190115           Clone not available
                     12.1.0.2.190115           Clone not available
                     12.2.0.1.190115           12.2.0.1.190115
                     18.5.0.0.190115           Clone not available

18.4.0.0.0           11.2.0.4.181016           Clone not available
                     12.1.0.2.181016           Clone not available
                     12.2.0.1.181016           Clone not available
                     18.4.0.0.181016           Clone not available

18.2.1.0.0           11.2.0.4.180417           Clone not available
                     12.1.0.2.180417           Clone not available
                     12.2.0.1.180417           Clone not available

12.2.1.4.0           11.2.0.4.180417           Clone not available
                     12.1.0.2.180417           Clone not available
                     12.2.0.1.180417           Clone not available

Useful aliases for ODA Patching

Patching an Oracle Database Appliance can be tiring. The ODA patching commands are quite long and there is always a jobid to check… So I just created a set of alias that make these tasks a bit easier:

# List jobs run today - $ jt
alias jt="odacli list-jobs -o $(date +%Y-%m-%d)"

# Describe one job - $ j <id>
alias j="odacli describe-job -i $1"

# Describe last job created - $ lj
alias lj='odacli describe-job -i $(odacli list-jobs -tl 1 | sed -n 4p | cut -d" " -f1)'

# Describe last prepatch report - $ lpr
alias lpr='odacli describe-prepatchreport -i $(odacli list-jobs -tl 1 | sed -n 4p | cut -d" " -f1)'

# Show free space of / /u01 and /opt - $ dff
alias dff="df -h / /u01 /opt"

# Describe components - $ comp
alias comp="odacli describe-component"

# Tail -f DCS Agent log - $ tal
alias tal="tail -f /opt/oracle/dcs/log/dcs-agent.log"

Maybe you have other suggestions?