Oracle


Get history of Switchover and Failover of an Oracle database

For physical standbys, there is no DBA_* view that I know which tracks the history of switchover and failover. For Logical standby there is DBA_LOGSTDBY_HISTORY.

Ways to find the information are the Dataguard Broker log – when Broker is in use -, or using the database alert log and the following grep command:

$ grep -EB1 '^ALTER DATABASE (SWITCHOVER|FAILOVER) TO' alert_*.log

2023-03-06T14:12:38.905705+01:00
ALTER DATABASE SWITCHOVER TO 'mydb2'
--
2023-03-06T14:37:43.209653+01:00
ALTER DATABASE FAILOVER TO MYDB1
--
2023-03-06T14:38:03.352141+01:00
ALTER DATABASE SWITCHOVER TO PRIMARY (MYDB1)
--
2023-03-11T16:00:22.224218+01:00
ALTER DATABASE SWITCHOVER TO 'mydb2'


My ARDCI cheatsheet

Check last problems

adrci> show problem -last 10
adrci> show incident -last 10

adrci> show incident -p "incident_id=180781" -mode detail

Create Package

adrci> ips create package problem 5
adrci> ips generate package 1 in "/tmp"

Purge

Value for control is hours

adrci> show control
adrci> set control (SHORTP_POLICY = 720)   /* 30 days */
adrci> set control (LONGP_POLICY= 8760)   /* 1 year */

Value for purge is minutes

adrci> purge -age 1440  /* 1 day */

Fix “No ADR base is set”

Sometimes also “DIA-48494: ADR home is not set”

SQL> select value from v$diag_info where name='ADR Base';
VALUE
------------------
/u01/app/oracle

Create a ${ORACLE_HOME}/log/diag/adrci_dir.mif file with:

$ mkdir -p ${ORACLE_HOME}/log/diag/
$ printf "%s\0" "/u01/app/oracle" > ${ORACLE_HOME}/log/diag/adrci_dir.mif

DIA-48448: This command does not support multiple ADR homes

Use set home diag/xxxxx (remove the ORACLE_BASE path)

adrci> show problems -last 10

ADR Home = /u01/app/oracle/diag/rdbms/mydb1/MYDB1:
...

adrci>  set home diag/rdbms/mydb1/MYDB1

Opatch now obfuscates its own backups – the new “opatch util Obfuscate” option explained

With OPatch version 12.2.0.1.36 for databases (and version 13.9.4.2.11 for Middleware), a new utility was included: obfuscate.

This utility was released to workaround the increased security needed around databases servers. We cannot escape having vulnerability scanners to run there. These vulnerability scanners sometimes do not distinguish between used and unused files.

When patching a database, backup copy of the modified files are kept in $ORACLE_HOME/.patch_storage. Their hash sometime trigger the vulnerability scanners and says – 🚨server not patched ⚠️. Which is misleading.

Starting with OPatch 12.2.0.1.36, released together with the January 2023 Release Update, the backup of patch files are automatically obfuscated.

The new “opatch util obfuscate” allows to do the same for older patches. Let’s see how it works.

Read More

New “OPatch util DeleteInactivePatches” tool for reducing the $ORACLE_HOME size

When you are a sensible person, you patch regularly your Oracle database. After a few years, maybe you noticed that the $ORACLE_HOME increased size from new ~7 GB to …. 26 GB!

$ du -hs /u00/app/oracle/product/19.18.0
26G     /u00/app/oracle/product/19.18.0

Looking better, you find the culprit is the hidden directory $ORACLE_HOME/.patch_storage :

$ du -hs /u00/app/oracle/product/19.18.0/.patch_storage
17G     /u00/app/oracle/product/19.18.0/.patch_storage

Today I show how a new option from OPatch allowed me to regain 11 GB of disk space in a couple of minutes. In a supported way.

Last year I had already presented other options to recover space from Oracle Patching. Below I remind them, but todays post is about the latest and improved way that OPatch tool has to keep only the usual needed patch backups.

Read More

How to workaround Oracle Text primary key limitations (and DRG-10528)

One of my clients had a quite easy desire to try Oracle Text for an existing application. Oracle Text allows to use “standard SQL to index, search, and analyze text and documents stored in the Oracle database, in files, and on the web”.

It seemed simple, until we tried to implement on table named after the city where I studied: T_COIMBRA

create table T_COIMBRA (COL_ID timestamp not null, COL_TXT varchar2(100));

create unique index COIMBRA_IDX_ID on T_COIMBRA(COL_ID);

alter table T_COIMBRA add constraint PK_COIMBRA primary key (COL_ID) using index COIMBRA_IDX_ID;

create index COIMBRA_IDX_TXT ON T_COIMBRA(COL_TXT) indextype is ctxsys.context;

Nothing special it seems. But we get an error:

create index COIMBRA_IDX_TXT ON T_COIMBRA(COL_TXT) indextype is ctxsys.context

Error report -
ORA-29855: error occurred in the execution of ODCIINDEXCREATE routine
ORA-20000: Oracle Text error:
DRG-10528: primary keys of type TIMESTAMP(6) are not allowed
ORA-06512: at "CTXSYS.DRUE", line 186
ORA-06512: at "CTXSYS.TEXTINDEXMETHODS", line 320
29855. 00000 -  "error occurred in the execution of ODCIINDEXCREATE routine"
*Cause:    Failed to successfully execute the ODCIIndexCreate routine.
*Action:   Check to see if the routine has been coded correctly.

Below I show how to workaround this and keep an “unsupported” column as unique identifier of the table.

Read More

Can ChatGPT steal our (Oracle DBA) job?

I would not say it will steal our job, but it can facilitate in some cases to write small pieces of code without having much to think. Well, help first to be lazy, before getting our job. Last example today:

Request ChatGPT to write a trigger that enables BCT after the database opens.

The power is amazing. After looking at the result, you think about a small detail, ask again and it just answers:

Ask ChatGPT to improve so it triggers only when a switchover happened.

But one must always test carefully the answers. Like this case, where the EVENT_TYPE column does not exist on V$DATAGUARD_STATUS table. No idea how he discovers it. I also got non existing syntax and complex solutions that could be easily made simpler.


The OPatch, .patch_storage and its space issues: the solutions!

[a new solution is available as of 2023. Read here about the new OPatch util DeleteInactivePatches option.]

I love database patching and apart of the tiring coordination work or the applications that keep to not automatically reconnect to the database, all is usually perfect and issue free. Well, almost. The most common error are space issues.

You can try to follow the Oracle guidelines and have a 100 GB partition for the $ORACLE_HOME(s). Initially it only uses 7 or 8 GB per home, but after few years you are fighting against the space pressure.

There are several strategies to prevent or act against this space problem when patching:

Solution 1 – Recreate separate Oracle Home from scratch

It is a clean solution, when you make really from scratch, meaning no home cloning, no opatchauto apply -outofplace, and then apply only the latest patch there. This solution is quite easy to do for DB home. However when you have Grid Home this is a bit more hassle.

Solution 2 – Use the opatch hidden “archive” feature

This feature allows to move out from .patch_storage folder some patches in a zip format. It was “documented” by Mike Dietrich in his blog. Unfortunately to have a common archive between different Oracle Homes you need to do some hand work: archive on one Oracle Home, delete the patches from remaining homes and copy the $ORACLE_HOME/OPatch/.patch_storage/.patch_archive_mapping.xml file to the other homes. Of course this works when all Homes have exactly the same features installed and patches. Keep in mind that before rollback you need to use the “unarchive” option and that the rollback procedure will restore the files that were changed, and this can vary depending on the state of the oracle home at the moment of patch. Use opatch util archive -help for more info.

Here we are not saving any space, just moving the problem away. The other partition can be a remote slower location, but the patching will also then be slower, as it will need to copy files there. Use: TARGET=<partition>/patch_storage ; mv $ORACLE_HOME/.patch_storage $TARGET ; ln -s $TARGET $ORACLE_HOME/.patch_storage

Solution 4 – Remove unneeded patches from .patch_storage

In $ORACLE_HOME/.patch_storage the whole history of patches you applied is kept. You can rollback one after the other and bring the Oracle Home several steps behind. However, the most of the cases you are ok to just be able to move one step backward. The older history of the home is past. If that is you case, then there is this nice Python script clean_patch_storage.py which is based on the premise of Oracle Doc ID 550522.1 which states you can “remove all the sub-directories from $ORACLE_HOME/.patch_storage that are not present in the list of installed patches”. The list directories you can delete is exactly what the script do.


Maybe you have other solutions or tricks, please share in the comments.


Useful aliases for ODA Patching

Patching an Oracle Database Appliance can be tiring. The ODA patching commands are quite long and there is always a jobid to check… So I just created a set of alias that make these tasks a bit easier:

# List jobs run today - $ jt
alias jt="odacli list-jobs -o $(date +%Y-%m-%d)"

# Describe one job - $ j <id>
alias j="odacli describe-job -i $1"

# Describe last job created - $ lj
alias lj='odacli describe-job -i $(odacli list-jobs -tl 1 | sed -n 4p | cut -d" " -f1)'

# Describe last prepatch report - $ lpr
alias lpr='odacli describe-prepatchreport -i $(odacli list-jobs -tl 1 | sed -n 4p | cut -d" " -f1)'

# Show free space of / /u01 and /opt - $ dff
alias dff="df -h / /u01 /opt"

# Describe components - $ comp
alias comp="odacli describe-component"

# Tail -f DCS Agent log - $ tal
alias tal="tail -f /opt/oracle/dcs/log/dcs-agent.log"

Maybe you have other suggestions?


gDBClone – Good and easy Oracle cloning tool with potential

Instead of writing and adapting cloning scripts for each client, I was looked today at gDBClone script, provided by Oracle note gDBClone Powerful Database Clone/Snapshot Management Tool (Doc ID 2099214.1).

Ruggero Citton did an excellent work and I believe that for most of situations and mainly in a development environment with the need of fast DB Snapshots this tool provides an alternative to developing new tools.

I find great that for database cloning it creates a temporary listener, and does not mixup with existing configuration. Another great thing is the possibility to use a pre-created passwordfile to clone remote DBs, without the need to provide any password at run time (or hardcode, or having a wallet).

It is a tool that works with DBs from Oracle 11.2 up, and also with RAC, RAC One node, it can perform upgrades, create standby, it works in ODA, etc.

The long set of pre-checks is very nice:

MacroStep1 - Getting information and validating setup...
INFO: 2022-01-13 15:40:09: Validating environment
INFO: 2022-01-13 15:40:09: Checking superuser usage
INFO: 2022-01-13 15:40:09: Checking if target database name 'xpto' is a valid name
INFO: 2022-01-13 15:40:09: Checking if target database home 'OraHome3' exists
INFO: 2022-01-13 15:40:09: Checking if Oracle Restart
INFO: 2022-01-13 15:40:09: Checking ping to host 'server27'
INFO: 2022-01-13 15:40:09: Getting ORACLE_BASE path from orabase
INFO: 2022-01-13 15:40:09: Checking if target database 'xpto' exists
INFO: 2022-01-13 15:40:09: Checking registered instance 'xpto'
INFO: 2022-01-13 15:40:12: Checking listener on 'server27:1521'
INFO: 2022-01-13 15:40:12: Checking ASM command options
INFO: 2022-01-13 15:40:15: Checking if '+U02' is a valid ASM diskgroup
INFO: 2022-01-13 15:40:15: Checking '+U02' RDBMS compatible
INFO: 2022-01-13 15:40:16: Checking if '+U01' is a valid ASM diskgroup
INFO: 2022-01-13 15:40:16: Checking '+U01' RDBMS compatible
INFO: 2022-01-13 15:40:16: Checking if '+U01' is a valid ASM diskgroup
INFO: 2022-01-13 15:40:17: Checking '+U01' RDBMS compatible
INFO: 2022-01-13 15:40:20: Checking source and target database version
INFO: 2022-01-13 15:40:21: Checking source database size
INFO: 2022-01-13 15:40:23: Checking source database role
INFO: 2022-01-13 15:40:23: Checking source log mode
INFO: 2022-01-13 15:40:24: Checking Flash Cache setting
SUCCESS: 2022-01-13 15:40:24: Environment validation complete

At the client I’m these days however I would be happy with some more flexibility.

  • The database unique name convention include underscores (xptodb_2 for instance). gDBClone only accepts alphanumeric elements to the clone database name.
  • gDBClone needs to run with root privileges. You can (recommended) configure in /etc/sudoers, however this is far too much for my client.
  • Cloning a dataguard protected database configured with broker failed at the end of the duplicate, as the clone automatically started the broker and connected back to the primary, failing then with ORA-16649: possible failover to another database prevents this database from being opened . The half-baked clone needs to be dropped manually.
  • It would be nice to have commands (or be part of clone) to create new ACFS mountpoints

At the end I get the impression that gDBClone was developed for specific customer needs and I’m sure it does it really well. For the needs and restrictions I’ve in different clients unfortunately I still need to use my set of tools. Luckily with the most recent Oracle versions the cloning, snapping, upgrades has become much easier and scripting it does not require many lines of code.


Datapatch and Tablespace Full

Yesterday on a simple 19.13 patching session, got a nice error during datapatch: ORA-01691: usable to extend lob segment in Tablespace SYSTEM ! Got a bit scared, but simple added more space to the tablespace, run datapatch again and all was ok. Maybe patch/datapatch should check also for free space in the SYSTEM Tablespace ?

Here the error:

DBD::Oracle::st execute failed: ORA-01691: unable to extend lob segment SYS.SYS_LOB0000323098C00008$$ by 128 in tablespace SYSTEM
ORA-06512: at line 2 (DBD ERROR: OCIStmtExecute) [for Statement "BEGIN
           INSERT INTO sys.dba_registry_sqlpatch_ru_info
             (patch_id,
              patch_uid,
              patch_descriptor,
              ru_version,
              ru_build_description,
              ru_build_timestamp,
              patch_directory)
           VALUES
             (:patch_id,
              :patch_uid,
              :patch_descriptor,
              :ru_version,
              :ru_build_description,
              TO_TIMESTAMP(:ru_build_timestamp, 'YYMMDDHH24MISS'),
              :patch_directory);
           COMMIT;
         END;" with ParamValues: :patch_descriptor=OCIXMLTypePtr=SCALAR(0x468b0a0), :patch_directory='PK........▒LOS▒.
▒. ..▒[..
...33192793.xml▒][.۸.~..
▒.v▒▒▒.x.▒▒q*.o▒▒*.'s▒▒▒(.-qL▒l.ji▒▒/@Im        8 .$.I▒<▒0j▒.>\.▒.▒~.▒▒▒K▒vE▒▒▒.~K..▒6▒▒b▒▒▒a▒?=F.▒..▒]▒\▒5▒▒▒▒.?|▒▒y4f▒▒{X▒▒▒..▒"▒▒|?d.▒..I▒.E.▒▒▒▒=▒▒▒▒▒..▒▒▒.▒▒5.Wg▒w.ۺ▒..▒v}Q%}.▒..▒▒.▒▒.-▒▒▒▒▒\%▒▒׿▒vE.▒▒xsU▒▒)%ħa@.r▒▒.▒Kۢ▒.▒}..y▒▒.~j2▒▒â.=▒▒▒.▒▒▒O.▒▒<▒wm▒▒▒d▒.▒vw▒▒▒▒....▒o޼.▒}▒▒▒.I.▒x▒.▒▒.▒▒▒▒.▒▒.▒x▒o▒)▒▒wo.vxۿ{|\▒,.i]5▒6▒▒..▒▒.b.g▒b▒▒7Ew▒▒▒▒./▒.▒▒o.▒/▒.▒.▒▒6▒#▒.▒.▒...▒▒..▒▒▒▒ߵ▒l▒▒▒▒...▒▒.▒.▒▒.a(▒.▒?|▒.▒
X▒▒{+▒▒v▒=.▒{▒▒▒▒9Íui▒▒.H.▒▒▒▒▒▒▒▒S.▒^▒|▒9.▒%▒,▒▒▒SU▒.▒. zZ.▒..c;.▒vحY▒B...▒▒▒▒/▒▒.▒▒▒▒.▒▒▒..H{.7}f.▒..▒j.▒mۿ▒.▒S.▒7▒.:§.▒+p▒%▒..w▒.▒.▒{7▒.▒k▒..▒6▒6▒▒>v▒bwMU▒..▒Y_▒+▒IG▒<.|4▒▒\▒4.▒i▒*▒▒;G▒-V.ε▒▒.p.▒▒▒▒
G̍1▒.▒]▒▒▒.1O▒v▒^.▒▒▒.▒:.Wy._q7▒▒.*▒&▒▒`/▒I߹...pW.
▒▒▒▒C.v▒j▒[▒▒.▒▒~▒▒.▒.bo▒▒.U▒..▒V▒▒
.`n▒▒N7▒.`nyup▒▒.w▒.]▒▒B▒.▒▒5D7▒.w.#w.▒.▒m▒▒ܬ▒.p▒.3p.▒▒.▒.l▒.▒▒Z;▒0.\bm▒J▒...', :patch_id="33192793", :patch_uid="24462514", :ru_build_description="Release_Update", :ru_build_timestamp="211004165050", :ru_version="19.13.0.0.0"] at /u01/oracle/db19s/sqlpatch/sqlpatch.pm line 5876, <LOGFILE> line 110.