Patchset Update

Warning: OPatchauto ignores disabled components – possible licensing issues

This entry was posted in OPatch Oracle Oracle 19c Patchset Update PSU on by

Since many years at my customer I’m using “opatchauto” to perform a out-of-place patching of Oracle Restart (GI+RDBMS).

My customer is concerned about database users using not licensed options, like partitioning. To avoid it, at the installation time the partitioning option is disabled using chopt, like described at Doc ID 948061.1.

Today during a check we noticed that Partitioning option was activated everywhere, which is not the client standard! We found out the origin of the problem was the out-of-place patching with “opatchauto”.

The big advantage of using “opatchauto” is that it allows easily either a single-step or a two-step Out-of-Place patching. We just write in a properties file the name of the new Oracle Homes and it does:

  • Clone current GI + RDBMS homes to new Homes (prepare clone)
  • Patches the new homes (prepare clone)
  • Stops GI and DBs (switch clone)
  • Switches GI and DBs from current homes to new Homes (switch clone)
  • Restart everything (switch clone)
  • Runs Datapatch on DBs if not standby (switch clone)

This allows to decrease the patching downtime without RAC to about 10 minutes, with the two-step (prepare clone + switch clone) operation.

Here the steps to reproduce de bug:

– install GI + RDBMS Oracle 19c (p.e 19.24) on server

– on RDBMS ORACLE_HOME disable partitioning component as described on Doc ID 948061.1:

$ chopt disable partitioning
Writing to /u00/app/oracle/product/19.24.0/install/disable_partitioning_2025-01-09_11-40-06AM.log...
/usr/bin/make -f /u00/app/oracle/product/19.24.0/rdbms/lib/ins_rdbms.mk part_off ORACLE_HOME=/u00/app/oracle/product/19.24.0
/usr/bin/make -f /u00/app/oracle/product/19.24.0/rdbms/lib/ins_rdbms.mk ioracle ORACLE_HOME=/u00/app/oracle/product/19.24.0

– Check – partition is disabled:

$ ar -tv /u00/app/oracle/product/19.24.0/rdbms/lib/libknlopt.a | grep -E 'kkpoban|ksnkkpo'
rw-r--r-- 1000/1001 5240 Apr 17 07:25 2019 ksnkkpo.o

– Perform out of place patching of DB from GI ORACLE_HOME (single-step example here):

$ cat /staging/clone.properties
/u00/app/grid/19.24.0=/u00/app/grid/19.25.0
/u00/app/oracle/product/19.24.0=/u00/app/oracle/product/19.25.0

$ echo $ORACLE_HOME
/u00/app/grid/19.24.0

$ sudo ${ORACLE_HOME}/OPatch/opatchauto apply -phBaseDir /staging/RU_19c_OCT2024/ -outofplace -silent /staging/clone.properties

OPatchauto session is initiated at Thu Jan  9 12:30:25 2025

System initialization log file is /u00/app/grid/19.24.0/cfgtoollogs/opatchautodb/systemconfig2025-01-09_12-30-30PM.log.

Session log file is /u00/app/grid/19.24.0/cfgtoollogs/opatchauto/opatchauto2025-01-09_12-30-34PM.log
The id for this session is GAEM

Executing OPatch prereq operations to verify patch applicability on home /u00/app/grid/19.24.0
Patch applicability verified successfully on home /u00/app/grid/19.24.0


Executing OPatch prereq operations to verify patch applicability on home /u00/app/oracle/product/19.24.0
Patch applicability verified successfully on home /u00/app/oracle/product/19.24.0


Executing patch validation checks on home /u00/app/grid/19.24.0
Patch validation checks successfully completed on home /u00/app/grid/19.24.0


Executing patch validation checks on home /u00/app/oracle/product/19.24.0
Patch validation checks successfully completed on home /u00/app/oracle/product/19.24.0


Verifying SQL patch applicability on home /u00/app/oracle/product/19.24.0
Skipping SQL patch step execution on standby database : C0201Z01_ENG1
No sqlpatch prereq operations are required on the local node for this home
No step execution required.........

Copying the files from the existing oracle home /u00/app/grid/19.24.0 to a new location. Please wait...

Copying the files from the existing oracle home /u00/app/oracle/product/19.24.0 to a new location. Please wait...
Clone of oracle home /u00/app/grid/19.24.0 is /u00/app/grid/19.25.0 on host vm01
Copying the files from the existing oracle home /u00/app/grid/19.24.0 to a new location is successful.

Clone of oracle home /u00/app/oracle/product/19.24.0 is /u00/app/oracle/product/19.25.0 on host vm01
Copying the files from the existing oracle home /u00/app/oracle/product/19.24.0 to a new location is successful.


Unlocking CRS clone home for home /u00/app/grid/19.24.0.
Prepatch operation log file location: /u00/app/oracle/crsdata/vm01/crsconfig/hapatch_2025-01-09_12-49-31AM.log
Unlocked CRS clone home successfully for home /u00/app/grid/19.24.0.


Creating clone for oracle home /u00/app/grid/19.24.0.
Clone operation successful for oracle home /u00/app/grid/19.24.0.


Creating clone for oracle home /u00/app/oracle/product/19.24.0.
Clone operation successful for oracle home /u00/app/oracle/product/19.24.0.


Performing post clone operation for oracle home /u00/app/grid/19.24.0.
Performing post clone operation was successful for oracle home /u00/app/grid/19.24.0.


Performing post clone operation for oracle home /u00/app/oracle/product/19.24.0.
Performing post clone operation was successful for oracle home /u00/app/oracle/product/19.24.0.


Start applying binary patch on home /u00/app/grid/19.25.0
Binary patch applied successfully on home /u00/app/grid/19.25.0


Start applying binary patch on home /u00/app/oracle/product/19.25.0
Binary patch applied successfully on home /u00/app/oracle/product/19.25.0


Running rootadd_rdbms.sh on home /u00/app/grid/19.25.0
Successfully executed rootadd_rdbms.sh on home /u00/app/grid/19.25.0


Running rootadd_rdbms.sh on home /u00/app/oracle/product/19.25.0
Successfully executed rootadd_rdbms.sh on home /u00/app/oracle/product/19.25.0


Update nodelist in the inventory for oracle home /u00/app/grid/19.25.0.
Update nodelist in the inventory is completed for oracle home /u00/app/grid/19.25.0.


Bringing down database service on home /u00/app/oracle/product/19.25.0
Database service successfully brought down on home /u00/app/oracle/product/19.25.0


Performing postpatch operations on CRS - starting CRS service on home /u00/app/grid/19.25.0
Postpatch operation log file location: /u00/app/oracle/crsdata/vm01/crsconfig/hapatch_2025-01-09_01-27-10PM.log
CRS service started successfully on home /u00/app/grid/19.25.0


Preparing home /u00/app/oracle/product/19.25.0 after database service restarted
No step execution required.........


Confirm that all resources have been started from home /u00/app/grid/19.25.0.
All resources have been started successfully from home /u00/app/grid/19.25.0.


Modifying  the database(s) to use new location home /u00/app/oracle/product/19.25.0.
All database(s) modified successfully to run from new location home /u00/app/oracle/product/19.25.0.


Trying to apply SQL patch on home /u00/app/oracle/product/19.25.0
Skipping SQL patch step execution on standby database : C0201Z01_ENG1
No SQL patch operations are required on local node for this home


OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:vm01
SIHA Home:/u00/app/grid/19.24.0
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /staging/RU_19c_OCT2024/36866740/36916690/36758186
Log: /u00/app/grid/19.24.0/cfgtoollogs/opatchauto/core/opatch/opatch2025-01-09_12-33-55PM_1.log
Reason: /staging/RU_19c_OCT2024/36866740/36916690/36758186 is not required to be applied to oracle home /u00/app/grid/19.24.0


==Following patches were SUCCESSFULLY applied:

Patch: /staging/RU_19c_OCT2024/36866740/36878697
Log: /u00/app/grid/19.25.0/cfgtoollogs/opatchauto/core/opatch/opatch2025-01-09_12-51-54PM_1.log

Patch: /staging/RU_19c_OCT2024/36866740/36916690/36912597
Log: /u00/app/grid/19.25.0/cfgtoollogs/opatchauto/core/opatch/opatch2025-01-09_12-51-54PM_1.log

Patch: /staging/RU_19c_OCT2024/36866740/36916690/36917397
Log: /u00/app/grid/19.25.0/cfgtoollogs/opatchauto/core/opatch/opatch2025-01-09_12-51-54PM_1.log

Patch: /staging/RU_19c_OCT2024/36866740/36916690/36917416
Log: /u00/app/grid/19.25.0/cfgtoollogs/opatchauto/core/opatch/opatch2025-01-09_12-51-54PM_1.log

Patch: /staging/RU_19c_OCT2024/36866740/36916690/36940756
Log: /u00/app/grid/19.25.0/cfgtoollogs/opatchauto/core/opatch/opatch2025-01-09_12-51-54PM_1.log


Host:vm01
SIDB Home:/u00/app/oracle/product/19.24.0
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /staging/RU_19c_OCT2024/36866740/36916690/36917397
Reason: This patch is not applicable to this specified target type - "oracle_database"

Patch: /staging/RU_19c_OCT2024/36866740/36916690/36758186
Reason: This patch is not applicable to this specified target type - "oracle_database"

Patch: /staging/RU_19c_OCT2024/36866740/36916690/36940756
Reason: This patch is not applicable to this specified target type - "oracle_database"


==Following patches were SUCCESSFULLY applied:

Patch: /staging/RU_19c_OCT2024/36866740/36878697
Log: /u00/app/oracle/product/19.25.0/cfgtoollogs/opatchauto/core/opatch/opatch2025-01-09_13-09-38PM_1.log

Patch: /staging/RU_19c_OCT2024/36866740/36916690/36912597
Log: /u00/app/oracle/product/19.25.0/cfgtoollogs/opatchauto/core/opatch/opatch2025-01-09_13-09-38PM_1.log

Patch: /staging/RU_19c_OCT2024/36866740/36916690/36917416
Log: /u00/app/oracle/product/19.25.0/cfgtoollogs/opatchauto/core/opatch/opatch2025-01-09_13-09-38PM_1.log


Patching session reported following warning(s):
_________________________________________________

[Note]: Please verify the database is running from the desired Oracle home, if not then manually execute
 $ORACLE_HOME/bin/srvctl modify database command to fix the problem


Out of place patching clone home(s) summary
____________________________________________
Host : vm01
Actual Home : /u00/app/grid/19.24.0
Version:19.0.0.0.0
Clone Home Path : /u00/app/grid/19.25.0

Host : vm01
Actual Home : /u00/app/oracle/product/19.24.0
Version:19.0.0.0.0
Clone Home Path : /u00/app/oracle/product/19.25.0


OPatchauto session completed at Thu Jan  9 13:32:41 2025
Time taken to complete the session 62 minutes, 16 seconds

– Check – partition component is enabled on the new Oracle Home:

$ ar -tv /u00/app/oracle/product/19.25.0/rdbms/lib/libknlopt.a | grep -E 'kkpoban|ksnkkpo'
rw-r--r-- 1000/1001 5144 Oct 11 13:17 2024 kkpoban.o

This problematic is also true for the other components: Advanced Analytics, OLAP and Real Application Testing.

And probably happens also when using Opatchauto with RAC (not tested), as described here at Doc ID 2419319.1.

I’ve open a Bug/SR today on Oracle Support site and will let here know when I’ve more information.

New in Oracle 19.12 – Log switch SCN in alertlog

This entry was posted in Oracle 19c Patchset Update and tagged oracle 19c on by

Just patched few DBs and noticed this change:

Before:

2021-08-26T05:30:52.463731+02:00
Thread 1 advanced to log sequence 101101 (LGWR switch)
  Current log# 1 seq# 101101 mem# 0: +U01/ANJODB_SITE1/ONLINELOG/group_1.282.988988971

And with 19.12:

2021-08-27T07:17:16.811288+02:00
Thread 1 advanced to log sequence 101176 (LGWR switch),  current SCN: 132131048333
  Current log# 1 seq# 101176 mem# 0: +U01/ANJODB_SITE1/ONLINELOG/group_1.282.988988971

Nice touch from Oracle. 🙂

Opatchauto does not work without DB created. And Oracle support… well

This entry was posted in My Oracle Support OPatch Oracle Oracle 19c Patchset Update on by

Few months ago I hit an interesting case. After installing Oracle Grid Infrastructure and Database software on a server, the new Release Update come out and I wanted to use opatchauto to apply it automatically on both homes.

It doesn’t work. 

sudo ${ORACLE_HOME}/OPatch/opatchauto apply -phBaseDir /tmp/RU_19c_JAN2021/32126842 -prepare-clone -silent /tmp/RU_19c_JAN2021/clone.properties
...
OPATCHAUTO-72128: Cannot execute out of place patching prepare session.
OPATCHAUTO-72128: Clone home properties /tmp/RU_19c_JAN2021/clone.properties specified has entry for non requested home(s) [/u00/app/oracle/product/19.8.0].
OPATCHAUTO-72128: Please correct properties file and re-run out of place patching prepare operation.
OPatchAuto failed.

The setup is all correct, opatchauto is the latest version (13.9.4.5.0, as of February 2021), the inventory has the two homes:

<HOME NAME="OraGI19Home1" LOC="/u00/app/grid/19.8.0" TYPE="O" IDX="1" CRS="true"/>
<HOME NAME="OraDB19Home1" LOC="/u00/app/oracle/product/19.8.0" TYPE="O" IDX="2"/>

The clone.properties file has the name of these homes:

/u00/app/grid/19.8.0=/u00/app/grid/19.10.0
/u00/app/oracle/product/19.8.0=/u00/app/oracle/product/19.10.0

The only “problem” is that no database is created. In fact, the workaround is just to register a fake DB on the CRS:

srvctl add db XX -oraclehome /u00/app/oracle/product/19.8.0

Then the opatchauto works.

I open a SR with Oracle. Inform about the OPatch bug and ask what is the meaning of the OPATCHAUTO-72128 error, because there is no documentation of Opatchauto.

The normal ping-pong with Oracle support starts, they ask me to run with maximum debug level before they can do something. I say that the complete use case is already described and they can try in-house and can/should open a bug.

No success. The support person says that if the customer doesn’t do more work, Oracle doesn’t care about the bug and problem will remain unfixed. I inform that I will blog about this bug, so that the community is aware.

Final answer from this SR:

“Knowledge content not created because the issue was resolved by the customer with no further information provided. Also no further ODM headings were used for the same reason.”

CRS-6706: Oracle Clusterware Release patch level (‘2565072065’) does not match Software patch level (‘277169099’)

This entry was posted in OPatch Oracle Oracle 12c Patchset Update PSU on by

Today on a dirt and quick patching session I end up with the following error when restarting the Oracle Restart Grid Infratructure.

The complete error was:

oracle@anjovm1: [+ASM] crsctl start has
CRS-6706: Oracle Clusterware Release patch level ('2565072065') does not match Software patch level ('277169099'). Oracle Clusterware cannot be started.
CRS-4000: Command Start failed, or completed with errors.

This was because I’ve installed the OJVM RU to the Grid home. The solution was simply:

oracle@anjovm1: ~oracle/ [+ASM] /u00/app/grid/18.0.0/OPatch/opatch rollback -id 28790647

And then I could restart:

oracle@anjovm1:~oracle/ [+ASM] crsctl start has
CRS-4123: Oracle High Availability Services has been started.

Install latest version of Oracle Goldengate 12

This entry was posted in AIX GoldenGate Linux OPatch Oracle Oracle 12c Patchset Update Uncategorized on by

With Oracle Goldengate 12 there is now a big change on the way to install: there is an installer!

Up to Oracle Goldengate 11 every new patch was a new complete version. With Oracle Goldengate 12 you need to install a base version and then the latest patch.

Here is a quick sum up of the instructions (done on AIX, but is similar on Linux):

  1. From Oracle website download the Goldengate 12.1.2.1.0 for your operating system
  2. From Oracle support note 1645495.1 download the latest patchset for Goldengate. Check the availability for your OS.
  3. Unzip both downloaded files
  4. Call the <download_dir>/ggs_AIX_ppc_shiphome/Disk1/runInstaller
  5. Follow the instructions by choosing the target DB version and install directory (I recommend $ORACLE_BASE/product/gg)

    6. Oracle GoldenGate 12.1.2.1.0 - Install Wizard - Step 1 of 5

    Oracle GoldenGate 12.1.2.1.0 - Install Wizard - Step 2 of 5

    Oracle GoldenGate 12.1.2.1.0 - Install Wizard - Step 3 of 5

    Oracle GoldenGate 12.1.2.1.0 - Install Wizard - Step 4 of 5

    Oracle GoldenGate 12.1.2.1.0 - Install Wizard - Step 5 of 5

  6. Check it works
    7. # set ORACLE_HOME to an existing Oracle database installation:
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0.4

# set LD_LIBRARY_PATH to use libraries from $ORACLE_HOME:
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_BASE/product/gg

# Start Goldengate:
cd $ORACLE_BASE/product/gg
./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 12.1.2.1.1 19797716 OGGCORE_12.1.2.1.1_PLATFORMS_141026.2215
AIX 6, ppc, 64bit (optimized), Oracle 11g on Oct 27 2014 00:19:39
Operating system character set identified as ISO-8859-1.

Copyright (C) 1995, 2014, Oracle and/or its affiliates. All rights reserved.

GGSCI (anjovm1) 1> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER STOPPED
GGSCI (anjovm1) 2> exit
  7. Install the latest patchset with OPatch
    8. cd <download_dir>/21785294
$ORACLE_BASE/product/gg/OPatch/opatch apply -oh $ORACLE_BASE/product/gg

Invoking OPatch 11.2.0.1.7

Oracle Interim Patch Installer version 11.2.0.1.7
Copyright (c) 2011, Oracle Corporation.  All rights reserved.

Oracle Home       : /oracle/app/product/gg121
Central Inventory : /oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.7
OUI version       : 11.2.0.3.0
Log file location : /oracle/app/product/gg121/cfgtoollogs/opatch/opatch2015-11-11_08-32-55AM.log

Applying interim patch '21785294' to OH '/oracle/app/product/gg121'
Verifying environment and performing prerequisite checks...

Do you want to proceed? [y|n] y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/oracle/app/product/gg121')

Is the local system ready for patching? [y|n] y
User Responded with: Y
Backing up files...

Patching component oracle.oggcore.ora11g, 12.1.2.1.0...
Patch 21785294 successfully applied
Log file location: /oracle/app/product/gg121/cfgtoollogs/opatch/opatch2015-11-11_08-32-55AM.log

OPatch succeeded.
  8. Check it is updated 
# Start Goldengate:
cd $ORACLE_BASE/product/gg
./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 12.1.2.1.10 21604177 OGGCORE_12.1.2.1.0OGGBP_PLATFORMS_150902.1337
AIX 6, ppc, 64bit (optimized), Oracle 11g on Sep  2 2015 16:29:21
Operating system character set identified as ISO-8859-1.

Copyright (C) 1995, 2015, Oracle and/or its affiliates. All rights reserved.

GGSCI (anjovm1) 1>

We have now Oracle Goldengate 12.1.2.1.10 installed.

Interesting enough is that Oracle Goldengate 12.1 still uses the OPatch version 11.2.0.1.7, taken probably from Oracle 11.2.0.3, as we can see from the line “OUI version : 11.2.0.3.0”.

Check which Oracle PSU are installed

This entry was posted in OPatch Oracle Patchset Update PSU on by

With latest version of OPatch:

$ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i ‘DATABASE PSU’

or

set lines 255

col action_time for a40

col action for a8

col namespace for a7

col version for a11

col comments for a40

select substr(ACTION_TIME,1,40) action_time,substr(ACTION,1,8) action,substr(NAMESPACE,1,7) namespace,

substr(VERSION,1,11) version,ID,substr(COMMENTS,1,40) comments,BUNDLE_SERIES

from registry$history;

To install latest OPatch, after download (make sure you only have OPatch zip for the correct Oracle version: 10.2, 11.1, 11.2):

  • mv $ORACLE_HOME/OPatch $ORACLE_HOME/OPatch.old
  • unzip p6880880* -d $ORACLE_HOME