Slow starting impdp from NFS Share   Recently updated !

This entry was posted in Datapump DNS Engineer Systems ExaCC on by

Unfortunately I lost the logs for this issue, but I try to document for information.

My customer has ExaCC with various 2-node clusters.

  • Export ACFS mount point as NFS from cluster1
  • Mount NFS mount point on cluster2, cluster3 and cluster4

He did an export from cluster1 to the ACFS mount point.

All was working fine until mid December, when impdp reading a dumpfile from the NFS mount point seems hanging when was called from cluster3 and cluster4. From cluster2 it was still fine.

Few days later, the impdp was slow everywhere, except locally on cluster1.

The behavior was very bizarre:

  • impdp starting showing timestamp
  • exactly 5 minutes later first output comes “W-1 Startup took 1 second”
  • exactly 5 minutes after comes second line “W-1 Master table … successfully loaded/unloaded”
  • and 5 minutes later runs the rest, quickly.

The NFS mount point seemed ok, ‘dd’ command tests did not show any slowness.

I started to investigating by enabling the DataPump tracing, as explained by Daniel Hansen on his Databases are Fun blog:

alter system set events 'sql_trace {process: pname = dw | process: pname = dm} level=8';

The trace files generated on Diagnostics directory did not help much – they are mostly for performance problems.

Then I did start a “strace” on the PID of the impdp

strace -p <pid> -o /tmp/strace.out

There I could see some “ECONNREFUSED” to one of the IPs of the Cluster1. But few lines above, there was the same connection without error.

Quite strange. Finally with the help of one system administrator, we found out that the nfs-server was not running on one of the cluster1 nodes. And the NFS mount point was using a hostname which dynamically would go either to one or another node of the cluster1. After making sure nfs-server was running on both nodes from cluster1, the problem was solved and impdp was fast to start again.

Learnings:

  • Use the clusterware to manage exportfs – srvctl add exportfs
  • Make use of VIPs which move from one node to another instead of round-robin DNS entries.

ORA-64307 when creating compressed table and /home

This entry was posted in Engineer Systems ExaCC Exadata Oracle 19c on by

My customer running on ExaCC (Exadata Cloud@Customer) was getting “ORA-64307: Exadata Hybrid Columnar Compression is not supported for tablespaces on this storage type” on one of his test databases.

I did test connecting to SYS and no problem. Then I try to do using his tablespace and indeed, I get the error:

Quite going around, to check what was different on the user tablespace than on others. I test a self created tablespace and it works.

Strange. Until I found that… some datafiles were not in ASM!

Seems the ASM Diskgroup is almost full and the client DBA just put the datafiles somewhere else!

JumpHost Matryoshka

This entry was posted in Other on by

My client just added an extra jumphost before arriving to the server. So now I’ve to connect, to connect, to connect and then open the connection. 🙂

Warning: OPatchauto ignores disabled components – possible licensing issues

This entry was posted in OPatch Oracle Oracle 19c Patchset Update PSU on by

Since many years at my customer I’m using “opatchauto” to perform a out-of-place patching of Oracle Restart (GI+RDBMS).

My customer is concerned about database users using not licensed options, like partitioning. To avoid it, at the installation time the partitioning option is disabled using chopt, like described at Doc ID 948061.1.

Today during a check we noticed that Partitioning option was activated everywhere, which is not the client standard! We found out the origin of the problem was the out-of-place patching with “opatchauto”.

The big advantage of using “opatchauto” is that it allows easily either a single-step or a two-step Out-of-Place patching. We just write in a properties file the name of the new Oracle Homes and it does:

  • Clone current GI + RDBMS homes to new Homes (prepare clone)
  • Patches the new homes (prepare clone)
  • Stops GI and DBs (switch clone)
  • Switches GI and DBs from current homes to new Homes (switch clone)
  • Restart everything (switch clone)
  • Runs Datapatch on DBs if not standby (switch clone)

This allows to decrease the patching downtime without RAC to about 10 minutes, with the two-step (prepare clone + switch clone) operation.

Here the steps to reproduce de bug:

– install GI + RDBMS Oracle 19c (p.e 19.24) on server

– on RDBMS ORACLE_HOME disable partitioning component as described on Doc ID 948061.1:

$ chopt disable partitioning
Writing to /u00/app/oracle/product/19.24.0/install/disable_partitioning_2025-01-09_11-40-06AM.log...
/usr/bin/make -f /u00/app/oracle/product/19.24.0/rdbms/lib/ins_rdbms.mk part_off ORACLE_HOME=/u00/app/oracle/product/19.24.0
/usr/bin/make -f /u00/app/oracle/product/19.24.0/rdbms/lib/ins_rdbms.mk ioracle ORACLE_HOME=/u00/app/oracle/product/19.24.0

– Check – partition is disabled:

$ ar -tv /u00/app/oracle/product/19.24.0/rdbms/lib/libknlopt.a | grep -E 'kkpoban|ksnkkpo'
rw-r--r-- 1000/1001 5240 Apr 17 07:25 2019 ksnkkpo.o

– Perform out of place patching of DB from GI ORACLE_HOME (single-step example here):

$ cat /staging/clone.properties
/u00/app/grid/19.24.0=/u00/app/grid/19.25.0
/u00/app/oracle/product/19.24.0=/u00/app/oracle/product/19.25.0

$ echo $ORACLE_HOME
/u00/app/grid/19.24.0

$ sudo ${ORACLE_HOME}/OPatch/opatchauto apply -phBaseDir /staging/RU_19c_OCT2024/ -outofplace -silent /staging/clone.properties

OPatchauto session is initiated at Thu Jan  9 12:30:25 2025

System initialization log file is /u00/app/grid/19.24.0/cfgtoollogs/opatchautodb/systemconfig2025-01-09_12-30-30PM.log.

Session log file is /u00/app/grid/19.24.0/cfgtoollogs/opatchauto/opatchauto2025-01-09_12-30-34PM.log
The id for this session is GAEM

Executing OPatch prereq operations to verify patch applicability on home /u00/app/grid/19.24.0
Patch applicability verified successfully on home /u00/app/grid/19.24.0


Executing OPatch prereq operations to verify patch applicability on home /u00/app/oracle/product/19.24.0
Patch applicability verified successfully on home /u00/app/oracle/product/19.24.0


Executing patch validation checks on home /u00/app/grid/19.24.0
Patch validation checks successfully completed on home /u00/app/grid/19.24.0


Executing patch validation checks on home /u00/app/oracle/product/19.24.0
Patch validation checks successfully completed on home /u00/app/oracle/product/19.24.0


Verifying SQL patch applicability on home /u00/app/oracle/product/19.24.0
Skipping SQL patch step execution on standby database : C0201Z01_ENG1
No sqlpatch prereq operations are required on the local node for this home
No step execution required.........

Copying the files from the existing oracle home /u00/app/grid/19.24.0 to a new location. Please wait...

Copying the files from the existing oracle home /u00/app/oracle/product/19.24.0 to a new location. Please wait...
Clone of oracle home /u00/app/grid/19.24.0 is /u00/app/grid/19.25.0 on host vm01
Copying the files from the existing oracle home /u00/app/grid/19.24.0 to a new location is successful.

Clone of oracle home /u00/app/oracle/product/19.24.0 is /u00/app/oracle/product/19.25.0 on host vm01
Copying the files from the existing oracle home /u00/app/oracle/product/19.24.0 to a new location is successful.


Unlocking CRS clone home for home /u00/app/grid/19.24.0.
Prepatch operation log file location: /u00/app/oracle/crsdata/vm01/crsconfig/hapatch_2025-01-09_12-49-31AM.log
Unlocked CRS clone home successfully for home /u00/app/grid/19.24.0.


Creating clone for oracle home /u00/app/grid/19.24.0.
Clone operation successful for oracle home /u00/app/grid/19.24.0.


Creating clone for oracle home /u00/app/oracle/product/19.24.0.
Clone operation successful for oracle home /u00/app/oracle/product/19.24.0.


Performing post clone operation for oracle home /u00/app/grid/19.24.0.
Performing post clone operation was successful for oracle home /u00/app/grid/19.24.0.


Performing post clone operation for oracle home /u00/app/oracle/product/19.24.0.
Performing post clone operation was successful for oracle home /u00/app/oracle/product/19.24.0.


Start applying binary patch on home /u00/app/grid/19.25.0
Binary patch applied successfully on home /u00/app/grid/19.25.0


Start applying binary patch on home /u00/app/oracle/product/19.25.0
Binary patch applied successfully on home /u00/app/oracle/product/19.25.0


Running rootadd_rdbms.sh on home /u00/app/grid/19.25.0
Successfully executed rootadd_rdbms.sh on home /u00/app/grid/19.25.0


Running rootadd_rdbms.sh on home /u00/app/oracle/product/19.25.0
Successfully executed rootadd_rdbms.sh on home /u00/app/oracle/product/19.25.0


Update nodelist in the inventory for oracle home /u00/app/grid/19.25.0.
Update nodelist in the inventory is completed for oracle home /u00/app/grid/19.25.0.


Bringing down database service on home /u00/app/oracle/product/19.25.0
Database service successfully brought down on home /u00/app/oracle/product/19.25.0


Performing postpatch operations on CRS - starting CRS service on home /u00/app/grid/19.25.0
Postpatch operation log file location: /u00/app/oracle/crsdata/vm01/crsconfig/hapatch_2025-01-09_01-27-10PM.log
CRS service started successfully on home /u00/app/grid/19.25.0


Preparing home /u00/app/oracle/product/19.25.0 after database service restarted
No step execution required.........


Confirm that all resources have been started from home /u00/app/grid/19.25.0.
All resources have been started successfully from home /u00/app/grid/19.25.0.


Modifying  the database(s) to use new location home /u00/app/oracle/product/19.25.0.
All database(s) modified successfully to run from new location home /u00/app/oracle/product/19.25.0.


Trying to apply SQL patch on home /u00/app/oracle/product/19.25.0
Skipping SQL patch step execution on standby database : C0201Z01_ENG1
No SQL patch operations are required on local node for this home


OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:vm01
SIHA Home:/u00/app/grid/19.24.0
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /staging/RU_19c_OCT2024/36866740/36916690/36758186
Log: /u00/app/grid/19.24.0/cfgtoollogs/opatchauto/core/opatch/opatch2025-01-09_12-33-55PM_1.log
Reason: /staging/RU_19c_OCT2024/36866740/36916690/36758186 is not required to be applied to oracle home /u00/app/grid/19.24.0


==Following patches were SUCCESSFULLY applied:

Patch: /staging/RU_19c_OCT2024/36866740/36878697
Log: /u00/app/grid/19.25.0/cfgtoollogs/opatchauto/core/opatch/opatch2025-01-09_12-51-54PM_1.log

Patch: /staging/RU_19c_OCT2024/36866740/36916690/36912597
Log: /u00/app/grid/19.25.0/cfgtoollogs/opatchauto/core/opatch/opatch2025-01-09_12-51-54PM_1.log

Patch: /staging/RU_19c_OCT2024/36866740/36916690/36917397
Log: /u00/app/grid/19.25.0/cfgtoollogs/opatchauto/core/opatch/opatch2025-01-09_12-51-54PM_1.log

Patch: /staging/RU_19c_OCT2024/36866740/36916690/36917416
Log: /u00/app/grid/19.25.0/cfgtoollogs/opatchauto/core/opatch/opatch2025-01-09_12-51-54PM_1.log

Patch: /staging/RU_19c_OCT2024/36866740/36916690/36940756
Log: /u00/app/grid/19.25.0/cfgtoollogs/opatchauto/core/opatch/opatch2025-01-09_12-51-54PM_1.log


Host:vm01
SIDB Home:/u00/app/oracle/product/19.24.0
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /staging/RU_19c_OCT2024/36866740/36916690/36917397
Reason: This patch is not applicable to this specified target type - "oracle_database"

Patch: /staging/RU_19c_OCT2024/36866740/36916690/36758186
Reason: This patch is not applicable to this specified target type - "oracle_database"

Patch: /staging/RU_19c_OCT2024/36866740/36916690/36940756
Reason: This patch is not applicable to this specified target type - "oracle_database"


==Following patches were SUCCESSFULLY applied:

Patch: /staging/RU_19c_OCT2024/36866740/36878697
Log: /u00/app/oracle/product/19.25.0/cfgtoollogs/opatchauto/core/opatch/opatch2025-01-09_13-09-38PM_1.log

Patch: /staging/RU_19c_OCT2024/36866740/36916690/36912597
Log: /u00/app/oracle/product/19.25.0/cfgtoollogs/opatchauto/core/opatch/opatch2025-01-09_13-09-38PM_1.log

Patch: /staging/RU_19c_OCT2024/36866740/36916690/36917416
Log: /u00/app/oracle/product/19.25.0/cfgtoollogs/opatchauto/core/opatch/opatch2025-01-09_13-09-38PM_1.log


Patching session reported following warning(s):
_________________________________________________

[Note]: Please verify the database is running from the desired Oracle home, if not then manually execute
 $ORACLE_HOME/bin/srvctl modify database command to fix the problem


Out of place patching clone home(s) summary
____________________________________________
Host : vm01
Actual Home : /u00/app/grid/19.24.0
Version:19.0.0.0.0
Clone Home Path : /u00/app/grid/19.25.0

Host : vm01
Actual Home : /u00/app/oracle/product/19.24.0
Version:19.0.0.0.0
Clone Home Path : /u00/app/oracle/product/19.25.0


OPatchauto session completed at Thu Jan  9 13:32:41 2025
Time taken to complete the session 62 minutes, 16 seconds

– Check – partition component is enabled on the new Oracle Home:

$ ar -tv /u00/app/oracle/product/19.25.0/rdbms/lib/libknlopt.a | grep -E 'kkpoban|ksnkkpo'
rw-r--r-- 1000/1001 5144 Oct 11 13:17 2024 kkpoban.o

This problematic is also true for the other components: Advanced Analytics, OLAP and Real Application Testing.

And probably happens also when using Opatchauto with RAC (not tested), as described here at Doc ID 2419319.1.

I’ve open a Bug/SR today on Oracle Support site and will let here know when I’ve more information.

Solve “OGG-08224 Error: CONTAINER option was specified though the database does not support containers” error

This entry was posted in GoldenGate Oracle 19c on by

Quick post to add info about the following Goldengate error:

OGG (http://localhost:9300 test1 as ogg_pdb1@CDB2) 10> REGISTER EXTRACT E_TEST1 DATABASE CONTAINER (pdb1)

2024-12-08T17:16:58Z ERROR OGG-08224 Error: CONTAINER option was specified though the database does not support containers.

This means that you are connected directly to the PDB, and not to CDB$ROOT.

To register Goldengate 21 extracts you need to connect to the Root container with a common user.

OGG (http://localhost:9300 test1 as ogg_pdb1@CDB2) 12> DBLOGIN USERIDALIAS ogg_cdb2
Successfully logged into database CDB$ROOT.

OGG (http://localhost:9300 test1 as ogg_cdb2@CDB2/CDB$ROOT) 13> REGISTER EXTRACT E_TEST1 DATABASE CONTAINER (pdb1)
2024-12-08T17:20:36Z  INFO    OGG-02003  Extract group E_TEST1 successfully registered with database at SCN 8039188.

Well, in the future this is not anymore true, as new version from Goldengate and DBs will work only at PDB level.

Using AI to confirm a wrongly cabled Exadata switch – or how to fix verify_roce_cables.py script for Python3.

This entry was posted in Engineer Systems Exadata Oracle on by

One of the preparation steps when installing an Exadata X10M is to verify that the cabling of the RoCE switches is correctly done. The next step is to upgrade the Cisco switches with the latest firmware. During my intervention for Tradeware at the customer, the first didn’t work as the provided script is not compatible with Python3 and the latter complained about wrong cabling.

Here I show how studied the wrong cabling of the X10M switches and how I use Claude.ai (ChatGPT and other AI tools probably also work) to quickly fix the Python script provided by Oracle.

(more…)

Oracle postpones release of 23ai on-premises to 2H CY2024

This entry was posted in Exadata Oracle 23ai on by

Oracle just updated the Release Schedule of Current Database Releases (Doc ID 742060.1) and changed the release date of database version 23ai on-premises to next half-year. Lets see how many months and bug fixing that means. 🙂

Update on 20.06.2024 – “Added new release dates for Oracle Autonomous Database – Dedicated Exadata Infrastructure, Autonomous Database on Exadata Cloud@Customer, ODA, Exadata and Linux-x86 64”

The DBT-16051 when creating a standby database using DBCA is still around. 7 years after.

This entry was posted in DataGuard Oracle Oracle 19c on by

Sometimes I ask myself why some bugs are not solved. When looking for DBT-16071 we find a blog post from Frank Pachot from more than 7 years ago. He shows that with Oracle 12.2 you can “create” standby databases directly with dbca. But that the script does only a duplicate for standby and nothing more.

I decided to try with 19.22 to see how the situation evolved. It didn’t.

The first thing I got was a DBT-16051 error:

$ dbca -createDuplicateDB -gdbName anjodb01 -primaryDBConnectionString "anjovm01.local.wsl/anjodb01_s1.local.wsl" -sid anjodb01 -createAsStandby -dbUniqueName anjodb01_s2 -silent
Enter SYS user password:
*****
[FATAL] [DBT-16051] Archive log mode is not enabled in the primary database.
   ACTION: Primary database should be configured with archive log mode for creating a duplicate or standby database.

Quick check shows the primary is correctly in archivelog mode. The problem is the Easy Connect string. The string I gave “anjovm1.local.wsl/anjodb1_s1.local.wsl” works well on sqlplus, but not with dbca. There you need to specify the port, also when you are just using the default one:

$ dbca -createDuplicateDB -gdbName anjodb01 -primaryDBConnectionString "anjovm01.local.wsl:1521/anjodb01_s1.local.wsl" -sid anjodb01 -createAsStandby -dbUniqueName anjodb01_s2 -silent
Enter SYS user password:
*****
[WARNING] [DBT-10331] Specified SID Name (anjodb01) may have a potential conflict with an already existing database on the system.
   CAUSE: The specified SID Name without the trailing numeric characters ({2}) may have a potential conflict with an already existing database on the system.
   ACTION: Specify a different SID Name that does not conflict with existing databases on the system.
Prepare for db operation
22% complete
Listener config step
44% complete
Auxiliary instance creation
67% complete
RMAN duplicate
89% complete
Post duplicate database operations
100% complete

The warning DBT-10331 appears because I’ve a “anjodb02” in the same VM, and this could create a problem, as they share the prefix “anjodb”. I don’t expect on a single instance environment that to be a problem though.

And it starts the new standby in ‘read only’ mode, which requires adequate licenses. 

SQL> select name, db_unique_name, database_role, open_mode, dataguard_broker from v$database;

NAME      DB_UNIQUE_NAME                 DATABASE_ROLE    OPEN_MODE            DATAGUAR
--------- ------------------------------ ---------------- -------------------- --------
ANJODB01 ANJODB02_S2                  PHYSICAL STANDBY READ ONLY            DISABLED

For the moment, I’ll stay with my set of scripts which do the operations in the right way.

Change language and prompt in SQLcl for VS Code

This entry was posted in SQLcl VSCode on by

Windows and language settings is a common problem for users outside English speaking countries. The windows language on my laptop is English, but time to time I use another laptop in Portugal with the same Microsoft account as login. This laptop is configured in Portuguese. My work laptop’s keyboard is Swiss, locale also set for Switzerland. Windows just gets confused and shows some applications in English, others in German, others in Portuguese.

Today I was testing SQLcl inside VS Code. I tried the help function and get answers in German, plus with the öäü signs wrong.

So I needed to change it to English. Based on SQLcl documentation it reads a file startup.sql once when it starts. And on VS Code, to know where it is currently running, you just have to run the ‘pwd’ command:

SQL> pwd
C:\Users\migue\

Then, on this path, you can create a startup.sql file and change the prompt, and language. My startup.sql starts by changing the display language, then the SQLcl layout and the prompt:

set feedb off termout off
-- changes the display language
alter session set nls_language=american;

-- Changes SQLcl to have word highlight and statusbar
set highlighting on
set highlighting keyword foreground green
set highlighting identifier background blue
set highlighting string foreground yellow
set highlighting number foreground cyan
set highlighting comment foreground white
set statusbar on
set statusbar add editmode
set statusbar add txn
set statusbar add timing

-- Changes prompt to show the instance and pdb name
DEFINE prompt="SQL"
COLUMN col_prompt NEW_VALUE prompt
SELECT UPPER(USER || '@' || SYS_CONTEXT('userenv', 'instance_name')) ||'.'|| SYS_CONTEXT('userenv', 'con_name') col_prompt FROM dual;
set sqlprompt "&prompt> " ;
set termout on feed on

At the ends it looks like this:

How to use Zabbix to monitor Oracle DB – a quick test run

This entry was posted in Oracle on by

Currently I’m testing different solutions for monitoring (mostly) Oracle databases. One solution in the shortlist is Zabbix.

Zabbix is open-source and currently has a quite active community helping out.

On Zabbix website there is a nice quick download and setup instructions:

I’ll use a Oracle Linux 9 VM under Windows WSL2 for installing Zabbix. For the exercise, I’ll configure it with PostgreSQL. The database installation step is missing. So, here are all the steps I’ve done.

(more…)