Miguel Anjo


DBCA doesn’t like personal SQL*Plus prompt

The database creation (19c) using DBCA was hanging at 33%:

[ 2024-04-24 10:31:41.770 CEST ] Prepare for db operation
DBCA_PROGRESS : 8%
[ 2024-04-24 10:31:41.808 CEST ] Creating and starting Oracle instance
DBCA_PROGRESS : 11%
DBCA_PROGRESS : 12%
DBCA_PROGRESS : 17%
[ 2024-04-24 10:32:11.412 CEST ] Creating database files
DBCA_PROGRESS : 18%
DBCA_PROGRESS : 25%
[ 2024-04-24 10:32:11.702 CEST ] Creating data dictionary views
DBCA_PROGRESS : 28%
DBCA_PROGRESS : 33%

The catproc0.log shows many connection errors:

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Apr 24 11:36:32 2024
Version 19.21.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.

connect SYS/SP2-0640: Not connected
connect SYS/  2  SP2-0640: Not connected
connect SYS/SP2-0640: Not connected
connect SYS/connect SYS/  2  SP2-0640: Not connected
connect SYS/connect SYS/  2  SP2-0640: Not connected
connect SYS/connect SYS/
connect SYS/connect SYS/  2  SP2-0640: Not connected
connect SYS/connect SYS/SP2-0640: Not connected
connect SYS/SP2-0640: Not connected
connect SYS/  2  SP2-0640: Not connected
connect SYS/connect SYS/connect SYS/connect SYS/connect SYS/connect SYS/connect SYS/connect SYS/connect SYS/connect SYS/connect SYS/connect SYS/connect SYS/connect SYS/SQL> SQL> SQL> SQL> SQL> SQL>   2  SP2-0640: Not connected
...

The generated trace file shows at the end:

[progressPage.flowWorker] [ 2024-04-24 10:32:17.334 CEST ] [PerlEngine.execute:220]  /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/admin/catproc.sql
[progressPage.flowWorker] [ 2024-04-24 10:32:17.334 CEST ] [PerlEngine.execute:232]  Execing PERL process...
[progressPage.flowWorker] [ 2024-04-24 10:32:17.334 CEST ] [PerlEngine.execute:274]  m_bReaderStarted: false
[progressPage.flowWorker] [ 2024-04-24 10:32:17.334 CEST ] [PerlEngine.execute:278]  Starting Reader Thread... 
[progressPage.flowWorker] [ 2024-04-24 10:32:17.334 CEST ] [PerlEngine.execute:332]  Inputing secret arg: 0
[progressPage.flowWorker] [ 2024-04-24 10:32:17.334 CEST ] [PerlEngine.execute:332]  Inputing secret arg: 1
[Thread-233] [ 2024-04-24 10:32:17.379 CEST ] [PerlEngine.readPerlOutput:597]  perl engine output:Enter Password: 
[Thread-233] [ 2024-04-24 10:32:17.379 CEST ] [PerlEngine.readPerlOutput:597]  perl engine output:Enter Password: 

This was part of an automation script I wrote, which was started during the VM installation using ansible. It was working fine until not long ago. What might have happened?

I end up opening an SR and searching a bit more, until I found out that the culprit was that in the previous steps there are a ORACLE_PATH variable added to .bash_profile . And recently I changed the login.sql to show the name of DB and PDB where the user is connected to. However I forgot to test for the cases when the database is down, on which it was simply showing:

$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Apr 24 11:36:32 2024
Version 19.21.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.

>

for some reason dbca did not like it. Ok, partly was my fault, lack of tests. Solution was to make sure that even with the database down (or without database), the sqlplus shows a meaningful prompt.

In a next blog post I’ll show how to personalize SQL*Plus and SQLcl prompt, which is compatible to dbca.


Oracle Engineering Systems (Community) Day

On a full room in the centre of Zurich, Switzerland, two great events dedicated to ODA, Exadata, ExaCC, Exadata at OCI or Azure, PCA and other derivatives took place on March 20th 2024. In the morning organized and sponsored by Tradeware AG was the 21st Oracle Engineering Systems Community meeting, followed by 11th Swiss Oracle Engineering Systems day, organized by Oracle.

Manfred Drodz from Peakmarks showed the results of his performance benchmarks with the new Exadata X10M and how it improves up to 35% compared with X7. Peakmarks is specialized on benchmarking Oracle workloads, helping to choose the right number of CPUs needed when migrating to a new platform.

In Switzerland, Twint is a relatively new payment platform, meanwhile used by more than 50% from the population at least once a month! Erich Steiger showed us how on Exadata performance dilutes application problems, making them quite difficult to spot. But, after convincing the developers to avoid doing unnecessary queries, finally two application bugs could be found, making the existing Exadata resistant for some more years. Currently Twint is looking for the ExaCC solution and its dynamic scaling to absorb the midday peaks.

Mike Dietrich from Oracle presented the new M5 cross-platform migration script and how it was used for migrating a 200TB database from Solaris Supercluster to Exadata. More information on Mike’s website.

After lunch, Ashish Ray presented the Oracle strategy in terms of Exadata, ExaCC and AI. It presented how Oracle look forward with future ExaScale architecture which might remove the limited amount of VMs in a Exadata with its elastic infrastructure.

Swisscom – a major Swiss infrastructure provider – had Roger Fehr and Roger Staubli presenting their ExaCC strategy and how they are together with two core banking solutions (Avaloq and Finova) doing its certification against the Cloud@Customer solution.

Later in the day Marcus Praetzas from Deutsche Bank showed the converged strategy they follow and how ExaCC and Autonomous DB on Exadata allows them to have Oracle as a main database for almost all types of usage.

As a Recap of this intensive day, it was clear that many Swiss and German customers are happy with the Oracle’s Exadata Cloud@Customer solution and pretend to continue using it in their strategy.


Install Oracle Linux 9 and Oracle Database 19c on Windows WSL

Since a few days I’ve a Oracle Database 19c running on my laptop (4 year old i5 processor with 16GB memory) under the Windows Subsystem for Linux with the latest Oracle Linux 9.3. It is working great for testing functionalities. It survives without problem sleep and restart without first shutting down correctly the database.

While there are no snapshot possibility like with Virtualbox, it is possible to export the running image and re-import later.

Below I describe the main steps to quickly install the system.

(more…)

Swiss OUC – Database Circle Conference 2024

I had the pleasure to assist to the first Swiss Oracle User Community conference in Zurich the last two days. Great event the the Oracle offices in the “Circle” at the Zurich Airport. It was opportunity to see some ex-Trivadis colleagues – Ludo, Frank, Dani and meet all the Oracle conference addicts.

The two parallel sessions were creating me every hour a dilemma: which presentation will I miss.

Here some of the highlights I noticed – this are what for me was important or new, certainly I will not cover all the knowledge that was in the air.

Swiss OUC day 1

Dominic Giles (Oracle) gave an overview of 23c new Generative AI features and more:

  • New datatype VECTOR for AI data models was introduced and delayed the 23c release. It allows to use ML models and calculate distance between vectors using different methods.
  • AI Assist will soon allow to use natural language to create SQL specific for the data model of “our” database. Will also be available on premises – similar but easier than “Select AI” from Autonomous DB
  • SQL Developer extension for VS Code allows using DB connections directly on VS Code – already available:

Martin Klier explained well and visually how networks, subnets, routers work and what happens when the way to the destination is cut. Really good reminder.

Øyvind Isene shared his story of being the sole developer of an DB automation API leaving the following messages:

  • “You don’t have the time to do all the errors again. Learn from other errors.” –> share your errors with others
  • Use Logger when scripting; Use ORDS for DB automation

Neil Chandler reminded some Oracle Security fundamentals, like: always enable Oracle Network Encryption in sqlnet.ora and add Password Verify Function to default database profile.

Again Martin Klier said that Performance is rarely result of an accident. For this, when doing performance analysis, document every action you do, even if will just show that the problem is/was not there; do not use averages and pay attention to skewness. 

Chris Saxon and Jasmin Fluri did a small theater play around SQL version management and explained why they use either use Flyway or Liquidbase (integrated into SQLcl) for that.

Swiss OUC day 2

Chris Saxon on a plenum session did a example based overview of the 23c new features for developers:

  • new DB_DEVELOPER_ROLE – combination of CONNECT+RESOURCE+some other privs.
  • create if not exists, drop if exist DDLs to avoid errors on multiple execution (might hide things with create if exists, but different)
  • No more need of “from dual”
  • Domain datatype extensions – allow named column types with integrated constraints and defaults used schema wide, with extension to have a display and order specific functions (for instance INTERVAL DAY datatype that displays in minutes)
  • Annotations for columns/tables/domains – kind of extension of comments in name-value pair format
  • Multiple insert values in a single command – also helpful to generate data
  • Group by using column alias
  • sum/floor/ceil/avg on interval and other datatypes
  • fuzzy_match function to replace like when looking  for similar results (without need of Oracle Text)
  • json converted to insert
  • boolean usage everywhere (eg: function returns boolean then possible to do “select … from … where func_name(input)”
  • SQL_TRANSPILER parameter injects content of function directly in SQL, allowing the optimizer to better improve query (default false)
  • GRANT … ANY … on Schema
  • Remind to use NOAUTHENTICATION for owner schema and proxy users instead of share passwords.

Mike Dietrich went on with 23c new features on Data Pump:

  • New dumpfile format with special functionalities for OCI Object Storage
  • INCLUDE and EXCLUDE can be combined in the same impdp/expdp
  • All transportable export/import can run now in parallel
  • Checksum for dumpfiles
  • REMAP_TABLESPACE=%:USERS (wildcard usage)
  • Recommendation to gather SYS+SYSTEM schema stats instead of using “gather_dictionary_stats”
  • Always install DATAPUMP merge patch. It is not part of an RU because it it not RAC Rolling installable

Neil Chandler shared the story of a bank that wanted to replace end-of-life Exadata with something cheaper. Neil spend some time investigating all resource usage – mainly the CPU from storage servers + saved IO from cell offloading and then calculating costs. Conclusion: if they want to keep the performance, there is no cheaper option by using commodity hardware or open source DB.

Frank Pachot on a tmux terminal session explained that the ‘load’ high values in the ‘top’ command from Linux do not mean the machine is overloaded. Basically my get ways were :

  • There is in recent kernel the PSI tool, not enabled by default, which can show better weather there is pressure and where this is happening.
  • Different types of IO – specifically a type of direct io can count for the load avg. 

Erik van Roon is a fan of SQLcl and convinced the audience by showing why that is superior to SQL*Plus. Check many of its features here or here or here.

Fernando Simon moved from Exadata to ExaCC and shared his thoughts about the new experience:

  • ExaCC offers good automation, click type of actions for inexperienced DBAs
  • ExaCC lacks basic checks like if there is enough space to backup OH before a patch [why it backups OH?, does not do out-of-place?]
  • No integration of created OH images
  • CPU scale up works well and is immediate, but scale down has a hard minimum wait time of 2h since last scale up
  • There is no documented tool that allows create a DB with all parameters (block size, charset, silent mode with password)
  • Even Zero Downtime Migration tool makes as workaround to create a dummy DB, drop it manually and create manually again using DBCA. DBs created directly with DBCA, will not be seen in the OCI, there is no ‘discovery’ command
  • No possibility to use DBs with different domain in the name in the same ExaCC without unofficial workarounds
  • ExaCC makes all admin operations serial which can be slow

Dani Schneider showed the new SQL Analysis Report and how it allows to get hints of bad sql when running explain plan.

Kamil Stawiarski presented “Jas-min” – a word play with the name of one conference organizer – JSON AWR Statspack Minier. And how this tool can help in cases like you have flashback queries and cursor: pin s wait on x. Basically, do not use use flashback queries in the application if the query will be use concurrently

Thanks a lot Gianni, Jasmin and Julian from Swiss Oracle User Community for this great conference. Hope to see you again next year!


Nice table for Oracle OLVM: VMs, hosts, clusters and vcpu pinning 2

On my client infrastructure, for improved performance, we need to make sure each VM is pinning to different physical cores.

We do the changes for the moment using the web interface. However, to read easily what the current status is, I’ve wrote a python script that draws a nice table showing where each VM is running and to which cores its vcpu are pinned.

The result is something like this:

(more…)

We did changed nothing. How to set up AES encrypted username password in DB2 client?

It all started with the connections from Goldengate to DB2 not working anymore:

2023-09-11 15:31:27  WARNING OGG-00552  Database operation failed: Couldn't connect to DB2_DB1. ODBC error: SQLSTATE 08001 native database error -30073. [IBM][CLI Driver] SQL30073N  "0x0000" Parameter value "0x0000" is not supported.  SQLSTATE=58017.
Failed to open data source DB2_DB1for user GG01.

The DB2 administrators say “We changed nothing. Maybe you changed something? And the DB2 documentation says nothing for this error.

Next email from DB2 admins “well, we did activated the option TCPALVER=SERVER_ENCRYPT and maybe that is the problem”

Indeed, that was the problem. With that option on the server side, then:

SERVER_ENCRYPT (recommended)


A user ID and password are required for connection requests. Kerberos tickets are also accepted. Also, one of the following conditions must be true:

The user ID and password is AES (Advanced Encryption Standard)-encrypted.

Non-encrypted security credentials are not accepted unless the connection is secured by the TCP/IP network. DES-based (Data Encryption Standard) encryption is also considered insecure.

Ok, I needed to activate on my client side the AES encrytion for username and password. Not so difficult:

db2cli writecfg add -dsn DB2_DB1  -parameter "Authentication=SERVER_ENCRYPT_AES"

And voilà, the connection to DB2 works again, and it is secured.


Get history of Switchover and Failover of an Oracle database

For physical standbys, there is no DBA_* view that I know which tracks the history of switchover and failover. For Logical standby there is DBA_LOGSTDBY_HISTORY.

Ways to find the information are the Dataguard Broker log – when Broker is in use -, or using the database alert log and the following grep command:

$ grep -EB1 '^ALTER DATABASE (SWITCHOVER|FAILOVER) TO' alert_*.log

2023-03-06T14:12:38.905705+01:00
ALTER DATABASE SWITCHOVER TO 'mydb2'
--
2023-03-06T14:37:43.209653+01:00
ALTER DATABASE FAILOVER TO MYDB1
--
2023-03-06T14:38:03.352141+01:00
ALTER DATABASE SWITCHOVER TO PRIMARY (MYDB1)
--
2023-03-11T16:00:22.224218+01:00
ALTER DATABASE SWITCHOVER TO 'mydb2'


Goldengate 21 for DB2 on z/OS and segmentation fault

These days I’ve been trying Goldengate 21 for DB2 on z/OS (Extract). And still surprises!

Last year, with Goldengate 21.6, it would simply corrupt trailfiles if the extract abended.

This year, with Goldengate 21.10, After adding a new extract,the extract abends without any error or explanation. The report file would remain with this:

2023-06-21 15:11:31 INFO OGG-25184

Using DB2 subsystem DBZOS.

Only in the /var/log/messages one could see:

Jun 21 15:11:31 oraclevm1 kernel: extract[281329]: segfault at 0 ip 00007f1c5cca609f sp 00007ffcabd65c28 error 4 in libc-2.17.so[7f1c5cc0b000+1c4000]

SR open and Oracle says, it is fixed with Goldengate 21.11, which is released in two weeks. Well… almost. With Goldengate 21.11 the extract still crashes but only about five minutes after the start command. And the error on /var/log/messages is little different:

Aug 28 13:29:50 oraclevm1 kernel: extract[154775]: segfault at 170 ip 00005569d4916a82 sp 00007ffe69d1e600 error 4 in extract[5569d4415000+aaa000]

Oracle says – yes, sorry. Please try the workaround to create the extract with:

ADD EXTRACT e_ab, TRANLOG, EOF

When I check the Goldengate 21 documentation for “ADD EXTRACT” , this option is not valid for DB2 for z/OS, but ok. I try and it works.

Checking with Oracle, the information is that when doing:

ADD EXTRACT e_ab, TRANLOG, BEGIN NOW

It stores the time we run the command and, when we start the extract, it will scan the DB2 log to find the LSN (Log Sequence Number or the CSN from DB2) corresponding to that time. In this scan operation is the bug that makes the segmentation fault. Using EOF, it will always use the end of the Transaction Log at the time we start the Extract.

The Goldengate documentation will be updated soon to add that EOF is also valid for DB2 for z/OS.


My ADRCI cheatsheet

Check last problems

adrci> show problem -last 10
adrci> show incident -last 10

adrci> show incident -p "incident_id=180781" -mode detail

Create Package

adrci> ips create package problem 5
adrci> ips generate package 1 in "/tmp"

Purge

Value for control is hours

adrci> show control
adrci> set control (SHORTP_POLICY = 720)   /* 30 days */
adrci> set control (LONGP_POLICY= 8760)   /* 1 year */

Value for purge is minutes

adrci> purge -age 1440  /* 1 day */

Fix “No ADR base is set”

Sometimes also “DIA-48494: ADR home is not set”

SQL> select value from v$diag_info where name='ADR Base';
VALUE
------------------
/u01/app/oracle

Create a ${ORACLE_HOME}/log/diag/adrci_dir.mif file with:

$ mkdir -p ${ORACLE_HOME}/log/diag/
$ printf "%s\0" "/u01/app/oracle" > ${ORACLE_HOME}/log/diag/adrci_dir.mif

DIA-48448: This command does not support multiple ADR homes

Use set home diag/xxxxx (remove the ORACLE_BASE path)

adrci> show problems -last 10

ADR Home = /u01/app/oracle/diag/rdbms/mydb1/MYDB1:
...

adrci>  set home diag/rdbms/mydb1/MYDB1

Opatch now obfuscates its own backups – the new “opatch util Obfuscate” option explained

With OPatch version 12.2.0.1.36 for databases (and version 13.9.4.2.11 for Middleware), a new utility was included: obfuscate.

This utility was released to workaround the increased security needed around databases servers. We cannot escape having vulnerability scanners to run there. These vulnerability scanners sometimes do not distinguish between used and unused files.

When patching a database, backup copy of the modified files are kept in $ORACLE_HOME/.patch_storage. Their hash sometime trigger the vulnerability scanners and says – 🚨server not patched ⚠️. Which is misleading.

Starting with OPatch 12.2.0.1.36, released together with the January 2023 Release Update, the backup of patch files are automatically obfuscated.

The new “opatch util obfuscate” allows to do the same for older patches. Let’s see how it works.

Read More