Oracle 19c

High IO due space manager (W00x process) doing db sequential read all the time – KTSJ Slave bug? 1

This entry was posted in bug Oracle Oracle 19c on by

On a client DB, after migrating it to OL8, the application started to complaint the DB seems slower. Checking the OEM, there is not much done by the application, but I see a good blue stripe from W00x processes doing sequential reads. Nonstop. This is a 19.31 single-instance database.

After tracing these sessions, they are reading two distinct LOB segments, each between 500 and 700GB, where a lot of deletes happened. All the time “db file sequential read”, up to 3 or 4 W00x parallel processes.

WAIT #0: nam='db file sequential read' ela= 199 file#=476 block#=2460662 blocks=1 obj#=53176 tim=1024371028390

I open an SR with Oracle. They ask more traces and session short stack. The short_stack looks like:

SQL> oradebug setospid 15297
Oracle pid: 28, Unix process pid: 15297, image: oracle@srv0nnnn.xxx.ch (W000)
SQL> oradebug unlimit
Statement processed.
SQL>  oradebug short_stack

ksedsts()+426<-ksdxfstk()+63<-ksdxcb()+872<-sspuser()+236<-__sighandler()<-semtimedop()+14<-sskgpwwait()+245<-skgpwwait()+187<-ksliwat()+2296<-kslwaitctx()+188<-ktsj_slave_main()+944<-ksvrdp_int()+1906<-opirip()+579<-opidrv()+581<-sou2o()+165<-opimai_real()+173<-ssthrdmain()+417<-main()+256<-__libc_start_main()+229<-_start()+46

No underscore parameters are set and changing “_enable_space_preallocation” to 0 has no immediate effect.

The tablespace is composed of several smallfiles, EXTENT_MANAGEMENT=LOCAL, ALLOCATION_TYPE=SYSTEM, SEGMENT_SPACE_MANAGEMENT=AUTO. All standard.

The LOB segment is securefile, configured in-row, with several larger values which go out of row.

I ask several times to Oracle: what triggers the KTSJ Slaves processes to fully read huge LOB segments? I do not get a satisfactory answer.

We do not have this problem with all other LOB segments of the DB. The workaround at the end was to defragment the LOBs, moving them to a new tablespace (which we wanted to do anyway). The LOB segment reduced from >500GB to 75G and the Space Manager processes disappeared completely. This lead to the next problem (next post).

I leave this post here just in case someone encounters the same.

Oracle Update Advisor in 19c – tutorial of the new easy way to find and download Oracle DB release updates

This entry was posted in Oracle Oracle 19c Oracle AI 26ai Patchset Update and tagged Release Updates on by

Oracle released a new version of its “Oracle Update Advisor”. This tool is part of Oracle AI Database 26ai, allows to anyone to check if their Oracle installation is up to date and download a new Gold Image updated to the latest version.

When using it with Oracle 19c, you need to download the tool “dbcactl“, which complements existing Database Creation Assistant – dbca – with the “managePatches” option – this is the Oracle Update Advisor.

To download dcbactl, you need to go to https://updates.oracle.com/download/6789999.html . While latest available version visible is 26.2.1, it is not downloadable. I did my tests using the version 25.4.1.

(more…)

How to Patch/Upgrade Java (JDK and JRE) in both Oracle Home and OEM Agent 13.5 home to latest certified version

This entry was posted in Enterprise Manager Oracle Oracle 19c Patchset Update Security on by

A vulnerability scan from the customer pointed out many problems due to old patch backups and old JDK versions installed in a Oracle VM.

Many of the problems were with the files:

/u01/app/oracle/product/oem13c/agent/agent_13.5.0.0.0/oracle_common/jdk/jre/lib/rt.jar
/u01/app/oracle/product/19.0.0/dbhome_1/jdk/jre/lib/rt.jar

What I learned:

  • OEM Agent 13.5 default java version is 1.8.0_261, also after patching to the latest RU!
  • Oracle OPatch has its own JRE and this is not updated when applying latest JDK patch for Oracle Home
  • Upgrade JDK in Oracle Home is online

What I already knew:

Below how I “cleanup” the vulnerabilities at my customer VM. This instructions are for java version 1.8.0_481, latest certified for Oracle in January 2026 :

(more…)

Follow the progress of a PDB remote cloning

This entry was posted in cloning Multitenant Oracle Oracle 19c on by

For a PDB migration, I’ve configured a Refresh PDB clone.

SQL> CREATE PLUGGABLE DATABASE P1QXPTO from P1QXPTO@C1Q_OLDCDB REFRESH MODE EVERY 2 HOURS;

The PDB is 16TB and copying will take a few hours. To get the progress, it is not so straight forward:

  • there is no size in V$PDBs until the end
  • the ASM space is reserved from the beginning
  • there are no files in V$DATAFILE or V$ASM_FILE during the copy

The only place to follow the PDB clone is using V$SQL_LONGOPS, as explained at KB135098 – How to Monitor PDB Clone / Move On ‘Create Pluggable Database’ with ‘COPY’ Clause Statement Execution.

(more…)

Classic My Oracle Support is gone 1

This entry was posted in My Oracle Support Oracle 19c on by

This morning I wanted to look for a note and to my surprise the old Metalink, then My Oracle Support portal, did not exist anymore.

All is now merged in the new “My Oracle Support” site, previously known as “My Oracle Cloud Support”.

The link to bookmark is https://support.oracle.com/support/

New DocID, now is KB

Important to notice is, The  DocID number were changed. For instance, to find the following note:
    ”How to add an Additional Listener in RAC setup (Doc ID 2456650.1)”

You need to look for the title. The document has now a new ID, starting with KB (Knowledge Base):

The links from Google are still not updated and do not work.

The solution is to search for the title in the new Oracle Support site.

Or use the MOS DocId to KB link translation I’ve compiled.

Old SRs, new SRs

The SRs from the “classic” portal are now at the end of the list as “Archived Service Requests”.

There are three headings now:

  1. These service requests need your attention
  2. Oracle is working on these service requests for you
  3. Archived Service Requests (last 2 years from old portal)

To create new SRs for On-premises, you need to search for “Oracle Database”:

Feedback

Unfortunately there is not anymore the possibility to:

  • see last searches
  • see last views documents/notes
  • favorites
  • use old DocID in the search (would be enough to keep in the text of the new note!)
  • No wget script to download patches
  • Search engine does not prioritize most viewed notes

I’ll get used to the new portal, but as everytime, changes are not easy.

Status 10.12.2025:

Extend Swap in Oracle Linux

This entry was posted in Linux Oracle 19c and tagged Oracle Linux 9 on by

The client has also configured a small Swap mountpoint, when looking at all memory available on the server. Oracle documentation says this:

The VM has 116GB memory, and 32GB reserved for HugePages. The actual swap mountpoint is 4GB big and, based on the Oracle Server Configuration Checklist for 19c, the swap mountpoint should be in this case 16GB.

I’m using Oracle Linux 9, xfs filesystems. I’ll increase the swap mountpoint online, with the databases running.

(more…)

TDE misconfigured and ORA-28353: failed to open wallet

This entry was posted in Oracle Oracle 19c Oracle 23ai Security TDE on by

I did script the TDE wallet configuration for my client.

  • Script 1 – Set the static parameters TABLESPACE_ENCRYPTION and WALLET_ROOT in the spfile.
  • Script 2 – Restart the database
  • Script 3 – Create the wallet, open the wallet, set the Masterkey, create the autologin wallet.

When running this 3rd script on a database it failed on the second command:

SQL> ADMINISTER KEY MANAGEMENT CREATE KEYSTORE IDENTIFIED BY  "wallet_password";
keystore altered.

SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY "wallet_password";
ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY "wallet_password"
*
ERROR at line 1:
ORA-28353: failed to open wallet
(more…)

Recover missing Masterkey – the famous ORA-28374 2

This entry was posted in Oracle Oracle 19c Oracle 23ai Security on by

The client plugged a non-encrypted PDB into a CDB with TDE Encryption Wallet enabled. All works fine, we can read the data. However when trying to create a tablespace we get:

CDB1 SQL> create tablespace TBS_NEW;
Error starting at line : 1 in command -
create tablespace t1
Error report -
ORA-28374: typed master key not found in wallet
28374. 0000 -  "typed master key not found in wallet"
*Cause:    You attempted to access encrypted tablespace or redo logs with a typed master key not existing in the wallet.
*Action:   Copy the correct Oracle Wallet from the instance where the tablespace was created.

What is wrong?

(more…)

Encrypt the whole Oracle database online with TDE

This entry was posted in Oracle Oracle 19c Security on by

Some programs are part of Oracle ISV – Independent Software Vendors – program and include various types of Oracle licenses. This allows to install the application database in various Oracle configurations. At my customer, the software includes Advance Security Option Oracle license. This was the reason we decided, even before moving to the cloud, to encrypt the database.

Here is a summary on how to perform full online encryption (TDE) of a database. The process is quite simple, but there are known surprises you might want to avoid.

(more…)

Oracle TDE – the basic information you need to know when encrypting the whole database

This entry was posted in Cloud Engineering Systems ExaCC Exadata OCI Oracle Oracle 19c and tagged OCI oracle 19c tde on by

TDE – Transparent Data Encryption – is the Oracle solution for protecting data at rest. This refers, protecting data that is stored in one file or one disk. This data will be encrypted. Only after the database is open and the wallet password is given, you can query the data and see it.

Recently I’ve been working more with ExaCC and migrating databases from on-premises to the Cloud. Took quite some days to get into TDE and sometimes confusing terms used in the documentation and on the web.

This post summarizes all the concepts to have in mind when working with Oracle encrypted databases, which is the default in the Cloud and Exadata systems.

(more…)