Miguel Anjo


On DB 19.31, ORA-600 [kaf lob buffer overwrite]   Recently updated !

After Oracle support did not accepted that there is bug with huge LOB segments and High IO due space manager KTSJ, this time it worked faster.

The move of the Table and LOB to a new Tablespace went ok, but soon we started seeing ORA-600 [kaf lob buffer overwrite] on the alertlog. These happened either from the application or also from DBMS_STATS daily process.

The culprit seems to be to parallel access to the segment. First I noticed the table had degree 4. After setting it back to 1, there were no more problems from the application, but only from DBMS_STATS – which is configured to run parallel. However setting it to run without parallel, does not help.

After sending all logs and a 10046 trace to Oracle, an automatic answer come:

Oracle has matched the symptom ‘ORA-600 [kaf lob buffer overwrite]’ from trace file xxxxx_p000_459134_i152080.trc in SR for version 19.31.0.0.0 to BUG 39180642 with a high level of confidence.

BUG 39180642 (unpublished) is still under investigation by Oracle Development. Update this SR if you wish to be kept informed of the progress of the investigation.

The error seems similar to: “KB348966 – Automatic Stats collection Task failing with ORA-20011: Approximate NDV failed: ORA-00600: internal error code, arguments: [kafger buffer overwrite]”, where the issue is some kind of corruption or invalid data.

Let’s see what Oracle says.


High IO due space manager (W00x process) doing db sequential read all the time – KTSJ Slave bug?

On a client DB, after migrating it to OL8, the application started to complaint the DB seems slower. Checking the OEM, there is not much done by the application, but I see a good blue stripe from W00x processes doing sequential reads. Nonstop. This is a 19.31 single-instance database.

After tracing these sessions, they are reading two distinct LOB segments, each between 500 and 700GB, where a lot of deletes happened. All the time “db file sequential read”, up to 3 or 4 W00x parallel processes.

WAIT #0: nam='db file sequential read' ela= 199 file#=476 block#=2460662 blocks=1 obj#=53176 tim=1024371028390

I open an SR with Oracle. They ask more traces and session short stack. The short_stack looks like:

SQL> oradebug setospid 15297
Oracle pid: 28, Unix process pid: 15297, image: oracle@srv0nnnn.xxx.ch (W000)
SQL> oradebug unlimit
Statement processed.
SQL>  oradebug short_stack

ksedsts()+426<-ksdxfstk()+63<-ksdxcb()+872<-sspuser()+236<-__sighandler()<-semtimedop()+14<-sskgpwwait()+245<-skgpwwait()+187<-ksliwat()+2296<-kslwaitctx()+188<-ktsj_slave_main()+944<-ksvrdp_int()+1906<-opirip()+579<-opidrv()+581<-sou2o()+165<-opimai_real()+173<-ssthrdmain()+417<-main()+256<-__libc_start_main()+229<-_start()+46

No underscore parameters are set and changing “_enable_space_preallocation” to 0 has no immediate effect.

The tablespace is composed of several smallfiles, EXTENT_MANAGEMENT=LOCAL, ALLOCATION_TYPE=SYSTEM, SEGMENT_SPACE_MANAGEMENT=AUTO. All standard.

The LOB segment is securefile, configured in-row, with several larger values which go out of row.

I ask several times to Oracle: what triggers the KTSJ Slaves processes to fully read huge LOB segments? I do not get a satisfactory answer.

We do not have this problem with all other LOB segments of the DB. The workaround at the end was to defragment the LOBs, moving them to a new tablespace (which we wanted to do anyway). The LOB segment reduced from >500GB to 75G and the Space Manager processes disappeared completely. This lead to the next problem (next post).

I leave this post here just in case someone encounters the same.


Database specific password wallet with 26ai client: the new SEPS_WALLET_LOCATION parameter

One inconvenient I had in the past with the usage of password wallets for Oracle users was their maintenance. With the years, there was just a lot of old entries. At the end you would do a script that would test all credentials and delete the ones which did not work or there was no more network alias equivalent.

In OCI, to connect to Autonomous Databases, there is since few years a possibility to have a wallet parameter defined at the network connection or tnsnames, using “(SECURITY=(wallet_location=/home/oracle/wallets/databases)))” – sometimes also seen using “my_wallet_directory”. But this only works for TLS connections and using certificates.

At my client I wanted to have a wallet per database, so that at the time of decommissioning, we just delete the whole wallet.

After looking around, I finally found out that using a 26ai client there are new parameters for the connection string and one of them is SEPS_WALLET_LOCATION, which allows to specify the location of the wallet for a specific entry.

This way I can do:

# My DB is ANJO_DB
$ echo $ORACLE_SID
ANJO_DB

# Create wallet
orapki wallet create -wallet $ORACLE_BASE/admin/$ORACLE_SID/wallet -pwd <wallet_pwd> -auto_login_local

# Add credential
orapki secretstore create_credential -wallet $ORACLE_BASE/admin/$ORACLE_SID/wallet -pwd <wallet_pwd> -connect_string $ORACLE_SID -username sys -password <sys_pwd>

# Add the wallet location in tnsnames
$ grep $ORACLE_SID $TNS_ADMIN/tnsnames.ora
ANJO_DB = (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=anjovm1)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=ANJO_DB.WSL.HOME))(security=(SEPS_WALLET_LOCATION=/u00/app/oracle/admin/ANJO_DB/wallet)))

# Check with tnsping
tnsping ANJO_DB

TNS Ping Utility for Linux: Version 23.26.2.0.0 - Production on 30-JUN-2026 14:10:10

Copyright (c) 1997, 2026, Oracle.  All rights reserved.

Used parameter files:
/u00/app/oracle/network/admin/sqlnet.ora

Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=anjovm1)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=ANJO_DB.WSL.HOME))(security=(SEPS_WALLET_LOCATION=/u00/app/oracle/admin/ANJO_DB/wallet)))
OK (10 msec)

# Connect
$ sqlplus /@ANJO_DB
SQL*Plus: Release 23.26.2.0.0 - Production on Tue Jun 30 14:11:40 2026
Version 23.26.2.0.0

Copyright (c) 1982, 2026, Oracle.  All rights reserved.

Last Successful login time: Tue Jun 30 2026 13:52:44 +02:00

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.30.0.0.0

SQL>

The documentation about this and other new elements of the tnsnames.ora file in Oracle 26ai are at https://docs.oracle.com/en/database/oracle/oracle-database/26/netrf/local-naming-parameters-in-tns-ora-file.html


ASM diskgroup space calculator for ExaCC

In ExaCC and probably on Exadata in OCI it is sometimes a bit complex to calculate how much space you need to add to the ASM, as you can only change the total space and the distribution.

When you just want to add 10TB (or GB) space to DATA diskgroup, you need to play a while or do an excel.

With the calculator below, you can now do it easily:

Edit any cell — usable, raw (×3), % of total, or the total row itself. Other values update accordingly.

Lock Disk group Usable space Raw space (×3) % of total
DATA
TB TB %
RECO
TB TB %
SPARSE
TB TB %
Total
TB TB 100%

When looking inside asmcmd, you see the “Raw space”. This is how much from disk space is used. The Usable space is what the users see and what you need to consider.


How to create an audit policy based on user profile? Example using a context

The customer wanted to have some audit policies that would be applied only to users who have a certain profile – lets say the profile APP_PROFILE.

Using unified auditing, it is possible to enable a policy for certain roles using AUDIT POLICY policy_name BY USERS WITH GRANTED ROLES role_name;

And you can use context on audit policies using CREATE AUDIT POLICY policy_name ACTIONS <action_list> WHEN q'!SYS_CONTEXT('<context_name>','<context_attr>') = <value>!' EVALUATE BY SESSION;

The user profile is unfortunately not automatically available in the default USERENV context.

The way to have audit policy on users with a certain profile, needs to go via a logon trigger, which either sets a ROLE to a user, or sets a CONTEXT.

I’ve decided on the latter option. For this I need few steps – this is done inside the PDB:

1. Create a context – AUDIT_CONTEXT -, which must say what program is allowed to set it

CREATE OR REPLACE CONTEXT audit_context USING audsys.proc_set_audit_flag;

2. Create the procedure referenced above – AUDSYS.PROC_SET_AUDIT_FLAG, that sets the context

CREATE OR REPLACE PROCEDURE audsys.proc_set_audit_flag IS
    v_profile VARCHAR2(128);
  BEGIN
    SELECT profile INTO v_profile
      FROM dba_users
     WHERE username = SYS_CONTEXT('USERENV', 'SESSION_USER');

    IF v_profile = 'APP_PROFILE' THEN
    dbms_output.put_line('Set context AUDIT Y');
      DBMS_SESSION.SET_CONTEXT('AUDIT_CONTEXT', 'AUDIT', 'Y');
    ELSE
      dbms_output.put_line('Set context AUDIT N');
      DBMS_SESSION.SET_CONTEXT('AUDIT_CONTEXT', 'AUDIT', 'N');
    END IF;
  EXCEPTION
    WHEN OTHERS THEN
      DBMS_SESSION.SET_CONTEXT('AUDIT_CONTEXT', 'AUDIT', 'N');
END proc_set_audit_flag;
/

3. Grant access to the procedure

The procedure will be called by low privilege users. So the procedure needs access to DBA_USERS, this is done via an intermediary role ROLE_SET_AUDIT_FLAG; and any user needs to be able to execute the procedure, so it needs to be available to PUBLIC.

CREATE ROLE IF NOT EXISTS role_set_audit_flag;
GRANT select ON dba_users TO role_set_audit_flag;
GRANT role_set_audit_flag TO PROCEDURE audsys.proc_set_audit_flag;
GRANT execute ON audsys.proc_set_audit_flag TO public;

4. Create a logon trigger – TRIGGER_SET_AUDIT_CONTEXT – calling this procedure

CREATE OR REPLACE TRIGGER trigger_set_audit_context
  AFTER LOGON ON DATABASE
BEGIN
  audsys.proc_set_audit_flag;
EXCEPTION
  WHEN OTHERS THEN NULL;
END;
/

5. Use this construct within an audit policy – AUDIT_APP_USERS

CREATE AUDIT POLICY audit_app_users 
ACTIONS PRIVILEGES CREATE ANY TABLE, DROP ANY TABLE 
WHEN q'!SYS_CONTEXT('AUDIT_CONTEXT', 'AUDIT') = 'Y'!' EVALUATE PER SESSION;

AUDIT POLICY audit_app_users;

Conclusion

Now, when I connect with a user having the APP_USER profile, I can see I’ve the right context

MIGUEL@ANJO.PDB1> SELECT * FROM SESSION_CONTEXT;

       NAMESPACE    ATTRIBUTE    VALUE
________________ ____________ ________
AUDIT_CONTEXT      AUDIT      Y

1 row selected.

And on the audit trail we can now see the entries from that policy:

SYS@ANJO.PDB1> select  EVENT_TIMESTAMP, DBUSERNAME, UNIFIED_AUDIT_POLICIES from audsys.unified_audit_trail order by 1 desc fetch first 1 rows only;

                  EVENT_TIMESTAMP    DBUSERNAME            UNIFIED_AUDIT_POLICIES                                                                            
_________________________________ _____________ _________________________ 
29-APR-2026 16:03:46.841520000    MIGUEL           AUDIT_APP_USERS 

Oracle Update Advisor in 19c – tutorial of the new easy way to find and download Oracle DB release updates

Oracle released a new version of its “Oracle Update Advisor”. This tool is part of Oracle AI Database 26ai, allows to anyone to check if their Oracle installation is up to date and download a new Gold Image updated to the latest version.

When using it with Oracle 19c, you need to download the tool “dbcactl“, which complements existing Database Creation Assistant – dbca – with the “managePatches” option – this is the Oracle Update Advisor.

To download dcbactl, you need to go to https://updates.oracle.com/download/6789999.html . While latest available version visible is 26.2.1, it is not downloadable. I did my tests using the version 25.4.1.

(more…)

Alter system kill session “force timeout 0” ? new form of “kill immediate”?

The other day I saw that besides the catcon.pl, there is a catcon_kill_sess_gen.sql:

ls -1 $ORACLE_HOME/rdbms/admin/catcon*
/u00/app/oracle/product/19.30.0/rdbms/admin/catcon_kill_sess_gen.sql
/u00/app/oracle/product/19.30.0/rdbms/admin/catcon.pl
/u00/app/oracle/product/19.30.0/rdbms/admin/catcon.pm
/u00/app/oracle/product/19.30.0/rdbms/admin/catconst.pm
/u00/app/oracle/product/19.30.0/rdbms/admin/catcont.sql

When looking at this file, I noticed it kills catcon session using a “ALTER SYSTEM KILL SESSION ‘sid, serial#’ FORCE TIMEOUT 0”, except if on Oracle 12.1, where still does “IMMEDIATE” instead:

This new syntax “FORCE TIMEOUT x” is first described in Oracle AI 26ai DB documentation (but it seems to work from version 12.2). Basically it theoretically allows the draining of sessions by setting a timeout until the session terminates some task, before killing it.

There are two new elements here: “FORCE” and “TIMEOUT x”.

(more…)

SQL and how Oracle filter the resultset

While reading the very interesting Oracle AI Database 26ai New Features Guide, I got to know that there is a new “QUALIFY” filter clause (there is also a new “FILTER” clause). However, what I found more interesting in the documentation was this sentence:

The order of operations is as follows: FROM → WHERE → GROUP BY → HAVING → WINDOW → QUALIFY → DISTINCT → ORDER BY → FETCH FIRST.

This is somehow a back to the roots and basic SQL information that I had in the back of my head, but never though really end-to-end about.

Here an example of query using all except DISTINCT elements of the operations order:

SELECT region, city, AVG(temperature), AVG(AVG(temperature)) OVER w AS avg_temp_region 
    FROM city, regions 
    WHERE city.region_id = region.region_id 
    GROUP BY region, city 
    HAVING region NOT LIKE 'LISBON' 
    WINDOW w AS (PARTITION BY region)
    QUALIFY avg_temp_region > 10
    ORDER BY region
    FETCH FIRST 5 ROWS ONLY;

I don’t think I could write a SQL like that, but at least now I know how it looks like.


How to Patch/Upgrade Java (JDK and JRE) in both Oracle Home and OEM Agent 13.5 home to latest certified version

A vulnerability scan from the customer pointed out many problems due to old patch backups and old JDK versions installed in a Oracle VM.

Many of the problems were with the files:

/u01/app/oracle/product/oem13c/agent/agent_13.5.0.0.0/oracle_common/jdk/jre/lib/rt.jar
/u01/app/oracle/product/19.0.0/dbhome_1/jdk/jre/lib/rt.jar

What I learned:

  • OEM Agent 13.5 default java version is 1.8.0_261, also after patching to the latest RU!
  • Oracle OPatch has its own JRE and this is not updated when applying latest JDK patch for Oracle Home
  • Upgrade JDK in Oracle Home is online

What I already knew:

Below how I “cleanup” the vulnerabilities at my customer VM. This instructions are for java version 1.8.0_481, latest certified for Oracle in January 2026 :

(more…)

New My Oracle Support – updates and issues 1

Oracle is taking seriously the confusion created with the abrupt migration to the new My Oracle Support (MOS) portal. Today there was a meeting between several VPs and other responsible and the Oracle ACE community. Thanks a lot!

We see many things are being improved and solved and the team is working hard to get the trust back from the community. We are here to help!

My top issues today (15.Dec.2025) with the new MOS are here:

Content related issues

  1. A list with “Last viewed Docs” + “Last searches” sections would very useful to find again documents. The list should include the last updated date in a column
  2. In “My Favorites” section, it is important to see the “last updated date” as a column. Old DocId may be removed. Folders/Categories were useful
  3. The “Favorites” were the base for the “Oracle Support HOT Topics” daily email, and this was where users can get information if any of our favorites was updated. This email seems not to exist anymore.
  4. Bugs are not part of the search – it is useful to know the bug exists, even if it is marked “not public”. This was the case in old MOS;
  5. Missing wget script to download patch – this was important, as usually we do not have browsers where we need the patch;
  6. SRs in “Pending Final Closure” do not appear any where (neither open, nor close). Should be in “need your attention” part. (Example SR 4-0001431240)
  7. There are still too many “Request Header Or Cookie Too Large” errors!
  8. Login timeout even when actively using the portal!
  9. Search results do not prioritize recent and most viewer content (example, search for “CPU October 2025” – it does not give KB795208 which is a top50 document)
  10. Internal links are not working (low priority)

UX related issues

  1. Choose sections to have in front page
    • My Favorites + My last viewed searches are important for me in first page
    • Archived Service Requests are not important to me in first page
  2. The Search box should be visible on every page, without need to return to the Home!
  3. Some text overflows the defined column and cannot be read (see KI35841)
  4. Instead of having two columns (left column with Title, DocID, Last Updated, Service | right with Doc text)
    • better would be two cells, top cell with content of the left column, followed by bottom cell with text
    • both cells would have 1296px width, which would give more place for tables (and the document title in top)

E-Mail related issues

  1. SR update emails do not have the “Title” of SR neither information the “Tenancy” the SR relates. As consultant I’ve about 50 CSI/Tenancies and this is important to prioritize my work
  2. SR Creation and Update emails do not have the information about the “Tenancy”.

I’m sending this information also by email to my contacts at MOS, hoping it helps prioritizing the issues.