Using Enterprise Manager to gather extra DB information using Monitoring Extensions

My customer wants to get some specific information from each database on regular basis. I was sure to be able doing that with Enterprise Manager, so the information would be stored in the repository.

Here how it is done:

On OEM (version 13.5 here), as a high privileged user go to Enterprise Menu – Monitoring – Metric Extensions.

Click Create – Metric Extension.

Target Type can be “Database Instance” if you want to query on CDB$ROOT level only, or “Pluggable Database” to query on PDB level. For some reason the “Pluggable Database” did not work at a later step, so the example here will be with “Database Instance”.

On “Select Advanced Properties” you can normally choose which categories this metric should be applied: if should run on “DB Instance only”, “RAC Instance only”, “Exclude CDB”, or based on Data Guard Status. Also here, when I chose “Primary”, later in the configuration I was getting errors. So at the end I did left all unselected (mean apply on all).

Then set the Metric Extension name, which will be the value of column “metric_name” in MGMT$METRIC_CURRENT table. The Adapter for my case is “SQL”

The collection schedule should be Enable, with Data Upload, using Metric for “Alerting and Historical Trending”, which allows to have some history. As my query is light and I wanted to have results gathered quite soon, I set the “Upload Interval” to 24 and collection frequency every 30 minutes. So I could have the first results in 30 minutes, next will come after 12 hours.

On the next Screen – Adapter – I wrote the SQL Query. In my case it was:

select con_id_to_con_name(con_id) pdb_name, owner, count(*) n_obj, count(distinct tablespace_name) n_tbs, round(sum(bytes)/1024/1024) mb_used 
from cdb_segments 
group by con_id, owner;

This makes the workaround to query on cdb$root level (Database Instance) only and get the results for all PDBs.

Then one has to define the output columns. In my case I’ve two “Key Columns”, which identify the row – pdb_name and owner, both of type String. Then I’ve three “Data Column” – n_obj, n_tbs and mb_used of type Number.

The following screen defines which database credentials to be used. In my case the “Default Monitoring Credentials” which are from dbsnmp user are enough.

And before finishing, one can test the collection on a few targets, to make sure all works as expected. Then we can click “Finish”.

Now that the Metric Extension is created there we have perform several Actions (select metric and use the “Actions” button):

  • Save as Deployable Draft
  • Public Metric Extension
  • Deploy to Targets

I don’t go through it as it is quite easy to understand how it works.

At my customer they use “Monitoring Templates”, in order to have a set of metrics automatic deployed to new targets of a certain type. One needs to add the Metric Extension to the Monitorig Template. For that, we go to Enterprise Menu – Monitoring – Monitoring Templates. We select the template concerning Database Instances, click “Edit”, go to “Other Collected Items” tab and click “Add Metrics to Template”. Select “Metric Extensions/Configuration Extensions” and our created metric should appear. We select and click Continue and then OK.

Now is just matter of waiting up to 30 minutes and the result from each database will be uploaded to the OEM Repository and can be queried using the following SQL – for the example above:

select target_name, key_value as pdb_name, key_value2 as owner,
   max(case when metric_column='n_obj' then value end) as n_obj,
   max(case when metric_column='n_tbs' then value end) as n_tbs, 
   max(case when metric_column='mb_used' then value end) as mb_used     
from MGMT$METRIC_CURRENT
where metric_name ='ME$<metric_name>' 
group by target_name,key_value,key_value2
order by 1,2,3,4;

Historical details should come in MGMT$METRIC_DETAILS, using the column collection_timestamp.


New mandatory unified audit policy on 19.26

This feature was just backported from Oracle 23ai. The new ORA$MANDATORY audit policy was added with the Oracle 19.26 RU. This policy is not visible at UNIFIED_AUDIT_POLICIES or AUDIT_UNIFIED_ENABLED_POLICIES.

After patching the database to 19.26, then you see entries on UNIFIED_AUDIT_TRAIL:

SYS@CDB2.CDB$ROOT> select EVENT_TIMESTMAP, SYSTEM_PRIVILEGE_USED, ACTION_NAME 
from UNIFIED_AUDIT_TRAIL 
where UNIFIED_AUDIT_POLICIES='ORA$MANDATORY' 
order by EVENT_TIMESTMAP;

                  EVENT_TIMESTAMP     SYSTEM_PRIVILEGE_USED       ACTION_NAME
_________________________________ _________________________ _________________
02-FEB-2025 21:54:56.192982000    SYSDBA                    LOGON
02-FEB-2025 21:54:56.216549000    SYSDBA                    SELECT
02-FEB-2025 21:55:00.381577000    SYSDBA, ALTER DATABASE    ALTER DATABASE
02-FEB-2025 21:55:00.393882000    SYSDBA                    LOGOFF
...

The actions that are audited by ORA$MANDATORY policy are described on Oracle 23ai documentation.

What I find interesting, is that the “ALTER DATABASE MOUNT” during startup is audited, so we can have a good history of database startups.

(more…)

How to change Goldengate Adminclient default editor permanently

Goldengate Microservices architecture replaced the “ggsci” tool with “adminclient”. This new client has few limitations and does not work well with “rlwrap” – my favorite tool to have history between sessions.

The Adminclient provides some options you can easily change after starting the tool:

$ ./adminclient
...
OGG (not connected) 1> show
Current directory: /home/oracle
COLOR            : OFF
DEBUG            : OFF
EDITOR           : vi
PAGER            : more
VERBOSE          : OFF

OGG (not connected) 2> set color ON
OGG (not connected) 3> set pager less

OGG (not connected) 4> show

Current directory: /home/oracle
COLOR            : ON
DEBUG            : OFF
EDITOR           : vi
PAGER            : less
VERBOSE          : OFF

However to keep the settings across sessions it is not very straight forward. The way to do it is to set variables:

$ export ADMINCLIENT_COLOR=ON  # ON, OFF in uppercase!

And ADMINCLIENT_DEBUG and ADMINCLIENT_VERBOSE for DEBUG and VERBOSE respectively

For the editor and pager, the variables are simply:

export EDITOR=nano
export PAGER=less

Attention that the variable EDITOR is used also by other clients, like sqlplus “edit” command.

So the way I do it, is to set everything within an alias:

alias gg="cd $OGG_HOME/bin; EDITOR=nano PAGER=less ADMINCLIENT_COLOR=ON $RLWRAP ./adminclient ; cd -"

And inside .bash_profile or something that sets the environment:

RLWRAP="$(command -v rlwrap)" && RLWRAP="${RLWRAP} -c"

export OGG_HOME="/u00/app/oracle/product/ogg/21.15"
alias gg="cd $OGG_HOME/bin; EDITOR=nano PAGER=less ADMINCLIENT_COLOR=ON $RLWRAP ./adminclient ; cd -" 


Slow starting impdp from NFS Share

Unfortunately I lost the logs for this issue, but I try to document for information.

My customer has ExaCC with various 2-node clusters.

  • Export ACFS mount point as NFS from cluster1
  • Mount NFS mount point on cluster2, cluster3 and cluster4

He did an export from cluster1 to the ACFS mount point.

All was working fine until mid December, when impdp reading a dumpfile from the NFS mount point seems hanging when was called from cluster3 and cluster4. From cluster2 it was still fine.

Few days later, the impdp was slow everywhere, except locally on cluster1.

The behavior was very bizarre:

  • impdp starting showing timestamp
  • exactly 5 minutes later first output comes “W-1 Startup took 1 second”
  • exactly 5 minutes after comes second line “W-1 Master table … successfully loaded/unloaded”
  • and 5 minutes later runs the rest, quickly.

The NFS mount point seemed ok, ‘dd’ command tests did not show any slowness.

I started to investigating by enabling the DataPump tracing, as explained by Daniel Hansen on his Databases are Fun blog:

alter system set events 'sql_trace {process: pname = dw | process: pname = dm} level=8';

The trace files generated on Diagnostics directory did not help much – they are mostly for performance problems.

Then I did start a “strace” on the PID of the impdp

strace -p <pid> -o /tmp/strace.out

There I could see some “ECONNREFUSED” to one of the IPs of the Cluster1. But few lines above, there was the same connection without error.

Quite strange. Finally with the help of one system administrator, we found out that the nfs-server was not running on one of the cluster1 nodes. And the NFS mount point was using a hostname which dynamically would go either to one or another node of the cluster1. After making sure nfs-server was running on both nodes from cluster1, the problem was solved and impdp was fast to start again.

Learnings:

  • Use the clusterware to manage exportfs – srvctl add exportfs
  • Make use of VIPs which move from one node to another instead of round-robin DNS entries.


ORA-64307 when creating compressed table and /home

My customer running on ExaCC (Exadata Cloud@Customer) was getting “ORA-64307: Exadata Hybrid Columnar Compression is not supported for tablespaces on this storage type” on one of his test databases.

I did test connecting to SYS and no problem. Then I try to do using his tablespace and indeed, I get the error:

Quite going around, to check what was different on the user tablespace than on others. I test a self created tablespace and it works.

Strange. Until I found that… some datafiles were not in ASM!

Seems the ASM Diskgroup is almost full and the client DBA just put the datafiles somewhere else!


JumpHost Matryoshka

My client just added an extra jumphost before arriving to the server. So now I’ve to connect, to connect, to connect and then open the connection. 🙂


Warning: OPatchauto ignores disabled components – possible licensing issues

Since many years at my customer I’m using “opatchauto” to perform a out-of-place patching of Oracle Restart (GI+RDBMS).

My customer is concerned about database users using not licensed options, like partitioning. To avoid it, at the installation time the partitioning option is disabled using chopt, like described at Doc ID 948061.1.

Today during a check we noticed that Partitioning option was activated everywhere, which is not the client standard! We found out the origin of the problem was the out-of-place patching with “opatchauto”.

The big advantage of using “opatchauto” is that it allows easily either a single-step or a two-step Out-of-Place patching. We just write in a properties file the name of the new Oracle Homes and it does:

  • Clone current GI + RDBMS homes to new Homes (prepare clone)
  • Patches the new homes (prepare clone)
  • Stops GI and DBs (switch clone)
  • Switches GI and DBs from current homes to new Homes (switch clone)
  • Restart everything (switch clone)
  • Runs Datapatch on DBs if not standby (switch clone)

This allows to decrease the patching downtime without RAC to about 10 minutes, with the two-step (prepare clone + switch clone) operation.

Here the steps to reproduce de bug:

(more…)

Solve “OGG-08224 Error: CONTAINER option was specified though the database does not support containers” error

Quick post to add info about the following Goldengate error:

OGG (http://localhost:9300 test1 as ogg_pdb1@CDB2) 10> REGISTER EXTRACT E_TEST1 DATABASE CONTAINER (pdb1)

2024-12-08T17:16:58Z ERROR OGG-08224 Error: CONTAINER option was specified though the database does not support containers.

This means that you are connected directly to the PDB, and not to CDB$ROOT.

To register Goldengate 21 extracts you need to connect to the Root container with a common user.

OGG (http://localhost:9300 test1 as ogg_pdb1@CDB2) 12> DBLOGIN USERIDALIAS ogg_cdb2
Successfully logged into database CDB$ROOT.

OGG (http://localhost:9300 test1 as ogg_cdb2@CDB2/CDB$ROOT) 13> REGISTER EXTRACT E_TEST1 DATABASE CONTAINER (pdb1)
2024-12-08T17:20:36Z  INFO    OGG-02003  Extract group E_TEST1 successfully registered with database at SCN 8039188.

Well, in the future this is not anymore true, as new version from Goldengate and DBs will work only at PDB level.


Using AI to confirm a wrongly cabled Exadata switch – or how to fix verify_roce_cables.py script for Python3.

One of the preparation steps when installing an Exadata X10M is to verify that the cabling of the RoCE switches is correctly done. The next step is to upgrade the Cisco switches with the latest firmware. During my intervention for Tradeware at the customer, the first didn’t work as the provided script is not compatible with Python3 and the latter complained about wrong cabling.

Here I show how studied the wrong cabling of the X10M switches and how I use Claude.ai (ChatGPT and other AI tools probably also work) to quickly fix the Python script provided by Oracle.

(more…)

Oracle postpones release of 23ai on-premises to 2H CY2024

Oracle just updated the Release Schedule of Current Database Releases (Doc ID 742060.1) and changed the release date of database version 23ai on-premises to next half-year. Lets see how many months and bug fixing that means. 🙂

Update on 20.06.2024 – “Added new release dates for Oracle Autonomous Database – Dedicated Exadata Infrastructure, Autonomous Database on Exadata Cloud@Customer, ODA, Exadata and Linux-x86 64”