Slow starting impdp from NFS Share
Unfortunately I lost the logs for this issue, but I try to document for information.
My customer has ExaCC with various 2-node clusters.
- Export ACFS mount point as NFS from cluster1
- Mount NFS mount point on cluster2, cluster3 and cluster4
He did an export from cluster1 to the ACFS mount point.
All was working fine until mid December, when impdp reading a dumpfile from the NFS mount point seems hanging when was called from cluster3 and cluster4. From cluster2 it was still fine.
Few days later, the impdp was slow everywhere, except locally on cluster1.
The behavior was very bizarre:
- impdp starting showing timestamp
- exactly 5 minutes later first output comes “W-1 Startup took 1 second”
- exactly 5 minutes after comes second line “W-1 Master table … successfully loaded/unloaded”
- and 5 minutes later runs the rest, quickly.
The NFS mount point seemed ok, ‘dd’ command tests did not show any slowness.
I started to investigating by enabling the DataPump tracing, as explained by Daniel Hansen on his Databases are Fun blog:
alter system set events 'sql_trace {process: pname = dw | process: pname = dm} level=8';
The trace files generated on Diagnostics directory did not help much – they are mostly for performance problems.
Then I did start a “strace” on the PID of the impdp
strace -p <pid> -o /tmp/strace.out
There I could see some “ECONNREFUSED” to one of the IPs of the Cluster1. But few lines above, there was the same connection without error.
Quite strange. Finally with the help of one system administrator, we found out that the nfs-server was not running on one of the cluster1 nodes. And the NFS mount point was using a hostname which dynamically would go either to one or another node of the cluster1. After making sure nfs-server was running on both nodes from cluster1, the problem was solved and impdp was fast to start again.
Learnings:
- Use the clusterware to manage exportfs – srvctl add exportfs
- Make use of VIPs which move from one node to another instead of round-robin DNS entries.
