11gR2 RAC VIP and SCAN addresses not reachable








Restart a node in a 11.2.0.3 (GI and DB both) cluster and the node vip is not reachable from the external network, restart the other nodes too and now port 1521 on all the cluster vip addresses and scan addresses is not reachable.

This is a rather serious issue even in a dev cluster and all the cluster services are actually up and running including SCAN vip addresses, SCAN listeners, local listeners etc.

However the port 1521 on the node vip addresses is reachable from the other nodes of the cluster or from the same subnet to be more specific.

This took a while to troubleshoot involving security and sysadmin teams and finally the issue was due to a weird mac address entry at the firewall arp cache and clearing the cache resolved it and the root cause is in the Oracle clusterware due to a strange Oracle bug 13440962 which says





"After upgrading to 11.2.0.3, after vip failover, the ip address is
not pingable from a different subnet on Linux."

And the fix is to run the below:
After vip failover, run command

/sbin/arping -U -c 3 -I  public NIC for vip   vip ip address

to update the ARP table of router.



There's a one-off patch 13440962 available for Linux x86-64 platform but not sure if this is included in any recent PSU's.




sqlplus hangs....


Came across this bug today where sqlplus hangs while everything else on this old Linux 3 vm (an app server) appears to be running. Could this be due to some missing libraries.... or a memory leak...load appears normal.

strace shows infinite 'times(null)' system calls and it turns out to be a bug in the Oracle client oci in 10.2.0.1 and apparently this bug# 4612267 presents itself if the server uptime is more than 248 days and in this case, the uptime is exactly 249 days.

This is a legacy system so it's untouched for a long time, a good reminder to upgrade and patch systems regularly.

Finally, a simple restart of the app server fixed the issue and gave us another 248 days to think about the patch ;-)