Restart a node in a 11.2.0.3 (GI and DB both) cluster and the node vip is not reachable from the external network, restart the other nodes too and now port 1521 on all the cluster vip addresses and scan addresses is not reachable.
This is a rather serious issue even in a dev cluster and all the cluster services are actually up and running including SCAN vip addresses, SCAN listeners, local listeners etc.
However the port 1521 on the node vip addresses is reachable from the other nodes of the cluster or from the same subnet to be more specific.
This took a while to troubleshoot involving security and sysadmin teams and finally the issue was due to a weird mac address entry at the firewall arp cache and clearing the cache resolved it and the root cause is in the Oracle clusterware due to a strange Oracle bug 13440962 which says
"After upgrading to 11.2.0.3, after vip failover, the ip address is
not pingable from a different subnet on Linux."
And the fix is to run the below:
After vip failover, run command
/sbin/arping -U -c 3 -I
to update the ARP table of router.
There's a one-off patch 13440962 available for Linux x86-64 platform but not sure if this is included in any recent PSU's.
7 comments:
Thanks for posting this!!! After much hair pulling we had the same issue...
top man!
thanks, good to know that this info was of some help.
1000 1000000 1000000000 1000000000000 Thanks for you
you saved my upgrade project , as I faced this bug on first reboot after I finished 12 hours of migration efforts ........
Really Thank YOU <3
Tons of thanks , long debate with Network team resolved with this post.
Regards
Riyaz
Thanks Naji and Riyaz, good to hear that this post has helped you guys.
Your Post helped us to solve an important incident and still helping us. Thank you very much.
What's the PUBLIC NIC of VIP?
Post a Comment