HOW TO ADD new / redundant private interconnect to a Oracle RAC Cluster

Starting 11gR2, Oracle supports up to 4 redundant interconnects that will be automatically managed by the cluster for fail-over and load balancing.
Following procedure shows how to add a new private interconnect to an existing RAC cluster.
Version tested: 11.2.0.3.9/12.1.0.2

Check the current HAIP in the ASM instance:
select name,ip_address from v$cluster_interconnects;
select inst_id, name,ip_address from gv$cluster_interconnects order by 1;

•  Before making this change, check on all nodes if any resource is OFFLINE.
•  All the CRS resources should be online.
•  In one case, we had ADVM (12c) STABLE and we got errors:

• Check to see what private interfaces are already registered

xxxx681:+ASM1::/opt/app/oragrid/product/12.1.0>> oifcfg getif
bond0.3267  172.29.70.0  global  cluster_interconnect  => this is private
bond0.659  135.213.168.0  global  public               => this is public

So we only have one private interface in the cluster, We are now going to add
second one, which is bond0.3268 which has a separate subnet.

** IMPORTANT: Make sure your second/redundant Private has DIFFERENT SUBNET

• List all the interfaces on the server
• Make sure that the new private interface that you are going to add is correctly plumbed
• On all other nodes - it has to be on the same interface name ( bond0.3268, in our case)

xxxx681:+ASM1::/opt/app/oragrid/product/12.1.0>>  oifcfg iflist
bond1  172.29.72.0
bond0.3267  172.29.70.0
bond0.3267  169.254.0.0
bond0.3268  172.29.71.0
bond0.629  10.119.0.0
bond0.630  10.119.8.0
bond0.659  135.213.168.0

Get the SubNet:

xxxx681:+ASM1::/opt/app/oragrid/product/12.1.0>>
Fully-Qualified-Domain-Name           IP-Address        Interface   Source        Subnet
------------------------------------  ----------------  ----------  ------------  ----------------

xxxx681-priv1.ffdc.sbc.com            172.29.70.1       bond0.3267  /etc/hosts    172.29.70.0
xxxx681-priv2.ffdc.sbc.com            172.29.71.1       bond0.3268  /etc/hosts    172.29.71.0
xxxx681-priv3.dadc.sbc.com            172.29.72.1       bond1       /etc/hosts    172.29.72.0

Add the redundant private network to Cluster
xxxx681:+ASM1::/opt/app/oragrid/product/12.1.0>> oifcfg setif -global bond0.3268/172.29.71.0:cluster_interconnect
xxxx681:+ASM1::/opt/app/oragrid/product/12.1.0>>

=> If you encounter errors like:
PRIF-33: Failed to set or delete interface because hosts could not be discovered
  CRS-02307: No GPnP services on requested remote hosts.
PRIF-32: Error in checking for profile availability for host xxxx687
  CRS-02306: GPnP service on host "xxxx687" not found.
PRIF-32: Error in checking for profile availability for host xxxx688
  CRS-02306: GPnP service on host "xxxx688" not found.

In some cases, just kill/restart of gpnpd daemon does not solve the problem,
you will need to restart the init resource "ora.mdnsd"
It is safe to kill the gpnpd daemon, as CRS will spawn another gpnpd daemon if the one running terminates.

crsctl stop res ora.mdnsd -init
crsctl start res ora.mdnsd -init
kill gpnpd process:
ps -eaf | grep gpnpd
kill -9 <gpnpd_pid>

Check/Validation

xxxx681:+ASM1::/opt/app/oragrid/product/12.1.0>> oifcfg getif
bond0.3267  172.29.70.0  global  cluster_interconnect
bond0.659  135.213.168.0  global  public
bond0.3268  172.29.71.0  global  cluster_interconnect   => new redundant private is now added

xxxx681:+ASM1::/opt/app/oragrid/product/12.1.0>>

Contributor: Jagadish B