This is 3 node production cluster ( multi subnet cluster). It was working fine. I evicted the DR node and tried to add it back, it throws an error.
I am having issues only in adding the DR node. However I can able to add the DR node to my non-prod or Dev existing cluster.
FailoverCluster EventViewer
Log Name: Microsoft-Windows-FailoverClustering/Operational
Source: Microsoft-Windows-FailoverClustering
Date: 5/09/2019 3:39:06 PM
Event ID: 1281
Task Category: Security Manager
Level: Information
Keywords:
User: SYSTEM
Computer: DRNODE.orionhealth.saas
Description:
Joiner tried to Create Security Context using Package='Kerberos/NTLM' with Context Requirement ='0' and Timeout ='40000' for the target = 'akl-shrd-pdb1'
Log Name: Microsoft-Windows-FailoverClustering/Operational
Source: Microsoft-Windows-FailoverClustering
Date: 5/09/2019 3:38:53 PM
Event ID: 1650
Task Category: Cluster Virtual Adapter
Level: Information
Keywords:
User: SYSTEM
Computer: DRNODE.orionhealth.saas
Description:
Microsoft Failover Cluster Virtual Adapter (NetFT) has missed more than 40 percent of consecutive heartbeats.
Local endpoint: 10.13.6.200:~3343~
Remote endpoint: 10.10.6.190:~3343~
Error in powershell:-
The clustered role was not successfully created. For more information view the report file below.
Report file location: C:\Windows\cluster\Reports\Add Node Wizard 76cd451a-538a-4fbe-9c52-2f9498396d17 on 2019.09.05 At 15.35.42.htm
Add-ClusterNode : An error occurred while performing the operation.
An error occurred while adding nodes to the cluster 'CLUST'.
An error occurred while adding node 'NODE3' to cluster 'CLUST'.
This operation returned because the timeout period expired
DR Cluster log:
00000918.00001528::2019/09/05-00:59:08.498 DBG [NETFTAPI] Signaled NetftLocalConnect event for fe80::14:a91:8f79:5d8f
00000918.00001528::2019/09/05-00:59:08.498 DBG [NETFTEVM] FTI NetFT event handler got event: Local endpoint fe80::14:a91:8f79:5d8f:~0~ connected
00000918.00001230::2019/09/05-00:59:08.498 DBG [NETFTEVM] FTI NetFT event dispatcher pushing event: Local endpoint fe80::14:a91:8f79:5d8f:~0~ connected
00000918.00001230::2019/09/05-00:59:08.498 DBG [FTI][Initiator] Got Netft event Local endpoint fe80::14:a91:8f79:5d8f:~0~ connected
00000918.00001528::2019/09/05-00:59:08.498 DBG [NETFTEVM] TM NetFT event handler got event: Local endpoint fe80::14:a91:8f79:5d8f:~0~ connected
00000918.0000096c::2019/09/05-00:59:08.498 DBG [NETFTEVM] TM NetFT event dispatcher pushing event: Local endpoint fe80::14:a91:8f79:5d8f:~0~ connected
00000918.0000096c::2019/09/05-00:59:08.498 INFO [IM] got event: Local endpoint fe80::14:a91:8f79:5d8f:~0~ connected
00000918.00001528::2019/09/05-00:59:08.498 DBG [WM] Filtering event NETFT_LOCAL_CONNECT? 1
00000918.0000155c::2019/09/05-00:59:08.503 INFO [NODE] Node 1: New join with n4: stage: 'Send Current Membership Status for Join Policy'
00000918.0000155c::2019/09/05-00:59:08.503 INFO [MM] Node 1: Adding a stream to existing node 4
00000918.0000155c::2019/09/05-00:59:08.503 INFO [NODE] Node 1: n4 node object adding stream
00000918.0000155c::2019/09/05-00:59:08.503 DBG [NODE] Node 1: n4 node object got a channel
00000918.0000155c::2019/09/05-00:59:08.503 DBG [NODE] Node 1: Using new stream to n4, setting epoch to 1
00000918.0000155c::2019/09/05-00:59:08.503 DBG [NODE] Node 1: Done closing stream to n4
00000918.0000155c::2019/09/05-00:59:08.503 DBG [NODE] Node 1: My Fault Tolerant Session Id is now 8d14d294-1635-416f-9e7c-44450c2a9cce
00000918.0000155c::2019/09/05-00:59:08.503 INFO [NODE] Node 1: No reconnect in progress to n4, updating send queue based on new stream.
00000918.0000155c::2019/09/05-00:59:08.503 DBG [NODE] Node 1: Treating stream with n4 as new connection because epoch (1) is <= 1.
00000918.0000155c::2019/09/05-00:59:08.503 INFO [MQ-Node1] Clearing 0 unsent and 0 unacknowledged messages.
00000918.0000155c::2019/09/05-00:59:08.503 INFO [NODE] Node 1: Highest version with n4 = Major 9 Minor 1 Upgrade 8 ClusterVersion 0x00090008, lowest = Major 8 Minor 9600 Upgrade 3 ClusterVersion 0x00080003
00000918.0000155c::2019/09/05-00:59:08.503 INFO [NODE] Node 1: Done processing new stream to n4.
00000918.0000155c::2019/09/05-00:59:08.503 DBG [CHANNEL 10.10.6.190:~3343~] Close().
00000918.000012f0::2019/09/05-00:59:08.503 INFO [RGP] node 1: Node Connected 4 00000000000000000000000000000000000000000000000000000000000010010
00000918.000012f0::2019/09/05-00:59:08.503 INFO [RGP] sending to node(4) 1: 001(1) => 001(1) +() -() [()] , ()
00000918.0000155c::2019/09/05-00:59:08.503 INFO [PULLER NODE1] Just about to start reading from <refcounted count='3' typeid='.?AVSimpleSecureStream@mscs_security@@'/>
00000918.0000155c::2019/09/05-00:59:08.503 INFO [RGP] node 1: received new information from 4 starting the timer
00000918.00001528::2019/09/05-00:59:08.798 INFO [RGP] node 1: Tick
00000918.00001528::2019/09/05-00:59:08.798 INFO [RGP] node 1: selected partition 10903(3 4) as node 4 has quorum
00000918.00001528::2019/09/05-00:59:08.798 INFO [RGP] node 1: selected partition 10903(3 4) to join [using info from 4]
00000918.00001528::2019/09/05-00:59:08.798 INFO [RGP] node 1: cannot join yet. no connection to (3)
00000918.00001528::2019/09/05-00:59:08.798 INFO [RGP] sending to all nodes 1: 001(1) => 001(1) +() -() [()] , ()
00000918.00001528::2019/09/05-00:59:08.798 DBG [NODE] Node 1: eating message sent to the dead node 3
00000918.00000dbc::2019/09/05-00:59:08.798 INFO [RGP] node 1: received new information from 1 starting the timer
00000918.00001528::2019/09/05-00:59:09.111 INFO [RGP] node 1: Tick
00000918.00001528::2019/09/05-00:59:09.111 INFO [RGP] node 1: selected partition 10903(3 4) as node 4 has quorum
00000918.00001528::2019/09/05-00:59:09.111 INFO [RGP] node 1: selected partition 10903(3 4) to join [using info from 4]
00000918.00001528::2019/09/05-00:59:09.111 INFO [RGP] node 1: cannot join yet. no connection to (3)
00000918.00001528::2019/09/05-00:59:09.111 INFO [RGP] sending to all nodes 1: 001(1) => 001(1) +() -() [()] , ()
00000918.00001528::2019/09/05-00:59:09.111 DBG [NODE] Node 1: eating message sent to the dead node 3
00000918.00001524::2019/09/05-00:59:10.507 DBG [NETFTAPI] received NsiParameterNotification for 169.254.93.143 (IpDadStateInvalid)
00000918.000015a4::2019/09/05-00:59:10.507 DBG [NETFTAPI] received NsiDeleteInstance for 169.254.93.143
00000918.000015a4::2019/09/05-00:59:10.507 WARN [NETFTAPI] Failed to query parameters for 169.254.93.143 (status 0x80070490)
00000918.000015a4::2019/09/05-00:59:10.507 DBG [NETFTAPI] Signaled NetftLocalAdd event for 169.254.93.143
00000918.000015a4::2019/09/05-00:59:10.507 DBG [NETFTEVM] FTI NetFT event handler ignoring PnP add event for IPv4 LinkLocal address 169.254.93.143:~0~
00000918.000015a4::2019/09/05-00:59:10.507 DBG [NETFTEVM] TM NetFT event handler ignoring PnP add event for IPv4 LinkLocal address 169.254.93.143:~0~
00000918.000015a4::2019/09/05-00:59:10.507 DBG [WM] Filtering event NETFT_LOCAL_ADD? 1
00000918.000015a4::2019/09/05-00:59:10.509 WARN [NETFTAPI] Failed to query parameters for 169.254.93.143 (status 0x80070490)
00000918.000015a4::2019/09/05-00:59:10.509 DBG [NETFTAPI] Signaled NetftLocalRemove event for 169.254.93.143
00000918.000015a4::2019/09/05-00:59:10.509 DBG [NETFTEVM] FTI NetFT event handler ignoring PnP remove event for IPv4 LinkLocal address 169.254.93.143:~0~
00000918.000015a4::2019/09/05-00:59:10.509 DBG [NETFTEVM] TM NetFT event handler ignoring PnP remove event for IPv4 LinkLocal address 169.254.93.143:~0~
00000918.000015a4::2019/09/05-00:59:10.509 DBG [WM] Filtering event NETFT_LOCAL_REMOVE? 1
00000918.000015a4::2019/09/05-00:59:10.509 DBG [NETFTAPI] received NsiParameterNotification for 169.254.1.68 (IpDadStatePreferred)
00000918.000015a4::2019/09/05-00:59:10.509 DBG [NETFTAPI] Signaled NetftLocalAdd event for 169.254.1.68
00000918.000015a4::2019/09/05-00:59:10.509 DBG [NETFTEVM] FTI NetFT event handler ignoring PnP add event for IPv4 LinkLocal address 169.254.1.68:~0~
00000918.000015a4::2019/09/05-00:59:10.509 DBG [NETFTEVM] TM NetFT event handler ignoring PnP add event for IPv4 LinkLocal address 169.254.1.68:~0~
00000918.000015a4::2019/09/05-00:59:10.509 DBG [WM] Filtering event NETFT_LOCAL_ADD? 1
00000918.000015a4::2019/09/05-00:59:10.509 DBG [NETFTAPI] Signaled NetftLocalConnect event for 169.254.1.68
00000918.000015a4::2019/09/05-00:59:10.509 DBG [NETFTEVM] FTI NetFT event handler got event: Local endpoint 169.254.1.68:~0~ connected
00000918.00001230::2019/09/05-00:59:10.509 DBG [NETFTEVM] FTI NetFT event dispatcher pushing event: Local endpoint 169.254.1.68:~0~ connected
00000918.00001230::2019/09/05-00:59:10.509 DBG [FTI][Initiator] Got Netft event Local endpoint 169.254.1.68:~0~ connected
00000918.000015a4::2019/09/05-00:59:10.509 DBG [NETFTEVM] TM NetFT event handler got event: Local endpoint 169.254.1.68:~0~ connected
00000918.0000096c::2019/09/05-00:59:10.509 DBG [NETFTEVM] TM NetFT event dispatcher pushing event: Local endpoint 169.254.1.68:~0~ connected
00000918.0000096c::2019/09/05-00:59:10.509 INFO [IM] got event: Local endpoint 169.254.1.68:~0~ connected
00000918.000015a4::2019/09/05-00:59:10.509 DBG [WM] Filtering event NETFT_LOCAL_CONNECT? 1
00000918.000015a4::2019/09/05-00:59:10.510 DBG [NETFTAPI] received NsiAddInstance for fe80::5efe:169.254.1.68
00000918.000015a4::2019/09/05-00:59:10.510 DBG [NETFTAPI] received NsiParameterNotification for fe80::5efe:169.254.1.68 (IpDadStateDeprecated)
00000918.0000152c::2019/09/05-00:59:17.174 DBG [CORE] WriteVersionFunctor: beginning write attempts
00000918.00001530::2019/09/05-00:59:37.222 DBG [NETFT] FTI NetFT event handler deregistration successful.
00000918.00001530::2019/09/05-00:59:37.222 INFO [NODE] Node 1: New join with n3: stage: 'Wait for Heartbeats on Initial NetFT Route' status (1460) reason: '[FTI][Initiator] Aborting connection because NetFT route to node NODE2 on virtual IP fe80::35c4:f902:cbd4:33ef:~3343~
has failed to come up.'
00000918.00001530::2019/09/05-00:59:37.276 INFO [CORE] Node 1: Clearing cookie e7920b13-f4cf-46bb-84ba-79562d7745d8
00000918.00001530::2019/09/05-00:59:37.276 INFO [CORE] Node 1: Cookie Cache 465e1aa8-175f-4879-a473-0ad991998962 [NODE1]
00000918.00001530::2019/09/05-00:59:37.276 DBG [CHANNEL 10.10.6.191:~3343~] Close().
00000918.00001530::2019/09/05-00:59:37.329 WARN cxl::ConnectWorker::operator (): (1460)' because of '[FTI][Initiator] Aborting connection because NetFT route to node NODE2 on virtual IP fe80::35c4:f902:cbd4:33ef:~3343~ has failed to come up.'
00000918.00001530::2019/09/05-01:00:07.531 DBG [JPM] Node 1: contacts size for node NODE2 is 1, current index 0
00000918.00001530::2019/09/05-01:00:07.531 DBG [JPM] Node 1: Trying to connect to node NODE2 (IP: 10.10.6.191:~0~)
00000918.00001530::2019/09/05-01:00:07.531 DBG [HM] Trying to connect to NODE2 at 10.10.6.191:~3343~
00000918.00001524::2019/09/05-01:00:07.547 INFO [CONNECT] 10.10.6.191:~3343~: Established connection to remote endpoint 10.10.6.191:~3343~.
00000918.00001524::2019/09/05-01:00:07.547 INFO [SV] New real route: local (10.13.6.200:~49794~) to remote NODE2 (10.10.6.191:~3343~).
00000918.00001524::2019/09/05-01:00:07.547 INFO [SV] Got a new outgoing stream to NODE2 at 10.10.6.191:~3343~
00000918.00001524::2019/09/05-01:00:07.547 DBG [SM] Joiner: Initialized with SPN = NODE2, RequiredCtxAttrib = 0, HandShakeTimeout = 40000
00000918.0000154c::2019/09/05-01:00:07.547 DBG [SM] Handling auth handshake posted by thread id 5412
00000918.0000154c::2019/09/05-01:00:07.547 DBG [SM] Joiner: Versions: 1-10
00000918.0000154c::2019/09/05-01:00:07.547 DBG [SM] Joiner: ISC returned status = 590610 output Blob size 1723, service principal name HOST/NODE2, auth type MSG_AUTH_PACKAGE::KerberosAuth, attr: 83998
00000918.0000154c::2019/09/05-01:00:07.547 DBG [SM] Joiner: Sending SSPI blob of size 1723 to Sponsor
00000918.0000154c::2019/09/05-01:00:07.563 DBG [SM] Joiner: Switching to Schannel
00000918.00001524::2019/09/05-01:00:07.578 DBG [Schannel] Client: Chosen Cert's version = 2, serialNo = <vector len='16'>00000918.00001524::2019/09/05-01:00:07.735 INFO [SV] Authentication and authorization were successful
00000918.00001524::2019/09/05-01:00:07.735 INFO [VER] Got new TCP connection. Exchanging version data.
00000918.00001524::2019/09/05-01:00:07.735 DBG [VER] Calculated cluster versions: highest [Major 9 Minor 1 Upgrade 8 ClusterVersion 0x00090008], lowest [Major 8 Minor 9600 Upgrade 3 ClusterVersion 0x00080003] with exclude node list: (3)
00000918.00001524::2019/09/05-01:00:07.735 INFO [VER] Checking version compatibility for node NODE2 id 3 with following versions: highest [Major 9 Minor 1 Upgrade 8 ClusterVersion 0x00090008], lowest [Major 8 Minor 9600 Upgrade 3 ClusterVersion 0x00080003].
00000918.00001524::2019/09/05-01:00:07.735 INFO [VER] Version check passed: node and cluster highest supported versions match. Other node still supports lower level, so joining in downlevel mode.
00000918.00001524::2019/09/05-01:00:07.735 INFO mscs::VersionManagerAgent::IsCompatible: First run: setting CFL to 8.3 manually instead of looking for value in database
00000918.00001524::2019/09/05-01:00:07.735 DBG [CORE-Dbg] IsCompatible: setting operating version to 8.3 on first run
00000918.00001524::2019/09/05-01:00:07.750 INFO [SV] Negotiating message security level.
00000918.00001524::2019/09/05-01:00:07.750 INFO [SV] Already protecting connection with message security level 'Sign'.
00000918.00001524::2019/09/05-01:00:07.750 INFO [FTI] Got new raw TCP/IP connection.
00000918.00001524::2019/09/05-01:00:07.765 INFO [FTI][Initiator] This node (1) is initiator
00000918.00001524::2019/09/05-01:00:07.765 DBG [FTI][Initiator] Cookie for remote node is e7920b13-f4cf-46bb-84ba-79562d7745d8
00000918.00001524::2019/09/05-01:00:07.765 DBG [FTI] Stream already exists to node 3: false
00000918.00001524::2019/09/05-01:00:07.783 INFO [FTI][Initiator] Trying to select best endpoints among 169.254.1.68:~3343~, fe80::14:a91:8f79:5d8f:~3343~ (first pair) and 169.254.3.177:~3343~, fe80::35c4:f902:cbd4:33ef:~3343~ (second pair)
00000918.00001524::2019/09/05-01:00:07.785 INFO [HM] Marking route from realLocal 10.13.6.200:~49794~ -> realRemote 10.10.6.191:~3343~ as a cross-subnet route
00000918.00001524::2019/09/05-01:00:07.785 INFO [RouteDb] Route virtual fe80::14:a91:8f79:5d8f:~0~ to virtual fe80::35c4:f902:cbd4:33ef:~0~ added
00000918.00001524::2019/09/05-01:00:07.785 DBG [NETFT] Removing route <struct mscs::FaultTolerantRoute>
00000918.00001524::2019/09/05-01:00:07.785 DBG <realLocal>10.13.6.200:~3343~</realLocal>
00000918.00001524::2019/09/05-01:00:07.785 DBG <realRemote>10.10.6.191:~3343~</realRemote>
00000918.00001524::2019/09/05-01:00:07.785 DBG <virtualLocal>fe80::14:a91:8f79:5d8f:~0~</virtualLocal>
00000918.00001524::2019/09/05-01:00:07.785 DBG <virtualRemote>fe80::35c4:f902:cbd4:33ef:~0~</virtualRemote>
00000918.00001524::2019/09/05-01:00:07.785 DBG <Delay>1000</Delay>
Charles Peter