Hello,
I have a 3 node Cluster on Windows Server 2016.
For some weeks one of the nodes (node3, a quorum node) was not part of the domain when it should have been. The remaining nodes continued to work.
Node3 was re-added to the domain thinking that would resolve the errors, but alas no.
The cluster event on node3 shows error “Cluster failed to start. The latest copy of cluster configuration data was not available within the set of nodes attempting to start the cluster. Changes to the cluster occurred while the set of nodes were not in membership and as a result were not able to receive data updates.” and
“Cluster node “Node3” failed to join the cluster because it could not communicate over the network with any other node in the cluster. Verify network connectivity and configuration of any network firewalls”
Firewall is off. All nodes can ping and browse to folders on one another.
The Cluster service has been restarted on all nodes.
Running command “cluster node /status” on node3 returns:
Node1 Down
Node2 Down
Node3 Joining
Running the same command on Node1 and Node2 returns the status as being Up for all.
Couple more logs from the Event viewer:
“The Cluster Service service terminated with the following service-specific error: The wait operation timed out.” and “The Cluster Service service terminated unexpectedly. It has done this 53 time(s). The following corrective action will be taken
in 0 milliseconds: Restart the service.”
A generation of the cluster log file gives the same errors as above but with an added:
“Attempt to start the cluster service on all nodes in the cluster so that nodes with the latest copy of the cluster configuration data can first form the cluster. The cluster will be able to start and the nodes will automatically obtain the updated cluster
configuration data. If there are no nodes available with the latest copy of the cluster configuration data, run the 'Start-ClusterNode -FQ' Windows PowerShell cmdlet. Using the ForceQuorum (FQ) parameter will start the cluster service and mark this node's
copy of the cluster configuration data to be authoritative. Forcing quorum on a node with an outdated copy of the cluster database may result in cluster configuration changes that occurred while the node was not participating in the cluster to be lost.”
How can we add node3 back into the cluster successfully?