In part 1 we successfully migrated the data within the cluster to a second datacenter, completing this without downtime to the cluster.
Now we will look at decommissioning the original RS_UK datacenter. Firstly we need to ensure all the applications are now pointing directly to the new datacenter GL_EU.
Once we know all applications are writing direct to the new DC it is worth doing a full repair on each node in the new DC to just be sure all the data has migrated correctly from the original DC.
While the repair is happening, the cassandra.yaml needs to be updated to repoint the seeds to nodes within the GL_EU DC.
Once the repair is complete and the cassandra.yaml updated, it is time to alter the replication factor of all keyspaces ( including the system ones ) to only replicate the to the GL_EU DC:
ALTER KEYSPACE user_keyspace1 WITH replication = {'class': 'NetworkTopologyStrategy', ‘GL_EU': 3};
ALTER KEYSPACE user_keyspace2 WITH replication = {'class': 'NetworkTopologyStrategy', ‘GL_EU': 3};
ALTER KEYSPACE user_keyspace3 WITH replication = {'class': 'NetworkTopologyStrategy', ‘GL_EU': 3};
We now have the data disconnected, and data written to GL_EU will not replicate back to RS_UK.
At this stage the nodes are still connected together across the datacenters, even though the data is not flowing, so we need to disconnect and shutdown the nodes in RS_UK.
So shutdown in turn each node in the RS_UK datacenter, then use the nodetool removenode command to remove the nodes from the cluster. This should be run on one of the nodes within the GL_EU cluster.
The nodetool removenode requires the host ID of the node to be removed, this can be found by running nodetool status
Datacenter: RS_UK
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.29.30.29 11.66 GB 256 ? ab479afd-c754-47f7-92fb-47790d734ac9 rack1
UN 10.29.30.33 12.32 GB 256 ? 9aa1c5c5-c6cd-4267-ba68-c6bd8b2ac460 rack2
UN 10.29.30.34 12.16 GB 256 ? db454258-ac73-4a8a-9c75-226108c66889 rack3
Datacenter: GL_EU
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.131.134.35 13.19 GB 256 ? 114b4a37-7d69-40e5-988b-a4c998e7a02a rack1
UN 10.131.134.39 12.14 GB 256 ? 4173fc2a-e65c-43aa-baa4-a5eefe0ceb60 rack2
UN 10.131.134.42 12.97 GB 256 ? 8b5dde02-1ff1-48cc-9900-6d8f2bb339bf rack3
This shows the host id for each of the RS_UK nodes that need to be removed.
nodetool removenode ab479afd-c754-47f7-92fb-47790d734ac9
nodetool removenode 9aa1c5c5-c6cd-4267-ba68-c6bd8b2ac460
nodetool removenode db454258-ac73-4a8a-9c75-226108c66889
These commands should be run in turn, and once completed, the cluster will be a single DC cluster again, in the new datacenter.
The cluster has now been successfully migrated to the new Datacenter with the old datacenter shutdown.
In the third and final part of this series we will look at the problems that may occur.