≡ Menu

Moving Cassandra Clusters without Downtime – Part 2

In part 1 we successfully migrated the data within the cluster to a second datacenter, completing this without downtime to the cluster.

Now we will look at decommissioning the original RS_UK datacenter. Firstly we need to ensure all the applications are now pointing directly to the new datacenter GL_EU.

Once we know all applications are writing direct to the new DC it is worth doing a full repair on each node in the new DC to just be sure all the data has migrated correctly from the original DC.

While the repair is happening, the cassandra.yaml needs to be updated to repoint the seeds to nodes within the GL_EU DC.

Once the repair is complete and the cassandra.yaml updated, it is time to alter the replication factor of all keyspaces ( including the system ones ) to only replicate the to the GL_EU DC:

ALTER KEYSPACE user_keyspace1 WITH replication = {'class': 'NetworkTopologyStrategy', ‘GL_EU': 3};

ALTER KEYSPACE user_keyspace2 WITH replication = {'class': 'NetworkTopologyStrategy', ‘GL_EU': 3};

ALTER KEYSPACE user_keyspace3 WITH replication = {'class': 'NetworkTopologyStrategy', ‘GL_EU': 3};

We now have the data disconnected, and data written to GL_EU will not replicate back to RS_UK.

Data disconnected

At this stage the nodes are still connected together across the datacenters, even though the data is not flowing, so we need to disconnect and shutdown the nodes in RS_UK.

So shutdown in turn each node in the RS_UK datacenter, then use the nodetool removenode command to remove the nodes from the cluster. This should be run on one of the nodes within the GL_EU cluster.

The nodetool removenode requires the host ID of the node to be removed, this can be found by running nodetool status

Datacenter: RS_UK
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns    Host ID                               Rack
UN    11.66 GB   256          ?       ab479afd-c754-47f7-92fb-47790d734ac9  rack1
UN    12.32 GB   256          ?       9aa1c5c5-c6cd-4267-ba68-c6bd8b2ac460  rack2
UN    12.16 GB   256          ?       db454258-ac73-4a8a-9c75-226108c66889  rack3
Datacenter: GL_EU
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns    Host ID                               Rack
UN  13.19 GB   256          ?       114b4a37-7d69-40e5-988b-a4c998e7a02a  rack1
UN  12.14 GB   256          ?       4173fc2a-e65c-43aa-baa4-a5eefe0ceb60  rack2
UN  12.97 GB   256          ?       8b5dde02-1ff1-48cc-9900-6d8f2bb339bf  rack3

This shows the host id for each of the RS_UK nodes that need to be removed.

nodetool removenode ab479afd-c754-47f7-92fb-47790d734ac9
nodetool removenode 9aa1c5c5-c6cd-4267-ba68-c6bd8b2ac460
nodetool removenode db454258-ac73-4a8a-9c75-226108c66889

These commands should be run in turn, and once completed, the cluster will be a single DC cluster again, in the new datacenter.

Single Datacenter

The cluster has now been successfully migrated to the new Datacenter with the old datacenter shutdown.

In the third and final part of this series we will look at the problems that may occur.

{ 0 comments… add one }

Leave a Comment