≡ Menu

Moving Cassandra Clusters without Downtime – Part 1

This is a post to go with a recent presentation at the Datastax Accelerate conference, and describes the process used to move clusters from Rackspace to Google cloud.

Our requirement was to move multiple clusters, but one at a time, from the UK, to GCP in Belgium without any downtime.

Within this post the original datacenter is called RS_UK and the new datacenter will be GL_EU

Application Pre Requisites

Before the data move started we need to ensure the application was following these pre requisites:

  • Local consistency types for Reading and Writing, i.e: 
    • LOCAL_ONE
    • LOCAL_QUORUM 
  • Application Driver needs to DC Aware policy
  • Light Weight Transactions (LWT) must use LOCAL_SERIAL
Original Datacenter with 5 nodes

Step 1 – Alter system keyspaces

Before any new nodes were created in the new datacenter the first step was alter the system keyspaces to include the new datacenter

ALTER KEYSPACE system_auth WITH replication = {'class': 'NetworkTopologyStrategy', ‘RS_UK': 3, ‘GL_EU': 3};

This should be performed for all the system keyspaces:

  • system_auth
  • system_schema
  • dse_leases
  • system_distributed
  • dse_perf
  • system_traces
  • dse_security

Ignore the ones starting dse if this is an apache cassandra cluster rather than a Datastax version.

Step 2 – Create Nodes in New Datacenter 

When preparing for the creation of the node in the new DC the following configuration needs to be taken into account:

  1. The cluster_name in the cassandra.yaml must be the same as the cluster_name in the old DC
  2. The seeds should point to the seeds in the old DC
  3. Assuming the old noes were using the GossipingPropertyFileSnitch, then continue using the GossipingPropertyFileSnitch and set the dc in the cassandra-rackdc.properties file to be the new DC.

Create the new nodes one at a time, waiting for each one to complete joining the cluster before moving to the next. Each node should join quickly because at this stage only the system keyspaces are being streamed to the new DC

New GL_EU nodes created

So now we are in the situation where the system keyspaces are replicated to the new DC, but the user keyspaces are not. So at this stage it is important that the application keeps connecting the the old DC.

System Keyspaces replicated

Step 3 – Alter Replication for User Keyspaces

Now it is time to alter the replication of the user keyspaces, to allow replication of all data to the new DC. Run this statement for each keyspace:

ALTER KEYSPACE user_keyspace1 WITH replication = {'class': 'NetworkTopologyStrategy', ‘RS_UK': 3, ‘GL_EU': 3};

ALTER KEYSPACE user_keyspace2 WITH replication = {'class': 'NetworkTopologyStrategy', ‘RS_UK': 3, ‘GL_EU': 3};

ALTER KEYSPACE user_keyspace3 WITH replication = {'class': 'NetworkTopologyStrategy', ‘RS_UK': 3, ‘GL_EU': 3};

Once this is completed, all new inserted data is replicated to GL_EU however the old data has not yet been streamed across so it is still not safe to connect to GL_EU.

Step 4 – Rebuild Nodes

On each node in turn the following nodetool command should be run:

nodetool rebuild  RS_UK

This will take sometime for each node to complete, as all the data required for each node is now streamed from the RS_UK datacenter. It is often a good idea to script this part to run on each node in turn.

It is possible that a nodetool rebuild will fail, I will go over some of the reasons we had failures in part 3, however, if a node fails you can just re run the command again.

Once all the new node have been rebuilt the cluster is now working as a multi DC cluster, and the applications can connect to either DC, with data flowing automatically between the DC’s.

Data flows automatically between the Datacenters

In Part 2 we will look at how to safely decommission the RS_UK datacenter.

{ 0 comments… add one }

Leave a Comment