Product Solutions Support
Try MemSQL

Online Upgrade

Warning

This page is part of Upgrading MemSQL. Start on that page.

This page describes one option for Upgrading MemSQL. Follow the instructions on that page for previous steps in the upgrade process.

This page describes how to upgrade MemSQL from an earlier version while keeping the cluster online and available. This option is only available for a cluster operating in High Availability. In this procedure, you upgrade individual nodes in the cluster while other nodes in the cluster are running to maintain cluster availability throughout the upgrade process.

Familiarize yourself with the entire procedure before beginning.

You must be running MemSQL Ops 5.7 or later. Upgrade MemSQL Ops to the latest version prior to upgrading the MemSQL database, following the instructions on the previous page Upgrading MemSQL. Verify that your MemSQL Ops version is 5.7 or later by running:

$ memsql-ops version
Version: 5.7.1

Ensure your cluster is running with High Availability (Redundancy Level 2). The redundancy_level variable should be 2.

memsql> show variables like 'redundancy_level';
+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| redundancy_level | 2     |
+------------------+-------+

Ensure your cluster is healthy, and in particular ensure that all nodes are online and there is both a master and slave for each partition. See EXPLAIN REBALANCE PARTITIONS. Otherwise, you will be unable to keep the cluster available throughout the upgrade procedure.

It is strongly recommended to avoid running CREATE DATABASE, DROP DATABASE, or ALTER TABLE during the upgrade procedure.

Warning

Follow the upgrade sequence exactly as described in this section: leaf nodes first, then child aggregator nodes, and finally the master aggregator.

Warning

This procedure is NOT safe for upgrades from MemSQL 3.2 and earlier to MemSQL 4.0 and later. For that, see Upgrading MemSQL from 3.2 and earlier.

Part 1: Prepare for Upgrade

  1. Retrieve the list of all databases by running SHOW DATABASES, excluding system databases (memsql, sharding, information_schema). These system databases are also excluded in all following instructions.

  2. (Recommended but optional) Prior to upgrading, you may run SNAPSHOT DATABASE on any or all databases, which compacts the transaction logs, reducing database recovery time on startup.

    memsql> SNAPSHOT DATABASE database
    
  3. Enable manual control in MemSQL Ops: In the web interface, go to Settings > Manual Cluster Control and check “Enable Manual Control”. Or with the command line interface:

    $ memsql-ops cluster-manual-control --enable
    
  4. On the master aggregator, disable auto-attach and leaf failure detection:

    memsql> SET GLOBAL auto_attach = OFF;
    memsql> SET GLOBAL leaf_failure_detection = OFF;
    
  5. On all aggregators, disable aggregator failure detection:

    $ memsql-ops memsql-list -q -r aggregator master | xargs -L 1 memsql-ops memsql-update-config --key aggregator_failure_detection --value off --set-global
    

Part 2: Upgrade Leaf Nodes

Determine which availability groups each of your leaves are in with SHOW LEAVES.

The simplest way to upgrade the leaf nodes is to upgrade all leaves in Availability Group 1, then all leaves in Availability Group 2. The simple instructions for this method are below, and more complex instructions for alternative options are discussed later in the section Appendix: Additional instructions for Part 2: Upgrade Leaf Nodes.

For an example, see the section Appendix: Additional instructions for Part 2: Upgrade Leaf Nodes.

Part 3: Upgrade Child Aggregator Nodes

Upgrade each child aggregator node according to the instructions in the Upgrading a node section.

You may upgrade child aggregators in parallel as desired. Keep enough aggregators online at any given time to continue running client queries smoothly.

Info

When upgrading a child aggregator, you will be unable to run queries against that node. Furthermore, it will not see any operations take effect until it is upgraded, so to minimize risks, avoid running DDL operations such as ALTER TABLE while upgrading any of your aggregators.

Part 4: Upgrade the Master Aggregator Node

Upgrade the master aggregator node according to the instructions in the Upgrading a node section.

Info

When upgrading the master aggregator, you will be unable to run queries against that node. This means you will be unable to run DDL operations, DML operations against reference tables, as well as database-level and cluster-level operations.

Part 5: Re-enable Auto Cluster Operations

Restore the auto-cluster operations settings to their initial state.

  1. On the master aggregator, re-enable auto-attach and leaf failure detection:

    memsql> SET GLOBAL auto_attach = ON;
    memsql> SET GLOBAL leaf_failure_detection = ON;
    
  2. On all aggregators, re-enable aggregator failure detection by resetting it to the default value ON:

    $ memsql-ops memsql-list -q -r aggregator master | xargs -L 1 memsql-ops memsql-update-config --key aggregator_failure_detection --value on --set-global
    $ memsql-ops memsql-list -q -r aggregator master | xargs -L 1 memsql-ops memsql-update-config --key aggregator_failure_detection --delete
    
  3. Disable manual control in MemSQL Ops: In the web interface, go to Settings > Manual Cluster Control and uncheck “Enable Manual Control”. Or with the command line interface:

    $ memsql-ops cluster-manual-control --disable
    

This completes the upgrade procedure.

Appendix: Additional instructions for Part 2: Upgrade Leaf Nodes

Example

For example: assume we have the following cluster configuration, with 8 leaf nodes in redundancy 2:

There are two availability groups:

There are four pairs of leaf nodes:

image

To upgrade these leaf nodes:

  1. Upgrade Availability Group 1: leaf nodes L1, L2, L3, and L4
    1. Detach leaf nodes L1, L2, L3, and L4
    2. Upgrade and start leaf nodes L1, L2, L3, and L4 according to the instructions in the Upgrading a node section
    3. Attach leaf nodes L1, L2, L3, and L4
  2. Run RESTORE REDUNDANCY on every database
  3. Upgrade Availability Group 2: leaf nodes L5, L6, L7, and L8
    1. Detach leaf nodes L5, L6, L7, and L8
    2. Upgrade and start leaf nodes L5, L6, L7, and L8 according to the instructions in the Upgrading a node section
    3. Attach leaf nodes L5, L6, L7, and L8
  4. Run RESTORE REDUNDANCY on every database
  5. Run REBALANCE PARTITIONS on every database

Upgrading alternate groups of leaves in parallel

To keep the cluster online, at least one leaf in each leaf pair must be online at any given time, so you must upgrade the two nodes in a leaf pair at different times. But upgrading nodes in different leaf pairs may be done in parallel.

The simplest and generally fastest procedure to achieve this is to upgrade all the leaves in one availability group, then the other availability group, as described above. But you can also upgrade leaf nodes in a different order - you do not have to upgrade all nodes in one availability group before the other - as long as you ensure at least one leaf node in each pair is online at all times.

You may split all leaf nodes into multiple groups of leaf nodes, and upgrade group 1, then group 2, and so on, as long as the two leaves in each leaf pair are in different groups. Run RESTORE REDUNDANCY on every database in after each group of upgrades, and run REBALANCE PARTITIONS on every database at the end.

Was this article useful?