Product Solutions Support
Try MemSQL

START PIPELINE

The START PIPELINE statement starts an existing pipeline in a MemSQL database. An error will occur if the pipeline is already running.

Syntax

START PIPELINE pipeline_name [FOREGROUND [LIMIT count BATCHES] ];

Arguments

pipeline_name

The name of the pipeline.

count

An integer that represents the number of batches to ingest.

Remarks

There are two ways to start a pipeline: in the foreground, or in the background. When a START PIPELINE mypipeline statement is executed, the client will not report the number of affected rows. All pipeline ingest behavior will occur internally, or in the background.

When a pipeline is started in the foreground, the number of affected rows will be displayed in the client. Two additional options are provided to start a pipeline in the foreground:

When a batch limit is specified, the pipeline will ingest only the specified number of batches. Note that a batch may contain more than one row of ingested data.

The FOREGROUND clause can also be used to help debug issues with a pipeline. When specified, any errors that occur will be reported to the client.

If a LIMIT clause is not specified, the pipeline will continue running until each offset from the data source has been ingested.

Examples

START PIPELINE in Background

The following example demonstrates how to start a pipeline with no additional clauses specified:

memsql> START PIPELINE mypipeline;
Query OK, 4 rows affected (0.04 sec)

To see if the pipeline was started successfully, you can execute SHOW PIPELINES:

memsql> SHOW PIPELINES;
+-------------------+---------+
| Pipelines_in_mydb | State   |
+-------------------+---------+
| mypipeline        | Running |
+-------------------+---------+
1 row in set (0.00 sec)

START PIPELINE in Foreground

The following example demonstrates how to start a pipeline in the foreground, which displays the number of rows written to the destination table:

memsql> START PIPELINE mypipeline FOREGROUND;
Query OK, 4 rows affected (1.39 sec)

To see if the pipeline was started successfully, you can execute SHOW PIPELINES:

memsql> SHOW PIPELINES;
+-------------------+---------+
| Pipelines_in_mydb | State   |
+-------------------+---------+
| mypipeline        | Running |
+-------------------+---------+
1 row in set (0.00 sec)

START PIPELINE in Foreground with Batch Limit

The following example demonstrates how to start a pipeline in the foreground with a specified limit, which displays the number of rows written to the destination table:

memsql> START PIPELINE mypipeline FOREGROUND LIMIT 5 BATCHES;
Query OK, 37 rows affected (0.71 sec)

Unlike a pipeline that is started without a batch limit, the pipeline is immediately stopped after ingesting the specified number of batches. You can verify that the pipeline is in a stopped state by executing SHOW PIPELINES:

memsql> SHOW PIPELINES;
+-------------------+---------+
| Pipelines_in_mydb | State   |
+-------------------+---------+
| mypipeline        | Stopped |
+-------------------+---------+
1 row in set (0.00 sec)
Was this article useful?