Product Solutions Support
Try MemSQL

ALTER PIPELINE

The ALTER PIPELINE clause changes an existing pipeline’s configuration.

ALTER PIPELINE pipeline_name
  [SET
    [OFFSETS
      [EARLIEST | LATEST | json_source_partition_offset]
    ]
    [TRANSFORM ('uri', ['executable', 'arguments [...]'])]
    [BATCH_INTERVAL milliseconds]
  ]
  [FIELDS | COLUMNS]
    [TERMINATED BY 'string'
      [[OPTIONALLY] ENCLOSED BY 'char']
      [ESCAPED BY 'char']
    ]

Each of the clauses in a ALTER PIPELINE statement are described below.

ALTER PIPELINE SET

You can set a pipeline’s offsets, transform, or batch interval by using the SET clause.

ALTER PIPELINE pipeline_name
  [SET
    [OFFSETS
      [EARLIEST | LATEST | json_source_partition_offset]
    ]
    [TRANSFORM]
    [BATCH_INTERVAL milliseconds]
  ]   

ALTER PIPELINE SET OFFSETS

A pipeline’s current starting offset can be altered by using the SET OFFSETS clause. When a new offset is set, the pipeline will begin extracting data from the specified offset, regardless of any previous offsets that have or have not been extracted. There are three offset options:

SET OFFSETS EARLIEST: Configures the pipeline to start reading from the earliest (or oldest) available offset in the data source.

memsql> ALTER PIPELINE mypipeline SET OFFSETS EARLIEST;
Query OK, 0 rows affected (0.01 sec)

SET OFFSETS LATEST: Configures the pipeline to start reading from the latest (or newest) available offset in the data source.

memsql> ALTER PIPELINE mypipeline SET OFFSETS LATEST;
Query OK, 0 rows affected (0.01 sec)

SET OFFSETS '{"<source-partition>": <partition-offset>}': Configures the pipeline to start reading from specific data source partitions and offsets. When you manually specify which source partition and offset to start extracting from, there are a few important things to consider:

ALTER PIPELINE SET TRANSFORM

You can configure an existing pipeline to use a transform by using the SET TRANSFORM clause. The syntax for applying a transform to a pipeline is identical to the WITH TRANFORM syntax that is used when creating a new pipeline.

memsql> ALTER PIPELINE mypipeline SET TRANSFORM('http://memsql.com/my-transform-tarball.tar.gz', 'my-executable.py', '-arg1 -arg1');
Query OK, 0 rows affected (0.01 sec)

ALTER PIPELINE SET BATCH_INTERVAL

You can alter the batch interval for an existing pipeline by using the SET BATCH_INTERVAL clause. A batch interval is the time duration between the end of a batch operation and the start of the next one. The syntax for setting a batch interval is identical to the BATCH_INTERVAL syntax that is used when creating a new pipeline.

memsql> ALTER PIPELINE mypipeline SET BATCH_INTERVAL 0;
Query OK, 0 rows affected (0.01 sec)
Was this article useful?