4.1 Release Notes
MemSQL 4.1 features the new MemSQL Streamliner and columnstore enhancements that simplifies the creation and management of real-time data pipelines running in Apache Spark.
The MemSQL Streamliner deploys Apache Spark and runs user-defined, real-time data pipelines in Spark. A data pipeline is a workflow that extracts information from a real-time data source such as Kafka, performs a transformation on the data, and loads it into MemSQL. All Streamliner configuration can be set up within MemSQL Ops.
Streamliner opens the door for a whole new set of data streaming use cases into MemSQL, and integrates Apache Spark more tightly with MemSQL. Using Streamliner, users can deploy a Spark cluster - master, workers and the MemSQL Spark Interface - with one click, then immediately create data pipelines with configurable extract, transform and load phases, all processed within Apache Spark. Data is loaded as micro-batches into MemSQL, and Streamliner gives visibility into the full data pipeline.
For more information about the MemSQL Streamliner, see the Spark Management section.
Enhanced Columnstore Functionality
Faster streaming into columnstore tables
MemSQL now puts a memory-optimized data structure in front of each columnstore table which groups small inserts into larger batches for increased insert efficiency. Rows are available for queries immediately after the commit and do not wait for the write to disk. As with the MemSQL rowstore, inserts are fully transactional and durable.
Alter table for columnstore
MemSQL now supports the
ALTER TABLE command for columnstore tables.
Increased columnstore data compression
JSON columns in a columnstore table are now represented in a more compressed format on disk.
Support for non-recursive CTEs
MemSQL now supports non-recursive common table expressions (CTEs). CTEs begin with a
WITH clause prior to a
DELETE operation. Non-recursive CTEs do not reference themselves within the CTE.
The skip_name_resolve variable now has 3 options - auto, true, and false. The default is auto, which tries to do a reverse DNS lookup. If the lookup succeeds, it uses rules defined for ‘%',
Maintenance Release Changelog
2016-02-04 Version 4.1.3
- Fixed issue when writing columnar segments that end with a JSON number value
2015-12-01 Version 4.1.2
- Fixed issue where low memory conditions could cause the columnstore memory-to-disk flusher to stop flushing rows
- Fixed issue where the
check_systemutility always complains that CRC32 is unavailable
2015-11-04 Version 4.1.1
- Fixed an issue with
CREATE TABLE... LIKEfor columnstore tables
- Fixed a performance issue with Parquet compression for certain JSON schemas