Time Series Data Management

As a core component of the IoT solution, the EnOS Edge TSDB service provides reliable time-series data storage and efficient data access for large-scale enterprise-wide IoT data processing.

The EnOS Edge time-series database stores and manages second-level raw data, normalized minute-level data, device status data, power data, etc. Its main functions and architecture are shown in the following figure:

../_images/tsdb_arch.png

Usage Scenarios

EnOS Edge TSDB service provides large-scale IoT data storage and query services for IT/OT application developers.

Features

  • Lightweight And Scalable: High-performance TSDB based on the open-source InfluxDB.

  • Lower Storage Cost: Adopts a domain-driven data storage strategy, which provides high compression rate and supports two lossy compression algorithms (Swinging Door and Deadband).

  • Rich Data Services: Provides efficient and extensive data query and analysis services through the V2.0 and V2.1 APIs.

  • High Availability: High availability based on dual writing mode.

Performance

  • Data writing throughput up to 30,000 records per second

  • Second-level query response

Key Concepts

The EnOS Edge TSDB offers efficient storage capabilities and low-cost storage features and supports lossless and lossy compression. By default, the lossless compression algorithm is executed according to the device data type to save storage costs. For specific devices and measuring points, the lossy compression function can also be enabled, and a specific lossy compression algorithm can be selected to perform proprietary compression to further improve storage efficiency.

Lossless Compression

EnOS Edge TSDB can classify and store different types of data (such as AI raw data, AI minute-level normalized data, DI data, PI data, generic data, etc.), while automatically executing an efficient lossless compression algorithm for the same type of data.

Lossy Compression

EnOS Edge TSDB provides the capability to configure storage policies and supports the storage of data after lossy compression. Two data compression algorithms are supported, and one or more compression methods can be selected and used according to actual needs:

  • Deadband Compression: Determines whether to store the measurement point data by comparing the difference between adjacent measurement point data. If the deviation between the current measurement point data and the last saved data exceeds the assigned change threshold, the current measurement point data shall be saved, otherwise it shall be discarded.

  • Swinging Door Compression: The Swinging Door Trending (SDT) algorithm is a linear trend compression algorithm. In essence, it replaces a series of continuous data points with a straight line determined by the start and end points (see here for an example of implementation). The compression rate is based on a user-defined absolute error (i.e., the width of the door). The swingin-door algorithm has the advantages of small computational load, can track the process trend change and is suitable for compressing slowly changing data.