Stream Computing

Stream Processing Service provides a set of generic and domain-based operators, to help data developers to quickly compose data processing solutions with just a few clicks. It significantly lowers the entry barrier and improves the efficiency for data development.
 Compared with centralized deployment of cloud computing, Edge Stream Computing not only solves the problems of excessive delay and excessive aggregated traffic, but also provides better support for real-time and bandwidth-intensive services. EnOS Edge is committed to precipitating common algorithms for stream processing in the IoT field. It provides a complete set of underlying encapsulated StreamSets operators for developers to develop customized stream data processing tasks to meet complex business needs.

../_images/edge_computing_struc.png
  • Low latency, Stream Computing focuses on real-time, short-period data analysis, which can better support the real-time intelligent processing and execution of local services;
  • More secure, the data is only exchanged between the source data device and the edge device, and no longer uploads all to the cloud computing platform, preventing the risk of data leakage;
  • Relieve the flow pressure, filter and preprocess the data in advance at the edge to reduce the upload flow pressure; For model points, it is often necessary to do some preprocessing, or perform some simple calculations, which can be performed directly using the EnOS Stream Computing module.

Application

EnOS Stream Computing service can be applied to the following scenarios:

  • Rapid processing improves the timeliness of data analysis and processing
  • Computing nearby relieves the pressure of uploading all raw data to the cloud
  • Avoid the risk of data leakage, core data is only processed at the edge

Specific application scenarios such as:

  • Active power calculation of Solar site inverters, which supports the calculation of the 5-minute window average value of a single telemetry point of a single device;
  • The 10-minute average active power of the wind turbine supports the calculation of the 10-minute window average value of a single telemetry point of a single device;
  • Device status statistics, according to the device status (DI meaurement points) to count the number of device in a certain state.

Features

  • Support import and execution the existing pipelines exported from EnOS Cloud
  • Support low-code/drag-and-drop to generate the pipeline
  • Provide rich generic operators and high-performance costomized operators

Key Concepts

Streaming data processing flow

  • Raw data processing: After the original data of the measuring point is collected by the protocol, it is converted into a model measuring point according to the point mapping template, and then sent to the Stream Computing module. Stream Computing services filter data according to specified thresholds.
  • Data calculation: The data after threshold filtering is aggregated and calculated by the algorithm defined in the data processing strategy.
  • Output calculation result: The data processed by the streaming data processing module will flow into the downstream module, and will be recorded in the time series database (TSDB) or other target storage system according to the pre-configured storage strategy. Users can query the stored data through API.

Continuous real-time data stream

The data that the streaming data processing engine needs to process is real-time and continuous. The data stream is subscribed and consumed by the stream data processing service in chronological order. Data is generated continuously, so the data stream is continuously integrated into the stream data processing system. Therefore, streaming data is always real-time and continuous.

Continuous and efficient calculation

The computing mode of EnOS Stream Computing service is “event triggered”. The trigger is the continuous stream data mentioned earlier. Whenever new streaming data is sent to the streaming data processing system, the system immediately starts and executes calculation tasks.

Real-time streaming data integration

After stream data triggers stream data processing, the calculation result is continuously recorded in the target data storage according to the pre-configured storage strategy.

StreamSets operator

EnOS Stream Computing service provides a complete set of underlying encapsulated StreamSets operators for developers to develop customized stream data processing tasks based on business needs. For more information about the StreamSets operators supported by EnOS Edge, see The list of supported StreamSets operators.