Stream Processing Resource

Note

This section refers to Enterprise Data Platform > Stream Processing. If you are using an older version of this service, see Stream Processing Resource (Old Version)


The stream processing service can fully satisfy the processing needs of the asset and device real-time data as well as the historical data that is integrated through offline message channels.


Based on Apache Spark™ Streaming, customized and optimized by Envision, the EnOS stream processing service has high scalability, high throughput, and high fault tolerance. EnOS is also committed to adopting common algorithms for streaming processing in the IoT field, enabling developers to complete the development of streaming processing tasks through simple template configurations.


In addition, the streaming processing service has adopted multiple sets of calculation templates and general operators in the energy field, helping developers to develop data processing solutions quickly without the need to code, thereby greatly improving data development efficiency and lowering the development threshold.


For more information, see Stream Processing.

Resource Application Scenario

Before installing the stream processing templates and StreamSets calculator libraries, or before configuring stream processing jobs, you need to apply for the Stream Processing resource. There are three resource types for Stream Processing resources, namely: Stream Designing, Cluster Processing, and Standalone Processing.

  • Stream Designing: Stream Designing resources are used to create and design stream processing tasks. When performing native drag-and-drop stream task designing and debugging, you need to temporarily run related tasks or install the related library packages.
  • Cluster Processing: When the amount of stream set is large and the processing performance is high, you can use the Cluster Processing resource.
  • Standalone Processing: When the amount of stream set is small and the cost control is strict, you can use the Standalone Processing resource.


The stream processing resources can be requested based on the computing unit (CU). Different specifications under different resource types correspond to different data processing capabilities. In the same resource type, the higher the specification, the higher the processing efficiency, and the larger the amount of data processed per unit time. The resource specifications and corresponding data processing capabilities are as per the below.

Stream Designing

Specification Computing Capability Reference
Standard 1CU 1CU = 1 core CPU + 2GB Memory. 1CU stream designing resource supports the installation of 3 lib packages.
Standard X2 2CU 1CU = 1 core CPU + 2GB Memory. 1CU stream designing resource supports the installation of 6 lib packages.
Standard X3 3CU 1CU = 1 core CPU + 2GB Memory. 1CU stream designing resource supports the installation of 9 lib packages.
Standard X4 4CU 1CU = 1 core CPU + 2GB Memory. 1CU stream designing resource supports the installation of 12 lib packages.

Cluster Processing

Specification Computing Capability
Standard
  • Yarn Cluster：8CU
  • StreamSets Server：4CU
Standard X 2
  • Yarn Cluster：16CU
  • StreamSets Server：8CU
Standard X 3
  • Yarn Cluster：24CU
  • StreamSets Server：12CU
Standard X 4
  • Yarn Cluster：32 CU
  • StreamSets Server：16CU
Standard X 5
  • Yarn Cluster：40CU
  • StreamSets Server：20CU
Standard X 6
  • Yarn Cluster：48CU
  • StreamSets Server：24CU
Standard X 7
  • Yarn Cluster：56CU
  • StreamSets Server：28CU
Standard X 8
  • Yarn Cluster：64CU
  • StreamSets Server：32CU

Note

Reference: 1CU = 1 core CPU + 2GB Memory. For simple data processing jobs such as single-pipeline filtering and string conversion, 1CU cluster processing resource can process 4,000 - 11,000 records per second. 1CU cluster processing resource can run 1 pipeline.

Standalone Processing

Specification Computing Capability
Standard 4CU
Standard X 2 8CU
Standard X 3 12CU
Standard X 4 16CU
Standard X 5 20CU
Standard X 6 24CU
Standard X 7 28CU
Standard X 8 32CU

Note

Reference: 1CU = 1 core CPU + 2GB Memory. For simple data processing jobs such as single-pipeline filtering and string conversion, 1CU standalone processing resource can process 4,000 - 11,000 records per second. 1CU standalone processing resource can run 2 pipelines.