Stream Processing - Message Queue Resource¶

Stream Processing

The stream processing service can fully satisfy the processing needs of the asset and device real-time data as well as the historical data that is integrated through offline message channels.

Based on Apache Spark™ Streaming, customized and optimized by EnOS, EnOS stream processing service has high scalability, high throughput, and high fault tolerance. EnOS is also committed to adopting common algorithms for streaming processing in the IoT field, enabling developers to complete the development of streaming processing tasks through simple template configurations.

In addition, the streaming processing service has adopted multiple sets of calculation templates and general operators in the energy field, helping developers to develop data processing solutions quickly without the need to code, thereby greatly improving data development efficiency and lowering the development threshold.

For more information, see Stream Processing.

Message Queue

For advanced stream processing pipelines, the system pipeline outputs data records that are needed by the pipelines to different Kafka topics. The corresponding data input Kafka topic is selected in the configuration of downstream stream processing pipeline, for saving computing resources efficiently. For more information, see Developing Advanced Pipelines.

Resource Application Scenario¶

Before creating an advanced stream processing pipeline, ensure that your organization has requested for the Stream Processing - Message Queue resource through the Resource Management page.

Resource Specification¶

Note

The maximum number of resource instances that can be applied for under each OU is 20.

Field	Description
Resource Name	Resource Name. Resource Management will add the OU ID information after you submit the name to form the full Kafka Topic name.
Number of Partitions	Number of partitions for Kafka Topic. Streaming Processing - Message Queuing (Kafka) is used to split resource traffic in the clustered version. The application range of partition support is 1.0 to 64 partitions by default.