Kafka Sub


The Kafka Sub node establishes a Kafka client that can subscribe up to 5 Kafka topics.

Node Type

Input.

Input and Output Ability

This node does not have an entry point and has 1 exit point. The output can be any format and is the message this client retrieves from the subscribed topic. The Kafka topic is automatically saved to the output metadata ${metadata.topic}.

Node Properties

../../_images/kafka_sub.png


Name

The name for this node.


Host

The host address or domain name.


Authentication Method

The authentication method, Anonymous or Username/Password.

  • Anonymous: Does not need any authentication.

  • Username/Password: Enter the Username and Password used for authentication.


Topic

The Kafka topic to subscribe to. The maximum number of topics you can subscribe to is 5.


Concurrency

The number of processes to access the Kafka server at the same time. Setting this can enable the running of multiple processes concurrently and improve processing speed. The range is 1-8, and the default value is 1.


Concurrency depends on the partition of the Topic and the Instance of the flow. Partition is the smallest storage unit that holds the records. The number of partitions is specified when creating a topic. For more information on flow instance, see Allocating Runtime Resource.

If you allocate 1 instance for the flow, it is recommended that concurrency does not exceed the number of partitions. If you allocate multiple instances for the flow, it is recommended that concurrency × instance does not exceed the number of partitions.

  • When concurrency × instance ＜ partition, one process will process data from multiple partitions.

  • When concurrency × instance = partition, one process will process data from one partitions, which is in the best performance.

  • When concurrency × instance ＞ partition, some processes will have no data to process, which causes waste of resources.


Description

The description for the node.

Limitations

  • Number of connections: 1

  • Maximum number of topics: 5

  • Maximum number of concurrencies: 8

  • As the Kafka server is not managed by EnOS, connection problems could occur due to server downtime or other instances that are not within our control.

Samples

Input Sample

This node does not receive any input message.

Output Sample

Example of a JSON object response retrieved from the Kafka topic. Its topic information is stored in the metadata.

{
    "MetaData":{
        "topic":"topic1",
        "topic":"topic2"
    },
    "Body":{
        "externalId":"externalId",
        "timestamp":24214324324,
        "measurepoints":{
            "speed":32,
            "heat":40
        }
    }
}