Data Gap Backfill

Edge scenarios often face disconnection and weak network problems, which will cause communication obstruction, data backlog or loss. EnOS Edge supports Data Gap Backfill in various scenarios, that is, Edge caches data in a network disconnection and continues to transmit cached historical data after communication resumes. Taking the data gap backfill between EnOS Edge and EnOS cloud as an example, data gap backfill can avoid data loss caused by communication failures, and ensure that applications in EnOS cloud can calculate and analyze historical data over a period of time. The data processing method is shown in the following figure.

../_images/backfill.png

Application

It supports data gap backfill between the following systems:

  • EnOS Edge - EnOS Cloud

  • EnOS Edge - EnOS Edge

  • Edge SCADA - EnOS Edge

  • Third Party SCADA - EnOS Edge

  • EnOS Edge - Third Party SCADA

Features

EnOS Edge Data Gap Backfill has the following features:

  • Provide measurement tags to process historical data ( discard, re-filled through real-time topic or re-filled through historical topic)

  • Provide data compression function to restore more data

  • Support data feed strategy to insure real-time data priority

Configuration Items

The default values of configuration itmes for data gap backfill can meet the requirements of common scenarios. If there are additional usgae requirements, the following itmes can be changed during the deployment process.

Item

Description

Default Value

config-manager.box.data.history.switch

Whether to enable history channel (enabled by default after v2.3.0)

True

config-manager.box.break.point.realtime.data.topic.capacity

The capacity of real-time Topic is used to set the storage capacity

100 GB

config-manager.box.break.point.history.data.topic.capacity

The capacity of historical Topic is used to set the storage capacity

100 GB

config-manager.box.break.point.topic.timeout

The maximum cache time (historical and real-time data retention time when resuming a breakpoint）is used to set the storage capacity

720 hours

Note

It is recommended to set the storage capacity according to your actual situations. The upper limit of the actual storage Topic capacity takes the smallest value among the Topic capacity, disk size, and the maximum cache time.