Data Asset Management¶
In EnOS 2.1 Release, we added new functions and features to the Stream Processing, Time Series Data Management, Data Catalog, Data Subscription, and Data Archiving services. Details are as follows:
Stream Processing¶
StreamSets Calculator¶
In EnOS 2.1 Release, we added support of StreamSets Calculator Library V0.0.4, which contains operators for asset information query, history data query, data aggregation, HTTP request, running customized Python and JS scripts, and pipeline debugging. You can leverage the new calculator library to develop customized and advanced stream data processing jobs to meet the requirements of various business scenarios.
For more information, see Calculator Library 0.0.4 Documentation and Developing with StreamSets Operators Tutorial.
Templates for Electric Energy Calculation¶
In EnOS 2.1 Release, we added support of 2 stream processing templates for electric energy calculation by instant power and average power. If the power value in a specific time interval is relatively stable, use the Electric Energy Calculation by Instant Power template; if the power value in a time interval is not stable, use the Electric Energy Calculation by Average Power template. For more information, see Configuring an Electric Energy Calculation Job and Electric Energy Calculation Logic.
Time Series Data management¶
Data Cleanup¶
Added support of cleaning up the history data stored in TSDB and the latest asset data (informal data or useless business data) of specified model, measurement point, and asset for releasing storage resource and saving data storage costs.
The usage of Data Cleanup is shown as follows:
For more information, see Data Cleanup.
Downloading Queried Data in Data Insights¶
Added supporting of downloading queried data in Data Insights by simply clicking the Download Data icon next to the data chart or data list. In case of data exception, the queried data can be downloaded for further analysis offline. For more information, see Generating Time Series Data Chart.
Data Catalog¶
Added support of the Data Catalog service, for querying and managing the metadata information of data assets in your OU. You can quickly query the metadata of measuring points, assets, Hive tables, files, and services. Tag templates are provided for tagging data assets, thus improving the efficiency of data asset management.
For more information, see Data Catalog.
Data Subscription¶
Monitoring Running Statistics of Data Subscription Jobs¶
Added support of monitoring the running statistics of data subscription jobs (producer rates, consumer rates, offset, and lag) to ensure that subscribed data is consumed in time without delay. If there is delay of data consumption, you can run 2 consumer clients of the same consumer group to improve the data consumption efficiency.
The usage of the Data Subscription Job Monitoring feature is shown as follows:
Subscribing to Data from Offline Message Channel¶
We added support of subscribing to time series data from offline message channel. When analyzing key business data that is stored offline with an application, you may need to subscribe to time series data from both the real-time message channel and offline message channel. For more information, see Developing Data Subscription Jobs.
When consuming subscribed data from offline message channel, you need to use the 2.4.1 version of the Data Subscription SDK. For more information, see Consuming Subscribed Data.
Filtering Subscribed Alert Data with Asset Tree Tags¶
Support for filtering subscribed alert data with asset tree tags in configuring data subscription jobs. Alert data of the specific asset trees can be subscribed to with this feature enabled. For more information, see Developing Data Subscription Jobs.
Data Archiving¶
- In EnOS 2.1 Release, we added support of data flow limiting feature. Data archiving resource that is requested through Resource Management determines the amount of data records that can be archived per second by all the running data archiving jobs. Therefore, you need to request for different data archiving resource specification based on your business requirement.
- Added support of archiving real-time alert records, which is featured by:
- Support of archiving both history alert records and active alerts
- Support of archiving specified fields in alert records
- Added support of archiving data that is integrated from the offline message channel. When creating a data archiving job, select the Offline Message Channel as the data source.
- Added support of HDFS storage type to store archived files. To use the HDFS storage, you need to request the file storage HDFS resource through Resource Management.
For more information, see Archiving Data.