Enterprise Data Platform¶
Note
Features marked with “(Preview)” are selective opt-in features that are ready for you to test and evaluate but not recommended for production use. Contact your sales representative if you are interested in a Preview feature.
In EnOS 2.3 Release, we reclassified product menu of Enterprise Data Platform, and added new functions and features for Data Catalog, Data Federation,Time Series Data Management, Stream Processing, and Batch Processing. Detailed information is as follows:
Data Catalog¶
In EnOS 2.3 Release, we revamped Data Catalog service to better meets the needs of data governance.
Data Catalog connects data sources, data processing, data storage, data services, and data applications, and records the entire process of data from generation to consumption. In this process, Data Catalog will involve the management of business metadata, operational metadata, technical metadata, master data, etc.. Data Catalog provides collaboration capabilities of dealing with core data assets inside and outside the enterprise for the company’s data governance team, data scientists, data analysts, data developers, and application developers.
For more information, see Data Catalog.
Data Modeling¶
Supports standardized modeling of metadata objects, master data objects, and object relationships, and has built-in multiple data models.
Data Synchronization¶
Supports multiple meta/master data synchronization methods of synchronizing the data from the access side and other data sources, and supports mapping configuration and data quality verification during synchronization.
Data Asset Explorer¶
Supports data searching based on type, type attributes, tags, etc..
Tags/Glossary Management¶
Use tags/terms to help data users better understand and search data. Supports group management of tags/terms, and supports manual and automatic association of tags and terms to data objects.
Data Analysis¶
Supports viewing the relationship between data assets to help data analysis.
Stream Processing¶
Operator Library¶
EnOS Stream Processing Service provides multiple basic calculators for stream data processing, enabling developers to compose customized stream data processing jobs to meet the requirements of complex business scenarios. This library contains all the calculators in the 0.3.0 Library and provides the following features:
- Supports new format of data forwarding.
- Added calculator that supports new format (Functions are the same as the format of previous version).
- Supports tagging measurement points and assets (The information of tags is sourced from Data Catalog).
- Added data filter calculator, and supports renaming data after filtering by expression.
Advanced Stream Processing Pipelines¶
The Stream Processing service added support for advanced stream processing pipelines.
A general stream processing pipeline will consume all the messages in Kafka Internal Topic. Then, the Point Selector operator filters data records that are needed by the pipeline. The stream processing pipeline will consume much more computing resources.
For advanced stream processing pipelines, the system pipeline outputs data records that are needed by the pipelines to different Kafka topics. The corresponding data input Kafka topic is selected in the configuration of downstream stream processing pipeline, for saving computing resources efficiently.
For more information, see Developing Advanced Pipelines.
Custom Operator Library¶
Streaming Processing Service added support for custom operator library, which allows data developers to install and use custom operator library to develop customized streaming data processing tasks.
For more information, see Installing a Custom Operator Library.
Time Series Data Management¶
Storing Last Changed Record Data¶
The Time Series Data Management service added support for configuring storage policy for the last changed record data of measurement points, which can be used for determining the device connection status and duration of unchanged data.
For more information, see Storing Last Changed Record Data.
Deleting Last Changed Record Data¶
The Time Series Data Management service added support for deleting the last changed record data that is stored in TSDB to release storage resource and reduce data storage costs.
For more information, see Deleting Last Changed Record Data.
Unformatted Data Storage and Data Insights¶
For the raw data sent directly from the device side, some of them are non-core, without model and measurement point definition, and cannot be stored in the various data buckets of TSDB, but these data need to be stored, The Time Series Data Management Service added support for unformatted data storage policy with special buckets to store this unformatted data to support later analysis or standardization.
For more information, see Configuring Generic Data Storage Type and Start Unformatted Data Data Insights.
Data Federation¶
Optimized Channel Type and Channel Creating Process¶
When creating a data federation channel, you can choose more channel types (READ, WRITE, or DOWNLOAD) and also complete the channel authorization directly.
For more information, see Creating a Channel.
Query and Download data across OUs¶
Cross-OU access to HDFS and HIVE data sources is now supported. If there are other data sources under this data source category that have been licensed to the OU, all the data sources under the licensed OU can be query.
For more information, see Querying Data Online.
View and Download Channel Running Log¶
Support users to view and download logs when the channel is starting, running, or in an abnormal state including the containerName、k8sLog、podName and podLog information.
For more information, see Viewing Channel Log.
Batch Processing¶
Script Development¶
The Batch Processing service added support for the Shell script development. You can edit and debug Shell scripts online.
For more information, see Script Development.
Workflow Operation Optimization¶
We optimized the following features of workflow operation:
- Support for querying workflows and workflow instances by owners.
- Support for querying workflow instances by planned running date or by creating date.
- Support to view workflow instances within one year of the creation time of the instance.
Data Source Registration¶
Added ClickHouse Data Sources¶
The Data Source Registration service added support for registering ClickHouse data sources, for synchronizing data from EnOS Hive to external ClickHouse data sources.
For more information, see Registering a ClickHouse Data Source.
Added Open API Data Sources¶
The Data Source Registration service added support for registering Open API data sources to support accessing data by Open API.
For more information, see Registering an Open API Data Source。
Blob source adds SAS Token authentication method¶
The Data Source Registration service added support for authenticating Azure Blob Data Source by SAS Token.
For more information, see Registering an Azure Blob Data Source.
Data Synchronization¶
- The Data Synchronization service added support for synchronizing data from EnOS Hive to external ClickHouse data sources.
For more information, see Synchronizing Data From EnOS Hive to the Target Database.
- When exporting the configuration of data synchronization tasks, if the task configuration is not completed, the data synchronization configuration cannot be exported.
For more information, see Exporting Task Configuration.
Data Subscription¶
Subscribing to Device Event Reporting¶
The Data Subscription service added support for Device Event Reporting Subscription. Added subscription to device event data, includes selection of event type（Info, Warning or Error）by device model through model filtering.
For more information, see Creating a Subscription Job.
Data Sandbox¶
Data Sandbox services will be removed in EnOS version 2.3. When the product is upgraded to EnOS 2.3, sandboxes are taken offline, resources are released, and cannot be restored.