Enterprise Data Platform¶
In EnOS 2.1 Update Release, we renamed the product and its related services, and added two new services (Data Federation and Data Asset Permission), a new version for Stream Processing, and new functions and features for Data Asset Catalog, Time Series Data Management, Data Subscription, Data Archiving, and Data Source Registration.
Product/Service Renaming¶
Starting from EnOS 2.1 Update Release, the Data Asset Management is renamed to Enterprise Data Platform, and its related services are also renamed.
| Before | After |
|---|---|
| Data Catalog | Data Asset Catalog |
| Data Cleanup | Data Deletion |
| Data Integration | Data Synchronization |
| Data IDE | Batch Processing |
| Data Explorer | Data Sandbox |
| Data Connection | Data Source Registration |
| Data Asset Permission | Data Asset Authorization |
Data Asset Catalog (Preview)¶
You can now query and manage data asset metadata to achieve the quick querying of the measurement points, assets, Hive tables, services, filesets, and DTV service metadata. You can also use tag templates to label data assets to improve the efficiency of data asset management.
For more information, see Data Asset Catalog.
Data Federation (Preview)¶
The Data Federation service is a new service which enables application developers and data analysts to query data from or write data to multi-source heterogeneous data storage systems through unified SQL query. The service also supports OData and JDBC connections where you can quickly connect to mainstream BI analysis tools such as tableau, Excel, and PowerBI without worrying about the differences in the source data storage systems.
For more information, see Data Federation.
Stream Processing (Preview)¶
The new version of the Stream Processing service enables you to install the required StreamSets libraries and algorithm templates, design and create stream data processing jobs via a single process, and run stream data processing jobs by standalonemode.
Managing StreamSets Libraries and Algorithm Templates¶
You can now install and uninstall StreamSets libraries and algorithm templates based on your business needs. Before installing the libraries, ensure that your organization has requested for the Stream Designing resource via Resource Management in the EnOS Management Console.
Developing Stream Data Processing Jobs¶
The designing and developing of stream data processing jobs is now a unified process. By using the Stream Development feature, you can use the pre-installed StreamSets libraries or algorithm templates to design and develop stream data processing jobs easily.
Supporting the Standalone Mode¶
You can now run stream data processing jobs via standalone mode by using the Stream Operation feature. Before configuring the running resource for your jobs, ensure that your organization has requested the Standalone Processing resource via Resource Management in the EnOS Management Console.
Advanced Parameter Configuration in Stream Operation¶
Added support of configuring advanced parameters for running stream processing jobs by Cluster Mode. With the advanced features, you can better control the running efficiency of stream processing jobs.
For more information, see Stream Data Processing.
Time Series Data Management¶
Storage Policy Import and Export¶
You can now import and export TSDB storage policies to speed up the configuration process. When you need to migrate TSDB storage policy configuration across environments or across OUs, the existing storage policy configuration can be exported and imported into the new environment or OU for rapid configuration and deployment.
For more information, see Storage Policy.
New Version of TSDB Data Service API¶
Starting from EnOS 2.1 Update Release, the TSDB Data Service APIs are upgraded to V2.1 (the V2.0 version is still in service) with the following additions and changes.
- New request parameter
localTimeFormat: For specifying whether the returned local time value includes the time zone information of the devices. - New request parameter
itemFormat: For specifying the display format of the returned device data. - The request method of all TSDB data service APIs is now POST.
- Name changes for request and response parameters.
For more information about TSDB Data Service API V2.1, see TSDB Data Service Overview.
Data Subscription¶
Subscribing to Asset Event Data¶
You can now subscribe to asset event data. Event data is about events that happened within the EnOS IoT Hub service, such as device registration, deletion, topology relationship changes, etc. When creating a data subscription job for asset event data, you can configure the job for subscribing to the event data for models, device assets, and logic assets.
Data Subscription SDK for Python¶
You can now develop applications with Python to consume subscribed asset data or alert records.
- URL for installation: https://github.com/EnvisionIot/enos-subscription-service-sdk-python
- For the application development sample code, see Consuming Subscribed Data.
For more information, see Data Subscription.
Data Archiving¶
You can now archive historical data that is stored in TSDB. It is a one-time offline data archiving job and needs to be triggered manually.
For more information, see Archiving Data.
Data Source Registration¶
You can now query data from HDFS (EnOS) and Hive (EnOS) data sources.
- To query data from HDFS (EnOS), you need to first request for the File Storage HDFS resource. After the resource is allocated, the system will automatically register the HDFS (EnOS) data source.
- To query data from Hive (EnOS), you need to first request for the Data Warehouse Storage resource. After the resource is allocated, the system will automatically register the Hive (EnOS) data source.
For more information, see Data Source Registration.
Data Asset Permission (Preview)¶
Data asset permission is an important means of data governance which provides comprehensive security for the data asset management system.
In EnOS 2.1 Update Release, the Data Asset Permission service is added to assign HDFS file access permissions to users of other organizations by defining the permission policy, which achieves cross-organization access to HDFS files.
For more information, see Data Asset Authorization (Preview).