Data Management and Analytics


Note

Features marked with “(Preview)” are selective opt-in features that are ready for you to test and evaluate but not recommended for production use. Contact your sales representative if you are interested in a Preview feature.


In EnOS 2.4, we reclassified the product menu. Data Management and Analytics is divided into Data Integration, Data Management, and Data Analysis. We added new features for some products/modules, such as Device Data Integration Service, Data Subscription, Data Archiving, and Data Catalog.

Data Integration


In EnOS 2.4, we added a new Data Integration category which includes Device Data Integration Service, Data Source Registration, and Data Synchronization. And we added some new features for Device Data Integration Service.

Device Data Integration Service


The new features with Device Data Integration Service in EnOS 2.4 are as per the below.

  • General

    Added Data Integration Resource to view and manage the resources needed in Device Data Integration Service in EnOS Resource Management.

  • Connection Configurations

    Added S3 connection types in Connection Configurations to configure connections with AWS S3 servers.

  • Server Configurations

    Added the ability to configure Server Configurations instances, which ranges from 1 to 4.

  • Flow Designing

    • Added the ability of filtering flows by publish status, including To be published, Publishing, Running, Publish failed, Unpublished, and Abnormal.

    • Added an authentication step before creating a custom node. You need to enter a verification code received by the OU administrator to upload the zip file.

  • Flow Monitoring

    • Added Trace ID for each flow log and node log. You can use the trace ID in the node logs to search for the log that corresponds to the flow log.

    • Added Runtime for each flow log and node log to record the time that the flow/node processes each message.

    • Added the node change log in Audit Service to record the adding, editing, and deleting of nodes.

    • Added a new alert rule for cloud flows: Flow is idle for too long, which will trigger the alert when the flow does not have any data input/output in the specified period.

    • Added details for Failed Runs in Dashboard. Click Failed Runs to view specified/all failed cloud flows’ details.


      ../_images/dis_failed_details.gif

Nodes


In EnOS 2.4, we added 3 new nodes, and added some new features for some of the existing nodes. See the below for more details.

New Nodes


Node Type: Input

Node

Description

S3 File

Auto polls the bucket of an S3 server to download files.


Node Type: Logic

Node

Description

Split End

Marks the end of the process between Split node and this node, and merges and outputs all the processed messages to downstream.


Node Type: IoT Hub

Node

Description

Open API

Accesses resources within EnOS based on EnOS Open APIs.
Existing Nodes


Node

What’s New

Update Device Status

Added the option to update the Offline status of a device.

MQTT Sub

Added the highest Quality of Service level to subscribe messages from the MQTT server. Added Concurrency to improve processing speed.

Inject

Added the option to trigger the flow using a CSV file.

Timer

Added the option to set the timer interval in seconds.

Store Context

Added max limit 864000 seconds for TTL and max 20 key-value pairs in Key&Value.

Kafka Sub

Added Concurrency to run multiple processes concurrently and improve processing speed.

Subflow

Added the ability to duplicate, export, and import the flow with a subflow node.

File

Added the ability to read multiple files, and view and download the files saved by File node. Added the ability to enable appending text to an existing file.

Data Management


In EnOS 2.4, we added a new Data Management category which includes Data Catalog, Data Federation, Stream Processing, Batch Data Processing, Time Series Data Management, Data Subscription, and Data Archiving. And we added new features for Data Subscription, Data Archiving, and Data Catalog.

Data Subscription


For real-time data subscription, it now supports subscription to real-time data of Data Model, which enables you to subscribe to data based on tags. For more information, see Developing Data Subscription Jobs.

Data Archiving


For archiving source data types, both real-time and offline message channels now support archiving of Data Model. For data collected from devices, data processed by stream processing, and historical data under a specified tag, files can be generated in a custom format and stored in HDFS or an external Blob Storage. For more information, see Creating a Data Archiving Job.

Data Catalog


  • Metrics

    • Data Catalog supports viewing the basic information of metrics, including IDs, names, metric types, units, business objects, subjects, metric libraries, data sources, data types, and update information.


      ../_images/data_catalog_metric.png


    • Data Catalog supports viewing the tag groups and tags associated with the metrics.

    • Data Catalog supports viewing the audit logs of metrics.

  • MysqlTable

    • Data Catalog supports viewing the basic information of MysqlTables, including IDs, names, descriptions, data source names, data source descriptions, database names, and update information.


      ../_images/data_catalog_mysql.png


    • Data Catalog supports viewing the columns of MysqlTables, including column names, data types, descriptions, and checking if the objects have primary keys and foreign keys and if they are non-null.

    • Data Catalog supports viewing the tag groups and tags associated with MysqlTables.

    • Data Catalog supports to preview the data of MysqlTables.

    • Data Catalog supports viewing the audit logs of MysqlTables.

  • Object Types

    • Data Catalog supports creating the masterdata object types by type IDs, attributes, keys, and check rules for fields checking and editing, deleting, and viewing the masterdata object types.


      ../_images/data_catalog_object.png


    • Data Catalog supports creating the relationship between two masterdata object types. The relationships include one-to-one, one-to-many, many-to-one, and many-to-many and editing, deleting, and viewing the relationships between masterdata object types.

    • Data Catalog supports editing, deleting, and viewing the masterdata objects created in Object Type page.

    • Data Catalog supports Supports editing, deleting, and viewing the masterdata objects created in Object Type page.

    • Data Catalog supports viewing the lists of masterdata object types and masterdata object relationships.

  • Data Catalog supports creating glossaries by the standard attribute template and the generic terminology template.


For more information, see Data Catalog.

Data Analytics


In EnOS 2.4, we added new features for Dataset Management, AI Lab, AI Hub, AI Pipelines, Resource Configuration, and AI Analytics Suite (Preview).

Dataset Management


Dataset Management includes the following new sample datasets and features:

  • Added sample datasets for you to quickly get started on Load Forecasting and Price Forecasting in AI Analytics Suite.

  • You can now enter variables in the SQL query statement to get dynamic data when creating datasets based on MySQL or Hive data sources. For more information, see Creating Datasets from MySQL or Hive Data Sources.

AI Lab


AI Lab includes the following new features:

  • You can now view the actual consumed CPU and memory resources for Notebook instances.


    ../_images/notebook_resource_chart.png


  • You can now run or stop Notebook instances.


    ../_images/notebook_start.png


  • Notebook instances can now restart automatically to apply the saved changes.


For more information about Notebook instance management, see Managing Notebook Instances.

AI Hub


In EnOS 2.4, you can:

  • View logs of a model deployment instance, and filter the logs by time or ES query statements. For more information about deployment instance logs, see Viewing Deployment Instance Logs.


    ../_images/hub_version_log.png


  • Use Pre-Process to adjust incompatible model input formats and API Call Logging to record API call logs when deploying a model version. For more information about Pre-Process and API Call Logging, see Deploying a Model Version.


    ../_images/pre_post_process.png


  • Deactivate a deployed instance to release occupied resources in the deployment instance list. For more information, see Deactivating Deployment Instances.

AI Pipelines


In EnOS 2.4, we added new operators and a quick start section for AI Pipelines.

Operators


The following new operators are added:

  • Process Operators

    • ShellCode operator to process Shell codes

    • PythonCode operator to process Python codes

    • EnOS Mail operator to send notifications by mail

    • EnOS SMS operator to send notifications by SMS

  • Model Operators

    • Model Version Info operator to get model version information in multi-deployment scenarios

    • Model Log operator to get model logs by time or key words

Quick Start


You can now select i_qs on the Pipeline Design page to explore and learn pipeline canvas. The Quick Start includes:

  • A beginner’s tour to walk you through pipeline canvas

  • A documentation link on how to design pipelines

  • A learn more section to illustrate some advanced functions


../_images/pipeline_quick_start.gif

Resource Configuration


In EnOS 2.4, you can:

  • Create a data source connection to EnOS File Storage HDFS and Data Warehouse Storage resources.

  • Create external data source in API and APIM types.


For more information about data source connection, see Configuring Data Source Connections.

Tutorials


In EnOS 2.4, we added the following tutorials with corresponding sample data, codes, and pipelines to help you get familiar with AI Studio:

AI Analytics Suite (Preview)


EnOS 2.4 introduces Fault Diagnosis and new features for Price Forecasting in AI Analytics Suite.

Fault Diagnosis


Fault Diagnosis provides an out-of-the-box algorithm framework that can build models to detect and identify fault types based on equipment time series data. Labels can be assigned to unlabeled data using the preset clustering algorithms and the diagnosing models can learn from user modifications on the labels to improve the accuracy.

Price Forecasting


In Price Forecasting, you can:

  • Train models to forecast prices by configuring forecasting tasks, dataset fields, and training tasks.

  • Create a schedule to automate model training tasks.

  • Call service-level or application-level forecasting services in RESTful API style.