Unit 4. Designing a Pipeline¶
In this tutorial, you can predict the temperature rises of wind turbine engines according to the following logic.
This unit describes how to design a pipeline to predict the temperature rises of wind turbine engines. You can choose one of the following options to design a pipeline:
Option 1: design a pipeline based on a sample pipeline to get started quickly.
Option 2: design a pipeline from scratch to get familiar with the overall functions of AI Pipelines.
For more information on operators used in this tutorial, see Operator Reference.
Option 1. Designing a Pipeline Based on a Sample Pipeline¶
If you want to get a quick start on performing predicting tasks with operators in AI Pipelines, you can design a pipeline by importing the built-in sample pipeline for predicting the temperature rises of wind turbine engines.
Step 1. Exporting the Sample Pipeline¶
To design your own pipeline based on a sample pipeline, export the sample pipeline first by the following steps:
Step 2. Importing the Sample Pipeline¶
To reuse the sample pipeline information such as pipeline structure, operator, and global parameters, import the exported JSON files by the following steps:
Select AI Studio > AI Pipelines from the left navigation pane.
Select New Experiment on the Custom Pipeline tab.
Enter
temp-rise
as the name of the experiment in the pop-up window.Select OK to create the experiment, and you can see the canvas for designing a pipeline.
Select Import to import the exported sample pipeline configuration.
Step 3. Configuring Global Parameters¶
For a more efficient and simpler pipeline design process, you can set globally applicable parameters as global parameters to avoid repetitive configuration. The global parameters used in this tutorial are included in the sample pipeline configuration file, select Workflow setting to configure the global parameters.
Number |
Parameter |
Description |
---|---|---|
1 |
tasktype |
|
2 |
model |
Specify the model name to specify the model for the training or predicting task. |
3 |
instance |
Specify the instance name of the model specified in model. |
4 |
resourcepool |
Select the resource pool used for model deployment. |
5 |
enable_alert_email |
|
6 |
mysql_source |
Specify the MySQL data source to store the predicting results. |
7 |
Threshold |
Specify the temperature rise threshold. If the temperature rise exceeds the threshold, an alert email will be sent. |
8 |
prediction_model |
Specify the model to perform predicting tasks. |
9 |
prediction_instance |
Specify the instance of the model specified in prediction_model to perform predicting tasks. |
10 |
dataset_name |
Specify the name of the dataset for predicting and training tasks. |
11 |
sample_ratio |
Specify the ratio to split the dataset specified in dataset_name into a training set and a predicting set. |
12 |
receivers |
Select the user to receive alert notification email. |
13 |
copyto |
Select the user to send the copied alert notification email. |
14 |
locale |
|
15 |
incident |
Enter the texts for the alert notification email. |
16 |
priority |
Enter the priority of the alert notification email. |
Step 4. Viewing Operators in the Main Canvas¶
You can find 2 Condition operators in the main canvas:
Do Train performs training tasks when the tasktype parameter is
train
.Do Predict performs predicting tasks when the tasktype parameter is
perdict
.
Step 5. Viewing Operators for Training Tasks¶
Select the Expand icon of the Do Train operator to open its sub-canvas. You can find the following operators.
Operator Name |
Description |
---|---|
NotebookEx |
|
Model |
This Model operator is used to create a model. |
Mlflow Model Version Register |
This Mlflow Model Version Register operator is used to register an Mlflow model version for a specified model. |
Model Instance |
This Model Instance operator is used to create a model deployment instance. |
Single Model Deployment |
This Single Model Deployment operator is used to deploy a single model version based on the model version registered in the Mlflow Model Version Register operator. |
Step 6. Viewing Operators for Predicting Tasks¶
Select the Expand of the Do Predict operator to open its sub-canvas. You can find the following operators.
Operator Name |
Description |
---|---|
Get History Data to Predict the Future |
|
Predict |
This Service Prediction operator is used to perform predicting tasks with the trained model and deliver predicting results as an output. |
Process Results |
|
Check MySQL Config |
This PythonCode operator is used to check if a MySQL data source is available in the current OU to save the predicting results. |
Condition for Training Alert |
This Condition operator is used to send alert notification emails to the specified users if the enable_alert_email global parameter is true. Its sub-canvas includes an EnOS Mail operator to send alert notification emails to the specified users. |
Export to MySQL? |
|
Option 2. Designing a Pipeline from Scratch¶
You can also design a low-code pipeline with operators from scratch to predict the temperature rises of wind turbine engines and upload the prediction results into MySQL data source. For more information about the operators you use, see Operator Reference.
Step 1. Creating an Experiment¶
Select AI Studio > AI Pipelines from the left navigation pane.
Select New Experiment on the Custom Pipeline tab.
Enter
temp-rise
as the name of the experiment in the popup window.Select OK to create the experiment, and you can see the canvas for designing a pipeline.
Step 2. Adding Global Parameters¶
For a more efficient and simpler pipeline design process, you can set globally applicable parameters as global parameters to avoid repetitive configuration by the following steps:
On the power-loss pipeline canvas, select Workflow Setting to open the Workflow Setting panel.
Select Add Parameter to add the following global parameters in the Configuration Parameters section.
Number |
Parameter |
Type |
Description |
---|---|---|---|
1 |
tasktype |
string |
Specify the tasks type of the pipeline. Enter predict to run predicting tasks or train to run training tasks. |
2 |
model |
string |
Specify the model name to specify the model for the training or predicting task. |
3 |
instance |
string |
Specify the instance name of the model specified in model. |
4 |
resourcepool |
resourcepool |
Select the resource pool used for model deployment. |
5 |
enable_alert_email |
boolean |
Select whether to send alert notification by email. Enter true to send alert notification by email or False not to send the email. |
6 |
mysql_source |
mysql_source |
Specify the MySQL data source to store the predicting results. |
7 |
Threshold |
number |
Specify the temperature rise threshold. If the temperature rise exceeds the threshold, an alert email will be sent. |
8 |
prediction_model |
model_name |
Specify the model to perform predicting tasks. |
9 |
prediction_instance |
model_instance |
Specify the instance of the model specified in prediction_model to perform predicting tasks. |
10 |
dataset_name |
string |
Specify the name of the dataset for predicting and training tasks. |
11 |
sample_ratio |
number |
Specify the ratio to split the dataset specified in dataset_name into a training set and a predicting set. |
12 |
receivers |
user_contact_list |
Select the user to receive alert notification email. |
13 |
copyto |
user_contact_list |
Select the user to send the copied alert notification email. |
14 |
locale |
locale |
Select the language of the alert notification email. Select en-US for English or zh_CN for simplified Chinese. |
15 |
incident |
string |
Enter the texts for the alert notification email. |
16 |
priority |
string |
Enter the priority of the alert notification email. |
Step 3. Configuring Operators in Main Canvas¶
You need to add two Condition operators in the main canvas:
A Condition operator performing the training tasks when the tasktype parameter is
train
.A Condition operator performing the predicting tasks when the tasktype parameter is
perdict
.
Adding a Condition Operator for Training Tasks¶
Drag a Condition operator to the canvas from the operator list on the left.
Select the operator to configure the following operator information on the right panel:
Enter
Do Train
as Name in the Basic Info section.Configure
Reference
|tasktype
|==
|Declaration
|train
as Expression in the Configuration Parameter section.
Adding a Condition Operator for Predicting Tasks¶
Drag a Condition operator to the canvas from the operator list on the left.
Select the operator to configure the following operator information on the right panel:
Enter
Do Predict
as Name in the Basic Info section.Configure
Reference
|tasktype
|==
|Declaration
|predict
as Expression in the Configuration Parameter section.
Step 4. Configuring Operators for Training Tasks¶
The training process in this tutorial includes creating a model, training the model, creating a model instance, creating a model version, and deploying the model version. Double-select the Do Train operator to open its sub-canvas. You need to add the following 5 operators in the Do Train sub-canvas:
A NotebookEx operator to train the model based on the sample dataset and export the trained model.
A Model operator to create a model.
An Mlflow Model Version Register operator to register an Mlflow model version and export the model version.
A Model Instance operator to create a model deployment instance.
A Single Model Deployment operator to deploy a single model version based on a published model version.
Configuring a NotebookEx Operator for Model Training¶
On the Do Train sub-canvas, drag a NotebookEx operator from the operator list on the left.
Select this NotebookEx operator and configure the following operator parameters in the Input Parameter section on the right panel.
Parameter
Reference/Declaration
Description
workspace
Declaration
Select the Notebook instance folder you created in Unit3. Preparing Codes to specify the directory of the files required in the entrypoint and requirement parameters.
entrypoint
Declaration
Select
Train.ipynb
to specify the code file for model training.requirement
Declaration
Select
requirements_train.txt
to declare the required packages for model training.Select Add Parameter in the Input Parameter section for 3 times to add 3 input parameters and configure the following information.
Name
Type
Reference/Declaration
Value
Description
epochs
number
Declaration
10
Train the model with the training dataset and repeat by 10 times.
dataset
string
Reference
dataset_name
Configure the training dataset as the dataset specified in the dataset_name global parameter.
ratio
number
Reference
sample_ratio
Configure the ratio specified in the sample_ratio global parameter to split the dataset into a training set and a predicting set.
Configuring a Model Operator to Create a Model¶
On the Do Train sub-canvas, drag a Model operator from the operator list on the left.
Select this Model operator and configure the following operator parameters in the Input Parameter section on the right panel.
Parameter
Reference/Declaration
Description
category
Declaration
Select
Predictor
to create a model for predicting tasks.model_name
Reference
Select
model
to configure the model name as the value specified in the model global parameter.input_data_type
Declaration
Select
Tabular
as the input data type for the model.scope
Declaration
Select
Public
to make the model available for all AI Studio users in the current OU.technique
Declaration
Select
Regression
as the technique of the model.usecase
Declaration
Select
Wind
to explain that the model is suitable for wind-related scenarios.input_format
Declaration
Enter the codes in Model Input Format to specify the input format of the model.
interface
Declaration
Select
REST
as the API type of the model.output_format
Declaration
Enter the codes in Model Output Format to specify the output format of the model.
error_on_exist
Declaration
Select
false
to skip without reporting an error when the model name exists.
Model Input Format¶
The input format of the model you created in the Model operator should be as below.
[
{
"annotations": "",
"defaultValue": null,
"dtype": "float",
"ftype": "continuous",
"name": "gen_active_pw",
"range": [],
"repeat": 0
},
{
"annotations": "",
"defaultValue": null,
"dtype": "float",
"ftype": "continuous",
"name": "gen_speed",
"range": [],
"repeat": 0
},
{
"annotations": "",
"defaultValue": null,
"dtype": "float",
"ftype": "continuous",
"name": "torque",
"range": [],
"repeat": 0
},
{
"annotations": "",
"defaultValue": null,
"dtype": "float",
"ftype": "continuous",
"name": "tem_in",
"range": [],
"repeat": 0
},
{
"annotations": "",
"defaultValue": null,
"dtype": "float",
"ftype": "continuous",
"name": "tem_out",
"range": [],
"repeat": 0
},
{
"annotations": "",
"defaultValue": null,
"dtype": "float",
"ftype": "continuous",
"name": "wind_speed",
"range": [],
"repeat": 0
},
{
"annotations": "",
"defaultValue": null,
"dtype": "float",
"ftype": "continuous",
"name": "rotor_speed",
"range": [],
"repeat": 0
},
{
"annotations": "",
"defaultValue": null,
"dtype": "float",
"ftype": "continuous",
"name": "tem_var",
"range": [],
"repeat": 0
}
]
Model Output Format¶
The output format of the model you created in the Model operator should be as below.
[
{
"annotations": "",
"defaultValue": null,
"dtype": "float",
"ftype": "continuous",
"name": "tem_var",
"range": [],
"repeat": 0
}
]
Configuring an Mlflow Model Version Register Operator to Register a Model Version¶
On the Do Train sub-canvas, drag an Mlflow Model Version Register operator from the operator list on the left.
Connect the output ports of the Model operator and the NotebookEx operator to the input port of this Mlflow Model Version Register operator.
Select this Mlflow Model Version Register operator and configure the following operator parameters in the Input Parameter section on the right panel.
Parameter
Reference/Declaration
Description
input_data
Declaration
Enter the codes in Model Version Input Data to specify the input data of the model version.
version_rule
Declaration
Select
time
to name the model version according to the time when the operator runs.architecture
Declaration
Select
x86
as the basic hardware for running the model version.framework
Declaration
Select
tensorflow
as the calculation framework for running the model version.language
Declaration
Select
python3
as the development language of the model version.model_reference
Reference
Select
model.model_name_output
, the output of the Model operator, to specify the model that the model version belongs to.minio_paths
Reference
Select
NotebookEx.mlflow_model_file_paths
, the output of the NotebookEx operator, to specify the minio path of the model version.
Model Version Input Data¶
The input data of the model version you created in the Mlflow Model Version Register operator should be as below.
{
"data": {
"names": [
"gen_active_pw",
"gen_speed",
"torque",
"tem_in",
"tem_out",
"wind_speed",
"rotor_speed",
"tem_var"
],
"ndarray": [
[
19.646925,
963.533899,
9311.739885,
19.948558,
6.743743,
6.990860,
13.066677,
932.398765
],
[
20.020045,
1010.172848,
10268.291071,
19.698487,
6.722673,
7.189325,
13.955926,
1070.147035
]
]
}
}
Configuring a Model Instance operator to Create a Model Deployment Instance¶
On the Do Train sub-canvas, drag a Model Instance operator from the operator list on the left.
Connect the output port of the Model operator to the input port of this Model Instance operator.
Select this Model Instance operator and configure the following operator parameters in the Input Parameter section on the right panel.
Parameter
Reference/Declaration
Description
name
Reference
Select
instance
to name the model instance as the value of the instance global parameter.resource_pool
Reference
Select
resource_pool
to use the resource pool specified in the resource_pool global parameter for model instance deployment.model_name
Reference
Select
Model.model_name_output
, the output of the Model operator, to specify the model that the deployed instance belongs to.deploy_mode
Declaration
Select
ONLINE
as the deployment mode.error_on_exist
Declaration
Select
false
to skip without reporting an error when the model name exists.
Configuring a Single Model Deployment Operator to Deploy the Model Version¶
On the Do Train sub-canvas, drag a Single Model Deployment operator from the operator list on the left.
Connect the output ports of the Model Instance operator and the Mlflow Model Version Register operator to the input port of this Single Model Deployment operator.
Select this Single Model Deployment operator and configure the following operator parameters in the Input Parameter section on the right panel.
Parameter
Reference/Declaration
Description
model_version
Reference
Select
Mlflow Model Version Register.model_revision_name
, the output of the Mlflow Model Version Register operator, to specify the model version to deploy.instance_name
Reference
Select
Model Instance.instance_name_output
, the output of the Model Instance operator, to specify to deployment instance for model version deployment.request_CPU
Declaration
Enter
0.1
to specify the CPU resource required for the model version deployment.limit_CPU
Declaration
Enter
0.5
to specify the maximum CPU resource for the model version deployment.request_memory
Declaration
Enter
1.0
to specify the memory resource required for the model version deployment.limit_memory
Declaration
Enter
1.0
to specify the maximum memory resource for the model version deployment.
The Do Train sub-canvas should look as below.
Step 5. Configuring Operators for Predicting Tasks¶
The training process in this tutorial includes preparing data, predicting, processing results, storing results, and configuring alert notification. On the main canvas, double-select the Do Predict operator to open its sub-canvas. You need to add the following 6 operators on the Do Train sub-canvas:
A PythonEx operator to prepare data for predicting tasks.
A Service Prediction operator to perform predicting tasks.
A PythonEx operator to collect and process predicting results.
A PythonCode operator to check whether the MySQL data source is available to store predicting results.
A Condition operator to store the predicting results to MySQL data source if the MySQL data source is available.
A Condition operator to send alert notification.
Configuring a PythonEx operator to Prepare Predicting Data¶
On the Do Predict sub-canvas, drag a PythonEx operator from the operator list on the left.
Select this PythonEx operator and configure the following operator parameters in the Input Parameter section on the right panel.
Parameter
Reference/Declaration
Description
workspace
Declaration
Select the Notebook instance folder you created in Unit3. Preparing Codes to specify the directory of the files required in the entrypoint and requirement parameters.
entrypoint
Declaration
Select
prepare_predict_history_data.py
to specify the code file for model training.requirement
Declaration
Select
requirements_predict.txt
to declare the required packages for model training.Select Add Parameter in the Input Parameter section for 2 times to add 2 input parameters and configure the following information.
Name
Type
Reference/Declaration
Value
Description
dataset
string
Reference
dataset_name
Configure the predicting dataset as the dataset specified in the dataset_name global parameter.
ratio
number
Reference
sample_ratio
Refer to the ratio specified in the sample_ratio global parameter to split the dataset into a training set and a predicting set.
Select Add Parameter in the Output Parameter section for 2 times to add 2 output parameters and configure the following information.
Name
Type
data
file
predict_data
file
Configuring a Service Prediction Operator for Predicting Tasks¶
On the Do Predict sub-canvas, drag a Service Prediction operator from the operator list on the left.
Connect the output port of the PythonEx operator to the input port of this Service Prediction operator.
Select this Service Prediction operator and configure the following operator parameters in the Input Parameter section on the right panel.
Parameter
Reference/Declaration
Description
model
Reference
Select
prediction_model
to use the model specified in the prediction_model global parameter for predicting tasks.instance
Reference
Select
prediction_instance
to use the model instance specified in the prediction_instance global parameter for predicting tasks.namespace
Reference
Select
resourcepool
to use the resource pool specified in the resource_pool global parameter for predicting tasks.data_type
Declaration
Select
csv
as the type of the predicting data.data
Reference
Select
PythonEx.predict_data
, the output of the PythonEx operator, to use the data for predicting tasks.
Configuring a PythonEx Operator to Process Predicting Results¶
On the Do Predict sub-canvas, drag a PythonEx operator from the operator list on the left.
Connect the output ports of the first PythonEx operator and the Service Prediction operator to the input port of this PythonEx operator.
Select the PythonEx operator and configure the following operator parameters in the Input Parameter section on the right panel.
Parameter
Reference/Declaration
Description
workspace
Declaration
Select the Notebook instance folder you created in Unit3. Preparing Codes to specify the directory of the files required in the entrypoint and requirement parameters.
entrypoint
Declaration
Select
process_results.py
to specify the code file for model training.requirement
Declaration
Select
requirements_predict.txt
to declare the required packages for model training.Select Add Parameter in the Input Parameter section for 4 times to add 4 input parameters and configure the following information.
Name
Type
Reference/Declaration
Value
Description
predictions
file
Reference
Service Prediction.predictions
Get the predicting results from the Service Prediction operator.
data
file
Reference
PythonEx.data
Get the original temperature data of wind turbine engines.
threshold
number
Reference
threshold
Specify the temperature rise threshold. If the temperature rise exceeds the threshold, an alert email will be sent.
enable_alert
boolean
Reference
enable_alert_email
Enable or disable the alert notification function according to the value of the enable_alert_email global parameter.
Select Add Parameter in the Output Parameter section for 3 times to add 3 output parameters and configure the following information.
Name
Type
result
file
alert
boolean
content
string
Configuring a PythonCode Operator to Check MySQL Configuration¶
On the Do Predict sub-canvas, drag a PythonCode operator from the operator list on the left.
Select this PythonCode operator and configure the following operator parameters in the Input Parameter section on the right panel.
Parameter
Reference/Declaration
Description
code
Declaration
Enter the codes in MySQL Checking Codes to check MySQL configuration.
Select Add Parameter in the Input Parameter section to add an input parameter and configure the following information.
Name
Type
Reference/Declaration
Value
Description
mysql_source
mysql_source
Reference
mysql_source
Specify the MySQL data source to check whether it is available to store the predicting results.
Select Add Parameter in the Output Parameter section to add an output parameter and configure the following information.
Name
Type
export_to_mysql
boolean
MySQL Checking Codes¶
The codes to check whether MySQL data source is available to store predicting results should be as below.
# Sample imports
import json
import argparse
from pathlib import Path
# Define an ArgumentParser
parser = argparse.ArgumentParser()
parser.add_argument("--mysql_source", type=str, required=True)
parser.add_argument("--export_to_mysql", type=str, required=True)
# Parse arguments from command
args = parser.parse_args()
Path(args.export_to_mysql).parent.mkdir(parents=True, exist_ok=True)
with open(args.export_to_mysql, 'w') as f:
f.write('True' if args.mysql_source else 'False')
Configuring a Condition Operator to Store Results to MySQL Data Source¶
On the Do Predict sub-canvas, drag a Condition operator from the operator list on the left.
Connect the output ports of the second PythonEx operator and the PythonCode operator to the input port of this Condition operator.
Select the operator and configure the following expression for the Condition operator on the right panel.
Reference | PythonCode.export_to_mysql | == | Declaration | True
After configuring this Condition operator as above, you need to double-select the operator to open its sub-canvas and add the following 2 oeprators:
A Mysql operator to create a data table.
A Mysql Writer operator to write the forecast results to the data table created by the Mysql operator.
Configuring a Mysql Operator to Create a Data Table¶
On the Condition sub-canvas, drag a Mysql operator from the operator list on the left.
Select this Mysql operator and configure the following operator parameters in the Input Parameter section on the right panel.
Parameter
Reference/Declaration
Description
data_source_name
Reference
Select
mysql_source
to create a table in the MySQL data source specified in the mysql_source global parameter.sqls
Declaration
Enter the codes in Codes to Create a Data Table to create a data table.
Codes to Create a Data Table¶
The codes to create data table should be as below.
"CREATE TABLE if not exists predicted_demo (data_time_utc timestamp DEFAULT CURRENT_TIMESTAMP, predicted float, actual float, threshold float, is_abnormal TINYINT(1));"
Configuring a Mysql Writer Operator to Write Results into the Data Table¶
On the Condition sub-canvas, drag a Mysql Writer operator from the operator list on the left.
Select this Mysql Writer operator and configure the following operator parameters in the Input Parameter section on the right panel.
Parameter
Reference/Declaration
Description
data_source_name
Reference
Select
mysql_source
to write predicting results to the MySQL data source specified in the mysql_source global parameter.statement
Declaration
Enter the codes in MySQL Statement Codes to write resulting data into the data table created by the Mysql operator.
data
Reference
Select
PythonEX(1).result
, the output of the PythonEx operator to process results, as the data to write into the data table.
MySQL Statement Codes¶
The codes to write predicting results data into the data table should be as below.
INSERT IGNORE INTO `predicted_demo` (`data_time_utc`, `predicted`, `threshold`, `is_abnormal`) VALUES (%s, %s, %s, %s);
Configuring a Condition Operator to Determine Whether to Send Alert Emails¶
On the Do Predict sub-canvas, drag a Condition operator from the operator list on the left.
Connect the output port of the second PythonEx operator to the input port of this Condition operator on the right panel.
Select the Condition operator and configure the following expression.
Reference | PythonEx(1).alert | == | Declaration | True
After configuring this Condition operator as above, you need to double-select the operator to open its sub-canvas and add an EnOS Mail oeprator to send alert notification to EnOS users by email.
Configuring an EnOS Mail Operator to Send Alert Email¶
On the Condition(1) sub-canvas, drag an EnOS Mail operator from the operator list on the left.
Select this EnOS Mail operator and configure the following parameter information in the Input Parameter section on the right panel.
Parameter
Reference/Declaration
Description
receivers
Reference
Select
receivers
to send the alert notification mail to the emails of the users in the receivers global parameter.copyto
Reference
Select
copyto
to copy the alert notification mail to the emails of the users in the copyto global parameter.locale
Reference
Select
locale
to send the alert notification mail in the language in the locale global parameter.module
Declaration
Select
Model Hub
to explain that the alert is related to the AI Hub module.target
Reference
Select
model
to configure the object of the alert as the model in the model global parameter.incident
Reference
Select
incident
to configure the incident related to the alert as the contents in the incident global parameter.detail
Reference
Select
PythonEx(2).content
, the output of the PythonEx operator to process results, as the detailed content for the alert.priority
Reference
Select
priority
to configure the priority of the alert as the value of the priority global parameter.
After configuring the operators above, select Do Predict from breadcrumb navigation to go back to the Do Predict sub-canvas. The configured Do Predict sub-canvas should look as below.