Program Operators¶
The MI Pipelines provides the following operators related to task processing:
- Notebook Operator
- Python Operator
- Shell Operator
- Email Operator
- Notebookex Operator
- Pythonex Operator
- Shellex Operator
- Pipeline Trigger Operator
- APIM Operator
- ParallelFor Status List Operator
Notebook Operator¶
The Notebook operator is often used to process ipynb-type tasks that have been verified and saved in Notebook. The Notebook operator is often used in combination with the Git Directory operator. Usually, the developed model code files will be saved onto Git, and the Notebook operator can be used to get the code files from the Git Directory operator and run them. The typical scenario is to run Python tasks, run Python code files, and train machine learning models, and the generated model files will be recorded and exported through the logmodel method of MLflow.
The input and output parameters for the Notebook operator can be configured or sorted dynamically based on business needs.
Input Parameters Description¶
The following table shows the commonly used input parameters.
| Name | Required/Optional | Type | Description |
|---|---|---|---|
| workspace | Required | Directory | Specify the file directory where the code is located, which usually comes from the directory specified by the Git Directory operator. |
| entrypoint | Required | String | Specify the name of the entry program file, which should include the path (because there may be files with the same name in different directories). |
| requirements_file_path | Optional | String | Specify the file path of the dependent package to be installed. |
| env | Optional | List | Specify the list of parameters to be passed. |
Output Parameters Description¶
The following table shows the commonly used output parameters.
| Name | Type | Description |
|---|---|---|
| mlflow_model_file_paths | List | List of model file paths recorded and exported by MLflow’s logmodel method. |
Python Operator¶
The Python operator is used to handle Python script tasks and often used in combination with the Git Directory operator. The parameters of the Python operator are divided into fixed parameters and dynamic parameters. The fixed parameters cannot be deleted, while the dynamic parameters can be added, modified, deleted, or sorted based on your needs.
Input Parameters Description¶
The table below lists the fixed parameters of the Python operator.
| Name | Required/Optional | Type | Description |
|---|---|---|---|
| workspace | Required | Directory | Specify the file directory where the code is located, which usually comes from the Git Directory operator. |
| entrypoint | Required | String | Specify the name of the entry program file, which should include the path. |
| requirements_file_path | Optional | String | Specify the file path of the dependent package to be installed. |
Output Parameters Description¶
The Python operator does not have fixed output parameters, you can add parameters based on your needs.
Shell Operator¶
The Shell operator is used to process Shell script tasks, and its input and output parameters are the same as those of Python operator. See Python operator descriptions.
Email Operator¶
The Email operator is used to send email notifications.
Input Parameters Description¶
| Name | Required/Optional | Type | Description |
|---|---|---|---|
| mail_host | Required | String | IP address or domain name of the email server; e.g. smtp.163.com and smtp.office365.com. |
| mail_user | Required | String | Email service user name. |
| mail_pass | Optional | Password | Password corresponding to the user name. |
| sender | Required | String | Sender. |
| receivers | Required | List | Recipient list, which can be derived from the user list in the organization. |
| content | Required | String | Content of the sent email. |
| subject | Required | String | Subject of the sent email. |
| on_condition | Optional | Run_status | When the specified value is succeed, completed, or failed, this operator can be used as an exit operator. Once the running status of the pipeline matches with the specified value, the email sending will be triggered. |
Output Parameters Description¶
| Name | Type | Description |
|---|---|---|
| status | String | Email sending status. |
| content_out | String | Email content. |
NotebookEx Operator¶
The NotebookEx operator is often used to process ipynb-type tasks that have been verified and saved in Notebook. The Notebook operator is used to get the developed model code files form the internal storage and run them. The typical scenario is to run Python tasks, run Python code files, and train machine learning models, and the generated model files will be recorded and exported through the logmodel method of MLflow. For more information about uploading code files to the internal storage, see Uploading Model Code Files to the Internal Storage.
The input and output parameters for the NotebookEx operator can be configured or sorted dynamically based on business needs.
Input Parameters Description¶
The following table shows the commonly used input parameters.
| Name | Required/Optional | Type | Description |
|---|---|---|---|
| workspace | Required | notebook_dir | Specify the file directory. |
| entrypoint | Required | notebook_file | Specify the name of the entry program file, which should include the path (because there may be files with the same name in different directories). |
| requirements | Optional | notebook_file | Specify the dependent package to be installed. |
| env | Optional | List | Specify the list of parameters to be passed. |
Output Parameters Description¶
The following table shows the commonly used output parameters.
| Name | Type | Description |
|---|---|---|
| mlflow_model_file_paths | List | List of model file paths recorded and exported by MLflow’s logmodel method. |
PythonEx Operator¶
The PythonEx operator is used to handle Python script tasks that are saved in the internal storage. The parameters of the Python operator are divided into fixed parameters and dynamic parameters. The fixed parameters cannot be deleted, while the dynamic parameters can be added, modified, deleted, or sorted based on your needs.
Input Parameters Description¶
The following table shows the commonly used input parameters.
| Name | Required/Optional | Type | Description |
|---|---|---|---|
| workspace | Required | notebook_dir | Specify the file directory. |
| entrypoint | Required | notebook_file | Specify the name of the entry program file, which should include the path (because there may be files with the same name in different directories). |
| requirements | Optional | notebook_file | Specify the dependent package to be installed. |
Output Parameters Description¶
PythonEx operator does not have fixed output parameters, you can add parameters based on your needs.
ShellEx Operator¶
ShellEx operator is used to process Shell script tasks stored at the internal storage. Its input and output parameters are the same as those of PythonEx operator. See PythonEx operator descriptions.0
Pipeline Trigger Operator¶
Pipeline Trigger operator can only be used as an OnExit operator to trigger another pipeline under the same OU.
Input Parameters Description¶
| Name | Required/Optional | Type | Description |
|---|---|---|---|
| experiment | Required | pipeline_experiment | Select the pipeline to be triggered. |
Output Parameters Description¶
| Name | Type | Description |
|---|---|---|
| pipeline_run_id | String | The instance name of the triggered pipeline. |
| pipeline_run_info | String | The running information of the triggered pipeline. |
APIM Operator¶
APIM operator is used to call a specified API from APIM.
Input Parameters Description¶
| Name | Required/Optional | Type | Description |
|---|---|---|---|
| url | Required | String | Enter the address of the API to be called. |
| access_key | Required | String | Enter the AccessKey of the API. |
| secret_key | Required | password | Enter the SecretKey of the API. The SecretKey is hidden when you go to the pipeline design page and view this operator again after you enter or modify the value and save the pipeline. |
| http_method | Required | http_method | Select an HTTP method. Values include GET, POST, PUT, and DELETE. |
| body | Optional | String | Enter the request body. |
| headers | Optional | String | Enter the request header. |
Output Parameters Description¶
| Name | Type | Description |
|---|---|---|
| result | File | Results of the API call. |
ParallelFor Status List Operator¶
ParallelFor Status List operator is used to get results for each item of a ParallelFor operator.
Input Parameters Description¶
| Name | Required/Optional | Type | Description |
|---|---|---|---|
| run_id | Required | String | Enter the instance name, you can use instances under this OU. |
| parallelfor_path | Required | String | Enter the path of the ParallelFor operator. |
Output Parameters Description¶
| Name | Type | Description |
|---|---|---|
| result | List | Result information |