- Documentation
- Data Services
- Archive Service
Archive Service¶
As the amount of data increases, the data in the database on the Edge can be archived. Archiving can increase data storage time. Edge data archiving contains original data (AI, DI, PI, General), average data over 5 or 10 minutes and alert data archives. Supports standard data archiving format in the wind and solar domain.
..image::../media/data_archiving_en.png
Application¶
Enterprises who need economic cold data storage for data owners to get data prepared for future utilization by data users.
Features¶
Featured Capabilities:
- Provide data archiving configuration pages to customize archive strategy
- Support TSDB and alert history data archiving
- Provide API to query/search/download archive file
- Support sharing of archive file with others application
- Support deleting of expired archive file automatically
- Distribute the common archive format of the wind and solar domain
The types of data that can be archived include the following.
- Raw Data:
- Raw data received through analog input (AI)
- Raw data received through digital input (DI)
- Raw data received through pulse input
- Generic raw data
- Alerts
- Custom data:
- 5-minute telemetry data archiving
- 10-minute telemetry data archiving
Related Information¶
Raw Data Archiving¶
Archiving Configuration¶
| Archiving Service Parameter | Parameter Value |
|---|---|
| User | General |
| Archiving Start Time | Specified when the archiving is created and cannot be changed later. Starts from the day before the archiving is created by default. |
| Period | 1 day |
| Archiving Model and Measurement Point Selection | You can select the model and measurement points to be archived. |
| Path | The format of the path is: data_type/date/model_ID. For example, AI/20200202/EnOS_Solar_Inverter |
| Archived File Name | The file is named after the asset ID. |
| Encoding | UTF-8 |
| Filename Extension | .csv |
| Format | As described below. |
| Column Separator | Specified when the archiving is created and cannot be changed later. The default separator is the comma. |
| File Size | No upper limit. |
| Compressed / Not Compressed | Specified when the archiving is created and cannot be changed later. Compressed by default. |
| Compression Algorighm | GZIP |
| Merge / Do Not Merge Files of the Same Type | Does not merge by default. |
| Effective Period | The data will be deleted when the time is exceeded. 1 month by default. |
Format of AI, PI, Generic Archived Files¶
An example to illustrate the format of AI, PI, Generic archived files is as per the below, where the user decides to archive the data of the following measurement points.
| Measurement Point ID | Name (English) | Name (Chinese) |
|---|---|---|
| WTUR.GenActivePW | Reactive Power Production | 有功功率 |
| WTUR.WindSpeed | Wind Speed | 风速 |
In an archived file, the headers take up the first 3 rows as per the following.
| Asset ID | Asset Name | Data Time | Data Time UTC | WTUR.GenActivePW | WTUR.WindSpeed |
|---|---|---|---|---|---|
| Asset ID | Asset Name | Data Time | Data Time UTC | Reactive Power Production | Wind Speed |
| 资产 ID | 资产名称 | 时间 | UTC时间 | 有功功率 | 风速 |
| WindturbineA | Wind Turbine A in China | 2020-01-01 00:00:00 | 2020-01-01 00:00:00 | 123.123 | 4.444 |
In this sample, the headers are displayed using the ID, English name, and Chinese name. The first 4 columns are fixed and the measurement point data starts from the 5th column.
Measurement points are displayed in an ascending order based on their priority. You can configure the priority of a measurement point by specifying the priority in a tag. Measurement points without priority tags are placed last by default.
Format of DI Archived Files¶
An example to illustrate the format of DI archived files is as per the below, where the user decides to archive the data of the following measurement points.
| Measurement Point ID | Name (English) | Name (Chinese) |
|---|---|---|
| WTUR.STA1 | Status #1 | 状态1 |
| WTUR.STA2 | Status #2 | 状态2 |
In an archived file, the headers take up the first 3 rows as per the following.
| Asset ID | Asset Name | Data Time | Data Time UTC | Point ID | Point Value |
|---|---|---|---|---|---|
| Asset ID | Asset Name | Data Time | Data Time UTC | Point ID | Point Value |
| 资产 ID | 资产名称 | 时间 | UTC时间 | 测点ID | 数值 |
| WindturbineA | Windturbine A in China | 2020-01-01 00:00:00 | 2020-01-01 00:00:00 | WTUR.STA1 | 1 |
| WindturbineA | Windturbine A in China | 2020-01-01 00:00:00 | 2020-01-01 00:00:00 | WTUR.STA2 | 0 |
| WindturbineB | Windturbine B in China | 2020-01-01 00:00:00 | 2020-01-01 00:00:00 | WTUR.STA1 | 0 |
| WindturbineB | Windturbine B in China | 2020-01-01 00:00:00 | 2020-01-01 00:00:00 | WTUR.STA2 | 1 |
In this sample, the headers are displayed using the ID, English name, and Chinese name. The measurement point data starts from the 5th column.
The measurement points are displayed in the same manner as the AI, PI, and Generic data, i.e., in ascending order based on their priority, etc.
Alert Archiving¶
Archiving Configuration¶
| Archiving Service Parameter | Parameter Value |
|---|---|
| User | General |
| Archiving Start Time | Specified when the archiving is created and cannot be changed later. Starts from the day before the archiving is created by default. |
| Period | 1 day |
| Archiving Model and Measurement Point Selection | All models and measurement points. |
| Path | The format of the path is: data_type/date/model_ID. For example, AI/20190909, DI/20190909. |
| Archived File Name | The file is named after the asset ID. |
| Encoding | UTF-8 |
| Filename Extension | .csv |
| Format | As described below. |
| Column Separator | Specified when the archiving is created and cannot be changed later. The default separator is the comma. |
| File Size | No upper limit. |
| Compressed / Not Compressed | Specified when the archiving is created and cannot be changed later. Compressed by default. |
| Compression Algorighm | GZIP |
| Merge / Do Not Merge Files of the Same Type | Does not merge by default. |
| File Cleanup Policy | Specified when the archiving is created and cannot be changed later. 1% by default. |
Format of Archived Files¶
EnOS Edge uses the same method to archive historic alerts as EnOS Cloud.
| Archived Field | Raw Data | Note |
|---|---|---|
| eventId | eventId | The alert ID that is unique within an OU and cannot be changed. |
| orgId | orgId | The OU ID to which the asset belongs. |
| assetId | deviceId | The asset ID. |
| modelId | modelId | The model ID to which the asset belongs. |
| modelIdPath | modelIdPath | The path of this model. |
| measurepointId | pointId | The measurement point ID. |
| value | value | The measurement point value. |
| occurTime | time | The time when the alert is triggered in UTC time. |
| localOccurTime | localTime | The time when the alert is triggered in local time. A 13-digit timestamp is stored in Elasticsearch. The API provided by the alert service converts the timestamp into yyyy-mm-dd hh:mm:ss, rounded up to seconds. |
| recoverTime | recoverTime | The time when the alert is dismissed in UTC time. |
| recoverLocalTime | recoverLocalTime | The time when the alert is dismissed in local time. A 13-digit timestamp is stored in Elasticsearch. The API provided by the alert service converts the timestamp into yyyy-mm-dd hh:mm:ss, rounded up to seconds. |
| recoverReason | recoverReason | The reason why the alert is dismissed. |
| createTime | createTime | The time when the alert is archived in UTC time. |
| updateTime | updateTime | The time when the alert record is updated in UTC time. |
| severityId | severityId | The user-defined alert severity ID. |
| severityDesc | severityDesc | The description for the alert severity. |
| typeId | typeBid | The user-defined alert type ID. |
| typeDesc | typeDesc | The description for alert type. |
| subTypeId | subTypeBid | The user-defined alert sub-type ID. |
| subTypeDesc | subTypeDesc | The description for the alert sub-type. |
| contentId | contentBid | The user-defined alert content ID. |
| contentDesc | contentDesc | The description for the alert content. |
| eventType | eventType | The event type.
|
| tag | tag | The alert tag. The API provided by the alert service will convert the tag into standard JSON object [key1:v1,key2:v2]. |
| ruleDesc | ruleDesc | The description for the alert rule. |
| assetPaths | assetPath | The scope where the alert rule takes effect and the path of the device where the alert is triggered is returned. The format is [“treeId1:/assetId1/assetId2/assetIdx”, “treeId2:/assetId3/assetIdx”]. |
| maskedBy | maskedBy | If this alert is from a sub-device, the root alert that masks this alert is returned. If this alert has multiple root alerts, all the root alerts that can mask this alert is returned and an eventId is generated. The format is [“treeId1:eventId1”, “treeId1:eventId2”]. |
Note
- The timestamps for
localOccurTimeandrecoverLocalTimeare stored in MySQL. Alert service APIs will convert the timestamp to datetime-type data and archive the converted data. The format of the datetime data is yyyy-mm-dd hh:mm:ss. - Structs, such as {} or [], will be stored as string without braces {} or brackets [].
- If no data is archived during the specified period, a file that contains only headers will still be generated.
- The compression method is GZIP by default with the filename extension .gz.
Archiving 5-minute and 10-minute Telemetry¶
Archiving Configuration¶
| Archiving Service Parameter | Parameter Value |
|---|---|
| User | General |
| Archiving Start Time | Specified when the archiving is created and cannot be changed later. Starts from the day before the archiving is created by default. |
| Period | 1 day |
| Archiving Model and Measurement Point Selection | All models and measurement points. |
| Path | The format of the path is: data_type/date/. For example, AI/20190909, DI/20190909. |
| Archived File Name | The file is named after the asset ID. |
| Encoding | UTF-8 |
| Filename Extension | .csv |
| Format | As described below. |
| Column Separator | Specified when the archiving is created and cannot be changed later. The default separator is the comma. |
| File Size | No upper limit |
| Compressed / Not Compressed | Specified when the archiving is created and cannot be changed later. Compressed by default. |
| Compression Algorithm | GZIP |
| Merge / Do Not Merge Files of the Same Type | Does not merge by default. |
| File Cleanup Policy | Specified when the archiving is created and cannot be changed later. 1% by default. |
Format of Archived Files¶
An example to illustrate the format of the archived file is per the below.
| Asset ID | Asset Name | Data Time | Data Time UTC | WTUR.GenActivePW_AVG_5m | WTUR.WindSpeed_AVG_5m |
|---|---|---|---|---|---|
| Asset ID | Asset Name | Data Time | Data Time UTC | Reactive Power Production(avg 5m) | WindSpeed(avg 5m) |
| 资产 ID | 资产名称 | 时间 | UTC时间 | 有功功率五分钟平均值 | 风速五分钟平均值 |
| windTurbineC | windTurbine C in China | 2020-01-01 00:00:00 | 2020-01-01 00:00:00 | 123.123 | 4.444 |
In this sample, the headers are displayed using the ID, English name, and Chinese name. The measurement point data starts from the 5th column.
The measurement points are displayed in the same manner as AI, PI, and Generic data, i.e., in ascending order based on their priority, etc.
Default File Path¶
| Data Type | Path |
|---|---|
| AI Raw Data | /EDGE/ARCHIVE/ORIGINAL_AI/ |
| DI Raw Data | /EDGE/ARCHIVE/ORIGINAL_DI/ |
| PI Raw Data | /EDGE/ARCHIVE/ORIGINAL_PI/ |
| GENERAL Raw Data | /EDGE/ARCHIVE/ORIGINAL_GENERIC/ |
| AI 5-minute Data | /EDGE/ARCHIVE/FIVE_MIN_AI/ |
| AI 10-minute Data | /EDGE/ARCHIVE/TEN_MIN_AI/ |
| ALert | /EDGE/ARCHIVE/ALARM/ |