Data Archiving Reference¶

Data Archiving Logic¶

The storage directory for data from the real-time message channel can be generated based on the data event time or system time while the storage directory for data from the offline message channel and real-time alert record can be generated based on system time only. The generated archive files will be synchronized to the specified storage system as per the storage path information. When setting the storage path, you can determine the file generation mode by choosing to generate the directory by event time or system time.

If you choose to generate the directory by the event time, the data content will be parsed and the data event time will be obtained. Next, the data in the same time partition will be written to the same file. Finally, the generated files will be synchronized to the corresponding directory.

If you choose to generate the directory by the system time, the data in the same time partition will be written to the same file as per the system timestamp and the generated files will be synchronized to the corresponding directory.

Storage Path Partition Parameters¶

After specifying the root directory for storing the archive files and the path generation method, you can select different time partition parameter formats. Four time partition parameter formats are supported currently.

Parameter Format	Description	Example
YYYYMMDD	Generate directories by day.	/bucketName/samplePath/20190101/
YYYYMMDD/HH	Generate directories by day/hour.	/bucketName/samplePath/20190101/00/
YYYY/MM/DD	Generate directories by year/month/day.	/bucketName/samplePath/2019/01/01/
YYYY/MM/DD/HH	Generate directories by year/month/day/hour.	/bucketName/samplePath/2019/01/01/00/

Archiving Cycle¶

Currently, the data archiving policy supports an archiving cycle of 1 hour, 12 housr, or 24 hours. When a data archiving cycle starts, the system starts reading data from the specified message channel. If the archived data falls in the same archiving cycle, the data will be saved in 1 file and sliced by the specified file size if it exceeds the limit.

However, if no data is cached in a data archiving cycle, no archive file will be generated.

Different archiving cycles mean different scheduled starting time to trigger the archiving jobs as well as different data range. For example:

Archiving Cycle	Scheduled Job Starting Time	Archived Data
1 hour	00:00:00, 01:00:00, 02:00:00, …, 23:00:00	Taking 01:00:00 for example, data to be archived falls in [00:00:00, 01:00:00).
12 hours	00:00:00, 12:00:00	Taking 12:00:00 PM for example, data to be archived falls in [00:00:00, 12:00:00 PM).
24 hours	00:00:00	Taking 2019-01-02 00:00:00 for example, data to be archived falls in [2019-01-01 00:00:00, 2019-01-02 00:00:00)

Note

The data archiving cycle adopts the system time stamp. If the system time stamp of data is within the current archiving cycle, the data will be archived as per the policy configuration and synchronized to the corresponding storage path.

Generation of Archived Files¶

In an archiving cycle, the archiving job will be triggered to generate the archive file only when the first data record arrives. If no data arrives in the archiving cycle, no file or path will be created.

When Real-time Message Channel and Generate path by event time are selected in the archiving policy, if the event time of the uploaded data is 1 hour later than the system time or 360 hours earlier than the system time, the archived files will be stored in a folder named archive_recycling_${filename} under the specified root directory (in which filename is the specified name for the archived file in the policy).

The columns of the archived file are as follows:

Field Name	Field Description
orgId	The organization ID.
modelId	The model ID.
assetId	The asset ID.
measurepoints	The measurement point name.
timestamp	The event time of the measurement point.
value	The value of the measurement point.
quality	The data quality.