Data Format Specification¶
The main change in this operator version update is the addition of support for new data formats.
- Old data format:
- New data format:
As shown in the figure, the most major differences between the old and new data formats are:
- In the old data format, the operator usually combined the fields modelId and pointId in the data to filter the data.
- The new data format has a separate measurementId field that can be used to filter the data.
This measurementId field corresponds to the Measurement metadata in Data Catalog service. You can Create a Measurement and get measurementId in Data Catalog service.
For the Measurement metadata, you can model directly in the Data Catalog service without device modeling in IoT Hub. The data accessed to the platform can be consumed by other data products according to the specified data format.
In combination with the new Record Formatter operator, tagging of measurement points and assets is supported. Further, with the new Record Filter operator, expressions can be used to filter the data more flexibly based on the tag information of data.
Tens of old operators were adapted in order to support the new data format throughout the operator functionality.
The name of the adapted operator is the same as the old one, and the icon of the adapted operator in the new format is marked with a * in the upper right corner (the adapted operator name is also marked with a * in this document to make it easier to distinguish).
In terms of calculation logic, the operator remains basically unchanged after adaptation, except that the input/output configuration (Input/Output) uses measurementId instead of modelId::pointId, and for more details, please refer to the corresponding operator documentation.
Below we will further display the differences between the two old and new formats before.
Old Data Format (ModelId::PointId)¶
{
"orgId":"o1234567890",
"modelId":"inverter",
"modelIdPath":"/rootModel/inverter",
"pointId":"inverter",
"assetId":"zabPDuHq",
"time":1542609276270,
"value":23.4,
"quality":0,
"dq":0,
"attr": {}
}
| Fields | Description |
|---|---|
| orgId | Organization ID |
| modelId | Model ID |
| modelIdPath | The full path of Model |
| pointId | Measurement points ID on the model |
| assetId | Asset ID |
| time | Timestamp |
| value | Measurement point value |
| quality | The quality point on the measurement point, without a quality point the value defaults to 0 |
| dq | Optional field, Quality Filter use for data quality operators and other operators |
| attr | Used to store information such as operator calculation results, default is empty JSON Object |
New Data Format (MeasurementId)¶
{
"orgId":"o1234567890",
"modelId":"legacyModelId",
"modelIdPath":"/legacyModelId",
"pointId":"legacyPointId",
"assetId":"zabPDuHq",
"assetTags": {"DCMModel":{"DCMModel::ModelA":{}}},
"measurementId":"measurement1234",
"measurementTags": {"Haystack":{"Haystack::temp":{"temp-attr":"dummy"}}},
"time":1542609276270,
"value":23.4,
"quality":0,
"dq":0,
"attr": {}
}
| Fields | Description |
|---|---|
| orgId | Organization ID |
| modelId | Model ID，Legacy field. The field is no longer used by operators adapted to the new format, except for a few operators such as Record Formatter, where the correctness of the field value is guaranteed by the user. |
| modelIdPath | The full path to the Model ID. Legacy field, adapted to the new format of the operator, which is no longer used except for a few operators such as Record Formatter, where the correctness of the field value is guaranteed by the user. |
| pointId | Measurement point ID on Model, legacy field. This field is no longer used by operators that are adapted to the new format, except for a few operators such as Point Lookup*, where the correctness of the field value is guaranteed by the user. |
| assetId | Asset ID |
| assetTags | Tag associated with asset ID, optional field, tag found by Record Formatter operator |
| measurementId | Measurement point ID, which can be associated with Measurement metadata in the Data Asset Catalog |
| measurementTags | Tag associated with the measurement point ID, optional field, the tag is found by the Record Formatter operator. |
| time | Timestamp |
| value | Measurement point value |
| quality | The quality point on the measurement point, without a quality point the value defaults to 0 |
| dq | Optional field, Quality Filter use for data quality operators and other operators |
| attr | Used to store information such as operator calculation results, default is empty JSON Object |