Fix tables. Better text structure.
All checks were successful
Build Docker image / build (push) Successful in 1m12s
Build Golang packages / release (push) Has been skipped

This commit is contained in:
Pieter Hollander 2024-03-20 01:12:00 +01:00
parent ea057f09c1
commit 19aca3e910
Signed by: pieter
SSH key fingerprint: SHA256:HbX+9cBXsop9SuvL+mELd29sK+7DehFfdVweFVDtMSg

View file

@ -2,21 +2,22 @@
P1 logger is a tool designed to store meter data conforming to the [ESMR or DSMR (European / Dutch Smart Meter Requirements) standard](https://github.com/jvhaarst/DSMR-P1-telegram-reader/blob/master/documentation/Dutch%20Smart%20Meter%20Requirements%20v5.0.2%20Final%20P1.pdf) as efficiently as possible in a PostgreSQL database. P1 logger is a tool designed to store meter data conforming to the [ESMR or DSMR (European / Dutch Smart Meter Requirements) standard](https://github.com/jvhaarst/DSMR-P1-telegram-reader/blob/master/documentation/Dutch%20Smart%20Meter%20Requirements%20v5.0.2%20Final%20P1.pdf) as efficiently as possible in a PostgreSQL database.
## Data storage architecture
To store the data as efficiently as possible, a few design choices had to be made. To store the data as efficiently as possible, a few design choices had to be made.
All metrics are stored as INT and not NUMERIC or FLOAT to optimise their storage size. All metrics are stored as INT and not NUMERIC or FLOAT to optimise their storage size.
Dt1, Dt2, Rt1, Rt2, G, F and Fl are cumulative meter readings. Therefore, they function as counters that can only go up. By making them pointers to int, they can be set to 'nil' when the reading has not changed. This way, they are logged as NULL in the database when no change has happened, saving storage capacity. As an added benefit, this also make data retrieval, visualisation and analysis more light-weight. Dt1, Dt2, Rt1, Rt2, G, F and Fl are cumulative meter readings. Therefore, they function as counters that can only go up. By making them pointers to int, they can be set to 'nil' when the reading has not changed. This way, they are logged as NULL in the database when no change has happened, saving storage capacity. As an added benefit, this also make data retrieval, visualisation and analysis more light-weight.
Gas consumption is updated once per minute. Not saving intermittent values theoretically saves 4 bytes per value. An average month has 43 829.0639 minutes. Gas consumption is updated once per minute. Not saving intermittent values theoretically saves 4 bytes per value. An average month has 43,829.0639 minutes. 4*43,829.0639 = 175,316.26 bytes = 0.18 MB
4*43829,0639 = 175,316.26 bytes = 0.18 MB
In practice, storage will be even lower when gas is not being consumed continuously, as a new metric won't be available for every minute. In practice, storage will be even lower when gas is not being consumed continuously, as a new metric won't be available for every minute.
When Tariff 1 is active, Tariff 2 isn't. When energy is being imported, it is not being returned. When Tariff 1 is active, Tariff 2 isn't. When energy is being imported, it is not being returned.
Therefore, only updating their values on-change should save at least 75% storage capacity requirements Therefore, only updating their values on-change should save at least 75% storage capacity requirements
An average month has 2 629 743.83 seconds. An average month has 2,629,743.83 seconds.
This saves at least: This saves at least:
2,629,743.83 seconds * 4 bytes = 10,518,975.32 bytes = 10,518.98 kB = 10.52 MB 2,629,743.83 seconds * 4 bytes = 10,518,975.32 bytes = 10,518.98 kB = 10.52 MB
@ -26,36 +27,39 @@ This applies both to Dt1 and Dt2 as well as Rt1 and Rt2, only one is recorded at
In addition, many meters only update these metrics once every 10 seconds, meaning even more storage capacity is saved: In addition, many meters only update these metrics once every 10 seconds, meaning even more storage capacity is saved:
10.52 / 10 * 9 = 9,468 MB. 10.52 / 10 * 9 = 9,468 MB.
For D1-3 and R1-3, either Dx or Rx will be active. Therefore, their storage requirements can be sliced in half, saving 3 * 10.52 = 42.08 MB. This is different for D and R, as these contain the sum of each phase and are reported simultaneously. Not saving failures when there is no change, will save around 10.52MB per month per metric, so an additional 21.04MB. Theoretical storage requirements for a DSMR5 meter with metrics emitted every second: For D1-3 and R1-3, either Dx or Rx will be active. Therefore, their storage requirements can be sliced in half, saving 3 * 10.52 = 42.08 MB. This is different for D and R, as these contain the sum of each phase and are reported simultaneously. Not saving failures when there is no change, will save around 10.52MB per month per metric, so an additional 21.04MB.
Column Column size (bytes) Per month (MB) X Total (MB) ### Theoretical storage requirements for a DSMR5 meter with metrics emitted every second
---------------------------------------------------------------------------
Timestamp 8 bytes 021.04
Dt1, Dt2, Rt1, Rt2 4 bytes 001.05
D, R 4 bytes 010.52
R 4 bytes 010.52
F 4 bytes 000.00
Fl 4 bytes 000.00
G 4 bytes 000.18
V1-3, C1-3 4 bytes 010.52 6 063.12
D1-3, R1-3 4 bytes 010.52 3 031.56
---------------------------------------------------------------------------
Total 137.99
Compared to no optimisations: | Column | Column size (bytes) | Per month (MB) | X | Total (MB)|
|:--------------------|---------------------|---------------:|----|-----------:|
| Timestamp | 8 bytes | 21.04 | | |
| Dt1, Dt2, Rt1, Rt2 | 4 bytes | 1.05 | | |
| D, R | 4 bytes | 10.52 | | |
| R | 4 bytes | 10.52 | | |
| F | 4 bytes | 0.00 | | |
| Fl | 4 bytes | 0.00 | | |
| G | 4 bytes | 0.18 | | |
| V1-3, C1-3 | 4 bytes | 10.52 | 6 | 63.12 |
| D1-3, R1-3 | 4 bytes | 10.52 | 3 | 31.56 |
| **Total** | | | | **137.99** |
Column Column size (bytes) Per month (MB) X Total (MB) ### Theoretical storage requirements without optimisations:
---------------------------------------------------------------------------
Timestamps 8 bytes 21.04 | Column | Column size (bytes) | Per month (MB) | X | Total (MB) |
Values 4 bytes 10.52 21 220.92 |:--------------------|---------------------|---------------:|----|-----------:|
--------------------------------------------------------------------------- | Timestamps | 8 bytes | 21.04 | | |
Total 241.96 | Values | 4 bytes | 10.52 | 21 | 220.92 |
| **Total** | | | | **241.96** |
### Conclusion
Conclusion:
At the expense of having to create more complicated queries for analysis and visualisation, the optimisations save at least 103.97 MB or 1-137.99/241.96 = 43% in uncompressed storage space per month. At the expense of having to create more complicated queries for analysis and visualisation, the optimisations save at least 103.97 MB or 1-137.99/241.96 = 43% in uncompressed storage space per month.
### Columns ### Columns
Data is stored in a table containing the following columns
- timestamp - timestamp
- delivered_tariff1, delivered_tariff2, returned_tariff1, returned_tariff2 - delivered_tariff1, delivered_tariff2, returned_tariff1, returned_tariff2
- delivery_all - delivery_all
@ -67,3 +71,16 @@ At the expense of having to create more complicated queries for analysis and vis
- current_l1, current_l2, current_l3 - current_l1, current_l2, current_l3
- delivery_l1, delivery_l2, delivery_l3 - delivery_l1, delivery_l2, delivery_l3
- returning_l1, returning_l2, returning_l3 - returning_l1, returning_l2, returning_l3
## Input: P1 reader
P1 Logger relies on a JSON input provided by a P1 reader.
The reader should send its data to an MQTT broker in the following JSON structure
```json
```
## References
- <http://domoticx.com/p1-poort-slimme-meter-hardware/>