Improve README, add screenshot.
This commit is contained in:
parent
80db0a8a47
commit
56fc4b31a9
3 changed files with 7 additions and 59 deletions
13
README.md
13
README.md
|
@ -2,20 +2,21 @@
|
||||||
|
|
||||||
P1 logger is a tool designed to store meter data conforming to the [ESMR or DSMR (European / Dutch Smart Meter Requirements) standard](https://github.com/jvhaarst/DSMR-P1-telegram-reader/blob/master/documentation/Dutch%20Smart%20Meter%20Requirements%20v5.0.2%20Final%20P1.pdf) as efficiently as possible in a PostgreSQL database.
|
P1 logger is a tool designed to store meter data conforming to the [ESMR or DSMR (European / Dutch Smart Meter Requirements) standard](https://github.com/jvhaarst/DSMR-P1-telegram-reader/blob/master/documentation/Dutch%20Smart%20Meter%20Requirements%20v5.0.2%20Final%20P1.pdf) as efficiently as possible in a PostgreSQL database.
|
||||||
|
|
||||||
## Data storage architecture
|
![P1 logs visualised in Grafana](https://git.hollander.online/energy/p1-logger/raw/branch/main/screenshots/P1-data-grafana.png)
|
||||||
|
|
||||||
To store the data as efficiently as possible, a few design choices had to be made.
|
## Background: data storage architecture
|
||||||
|
|
||||||
|
P1 logger aims to store detailed meter data as efficiently as possible. Therefore, a few design choices were made.
|
||||||
|
|
||||||
All metrics are stored as INT and not NUMERIC or FLOAT to optimise their storage size.
|
All metrics are stored as INT and not NUMERIC or FLOAT to optimise their storage size.
|
||||||
|
|
||||||
Dt1, Dt2, Rt1, Rt2, G, F and Fl are cumulative meter readings. Therefore, they function as counters that can only go up. By making them pointers to int, they can be set to 'nil' when the reading has not changed. This way, they are logged as NULL in the database when no change has happened, saving storage capacity. As an added benefit, this also make data retrieval, visualisation and analysis more light-weight.
|
Dt1, Dt2, Rt1, Rt2, G, F and Fl are cumulative meter readings. Therefore, they function as counters that can only go up. By making them pointers to int, they can be set to 'nil' whenever a reading has not changed. This way, they are logged as NULL in the database when no change has happened, saving storage capacity. As an added benefit, this makes data retrieval, visualisation and analysis more light-weight.
|
||||||
|
|
||||||
Gas consumption is updated once per minute. Not saving intermittent values theoretically saves 4 bytes per value. An average month has 43,829.0639 minutes. 4*43,829.0639 = 175,316.26 bytes = 0.18 MB
|
Typical P1 meters update gas consumption once per minute. Not saving intermittent values theoretically saves 4 bytes per value. An average month has 43,829.0639 minutes. 4*43,829.0639 = 175,316.26 bytes = 0.18 MB
|
||||||
|
|
||||||
In practice, storage will be even lower when gas is not being consumed continuously, as a new metric won't be available for every minute.
|
In practice, storage will be even lower, as gas is usually not consumed continuously. Therefore, a new metric won't be available for every minute.
|
||||||
|
|
||||||
When Tariff 1 is active, Tariff 2 isn't. When energy is being imported, it is not being returned.
|
When Tariff 1 is active, Tariff 2 isn't. When energy is being imported, it is not being returned.
|
||||||
|
|
||||||
Therefore, only updating their values on-change should save at least 75% storage capacity requirements
|
Therefore, only updating their values on-change should save at least 75% storage capacity requirements
|
||||||
An average month has 2,629,743.83 seconds.
|
An average month has 2,629,743.83 seconds.
|
||||||
|
|
||||||
|
|
|
@ -1,53 +0,0 @@
|
||||||
# ChatGPT prompt
|
|
||||||
|
|
||||||
Please write a golang program that subscribes to an mqtt topic which outputs the following payload
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"t":"231205164749W","dt1":830959,"dt2":729319,"rt1":33727,"rt2":111841,"d":224,"r":0,"f":18,"fl":17,"g":426077,"v1":219,"v2":227,"v3":223,"c1":0,"c2":0,"c3":0,"d1":84,"d2":50,"d3":90,"r1":0,"r2":0,"r3":0}
|
|
||||||
```
|
|
||||||
|
|
||||||
T is the Timestamp in Timezone Europe/Amsterdam and format YYMMDDhhmmss. The letter at the end of the timestamp can be either "W" for Winter or "S" for Summer and indicates daylight savings time.
|
|
||||||
|
|
||||||
The data should be inserted into a configurable postgres database with structure
|
|
||||||
|
|
||||||
```sql
|
|
||||||
CREATE TABLE p1 (
|
|
||||||
timestamp TIMESTAMPTZ,
|
|
||||||
delivered_tariff1 INT,
|
|
||||||
delivered_tariff2 INT,
|
|
||||||
returned_tariff1 INT,
|
|
||||||
returned_tariff2 INT,
|
|
||||||
delivery_all INT,
|
|
||||||
returning_all INT,
|
|
||||||
failures INT,
|
|
||||||
long_failures INT,
|
|
||||||
gas INT,
|
|
||||||
voltage_l1 INT,
|
|
||||||
voltage_l2 INT,
|
|
||||||
voltage_l3 INT,
|
|
||||||
current_l1 INT,
|
|
||||||
current_l2 INT,
|
|
||||||
current_l3 INT,
|
|
||||||
delivery_l1 INT,
|
|
||||||
delivery_l2 INT,
|
|
||||||
delivery_l3 INT,
|
|
||||||
returning_l1 INT,
|
|
||||||
returning_l2 INT,
|
|
||||||
returning_l3 INT
|
|
||||||
);
|
|
||||||
CREATE EXTENSION IF NOT EXISTS timescaledb;
|
|
||||||
SELECT create_hypertable('p1', 'timestamp', if_not_exists => TRUE);
|
|
||||||
```
|
|
||||||
|
|
||||||
The connections should be configured using the following environment variables
|
|
||||||
|
|
||||||
```conf
|
|
||||||
MQTT_BROKER=tls://mqtt.example.com:8883
|
|
||||||
MQTT_TOPIC=p1/#
|
|
||||||
MQTT_USERNAME=your_mqtt_username
|
|
||||||
MQTT_PASSWORD=your_mqtt_password
|
|
||||||
|
|
||||||
PG_DB='host=localhost port=5432 user=p1 password=secret-replace dbname=p1 sslmode=disable'
|
|
||||||
```
|
|
||||||
|
|
||||||
The program should be usable in production and should automatically recover on database or mqtt service interruptions.
|
|
BIN
screenshots/P1-data-grafana.png
Normal file
BIN
screenshots/P1-data-grafana.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 895 KiB |
Loading…
Reference in a new issue