Improve README, add screenshot.
Some checks failed
Build Golang packages / release (push) Waiting to run
Build Docker image / build (push) Has been cancelled

This commit is contained in:
Pieter Hollander 2024-06-15 15:06:32 +02:00
parent 80db0a8a47
commit 56fc4b31a9
Signed by: pieter
SSH key fingerprint: SHA256:HbX+9cBXsop9SuvL+mELd29sK+7DehFfdVweFVDtMSg
3 changed files with 7 additions and 59 deletions

View file

@ -2,20 +2,21 @@
P1 logger is a tool designed to store meter data conforming to the [ESMR or DSMR (European / Dutch Smart Meter Requirements) standard](https://github.com/jvhaarst/DSMR-P1-telegram-reader/blob/master/documentation/Dutch%20Smart%20Meter%20Requirements%20v5.0.2%20Final%20P1.pdf) as efficiently as possible in a PostgreSQL database. P1 logger is a tool designed to store meter data conforming to the [ESMR or DSMR (European / Dutch Smart Meter Requirements) standard](https://github.com/jvhaarst/DSMR-P1-telegram-reader/blob/master/documentation/Dutch%20Smart%20Meter%20Requirements%20v5.0.2%20Final%20P1.pdf) as efficiently as possible in a PostgreSQL database.
## Data storage architecture ![P1 logs visualised in Grafana](https://git.hollander.online/energy/p1-logger/raw/branch/main/screenshots/P1-data-grafana.png)
To store the data as efficiently as possible, a few design choices had to be made. ## Background: data storage architecture
P1 logger aims to store detailed meter data as efficiently as possible. Therefore, a few design choices were made.
All metrics are stored as INT and not NUMERIC or FLOAT to optimise their storage size. All metrics are stored as INT and not NUMERIC or FLOAT to optimise their storage size.
Dt1, Dt2, Rt1, Rt2, G, F and Fl are cumulative meter readings. Therefore, they function as counters that can only go up. By making them pointers to int, they can be set to 'nil' when the reading has not changed. This way, they are logged as NULL in the database when no change has happened, saving storage capacity. As an added benefit, this also make data retrieval, visualisation and analysis more light-weight. Dt1, Dt2, Rt1, Rt2, G, F and Fl are cumulative meter readings. Therefore, they function as counters that can only go up. By making them pointers to int, they can be set to 'nil' whenever a reading has not changed. This way, they are logged as NULL in the database when no change has happened, saving storage capacity. As an added benefit, this makes data retrieval, visualisation and analysis more light-weight.
Gas consumption is updated once per minute. Not saving intermittent values theoretically saves 4 bytes per value. An average month has 43,829.0639 minutes. 4*43,829.0639 = 175,316.26 bytes = 0.18 MB Typical P1 meters update gas consumption once per minute. Not saving intermittent values theoretically saves 4 bytes per value. An average month has 43,829.0639 minutes. 4*43,829.0639 = 175,316.26 bytes = 0.18 MB
In practice, storage will be even lower when gas is not being consumed continuously, as a new metric won't be available for every minute. In practice, storage will be even lower, as gas is usually not consumed continuously. Therefore, a new metric won't be available for every minute.
When Tariff 1 is active, Tariff 2 isn't. When energy is being imported, it is not being returned. When Tariff 1 is active, Tariff 2 isn't. When energy is being imported, it is not being returned.
Therefore, only updating their values on-change should save at least 75% storage capacity requirements Therefore, only updating their values on-change should save at least 75% storage capacity requirements
An average month has 2,629,743.83 seconds. An average month has 2,629,743.83 seconds.

View file

@ -1,53 +0,0 @@
# ChatGPT prompt
Please write a golang program that subscribes to an mqtt topic which outputs the following payload
```json
{"t":"231205164749W","dt1":830959,"dt2":729319,"rt1":33727,"rt2":111841,"d":224,"r":0,"f":18,"fl":17,"g":426077,"v1":219,"v2":227,"v3":223,"c1":0,"c2":0,"c3":0,"d1":84,"d2":50,"d3":90,"r1":0,"r2":0,"r3":0}
```
T is the Timestamp in Timezone Europe/Amsterdam and format YYMMDDhhmmss. The letter at the end of the timestamp can be either "W" for Winter or "S" for Summer and indicates daylight savings time.
The data should be inserted into a configurable postgres database with structure
```sql
CREATE TABLE p1 (
timestamp TIMESTAMPTZ,
delivered_tariff1 INT,
delivered_tariff2 INT,
returned_tariff1 INT,
returned_tariff2 INT,
delivery_all INT,
returning_all INT,
failures INT,
long_failures INT,
gas INT,
voltage_l1 INT,
voltage_l2 INT,
voltage_l3 INT,
current_l1 INT,
current_l2 INT,
current_l3 INT,
delivery_l1 INT,
delivery_l2 INT,
delivery_l3 INT,
returning_l1 INT,
returning_l2 INT,
returning_l3 INT
);
CREATE EXTENSION IF NOT EXISTS timescaledb;
SELECT create_hypertable('p1', 'timestamp', if_not_exists => TRUE);
```
The connections should be configured using the following environment variables
```conf
MQTT_BROKER=tls://mqtt.example.com:8883
MQTT_TOPIC=p1/#
MQTT_USERNAME=your_mqtt_username
MQTT_PASSWORD=your_mqtt_password
PG_DB='host=localhost port=5432 user=p1 password=secret-replace dbname=p1 sslmode=disable'
```
The program should be usable in production and should automatically recover on database or mqtt service interruptions.

Binary file not shown.

After

Width:  |  Height:  |  Size: 895 KiB