diff --git a/README.md b/README.md index d35a1f3..2480edb 100644 --- a/README.md +++ b/README.md @@ -2,20 +2,21 @@ P1 logger is a tool designed to store meter data conforming to the [ESMR or DSMR (European / Dutch Smart Meter Requirements) standard](https://github.com/jvhaarst/DSMR-P1-telegram-reader/blob/master/documentation/Dutch%20Smart%20Meter%20Requirements%20v5.0.2%20Final%20P1.pdf) as efficiently as possible in a PostgreSQL database. -## Data storage architecture +![P1 logs visualised in Grafana](https://git.hollander.online/energy/p1-logger/raw/branch/main/screenshots/P1-data-grafana.png) -To store the data as efficiently as possible, a few design choices had to be made. +## Background: data storage architecture + +P1 logger aims to store detailed meter data as efficiently as possible. Therefore, a few design choices were made. All metrics are stored as INT and not NUMERIC or FLOAT to optimise their storage size. -Dt1, Dt2, Rt1, Rt2, G, F and Fl are cumulative meter readings. Therefore, they function as counters that can only go up. By making them pointers to int, they can be set to 'nil' when the reading has not changed. This way, they are logged as NULL in the database when no change has happened, saving storage capacity. As an added benefit, this also make data retrieval, visualisation and analysis more light-weight. +Dt1, Dt2, Rt1, Rt2, G, F and Fl are cumulative meter readings. Therefore, they function as counters that can only go up. By making them pointers to int, they can be set to 'nil' whenever a reading has not changed. This way, they are logged as NULL in the database when no change has happened, saving storage capacity. As an added benefit, this makes data retrieval, visualisation and analysis more light-weight. -Gas consumption is updated once per minute. Not saving intermittent values theoretically saves 4 bytes per value. An average month has 43,829.0639 minutes. 4*43,829.0639 = 175,316.26 bytes = 0.18 MB +Typical P1 meters update gas consumption once per minute. Not saving intermittent values theoretically saves 4 bytes per value. An average month has 43,829.0639 minutes. 4*43,829.0639 = 175,316.26 bytes = 0.18 MB -In practice, storage will be even lower when gas is not being consumed continuously, as a new metric won't be available for every minute. +In practice, storage will be even lower, as gas is usually not consumed continuously. Therefore, a new metric won't be available for every minute. When Tariff 1 is active, Tariff 2 isn't. When energy is being imported, it is not being returned. - Therefore, only updating their values on-change should save at least 75% storage capacity requirements An average month has 2,629,743.83 seconds. diff --git a/chatgpt prompt.md b/chatgpt prompt.md deleted file mode 100644 index ace44a4..0000000 --- a/chatgpt prompt.md +++ /dev/null @@ -1,53 +0,0 @@ -# ChatGPT prompt - -Please write a golang program that subscribes to an mqtt topic which outputs the following payload - -```json -{"t":"231205164749W","dt1":830959,"dt2":729319,"rt1":33727,"rt2":111841,"d":224,"r":0,"f":18,"fl":17,"g":426077,"v1":219,"v2":227,"v3":223,"c1":0,"c2":0,"c3":0,"d1":84,"d2":50,"d3":90,"r1":0,"r2":0,"r3":0} -``` - -T is the Timestamp in Timezone Europe/Amsterdam and format YYMMDDhhmmss. The letter at the end of the timestamp can be either "W" for Winter or "S" for Summer and indicates daylight savings time. - -The data should be inserted into a configurable postgres database with structure - -```sql -CREATE TABLE p1 ( -timestamp TIMESTAMPTZ, -delivered_tariff1 INT, -delivered_tariff2 INT, -returned_tariff1 INT, -returned_tariff2 INT, -delivery_all INT, -returning_all INT, -failures INT, -long_failures INT, -gas INT, -voltage_l1 INT, -voltage_l2 INT, -voltage_l3 INT, -current_l1 INT, -current_l2 INT, -current_l3 INT, -delivery_l1 INT, -delivery_l2 INT, -delivery_l3 INT, -returning_l1 INT, -returning_l2 INT, -returning_l3 INT -); -CREATE EXTENSION IF NOT EXISTS timescaledb; -SELECT create_hypertable('p1', 'timestamp', if_not_exists => TRUE); -``` - -The connections should be configured using the following environment variables - -```conf -MQTT_BROKER=tls://mqtt.example.com:8883 -MQTT_TOPIC=p1/# -MQTT_USERNAME=your_mqtt_username -MQTT_PASSWORD=your_mqtt_password - -PG_DB='host=localhost port=5432 user=p1 password=secret-replace dbname=p1 sslmode=disable' -``` - -The program should be usable in production and should automatically recover on database or mqtt service interruptions. diff --git a/screenshots/P1-data-grafana.png b/screenshots/P1-data-grafana.png new file mode 100644 index 0000000..dea3ce9 Binary files /dev/null and b/screenshots/P1-data-grafana.png differ