opendtu-logger/README.md

# OpenDTU logger

OpenDTU logger connects to the OpenDTU livedata websocket and captures metrics.
These metrics are inserted into a PostgreSQL database.
Optionally, TimescaleDB can be used.
Included in OpenDTU Logger are a number of Grafana dashboards, which can be used to provide useful visualisations and insights. The image below shows the PV dashboard of a given day in dark mode. Light mode is also supported. Hovering over the graphs will result in a tooltip, providing exact the values recorded at a given time.

![PV dashboard in dark mode](https://git.hollander.online/energy/opendtu-logger/raw/commit/a3debb78665641fe7101555d85df89eb0f0d2a79/screenshots/PV-overview-dark-mode.png)

## Install instructions

Docker Compose is the preferred way to install OpenDTU Logger, but using the binary is also possible.

### Docker

```sh
docker pull git.hollander.online/energy/opendtu-logger:0.0
```

Preferably, run the Docker image using the Docker compose examples provided in the `./docker` folder.

#### Docker Compose

The `docker` folder in this [repository](https://git.hollander.online/energy/opendtu-logger) contains example Docker compose files. The `compose.with-database-grafana.yml` file contains a full setup suitable for a standalone deployment. The other compose files are aimed at integration into existing environments.

To use the standalone deployment, take the following steps

Enter the `docker` folder

```sh
cd docker
```

Copy `example.with-database.env` to `.env`. E.g.:

```sh
cp example.with-database.env .env
```

In the `.env` file:

- Edit the `REMOTE_URL` to point to your OpenDTU.
- In `DB_URL` alter the `password` field. Set the same value in. `PG_PASSWORD`

Deploy the Docker containers by running the following command from the `docker` folder.

```sh
docker compose -f compose.with-database-grafana.yml up
```

Docker should now start downloading the required containers and bring up OpenDTU Logger, TimescaleDB and Grafana. If everything seems ok, press `CTRL + C` to stop the containers. Next, execute

```sh
docker compose -f compose.with-database-grafana.yml up -d
```

The `-d` flag ensures that the containers are started in the background, so you can safely close your terminal window. If desired, containers running in the background can be stopped by running compose again with the `down` command.

```sh
docker compose -f compose.with-database-grafana.yml down
```

#### Setting up Grafana

After deploying OpenDTU Logger with Docker Compose, Grafana should be available at <http://localhost:3000>.

The initial login username and password are `admin:admin`. After the initial login, Grafana will request you to change the password to something more secure.

##### Configuring the OpenDTU Logger data source

After logging in, go to `Connections` -> `Data sources` -> `Add new data source`.

- Under `SQL` select `PostgreSQL`.
- Change the following fields
  - `Host URL`: `timescaledb:5432`
  - `Database name`: `opendtu_logger`
  - `Username`: `postgres`
  - `Password`: The password you set in the `.env.` file in the previous step.
  - `TLS/SSL Mode`: `disable`
  - `TimescaleDB`: set the slider to `enable`
- Click `Save and test`
  - You should get a green box with a `Database Connection OK` message.

##### Adding the OpenDTU Logger dashboards

Now that the data source has been configured, we can add our dashboards.

- Go to `Dashboards` -> `New` -> `Import`.
- Upload `PV.json` from the `grafana` folder in this [repository](https://git.hollander.online/energy/opendtu-logger)
- After uploading, Grafana requests you to configure a data source. Select the data source configured in the previous step. Click `Import`. The dashboard should now be ready.
- If you want more dashboards, import the other dashboards from the `grafana` folder as well, or create your own.

### Binary

Go to the [releases page](https://git.hollander.online/energy/opendtu-logger/releases) and download the most recent stable release.

#### Running the binary as local user

```sh
REMOTE_URL="opendtu:80" DB_URL="host=localhost port=5432 user=postgres password=secret dbname=dtu sslmode=disable" TIMESCALEDB_ENABLED="true" TZ="Europe/Amsterdam" ./opendtu-logger
```

#### Creating a more permanent setup on Linux using systemd

```sh
sudo adduser --system --group --disabled-login --home /opt/opendtu-logger opendtu-logger

sudo mkdir /opt/opendtu-logger
sudo mkdir /opt/opendtu-logger/data
sudo mv ./opendtu-logger /opt/opendtu-logger/

sudo chown -R opendtu-logger:opendtu-logger /opt/opendtu-logger
```

```sh
sudo nano /etc/systemd/system/opendtu-logger.service
```

Add the contents of `systemd/opendtu-logger.service` and alter the `REMOTE_URL` and `DB_URL` environment variables to match your setup.

### PostgreSQL setup

To install PostgreSQL on Debian / Ubuntu run

```sh
sudo apt install postgresql
```

The OpenDTU logger logs to a PostgreSQL database and optionally supports TimescaleDB. Create a separate database and user using the following commands.

```sql
CREATE DATABASE opendtu_logger;
CREATE USER opendtu_logger WITH password 'SECRET';
GRANT ALL privileges ON DATABASE opendtu_logger TO opendtu_logger;
GRANT ALL ON SCHEMA public TO opendtu_logger;
```

Optional: enable the TimescaleDB extension. To make use of TimescaleDB, follow the steps outlined in [their documentation](https://docs.timescale.com/self-hosted/latest/install/installation-linux/). Then, enable TimescaleDB support for the `opendtu_logger` database by executing the following commands

```sql
\c opendtu_logger
CREATE EXTENSION OF NOT EXISTS timescaledb;
```

## Environment variables

The following environment variables are required for OpenDTU Logger to be configured.

```conf
REMOTE_URL="opendtu:80"
DB_URL="host=localhost port=5432 user=postgres password=secret dbname=dtu sslmode=disable"
TIMESCALEDB_ENABLED="true"
TZ="Europe/Amsterdam"
```

- `REMOTE_URL`is used to specify the IP address or hostname OpenDTU is running on. E.g.: `192.168.1.6` or `opendtu.internal`
- `DB_URL` specifies the credentials required to connect to the PostgreSQL database. All [connection parameters documented by golang's `lib/pq` package](https://pkg.go.dev/github.com/lib/pq#hdr-Connection_String_Parameters) can be used.
- `TIMESCALEDB_ENABLED` should be set to `true` or `false`.
- `TZ` is used to ensure data is recorded with the right timestamp. Choose the timezone valid for your location from [this Wikipedia page](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones).

## Grafana dashboards

The `grafana` folder contains example dashboards which can be imported into existing Grafana installs, or into the provided Docker Compose with Grafana setup.

- The `opendtu_` dashboards contain visualisations for each table the OpenDTU logger records to.
- `PV.json` is a dashboard which combines the visualisations of each table that I found to be most useful.
- `Zonnepanelen.json` is the same as `PV.json`, but in Dutch language.

## Known limitations

- The logger will currently log every event posted by OpenDTU to the websocket. Checks still need to be added to determine the uniqueness of an event. For more information, see also [this OpenDTU Github issue](https://github.com/tbnobody/OpenDTU/issues/1800).
- Upon restart of the OpenDTU, the OpenDTU Logger binary will need to be restarted as well. When using the provided `compose.yml` files with Docker, or using the `systemd` service file when using the binary, this will happen automatically.

## Feedback

If you'd like to provide feedback about this project, or in case you'd like to ask questions, please refer to [this page](https://pieterhollander.nl/author/pieter-hollander/) for contact information (e-mail, Github or LinkedIn), or use the [contact form](https://pieterhollander.nl/#contact) on my website. You can also respond by placing your remarks in [this discussion on Github](https://github.com/tbnobody/OpenDTU/discussions/1842).