CubelaPetar/notes
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

20250404

Browse Source
This commit is contained in:
Petar Cubela
2025-04-04 10:57:26 +02:00
parent e1d3a98fd2
commit 7becbea415
28 changed files with 1760 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files

300
projects/discopharma/20250320-manual-project.md Normal file
View File

@@ -0,0 +1,300 @@
---
title: "Metabase - Setup Manual"
author: Petar Cubela
date: March 20, 2025
geometry: margin=1.5cm
output: pdf_document
---
## Intro
Setting up a Metabase instance via Docker with a PostgreSQL application database and a secure web connection via https mediated by a public facing reverse proxy (nginx) and commercial TLS/SSL certificates.
### Goals and Requirements
### Software
- Google Cloud Platform (GCP)
- [Debain 12 (OS)](https://www.debian.org/download)
- [Docker (Containerization Platform)](https://docs.docker.com/engine/install/debian/)
- [NGINX (Web Server, Reverse Proxy)](https://docs.nginx.com/)
- [Postgres (as Container)](https://hub.docker.com/_/postgres)
- [Metabase (as Container)](https://hub.docker.com/r/metabase/metabase)
## VM Specs
### Metabase Server
- Name: Metabase Server
- OS: Debian 12
- hostname: mb-prod
- IP Address: `10.156.0.6/24`
- CPU: 2 core
- RAM: 2 GB (2048 MB)
- Storage: depends (30 GB)
- DNS entry: none
- Note: for every 20 concurrent users: needs 1CPU and 2GB of RAM more
### Reverse Proxy
- Name: Reverse Proxy
- OS: Debian 12
- hostname: rproxy
- IP Address: `10.156.0.7/24` + `<PUBLIC IP>` address (only activated in the end)
- CPU: 1 core
- RAM: 1 GB (1024 MB)
- Storage: depends (16 GB)
- DNS entry: metabase.discopharma.de -> `<PUBLIC IP>`
- Note: for every concurrent users: needs 1CPU and 2GB of RAM more
### Firewall
I list all necessary communications and respective ports needed:
Abbreviations:
- Metabse: mb-prod = `10.156.0.6`
- Metabse Dev: mb-dev = `10.156.0.8`
- ReverseProxy: rp = `10.156.0.7`
| Source | SourcePort | Destination | DestPort | Description |
| ------------- | ----------------------- | --------------- | ----------------------- | ------------------------------------------------------------------------------- |
| mb-prod | 3306/tcp | db | 3306/tcp | 3306 is the standard mysql port. Communication of mb-prod to db |
| rp | 3000/tcp </br> 3000/udp | mb-prod | 3000/tcp </br> 3000/udp | 3000 is the metabase web port. Reverse Proxy sends request via this port to mb. |
| rp | 3000/tcp </br>3000/udp | mb-dev | 3000/tcp </br> 3000/udp | 3000 is the metabase web port. Reverse Proxy sends request via this port to mb. |
| OPEN INTERNET | any | PUBLIC IP of rp | 443/tcp | 443 is the https port to communicate to rp over internet |
### Network Diagram
![diagram](/files/discopharma/discopharma-infra.drawio.png)
## Metabase Application Server and Database
### Administration
#### Update
In order to update the metabase containers change to the `~/metabase/` folder (where `compose.yml` file resides) and use the following command:
```bash
docker compose pull && docker compose up -d
```
Monitor the container logs to see if there are any errors by using the command:
```bash
docker compose logs -f
```
The `docker compose pull` command searches for images which are specified by a tag in the image variable in the `compose.yml` file:
`image: metabase/metabase:latest`
`latest` is here the tag and can also be changed to a version number which can be extracted from the [docker-hub](https://hub.docker.com/r/metabase/metabase/tags).
To simplify the process I wrote a simple bash script which updates the container images and removes old container images. The script is in the folder `/home/lukas_discopharma_de/scripts/metabase-update.sh`.
The update has to be done manually.
#### Backup
There is a script `/home/lukas_discopharma_de/db-backup.sh` which creates a database dump from the postgres instance running in the container and places the dump into the folder at `/home/lukas_discopharma_de/backup-db` including the current date in the filename.
The scripts runs weekly mondays at 2 a.m. via a cronjob. You should secure the backups/dumps to a secure location.
### Development Instance
Go step-by-step through the installation and setup of a development metabase instance.
#### 1. Setup VM
Setup the a new VM with specs as described in the [VM specs](#vm-specs) section. The OS we are using is Debian 12. The private ip address can be chosen as `10.156.0.8`
#### 2. Update pkgs and install docker and compose
After Installation of the OS perform a pkg update:
```bash
sudo apt update && sudo apt upgrade -y
```
In order to install docker engine we will follow the official [documentation](https://docs.docker.com/engine/install/debian/).
1. Set up Docker's `apt` repository
```bash
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
# Add the repository to Apt sources:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
```
2. Install the Docker packages (which includes docker compose)
```bash
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
```
3. Verify that the installation is successful by running the `hello-world` image
```bash
sudo docker run hello-world
```
It is possible to manage Docker as a non-root user. It the next steps we describe how to achieve this.
We need to create a `docker` group and add to user we wish to use:
1. Create the `docker` group
```bash
sudo groupadd docker
```
2. Add your user to the `docker` group.
```bash
sudo usermod -aG docker $USER
```
3. Log out and log back in so that your group membership is re-evaluated
4. Verify that you can run `docker` commands without `sudo`
```bash
docker run hello-world
```
#### 3. Create folder and compose file
After getting Docker Engine to work we can setup the necessary files and folders for the metabase container.
Create a metabase folder for the docker compose files in your home folder:
``` bash
mkdir -p ~/metabase/plugins
```
In addition create two files where the database user name and password will be placed:
```bash
touch ~/metabase/{db_user.txt,db_password.txt}
```
Create a `compose.yml` file which will be used to spin up the containers:
```yaml
---
services:
metabase:
image: metabase/metabase:latest
container_name: mb-dev
hostname: mb-dev
restart: unless-stopped
volumes:
- /dev/urandom:/dev/random:ro
- ./plugins:/plugins
ports:
- 3000:3000
environment:
JAVA_TIMEZONE: Europe/Berlin
MB_DB_TYPE: postgres
MB_DB_DBNAME: metabase
MB_DB_PORT: 5432
MB_DB_USER_FILE: /run/secrets/db_user
MB_DB_PASS_FILE: /run/secrets/db_password
MB_DB_HOST: postgres
networks:
- metanet1
secrets:
- db_password
- db_user
healthcheck:
test: curl --fail -I http://localhost:3000/api/health || exit 1
interval: 15s
timeout: 5s
retries: 5
postgres:
image: postgres:latest
container_name: postgres-dev
hostname: postgres-dev
restart: unless-stopped
environment:
POSTGRES_USER_FILE: /run/secrets/db_user
POSTGRES_DB: metabase
POSTGRES_PASSWORD_FILE: /run/secrets/db_password
networks:
- metanet1
secrets:
- db_password
- db_user
networks:
metanet1:
driver: bridge
secrets:
db_password:
file: db_password.txt
db_user:
file: db_user.txt
```
Choose a name for the database user and place it in the `db_user.txt` file, e.g.:
```bash
echo "metabase" > db_user.txt
```
and accordingly for the password:
```bash
echo "SecurePass" > db_password.txt
```
Change the permissions of the files such that they are read-only for your own user:
```bash
chmod 400 db_*.txt
```
#### 4. Pull images and start container
The pull of the container images and the start of the containers can be simply done by one command. Change the working directory to the metabase folder,
```bash
cd ~/metabase
```
and execute the command:
```bash
docker compose up -d
```
During the startup the log files for the containers should be monitored for possible errors by using the command:
```bash
docker compose logs -f
```
If you see now errors and if you have the possibility to reach the server you can visit the metabase instance using the URL `http://<private-ip-of-server>:3000`. Port 3000 has to be open and you have to be able to reache the server via its private ip address.
## Reverse Proxy
The software which is used on the reverse proxy server is called `nginx`. This is a standard common web server/reverse proxy. Its configuration files reside in the folder `/etc/nginx/` and its log files can be found in `/var/logs/nginx/`.
The configuration file which accomplishes the reverse proxying for your metabase instance is `/etc/nginx/sites-available/metabase.conf`:
```conf
server {
listen 80;
listen [::]:80;
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name metabase.discopharma.de;
ssl_certificate /etc/nginx/ssl/discopharma.de/discopharma_fullchain.cer;
ssl_certificate_key /etc/nginx/ssl/discopharma.de/discopharma_private.key;
if ($ssl_protocol = "") {
return 301 https://$server_name$request_uri;
}
location / {
proxy_pass http://10.156.0.6:3000;
proxy_set_header HOST $host;
}
}
```
In order to reverse proxy traffic to a development instance you can proceed as follows:
1. Create a nginx configuration file for the dev metabase instance by copying the existing config: `cp /etc/nginx/sites-available/metabase.conf /etc/nginx/sites-available/mb-dev.conf`
2. Open the new file using any text editor `nano /etc/nginx/sites-available/mb-dev.conf` and edit the `server_name` and `proxy_pass` variables to reflect your new dev instance, e.g.: `server_name mb-dev.discopharma.de;` (the corresponding dns entry for `mb-dev.discopharma.de` has to point to the public ip of the reverse proxy) and `proxy_pass http:<private-ip-of-server>:3000;`
3. Create a symbolic link (nignx reads the config files in `sites-enabled`):
```bash
ln -sf /etc/nginx/sites-available/mb-dev.conf /etc/nginx/sites-enabled/
```
4. Restart the `nignx` service: `systemctl restart nginx`
5. Setup your google firewall such that the reverse proxy can reach your dev metabase instance via port 3000.
6. Visit `https://mb-dev.discopharma.de`. The homepage should working ssl certificates which are configured in the `nginx` configuration file for mb-dev.