CubelaPetar/notes
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

]

Browse Source
This commit is contained in:
Petar Cubela
2025-07-15 15:36:24 +02:00
parent f6398bf45c
commit 252a91dbcc
162 changed files with 4584 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files

65
archive/discopharma/20250311-metabase-environment.md Normal file
View File

@@ -0,0 +1,65 @@
## VM Ressources and Setup
The listed IP Addresses are only example values here and can be chosen on your judgement. Important is that the machines can communicate with each other.
### MySQL Database
- Name: MySQL Database
- OS: Debian 12
- hostname: db.discopharma.de (unimportant)
- IP Address: 10.156.0.5/24
- CPU: 1 core
- RAM: 2 GB (2048 MB)
- Storage: depends (30 GB)
- DNS entry: none
- Note: for every 40 concurrent users: needs 1CPU and 1GB of RAM more
### Metabase Server
- Name: Metabase Server
- OS: Debian 12
- hostname: mb.discopharma.de (unimportant)
- IP Address: 10.156.0.6/24
- CPU: 1 core
- RAM: 1 GB (1024 MB)
- Storage: depends (30 GB)
- DNS entry: none
- Note: for every 20 concurrent users: needs 1CPU and 2GB of RAM more
### Reverse Proxy
- Name: Reverse Proxy
- OS: Debian 12
- hostname: rproxy.discopharma.de (unimportant)
- IP Address: 10.156.0.7/24 + \<PUBLIC IP\> address (only activated in the end)
- CPU: 1 core
- RAM: 1 GB (1024 MB)
- Storage: depends (16 GB)
- DNS entry: metabase.discopharma.de -> \<PUBLIC IP\>
- Note: for every concurrent users: needs 1CPU and 2GB of RAM more
## SSL/TSL certificates
- we need the discopharma wildcard certificate placed on the Reverse Proxy
- usually two files enough called `privkey.pem` and `fullchain.pem`
- you can put all the cert files on the reverse proxy and we will then use only the needed ones or convert them in the process if necessary
## Firewall Setup
I list all necessary communications and respective ports needed:
(Abbreviations:
- Databse: db = 10.156.0.5
- Metabse: mb = 10.156.0.6
- ReverseProxy: rp = 10.156.0.7)
| Source | SourcePort | Destination | DestPort | Description |
| ------------- | ----------------- | --------------- | ----------------- | ------------------------------------------------------------------------------------------- |
| mb | 3306/tcp | db | 3306/tcp | 3306 is the standard mysql port. Communication of mb to db |
| rp | 3000/tcp,3000/udp | mb | 3000/tcp,3000/udp | 3000 is the metabase web port (arbitrary). Reverse Proxy sends request via this port to mb. |
| OPEN INTERNET | any | PUBLIC IP of rp | 443/tcp | 443 is the https port to communicate to rp over internet |
You could also limit the access to the public ip such that only your company ip can reach it. The 443 port should be opened as the last thing when everything is done.
When the VMs are in the same private network, they should be able to openly communicate with each other; the first two entries in the table should be already open.

19
archive/discopharma/20250312-metabase-deployment.md Normal file
View File

@@ -0,0 +1,19 @@
## Metabase Instance
### Requirements
- [x] unattended-updates
- [x] docker
### Database
- name: metabase
- user: metabase
- pass: /E^bOu|<C{Y{bZu
### Reverse Proxy
- [x] unattended-updates
- [x] fail2ban
- [x] nginx

300
archive/discopharma/20250320-manual-project.md Normal file
View File

@@ -0,0 +1,300 @@
---
title: "Metabase - Setup Manual"
author: Petar Cubela
date: March 20, 2025
geometry: margin=1.5cm
output: pdf_document
---
## Intro
Setting up a Metabase instance via Docker with a PostgreSQL application database and a secure web connection via https mediated by a public facing reverse proxy (nginx) and commercial TLS/SSL certificates.
### Goals and Requirements
### Software
- Google Cloud Platform (GCP)
- [Debain 12 (OS)](https://www.debian.org/download)
- [Docker (Containerization Platform)](https://docs.docker.com/engine/install/debian/)
- [NGINX (Web Server, Reverse Proxy)](https://docs.nginx.com/)
- [Postgres (as Container)](https://hub.docker.com/_/postgres)
- [Metabase (as Container)](https://hub.docker.com/r/metabase/metabase)
## VM Specs
### Metabase Server
- Name: Metabase Server
- OS: Debian 12
- hostname: mb-prod
- IP Address: `10.156.0.6/24`
- CPU: 2 core
- RAM: 2 GB (2048 MB)
- Storage: depends (30 GB)
- DNS entry: none
- Note: for every 20 concurrent users: needs 1CPU and 2GB of RAM more
### Reverse Proxy
- Name: Reverse Proxy
- OS: Debian 12
- hostname: rproxy
- IP Address: `10.156.0.7/24` + `<PUBLIC IP>` address (only activated in the end)
- CPU: 1 core
- RAM: 1 GB (1024 MB)
- Storage: depends (16 GB)
- DNS entry: metabase.discopharma.de -> `<PUBLIC IP>`
- Note: for every concurrent users: needs 1CPU and 2GB of RAM more
### Firewall
I list all necessary communications and respective ports needed:
Abbreviations:
- Metabse: mb-prod = `10.156.0.6`
- Metabse Dev: mb-dev = `10.156.0.8`
- ReverseProxy: rp = `10.156.0.7`
| Source | SourcePort | Destination | DestPort | Description |
| ------------- | ----------------------- | --------------- | ----------------------- | ------------------------------------------------------------------------------- |
| mb-prod | 3306/tcp | db | 3306/tcp | 3306 is the standard mysql port. Communication of mb-prod to db |
| rp | 3000/tcp </br> 3000/udp | mb-prod | 3000/tcp </br> 3000/udp | 3000 is the metabase web port. Reverse Proxy sends request via this port to mb. |
| rp | 3000/tcp </br>3000/udp | mb-dev | 3000/tcp </br> 3000/udp | 3000 is the metabase web port. Reverse Proxy sends request via this port to mb. |
| OPEN INTERNET | any | PUBLIC IP of rp | 443/tcp | 443 is the https port to communicate to rp over internet |
### Network Diagram
![diagram](/files/discopharma/discopharma-infra.drawio.png)
## Metabase Application Server and Database
### Administration
#### Update
In order to update the metabase containers change to the `~/metabase/` folder (where `compose.yml` file resides) and use the following command:
```bash
docker compose pull && docker compose up -d
```
Monitor the container logs to see if there are any errors by using the command:
```bash
docker compose logs -f
```
The `docker compose pull` command searches for images which are specified by a tag in the image variable in the `compose.yml` file:
`image: metabase/metabase:latest`
`latest` is here the tag and can also be changed to a version number which can be extracted from the [docker-hub](https://hub.docker.com/r/metabase/metabase/tags).
To simplify the process I wrote a simple bash script which updates the container images and removes old container images. The script is in the folder `/home/lukas_discopharma_de/scripts/metabase-update.sh`.
The update has to be done manually.
#### Backup
There is a script `/home/lukas_discopharma_de/db-backup.sh` which creates a database dump from the postgres instance running in the container and places the dump into the folder at `/home/lukas_discopharma_de/backup-db` including the current date in the filename.
The scripts runs weekly mondays at 2 a.m. via a cronjob. You should secure the backups/dumps to a secure location.
### Development Instance
Go step-by-step through the installation and setup of a development metabase instance.
#### 1. Setup VM
Setup the a new VM with specs as described in the [VM specs](#vm-specs) section. The OS we are using is Debian 12. The private ip address can be chosen as `10.156.0.8`
#### 2. Update pkgs and install docker and compose
After Installation of the OS perform a pkg update:
```bash
sudo apt update && sudo apt upgrade -y
```
In order to install docker engine we will follow the official [documentation](https://docs.docker.com/engine/install/debian/).
1. Set up Docker's `apt` repository
```bash
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
# Add the repository to Apt sources:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
```
2. Install the Docker packages (which includes docker compose)
```bash
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
```
3. Verify that the installation is successful by running the `hello-world` image
```bash
sudo docker run hello-world
```
It is possible to manage Docker as a non-root user. It the next steps we describe how to achieve this.
We need to create a `docker` group and add to user we wish to use:
1. Create the `docker` group
```bash
sudo groupadd docker
```
2. Add your user to the `docker` group.
```bash
sudo usermod -aG docker $USER
```
3. Log out and log back in so that your group membership is re-evaluated
4. Verify that you can run `docker` commands without `sudo`
```bash
docker run hello-world
```
#### 3. Create folder and compose file
After getting Docker Engine to work we can setup the necessary files and folders for the metabase container.
Create a metabase folder for the docker compose files in your home folder:
``` bash
mkdir -p ~/metabase/plugins
```
In addition create two files where the database user name and password will be placed:
```bash
touch ~/metabase/{db_user.txt,db_password.txt}
```
Create a `compose.yml` file which will be used to spin up the containers:
```yaml
---
services:
metabase:
image: metabase/metabase:latest
container_name: mb-dev
hostname: mb-dev
restart: unless-stopped
volumes:
- /dev/urandom:/dev/random:ro
- ./plugins:/plugins
ports:
- 3000:3000
environment:
JAVA_TIMEZONE: Europe/Berlin
MB_DB_TYPE: postgres
MB_DB_DBNAME: metabase
MB_DB_PORT: 5432
MB_DB_USER_FILE: /run/secrets/db_user
MB_DB_PASS_FILE: /run/secrets/db_password
MB_DB_HOST: postgres
networks:
- metanet1
secrets:
- db_password
- db_user
healthcheck:
test: curl --fail -I http://localhost:3000/api/health || exit 1
interval: 15s
timeout: 5s
retries: 5
postgres:
image: postgres:latest
container_name: postgres-dev
hostname: postgres-dev
restart: unless-stopped
environment:
POSTGRES_USER_FILE: /run/secrets/db_user
POSTGRES_DB: metabase
POSTGRES_PASSWORD_FILE: /run/secrets/db_password
networks:
- metanet1
secrets:
- db_password
- db_user
networks:
metanet1:
driver: bridge
secrets:
db_password:
file: db_password.txt
db_user:
file: db_user.txt
```
Choose a name for the database user and place it in the `db_user.txt` file, e.g.:
```bash
echo "metabase" > db_user.txt
```
and accordingly for the password:
```bash
echo "SecurePass" > db_password.txt
```
Change the permissions of the files such that they are read-only for your own user:
```bash
chmod 400 db_*.txt
```
#### 4. Pull images and start container
The pull of the container images and the start of the containers can be simply done by one command. Change the working directory to the metabase folder,
```bash
cd ~/metabase
```
and execute the command:
```bash
docker compose up -d
```
During the startup the log files for the containers should be monitored for possible errors by using the command:
```bash
docker compose logs -f
```
If you see now errors and if you have the possibility to reach the server you can visit the metabase instance using the URL `http://<private-ip-of-server>:3000`. Port 3000 has to be open and you have to be able to reache the server via its private ip address.
## Reverse Proxy
The software which is used on the reverse proxy server is called `nginx`. This is a standard common web server/reverse proxy. Its configuration files reside in the folder `/etc/nginx/` and its log files can be found in `/var/logs/nginx/`.
The configuration file which accomplishes the reverse proxying for your metabase instance is `/etc/nginx/sites-available/metabase.conf`:
```conf
server {
listen 80;
listen [::]:80;
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name metabase.discopharma.de;
ssl_certificate /etc/nginx/ssl/discopharma.de/discopharma_fullchain.cer;
ssl_certificate_key /etc/nginx/ssl/discopharma.de/discopharma_private.key;
if ($ssl_protocol = "") {
return 301 https://$server_name$request_uri;
}
location / {
proxy_pass http://10.156.0.6:3000;
proxy_set_header HOST $host;
}
}
```
In order to reverse proxy traffic to a development instance you can proceed as follows:
1. Create a nginx configuration file for the dev metabase instance by copying the existing config: `cp /etc/nginx/sites-available/metabase.conf /etc/nginx/sites-available/mb-dev.conf`
2. Open the new file using any text editor `nano /etc/nginx/sites-available/mb-dev.conf` and edit the `server_name` and `proxy_pass` variables to reflect your new dev instance, e.g.: `server_name mb-dev.discopharma.de;` (the corresponding dns entry for `mb-dev.discopharma.de` has to point to the public ip of the reverse proxy) and `proxy_pass http:<private-ip-of-server>:3000;`
3. Create a symbolic link (nignx reads the config files in `sites-enabled`):
```bash
ln -sf /etc/nginx/sites-available/mb-dev.conf /etc/nginx/sites-enabled/
```
4. Restart the `nignx` service: `systemctl restart nginx`
5. Setup your google firewall such that the reverse proxy can reach your dev metabase instance via port 3000.
6. Visit `https://mb-dev.discopharma.de`. The homepage should working ssl certificates which are configured in the `nginx` configuration file for mb-dev.

149
archive/discopharma/Meetings/20250310-Next_Steps.md Normal file
View File

@@ -0,0 +1,149 @@
## Goal
Setup a metabase instance via docker with https support and a professional Deployment Pipeline
## Questions
- Separate Reverse Proxy or local Web Server enough??
- Exisiterende SSL Zertifikate nutzen?
- Kriege ich irgendwie Zugang?
### 20250311
- How many users?
- What is the old db software? Maybe we can reuse it? Are there backups of the old database ?
- DNS Verwaltung
- is the metabase version a requirement?
## Meeting-20250311
Teilnehmer: Lukas Maas, Milos Nikolic, Petar Cubela
### Answers
- DB: MySQL. Backup dump exist.
- Version needs to be 0.49.18
- 20 people
- Existing certs
- Use Reverse Proxy
- I will get access to the machines
### My Time/ Steps
1. Databse Instance MySQL (0.5h -1h)
2. Metabase (.50 h)
3. VM R2verse Proxy (.50 h)
4. Find and Test the recreation of the data/dashboard database (metabase.db/) (1-2h)
5. write overwivew network setup (ip address, open ports in firewall, metabase.discopharma.de -> public ip ) (1h)
6. Recreate in discopharma setup: (2-3h)
1. dns setup properly
2. network setup properly
3. creation of the VMs (oeither discopharma or me)
4. Installation process (db exist, docker deployment of metabase, reverse proxy)
5. Test
## List of requirements regarding Metabase deployment (discopharma)
1. Find or create backup of Metabase Dashboard data within Docker image on the old machine (marketplace image that was compromised, or a previous image of it)
2. Solution architecture that obeys to best practices of security, so that
- DISCO employees can connect to a DISCO-internal metabase application using a web browser and the URL “metabase.discopharma.de”
- The application is not exposed to the public
- All connections to the application are encrypted (https)
3. Solution architecture that includes a
- Productive instance (highest priority)
- Development/sandbox instance (lower priority)
- A process to deploy upgrades of the application (lower priority)
4. Metabase version 0.49.18
## Requirements
- properly configured and firewalled google cloud; VMs should only be able to communicate via private IPs!
- VM in google cloud for the metabase instance; Public IP address, port 80 and 443 forwarded; 1 cores, 2GB RAM (depends on user number)
- VM in google cloud for the metabase database instance; Private IP address; 1 cores, 1GB RAM (depends on user number); PostgreSQL
- Use existing SSL certs(??) with web server/reverse proxy like nginx/traefik/etc
## Software
- Debian 12
- Docker
- Metabase
- PostgreSQL
- Traefik/Nginx (depends)
## Notes
### 20250311
- <https://www.metabase.com/learn/metabase-basics/administration/administration-and-operation/metabase-in-production#metabase-application-server-size>
- Run separate database (PostgreSQL) and application server instances
#### Metabase application server size
- Metabase needs at least 1 core and 1GB of RAM
- For every 20 concurrent people it needs 1CPU and 2GB of RAM
#### Metabase application database server size
- Database needs at least 1 core and 2GB of RAM
- For every 40 concurrent people it needs 1CPU and 1GB of RAM
## docker-compose.yml example
```yml
services:
metabase:
image: metabase/metabase:latest
container_name: metabase
hostname: metabase
restart: unless-stopped
volumes:
- /dev/urandom:/dev/random:ro
- "./metabase-db:/metabase.db"
- ./plugins:/plugins
ports:
- 3000:3000
environment:
JAVA_TIMEZONE: Europe/Berlin
MB_DB_FILE=/metabase.db
MB_DB_TYPE: postgres
MB_DB_DBNAME: metabase
MB_DB_PORT: 5432
MB_DB_USER_FILE: /run/secrets/db_user
MB_DB_PASS_FILE: /run/secrets/db_password
MB_DB_HOST: postgres
networks:
- metanet1
secrets:
- db_password
- db_user
healthcheck:
test: curl --fail -I http://localhost:3000/api/health || exit 1
interval: 15s
timeout: 5s
retries: 5
postgres:
image: postgres:latest
container_name: postgres
hostname: postgres
restart: unless-stopped
environment:
POSTGRES_USER_FILE: /run/secrets/db_user
POSTGRES_DB: metabase
POSTGRES_PASSWORD_FILE: /run/secrets/db_password
networks:
- metanet1
secrets:
- db_password
- db_user
networks:
metanet1:
driver: bridge
secrets:
db_password:
file: db_password.txt
db_user:
file: db_user.txt
```

11
archive/discopharma/Meetings/20250317-finishing-meeting.md Normal file
View File

@@ -0,0 +1,11 @@
## To do's:
- [x] Cloud SQL dump load and user mgmt (Miloš)
- [x] Docker licensing (Lukas)
- [x] backup procedure for MB application db (Petar)
- Documentation/ manual (Petar)
- For example,
- how deployment works,
- what docker image to select
- how the routing in the reverse proxy is done

12
archive/discopharma/Meetings/20250324-meeting_in_prod.md Normal file
View File

@@ -0,0 +1,12 @@
## Members
- Lukas Maas, Petar Cubela
## Topics
- Manual
- Backups of database
- Updates of Container

31
archive/discopharma/reverse-proxy.md Normal file
View File

@@ -0,0 +1,31 @@
## Notes
- I need for testing a separate dns server
- build a test environment on our proxmox instance
## Init Meeting
- reverse proxy aufsetzen
- bestehende maschinen brauchen nicht angefasst zu werden
- ssl zertifikate bestehen
### Steps
- ansible script which sets up caddy with own certificates (research how to run against google cloud engine)
- create plan of reverse proxy function
- test against own infrastructure
- apply with lukas
- create vm with terraform
- provision it with ansible
- change dns entries to point to reverse proxy
- configure reverse proxy for desired domains
## After Meeting
- schreibe kalkulation fuer die umsetzung
## Ansible
- [ ] change caddy such that it uses existing ssl certs