Rename phy-z-srv-gpu01 to phy-srv-gpu01; fix share analysis scripts

- rename host/group/folder everywhere to match the server's actual
  hostname and physical label
- share-analysis.ps1: sanitize the output filename prefix — '-Paths "D:"'
  produced 'D:-file-types.csv' and Export-Csv failed with 'path format
  not supported', so no CSVs were written
- share-analysis.ps1: new -FolderDepth so folders can be aggregated at
  D:\Abteilungen\<Share> level, which matches the share layout
- list-shares.ps1: -WithSize walks local paths instead of UNC when run on
  the server itself (UNC was orders of magnitude slower and looked stuck)
  and prints progress every 50k files

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-09-03 14:47:07 +02:00
co-authored by Claude Opus 4.8
parent 9c38c60783
commit 75296ce58c
16 changed files with 81 additions and 34 deletions
+7 -7
View File
@@ -40,13 +40,13 @@ server/<hostname>/ # Host-specific docs, scripts, files
## Servers ## Servers
| Hostname | IP | Status | | Hostname | IP | Status |
| ---------------- | ------------- | ------------------------------- | | ---------------- | ------------- | ---------------------------------- |
| phy-z-srv-jira | 192.168.66.41 | configured | | phy-z-srv-jira | 192.168.66.41 | configured |
| phy-z-srv-cloud | 192.168.66.66 | configured (Nextcloud) | | phy-z-srv-cloud | 192.168.66.66 | configured (Nextcloud) |
| phy-z-srv-git | 192.168.66.67 | configured | | phy-z-srv-git | 192.168.66.67 | configured |
| phy-z-dmz-sftp01 | 192.168.99.68 | configured | | phy-z-dmz-sftp01 | 192.168.99.68 | configured |
| phy-z-srv-gpu01 | 192.168.66.69 | planned — setup is the next task| | phy-srv-gpu01 | 192.168.66.69 | in setup — base + GPU driver done |
## Conventions ## Conventions
+1 -1
View File
@@ -10,7 +10,7 @@ Document all the steps done for the initial generic setup of the server.
- [x] Document Server in IT-Glue configurations with all infos: IP address, SN, MAC, Hostname, Set as 'planned', ... - [x] Document Server in IT-Glue configurations with all infos: IP address, SN, MAC, Hostname, Set as 'planned', ...
- [x] Document iLO of the Server in IT-Glue configurations: IP address, SN, MAC, Hostname, .... - [x] Document iLO of the Server in IT-Glue configurations: IP address, SN, MAC, Hostname, ....
- [x] Document iLO license in TI-Glue under 'Licensing' - [x] Document iLO license in TI-Glue under 'Licensing'
- [x] Label physical server: `phy-z-srv-gpu01` - [x] Label physical server: `phy-srv-gpu01`
- [x] iLO setup: Hostname, snmp (v1 disable, configure v3), license was already inserted, - [x] iLO setup: Hostname, snmp (v1 disable, configure v3), license was already inserted,
- [x] RAID setup of server in BIOS - [x] RAID setup of server in BIOS
- [ ] OS Install - [ ] OS Install
+4 -4
View File
@@ -1,8 +1,8 @@
# TODO # TODO
## phy-z-srv-gpu01 ## phy-srv-gpu01
Per [projektplan](server/phy-z-srv-gpu01/notes/20260710-projektplan.md) (§ references below). Per [projektplan](server/phy-srv-gpu01/notes/20260710-projektplan.md) (§ references below).
Server is delivered; base setup + NVIDIA driver are done (see `SETUP.md`). Server is delivered; base setup + NVIDIA driver are done (see `SETUP.md`).
### Blocking the knowledge base (customer conversation) ### Blocking the knowledge base (customer conversation)
@@ -32,8 +32,8 @@ Server is delivered; base setup + NVIDIA driver are done (see `SETUP.md`).
- [ ] TLS: currently plain HTTP on `chat.phytron.local`; retrofit an internal CA - [ ] TLS: currently plain HTTP on `chat.phytron.local`; retrofit an internal CA
certificate (AD passwords travel in clear text until then) certificate (AD passwords travel in clear text until then)
- [ ] Hostname mismatch: server reports `phy-srv-gpu01`, repo/inventory/label use - [ ] Hostname mismatch: server reports `phy-srv-gpu01`, repo/inventory/label use
`phy-z-srv-gpu01` — align `phy-srv-gpu01` — align
- [ ] Finish base hardening in `SETUP.md` (ssh, updates) via `just run phy_z_srv_gpu01` - [ ] Finish base hardening in `SETUP.md` (ssh, updates) via `just run phy_srv_gpu01`
### Done ### Done
+2 -2
View File
@@ -10,5 +10,5 @@ phy-z-srv-git ansible_host=192.168.66.67
[phy_z_dmz_sftp01] [phy_z_dmz_sftp01]
phy-z-dmz-sftp01 ansible_host=192.168.99.68 phy-z-dmz-sftp01 ansible_host=192.168.99.68
[phy_z_srv_gpu01] [phy_srv_gpu01]
phy-z-srv-gpu01 ansible_host=192.168.66.69 phy-srv-gpu01 ansible_host=192.168.66.69
+1 -1
View File
@@ -1,5 +1,5 @@
--- ---
# Automates server/phy-z-srv-gpu01/manuals/20260714-nvidia-driver-install.md # Automates server/phy-srv-gpu01/manuals/20260714-nvidia-driver-install.md
# Requires Docker (geerlingguy.docker) for the container toolkit part. # Requires Docker (geerlingguy.docker) for the container toolkit part.
- name: Check Secure Boot state - name: Check Secure Boot state
+1 -1
View File
@@ -69,7 +69,7 @@
tags: base tags: base
- hosts: phy_z_srv_gpu01 - hosts: phy_srv_gpu01
become: yes become: yes
vars_files: vars_files:
- "group_vars/secrets.yml" - "group_vars/secrets.yml"
@@ -1,11 +1,11 @@
# phy-z-srv-gpu01 # phy-srv-gpu01
GPU server for AI/ML workloads. Hardware is ordered/assessed; OS setup and configuration are the next step. GPU server for AI/ML workloads. Hardware is ordered/assessed; OS setup and configuration are the next step.
| | | | | |
| - | - | | - | - |
| IP | 192.168.66.69 (planned) | | IP | 192.168.66.69 (planned) |
| Ansible group | `phy_z_srv_gpu01` | | Ansible group | `phy_srv_gpu01` |
| Status | planned — not yet configured | | Status | planned — not yet configured |
## Hardware ## Hardware
@@ -1,4 +1,4 @@
# Projektplan — Setup phy-z-srv-gpu01 (LLM-Server mit SMB-Wissensbasis) # Projektplan — Setup phy-srv-gpu01 (LLM-Server mit SMB-Wissensbasis)
Date: 2026-07-10 (Stand-Update: 2026-09-03) Date: 2026-07-10 (Stand-Update: 2026-09-03)
Status: **In Umsetzung** — Server geliefert, Basis-Setup + GPU-Treiber fertig Status: **In Umsetzung** — Server geliefert, Basis-Setup + GPU-Treiber fertig
@@ -144,7 +144,7 @@ und Indexgröße (§2.6).
### 2.4 Ansible vorbereiten (im Repo, testbar ohne GPU) ### 2.4 Ansible vorbereiten (im Repo, testbar ohne GPU)
- `run.yml`: Play für `phy_z_srv_gpu01` ergänzen - `run.yml`: Play für `phy_srv_gpu01` ergänzen
- Rollen-Skelett unter `ansible/roles/` (Details erst bei Umsetzung): - Rollen-Skelett unter `ansible/roles/` (Details erst bei Umsetzung):
- `nvidia_gpu` — Treiber (≥ 580), Container Toolkit, optional MIG, DCGM - `nvidia_gpu` — Treiber (≥ 580), Container Toolkit, optional MIG, DCGM
- `cifs_mounts` — ro-Mounts, Credentials aus `group_vars/secrets.yml` - `cifs_mounts` — ro-Mounts, Credentials aus `group_vars/secrets.yml`
@@ -152,7 +152,7 @@ und Indexgröße (§2.6).
Embedding-Server, Open WebUI + pgvector, Reverse Proxy (TLS), oikb-Timer Embedding-Server, Open WebUI + pgvector, Reverse Proxy (TLS), oikb-Timer
- Bestehendes nachnutzen: `geerlingguy.security` (Basis-Härtung), - Bestehendes nachnutzen: `geerlingguy.security` (Basis-Härtung),
`geerlingguy.docker` `geerlingguy.docker`
- `group_vars/phy_z_srv_gpu01.yml`: nur Overrides (Modellname, VRAM-Quote, - `group_vars/phy_srv_gpu01.yml`: nur Overrides (Modellname, VRAM-Quote,
Share-Liste, LDAP-Parameter) Share-Liste, LDAP-Parameter)
- Alles außer GPU-Rolle ist vorab in einer Wegwerf-VM testbar (`just run`-Pfad) - Alles außer GPU-Rolle ist vorab in einer Wegwerf-VM testbar (`just run`-Pfad)
@@ -210,7 +210,7 @@ wenn der Vollindex schlechte Treffer liefert.
| Phase | Inhalt | Ergebnis/Abnahme | | Phase | Inhalt | Ergebnis/Abnahme |
|---|---|---| |---|---|---|
| **1. Basis** (Woche 1) | Rack/Strom (600-W-GPU!), iLO, Firmware, RAID, Ubuntu 24.04 LTS, Eintrag in Ansible-Basis-Setup (Security, Pakete, Nutzer) | `just run phy_z_srv_gpu01` läuft grün | | **1. Basis** (Woche 1) | Rack/Strom (600-W-GPU!), iLO, Firmware, RAID, Ubuntu 24.04 LTS, Eintrag in Ansible-Basis-Setup (Security, Pakete, Nutzer) | `just run phy_srv_gpu01` läuft grün |
| **2. GPU-Stack** (Woche 12) | Rolle `nvidia_gpu`: Treiber, Container Toolkit, DCGM; **Burn-in unter Dauerlast** (SM120-Risiken, Deep Dive §1.6); MIG erst mal **aus** | `nvidia-smi` ok, 48 h-Lasttest ohne Reset | | **2. GPU-Stack** (Woche 12) | Rolle `nvidia_gpu`: Treiber, Container Toolkit, DCGM; **Burn-in unter Dauerlast** (SM120-Risiken, Deep Dive §1.6); MIG erst mal **aus** | `nvidia-smi` ok, 48 h-Lasttest ohne Reset |
| **3. Inference** (Woche 2) | vLLM-Container (NVIDIA-Build) mit gewähltem Modell, feste VRAM-Quote, Embedding-Server daneben; Benchmark mit Eval-Set | deutsche Antworten ok, Ziel-Parallelität erreicht | | **3. Inference** (Woche 2) | vLLM-Container (NVIDIA-Build) mit gewähltem Modell, feste VRAM-Quote, Embedding-Server daneben; Benchmark mit Eval-Set | deutsche Antworten ok, Ziel-Parallelität erreicht |
| **4. UI + Auth** (Woche 23) | Open WebUI + pgvector, LDAP-Login, Reverse Proxy + TLS, vorkonfigurierter „Phytron-Assistent" | Login mit AD-Konto, Chat läuft | | **4. UI + Auth** (Woche 23) | Open WebUI + pgvector, LDAP-Login, Reverse Proxy + TLS, vorkonfigurierter „Phytron-Assistent" | Login mit AD-Konto, Chat läuft |
@@ -16,8 +16,20 @@
.\list-shares.ps1 .\list-shares.ps1
.EXAMPLE .EXAMPLE
# remote, and with size per share (slow on large shares) # with size per share — run this ON the file server, it then walks the local
.\list-shares.ps1 -ComputerName Z-FILESERVER -WithSize # paths instead of the UNC paths (far faster). Still minutes to hours on
# millions of files; progress is printed every 50k files.
.\list-shares.ps1 -WithSize
.NOTES
Faster alternative when all shares live under one drive (as on Z-FILESERVER,
where everything sits under D:): run share-analysis.ps1 once with a matching
folder depth instead of measuring every share separately, e.g.
.\share-analysis.ps1 -Paths "D:" -FolderDepth 2
That produces size/count per D:\Abteilungen\<Share> in a single pass over
the disk, which is what the share list here maps onto.
#> #>
param( param(
[string]$ComputerName = $env:COMPUTERNAME, [string]$ComputerName = $env:COMPUTERNAME,
@@ -55,13 +67,28 @@ $result = foreach ($sh in $shares) {
$sizeGB = $null; $fileCount = $null $sizeGB = $null; $fileCount = $null
if ($WithSize) { if ($WithSize) {
Write-Host " measuring size ..." -ForegroundColor DarkGray # Walk the LOCAL path when we are on the server itself — going through
try { # the UNC path (\\server\share) routes every single file through the SMB
$m = Get-ChildItem -LiteralPath "\\$ComputerName\$($sh.Name)" -Recurse -File -Force -ErrorAction SilentlyContinue | # stack and is slower by orders of magnitude on large shares.
Measure-Object -Property Length -Sum $scanPath = if ($ComputerName -eq $env:COMPUTERNAME -and $sh.Path) {
$sizeGB = [Math]::Round($m.Sum / 1GB, 2) $sh.Path
$fileCount = $m.Count } else {
} catch { } "\\$ComputerName\$($sh.Name)"
}
$sw = [System.Diagnostics.Stopwatch]::StartNew()
$n = 0L; $bytes = 0L
Get-ChildItem -LiteralPath $scanPath -Recurse -File -Force -ErrorAction SilentlyContinue |
ForEach-Object {
$n++; $bytes += $_.Length
if ($n % 50000 -eq 0) {
Write-Host (" ... {0:N0} files, {1:N1} GB ({2:N0}s)" -f $n, ($bytes / 1GB), $sw.Elapsed.TotalSeconds) -ForegroundColor DarkGray
}
}
$sw.Stop()
$sizeGB = [Math]::Round($bytes / 1GB, 2)
$fileCount = $n
Write-Host (" {0:N0} files, {1:N1} GB in {2:N0}s" -f $n, ($bytes / 1GB), $sw.Elapsed.TotalSeconds) -ForegroundColor DarkGray
} }
[PSCustomObject]@{ [PSCustomObject]@{
@@ -26,7 +26,11 @@ param(
[string]$OutDir = (Join-Path (Get-Location) ("share-analysis-" + (Get-Date -Format "yyyyMMdd-HHmmss"))), [string]$OutDir = (Join-Path (Get-Location) ("share-analysis-" + (Get-Date -Format "yyyyMMdd-HHmmss"))),
# PDFs sampled per share for the text-layer check # PDFs sampled per share for the text-layer check
[int]$PdfSampleSize = 200 [int]$PdfSampleSize = 200,
# How many folder levels to aggregate in the *-toplevel-folders.csv.
# 1 = D:\Abteilungen, 2 = D:\Abteilungen\AA (matches the share layout here).
[int]$FolderDepth = 1
) )
$ErrorActionPreference = 'Continue' $ErrorActionPreference = 'Continue'
@@ -80,7 +84,17 @@ function Test-PdfTextLayer {
# --- per-share pass ---------------------------------------------------------- # --- per-share pass ----------------------------------------------------------
foreach ($root in $Paths) { foreach ($root in $Paths) {
# Label used for the output filenames. Must not contain characters that are
# illegal in a filename — e.g. -Paths "D:" would otherwise produce
# "D:-file-types.csv" and Export-Csv fails with "path format not supported".
$shareName = ($root.TrimEnd('\') -split '[\\/]')[-1] $shareName = ($root.TrimEnd('\') -split '[\\/]')[-1]
if (-not $shareName) { $shareName = 'root' }
foreach ($c in [System.IO.Path]::GetInvalidFileNameChars()) {
$shareName = $shareName.Replace($c, '_')
}
$shareName = $shareName.TrimEnd('_', '.', ' ')
if (-not $shareName) { $shareName = 'root' }
Write-Host "=== Analyzing '$root' ..." -ForegroundColor Cyan Write-Host "=== Analyzing '$root' ..." -ForegroundColor Cyan
if (-not (Test-Path -LiteralPath $root)) { if (-not (Test-Path -LiteralPath $root)) {
@@ -118,9 +132,15 @@ foreach ($root in $Paths) {
$extStats[$ext].Count++ $extStats[$ext].Count++
$extStats[$ext].Bytes += $_.Length $extStats[$ext].Bytes += $_.Length
# top-level folder relative to the share root # folder relative to the share root, aggregated at -FolderDepth levels
$rel = $_.FullName.Substring($rootLen).TrimStart('\') $rel = $_.FullName.Substring($rootLen).TrimStart('\')
$top = if ($rel.Contains('\')) { $rel.Split('\')[0] } else { '(root)' } $parts = $rel.Split('\')
if ($parts.Count -le 1) {
$top = '(root)'
} else {
$n = [Math]::Min($FolderDepth, $parts.Count - 1)
$top = ($parts[0..($n - 1)] -join '\')
}
if (-not $topStats.ContainsKey($top)) { $topStats[$top] = @{ Count = 0L; Bytes = 0L } } if (-not $topStats.ContainsKey($top)) { $topStats[$top] = @{ Count = 0L; Bytes = 0L } }
$topStats[$top].Count++ $topStats[$top].Count++
$topStats[$top].Bytes += $_.Length $topStats[$top].Bytes += $_.Length