Rename phy-z-srv-gpu01 to phy-srv-gpu01; fix share analysis scripts
- rename host/group/folder everywhere to match the server's actual hostname and physical label - share-analysis.ps1: sanitize the output filename prefix — '-Paths "D:"' produced 'D:-file-types.csv' and Export-Csv failed with 'path format not supported', so no CSVs were written - share-analysis.ps1: new -FolderDepth so folders can be aggregated at D:\Abteilungen\<Share> level, which matches the share layout - list-shares.ps1: -WithSize walks local paths instead of UNC when run on the server itself (UNC was orders of magnitude slower and looked stuck) and prints progress every 50k files Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -41,12 +41,12 @@ server/<hostname>/ # Host-specific docs, scripts, files
|
|||||||
## Servers
|
## Servers
|
||||||
|
|
||||||
| Hostname | IP | Status |
|
| Hostname | IP | Status |
|
||||||
| ---------------- | ------------- | ------------------------------- |
|
| ---------------- | ------------- | ---------------------------------- |
|
||||||
| phy-z-srv-jira | 192.168.66.41 | configured |
|
| phy-z-srv-jira | 192.168.66.41 | configured |
|
||||||
| phy-z-srv-cloud | 192.168.66.66 | configured (Nextcloud) |
|
| phy-z-srv-cloud | 192.168.66.66 | configured (Nextcloud) |
|
||||||
| phy-z-srv-git | 192.168.66.67 | configured |
|
| phy-z-srv-git | 192.168.66.67 | configured |
|
||||||
| phy-z-dmz-sftp01 | 192.168.99.68 | configured |
|
| phy-z-dmz-sftp01 | 192.168.99.68 | configured |
|
||||||
| phy-z-srv-gpu01 | 192.168.66.69 | planned — setup is the next task|
|
| phy-srv-gpu01 | 192.168.66.69 | in setup — base + GPU driver done |
|
||||||
|
|
||||||
## Conventions
|
## Conventions
|
||||||
|
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ Document all the steps done for the initial generic setup of the server.
|
|||||||
- [x] Document Server in IT-Glue configurations with all infos: IP address, SN, MAC, Hostname, Set as 'planned', ...
|
- [x] Document Server in IT-Glue configurations with all infos: IP address, SN, MAC, Hostname, Set as 'planned', ...
|
||||||
- [x] Document iLO of the Server in IT-Glue configurations: IP address, SN, MAC, Hostname, ....
|
- [x] Document iLO of the Server in IT-Glue configurations: IP address, SN, MAC, Hostname, ....
|
||||||
- [x] Document iLO license in TI-Glue under 'Licensing'
|
- [x] Document iLO license in TI-Glue under 'Licensing'
|
||||||
- [x] Label physical server: `phy-z-srv-gpu01`
|
- [x] Label physical server: `phy-srv-gpu01`
|
||||||
- [x] iLO setup: Hostname, snmp (v1 disable, configure v3), license was already inserted,
|
- [x] iLO setup: Hostname, snmp (v1 disable, configure v3), license was already inserted,
|
||||||
- [x] RAID setup of server in BIOS
|
- [x] RAID setup of server in BIOS
|
||||||
- [ ] OS Install
|
- [ ] OS Install
|
||||||
|
|||||||
@@ -1,8 +1,8 @@
|
|||||||
# TODO
|
# TODO
|
||||||
|
|
||||||
## phy-z-srv-gpu01
|
## phy-srv-gpu01
|
||||||
|
|
||||||
Per [projektplan](server/phy-z-srv-gpu01/notes/20260710-projektplan.md) (§ references below).
|
Per [projektplan](server/phy-srv-gpu01/notes/20260710-projektplan.md) (§ references below).
|
||||||
Server is delivered; base setup + NVIDIA driver are done (see `SETUP.md`).
|
Server is delivered; base setup + NVIDIA driver are done (see `SETUP.md`).
|
||||||
|
|
||||||
### Blocking the knowledge base (customer conversation)
|
### Blocking the knowledge base (customer conversation)
|
||||||
@@ -32,8 +32,8 @@ Server is delivered; base setup + NVIDIA driver are done (see `SETUP.md`).
|
|||||||
- [ ] TLS: currently plain HTTP on `chat.phytron.local`; retrofit an internal CA
|
- [ ] TLS: currently plain HTTP on `chat.phytron.local`; retrofit an internal CA
|
||||||
certificate (AD passwords travel in clear text until then)
|
certificate (AD passwords travel in clear text until then)
|
||||||
- [ ] Hostname mismatch: server reports `phy-srv-gpu01`, repo/inventory/label use
|
- [ ] Hostname mismatch: server reports `phy-srv-gpu01`, repo/inventory/label use
|
||||||
`phy-z-srv-gpu01` — align
|
`phy-srv-gpu01` — align
|
||||||
- [ ] Finish base hardening in `SETUP.md` (ssh, updates) via `just run phy_z_srv_gpu01`
|
- [ ] Finish base hardening in `SETUP.md` (ssh, updates) via `just run phy_srv_gpu01`
|
||||||
|
|
||||||
### Done
|
### Done
|
||||||
|
|
||||||
|
|||||||
+2
-2
@@ -10,5 +10,5 @@ phy-z-srv-git ansible_host=192.168.66.67
|
|||||||
[phy_z_dmz_sftp01]
|
[phy_z_dmz_sftp01]
|
||||||
phy-z-dmz-sftp01 ansible_host=192.168.99.68
|
phy-z-dmz-sftp01 ansible_host=192.168.99.68
|
||||||
|
|
||||||
[phy_z_srv_gpu01]
|
[phy_srv_gpu01]
|
||||||
phy-z-srv-gpu01 ansible_host=192.168.66.69
|
phy-srv-gpu01 ansible_host=192.168.66.69
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
---
|
---
|
||||||
# Automates server/phy-z-srv-gpu01/manuals/20260714-nvidia-driver-install.md
|
# Automates server/phy-srv-gpu01/manuals/20260714-nvidia-driver-install.md
|
||||||
# Requires Docker (geerlingguy.docker) for the container toolkit part.
|
# Requires Docker (geerlingguy.docker) for the container toolkit part.
|
||||||
|
|
||||||
- name: Check Secure Boot state
|
- name: Check Secure Boot state
|
||||||
|
|||||||
+1
-1
@@ -69,7 +69,7 @@
|
|||||||
tags: base
|
tags: base
|
||||||
|
|
||||||
|
|
||||||
- hosts: phy_z_srv_gpu01
|
- hosts: phy_srv_gpu01
|
||||||
become: yes
|
become: yes
|
||||||
vars_files:
|
vars_files:
|
||||||
- "group_vars/secrets.yml"
|
- "group_vars/secrets.yml"
|
||||||
|
|||||||
@@ -1,11 +1,11 @@
|
|||||||
# phy-z-srv-gpu01
|
# phy-srv-gpu01
|
||||||
|
|
||||||
GPU server for AI/ML workloads. Hardware is ordered/assessed; OS setup and configuration are the next step.
|
GPU server for AI/ML workloads. Hardware is ordered/assessed; OS setup and configuration are the next step.
|
||||||
|
|
||||||
| | |
|
| | |
|
||||||
| - | - |
|
| - | - |
|
||||||
| IP | 192.168.66.69 (planned) |
|
| IP | 192.168.66.69 (planned) |
|
||||||
| Ansible group | `phy_z_srv_gpu01` |
|
| Ansible group | `phy_srv_gpu01` |
|
||||||
| Status | planned — not yet configured |
|
| Status | planned — not yet configured |
|
||||||
|
|
||||||
## Hardware
|
## Hardware
|
||||||
+4
-4
@@ -1,4 +1,4 @@
|
|||||||
# Projektplan — Setup phy-z-srv-gpu01 (LLM-Server mit SMB-Wissensbasis)
|
# Projektplan — Setup phy-srv-gpu01 (LLM-Server mit SMB-Wissensbasis)
|
||||||
|
|
||||||
Date: 2026-07-10 (Stand-Update: 2026-09-03)
|
Date: 2026-07-10 (Stand-Update: 2026-09-03)
|
||||||
Status: **In Umsetzung** — Server geliefert, Basis-Setup + GPU-Treiber fertig
|
Status: **In Umsetzung** — Server geliefert, Basis-Setup + GPU-Treiber fertig
|
||||||
@@ -144,7 +144,7 @@ und Indexgröße (§2.6).
|
|||||||
|
|
||||||
### 2.4 Ansible vorbereiten (im Repo, testbar ohne GPU)
|
### 2.4 Ansible vorbereiten (im Repo, testbar ohne GPU)
|
||||||
|
|
||||||
- `run.yml`: Play für `phy_z_srv_gpu01` ergänzen
|
- `run.yml`: Play für `phy_srv_gpu01` ergänzen
|
||||||
- Rollen-Skelett unter `ansible/roles/` (Details erst bei Umsetzung):
|
- Rollen-Skelett unter `ansible/roles/` (Details erst bei Umsetzung):
|
||||||
- `nvidia_gpu` — Treiber (≥ 580), Container Toolkit, optional MIG, DCGM
|
- `nvidia_gpu` — Treiber (≥ 580), Container Toolkit, optional MIG, DCGM
|
||||||
- `cifs_mounts` — ro-Mounts, Credentials aus `group_vars/secrets.yml`
|
- `cifs_mounts` — ro-Mounts, Credentials aus `group_vars/secrets.yml`
|
||||||
@@ -152,7 +152,7 @@ und Indexgröße (§2.6).
|
|||||||
Embedding-Server, Open WebUI + pgvector, Reverse Proxy (TLS), oikb-Timer
|
Embedding-Server, Open WebUI + pgvector, Reverse Proxy (TLS), oikb-Timer
|
||||||
- Bestehendes nachnutzen: `geerlingguy.security` (Basis-Härtung),
|
- Bestehendes nachnutzen: `geerlingguy.security` (Basis-Härtung),
|
||||||
`geerlingguy.docker`
|
`geerlingguy.docker`
|
||||||
- `group_vars/phy_z_srv_gpu01.yml`: nur Overrides (Modellname, VRAM-Quote,
|
- `group_vars/phy_srv_gpu01.yml`: nur Overrides (Modellname, VRAM-Quote,
|
||||||
Share-Liste, LDAP-Parameter)
|
Share-Liste, LDAP-Parameter)
|
||||||
- Alles außer GPU-Rolle ist vorab in einer Wegwerf-VM testbar (`just run`-Pfad)
|
- Alles außer GPU-Rolle ist vorab in einer Wegwerf-VM testbar (`just run`-Pfad)
|
||||||
|
|
||||||
@@ -210,7 +210,7 @@ wenn der Vollindex schlechte Treffer liefert.
|
|||||||
|
|
||||||
| Phase | Inhalt | Ergebnis/Abnahme |
|
| Phase | Inhalt | Ergebnis/Abnahme |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| **1. Basis** (Woche 1) | Rack/Strom (600-W-GPU!), iLO, Firmware, RAID, Ubuntu 24.04 LTS, Eintrag in Ansible-Basis-Setup (Security, Pakete, Nutzer) | `just run phy_z_srv_gpu01` läuft grün |
|
| **1. Basis** (Woche 1) | Rack/Strom (600-W-GPU!), iLO, Firmware, RAID, Ubuntu 24.04 LTS, Eintrag in Ansible-Basis-Setup (Security, Pakete, Nutzer) | `just run phy_srv_gpu01` läuft grün |
|
||||||
| **2. GPU-Stack** (Woche 1–2) | Rolle `nvidia_gpu`: Treiber, Container Toolkit, DCGM; **Burn-in unter Dauerlast** (SM120-Risiken, Deep Dive §1.6); MIG erst mal **aus** | `nvidia-smi` ok, 48 h-Lasttest ohne Reset |
|
| **2. GPU-Stack** (Woche 1–2) | Rolle `nvidia_gpu`: Treiber, Container Toolkit, DCGM; **Burn-in unter Dauerlast** (SM120-Risiken, Deep Dive §1.6); MIG erst mal **aus** | `nvidia-smi` ok, 48 h-Lasttest ohne Reset |
|
||||||
| **3. Inference** (Woche 2) | vLLM-Container (NVIDIA-Build) mit gewähltem Modell, feste VRAM-Quote, Embedding-Server daneben; Benchmark mit Eval-Set | deutsche Antworten ok, Ziel-Parallelität erreicht |
|
| **3. Inference** (Woche 2) | vLLM-Container (NVIDIA-Build) mit gewähltem Modell, feste VRAM-Quote, Embedding-Server daneben; Benchmark mit Eval-Set | deutsche Antworten ok, Ziel-Parallelität erreicht |
|
||||||
| **4. UI + Auth** (Woche 2–3) | Open WebUI + pgvector, LDAP-Login, Reverse Proxy + TLS, vorkonfigurierter „Phytron-Assistent" | Login mit AD-Konto, Chat läuft |
|
| **4. UI + Auth** (Woche 2–3) | Open WebUI + pgvector, LDAP-Login, Reverse Proxy + TLS, vorkonfigurierter „Phytron-Assistent" | Login mit AD-Konto, Chat läuft |
|
||||||
+36
-9
@@ -16,8 +16,20 @@
|
|||||||
.\list-shares.ps1
|
.\list-shares.ps1
|
||||||
|
|
||||||
.EXAMPLE
|
.EXAMPLE
|
||||||
# remote, and with size per share (slow on large shares)
|
# with size per share — run this ON the file server, it then walks the local
|
||||||
.\list-shares.ps1 -ComputerName Z-FILESERVER -WithSize
|
# paths instead of the UNC paths (far faster). Still minutes to hours on
|
||||||
|
# millions of files; progress is printed every 50k files.
|
||||||
|
.\list-shares.ps1 -WithSize
|
||||||
|
|
||||||
|
.NOTES
|
||||||
|
Faster alternative when all shares live under one drive (as on Z-FILESERVER,
|
||||||
|
where everything sits under D:): run share-analysis.ps1 once with a matching
|
||||||
|
folder depth instead of measuring every share separately, e.g.
|
||||||
|
|
||||||
|
.\share-analysis.ps1 -Paths "D:" -FolderDepth 2
|
||||||
|
|
||||||
|
That produces size/count per D:\Abteilungen\<Share> in a single pass over
|
||||||
|
the disk, which is what the share list here maps onto.
|
||||||
#>
|
#>
|
||||||
param(
|
param(
|
||||||
[string]$ComputerName = $env:COMPUTERNAME,
|
[string]$ComputerName = $env:COMPUTERNAME,
|
||||||
@@ -55,13 +67,28 @@ $result = foreach ($sh in $shares) {
|
|||||||
|
|
||||||
$sizeGB = $null; $fileCount = $null
|
$sizeGB = $null; $fileCount = $null
|
||||||
if ($WithSize) {
|
if ($WithSize) {
|
||||||
Write-Host " measuring size ..." -ForegroundColor DarkGray
|
# Walk the LOCAL path when we are on the server itself — going through
|
||||||
try {
|
# the UNC path (\\server\share) routes every single file through the SMB
|
||||||
$m = Get-ChildItem -LiteralPath "\\$ComputerName\$($sh.Name)" -Recurse -File -Force -ErrorAction SilentlyContinue |
|
# stack and is slower by orders of magnitude on large shares.
|
||||||
Measure-Object -Property Length -Sum
|
$scanPath = if ($ComputerName -eq $env:COMPUTERNAME -and $sh.Path) {
|
||||||
$sizeGB = [Math]::Round($m.Sum / 1GB, 2)
|
$sh.Path
|
||||||
$fileCount = $m.Count
|
} else {
|
||||||
} catch { }
|
"\\$ComputerName\$($sh.Name)"
|
||||||
|
}
|
||||||
|
|
||||||
|
$sw = [System.Diagnostics.Stopwatch]::StartNew()
|
||||||
|
$n = 0L; $bytes = 0L
|
||||||
|
Get-ChildItem -LiteralPath $scanPath -Recurse -File -Force -ErrorAction SilentlyContinue |
|
||||||
|
ForEach-Object {
|
||||||
|
$n++; $bytes += $_.Length
|
||||||
|
if ($n % 50000 -eq 0) {
|
||||||
|
Write-Host (" ... {0:N0} files, {1:N1} GB ({2:N0}s)" -f $n, ($bytes / 1GB), $sw.Elapsed.TotalSeconds) -ForegroundColor DarkGray
|
||||||
|
}
|
||||||
|
}
|
||||||
|
$sw.Stop()
|
||||||
|
$sizeGB = [Math]::Round($bytes / 1GB, 2)
|
||||||
|
$fileCount = $n
|
||||||
|
Write-Host (" {0:N0} files, {1:N1} GB in {2:N0}s" -f $n, ($bytes / 1GB), $sw.Elapsed.TotalSeconds) -ForegroundColor DarkGray
|
||||||
}
|
}
|
||||||
|
|
||||||
[PSCustomObject]@{
|
[PSCustomObject]@{
|
||||||
+23
-3
@@ -26,7 +26,11 @@ param(
|
|||||||
[string]$OutDir = (Join-Path (Get-Location) ("share-analysis-" + (Get-Date -Format "yyyyMMdd-HHmmss"))),
|
[string]$OutDir = (Join-Path (Get-Location) ("share-analysis-" + (Get-Date -Format "yyyyMMdd-HHmmss"))),
|
||||||
|
|
||||||
# PDFs sampled per share for the text-layer check
|
# PDFs sampled per share for the text-layer check
|
||||||
[int]$PdfSampleSize = 200
|
[int]$PdfSampleSize = 200,
|
||||||
|
|
||||||
|
# How many folder levels to aggregate in the *-toplevel-folders.csv.
|
||||||
|
# 1 = D:\Abteilungen, 2 = D:\Abteilungen\AA (matches the share layout here).
|
||||||
|
[int]$FolderDepth = 1
|
||||||
)
|
)
|
||||||
|
|
||||||
$ErrorActionPreference = 'Continue'
|
$ErrorActionPreference = 'Continue'
|
||||||
@@ -80,7 +84,17 @@ function Test-PdfTextLayer {
|
|||||||
# --- per-share pass ----------------------------------------------------------
|
# --- per-share pass ----------------------------------------------------------
|
||||||
|
|
||||||
foreach ($root in $Paths) {
|
foreach ($root in $Paths) {
|
||||||
|
# Label used for the output filenames. Must not contain characters that are
|
||||||
|
# illegal in a filename — e.g. -Paths "D:" would otherwise produce
|
||||||
|
# "D:-file-types.csv" and Export-Csv fails with "path format not supported".
|
||||||
$shareName = ($root.TrimEnd('\') -split '[\\/]')[-1]
|
$shareName = ($root.TrimEnd('\') -split '[\\/]')[-1]
|
||||||
|
if (-not $shareName) { $shareName = 'root' }
|
||||||
|
foreach ($c in [System.IO.Path]::GetInvalidFileNameChars()) {
|
||||||
|
$shareName = $shareName.Replace($c, '_')
|
||||||
|
}
|
||||||
|
$shareName = $shareName.TrimEnd('_', '.', ' ')
|
||||||
|
if (-not $shareName) { $shareName = 'root' }
|
||||||
|
|
||||||
Write-Host "=== Analyzing '$root' ..." -ForegroundColor Cyan
|
Write-Host "=== Analyzing '$root' ..." -ForegroundColor Cyan
|
||||||
|
|
||||||
if (-not (Test-Path -LiteralPath $root)) {
|
if (-not (Test-Path -LiteralPath $root)) {
|
||||||
@@ -118,9 +132,15 @@ foreach ($root in $Paths) {
|
|||||||
$extStats[$ext].Count++
|
$extStats[$ext].Count++
|
||||||
$extStats[$ext].Bytes += $_.Length
|
$extStats[$ext].Bytes += $_.Length
|
||||||
|
|
||||||
# top-level folder relative to the share root
|
# folder relative to the share root, aggregated at -FolderDepth levels
|
||||||
$rel = $_.FullName.Substring($rootLen).TrimStart('\')
|
$rel = $_.FullName.Substring($rootLen).TrimStart('\')
|
||||||
$top = if ($rel.Contains('\')) { $rel.Split('\')[0] } else { '(root)' }
|
$parts = $rel.Split('\')
|
||||||
|
if ($parts.Count -le 1) {
|
||||||
|
$top = '(root)'
|
||||||
|
} else {
|
||||||
|
$n = [Math]::Min($FolderDepth, $parts.Count - 1)
|
||||||
|
$top = ($parts[0..($n - 1)] -join '\')
|
||||||
|
}
|
||||||
if (-not $topStats.ContainsKey($top)) { $topStats[$top] = @{ Count = 0L; Bytes = 0L } }
|
if (-not $topStats.ContainsKey($top)) { $topStats[$top] = @{ Count = 0L; Bytes = 0L } }
|
||||||
$topStats[$top].Count++
|
$topStats[$top].Count++
|
||||||
$topStats[$top].Bytes += $_.Length
|
$topStats[$top].Bytes += $_.Length
|
||||||
Reference in New Issue
Block a user