Rename phy-z-srv-gpu01 to phy-srv-gpu01; fix share analysis scripts
- rename host/group/folder everywhere to match the server's actual hostname and physical label - share-analysis.ps1: sanitize the output filename prefix — '-Paths "D:"' produced 'D:-file-types.csv' and Export-Csv failed with 'path format not supported', so no CSVs were written - share-analysis.ps1: new -FolderDepth so folders can be aggregated at D:\Abteilungen\<Share> level, which matches the share layout - list-shares.ps1: -WithSize walks local paths instead of UNC when run on the server itself (UNC was orders of magnitude slower and looked stuck) and prints progress every 50k files Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.8
@@ -41,12 +41,12 @@ server/<hostname>/ # Host-specific docs, scripts, files
|
||||
## Servers
|
||||
|
||||
| Hostname | IP | Status |
|
||||
| ---------------- | ------------- | ------------------------------- |
|
||||
| ---------------- | ------------- | ---------------------------------- |
|
||||
| phy-z-srv-jira | 192.168.66.41 | configured |
|
||||
| phy-z-srv-cloud | 192.168.66.66 | configured (Nextcloud) |
|
||||
| phy-z-srv-git | 192.168.66.67 | configured |
|
||||
| phy-z-dmz-sftp01 | 192.168.99.68 | configured |
|
||||
| phy-z-srv-gpu01 | 192.168.66.69 | planned — setup is the next task|
|
||||
| phy-srv-gpu01 | 192.168.66.69 | in setup — base + GPU driver done |
|
||||
|
||||
## Conventions
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ Document all the steps done for the initial generic setup of the server.
|
||||
- [x] Document Server in IT-Glue configurations with all infos: IP address, SN, MAC, Hostname, Set as 'planned', ...
|
||||
- [x] Document iLO of the Server in IT-Glue configurations: IP address, SN, MAC, Hostname, ....
|
||||
- [x] Document iLO license in TI-Glue under 'Licensing'
|
||||
- [x] Label physical server: `phy-z-srv-gpu01`
|
||||
- [x] Label physical server: `phy-srv-gpu01`
|
||||
- [x] iLO setup: Hostname, snmp (v1 disable, configure v3), license was already inserted,
|
||||
- [x] RAID setup of server in BIOS
|
||||
- [ ] OS Install
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
# TODO
|
||||
|
||||
## phy-z-srv-gpu01
|
||||
## phy-srv-gpu01
|
||||
|
||||
Per [projektplan](server/phy-z-srv-gpu01/notes/20260710-projektplan.md) (§ references below).
|
||||
Per [projektplan](server/phy-srv-gpu01/notes/20260710-projektplan.md) (§ references below).
|
||||
Server is delivered; base setup + NVIDIA driver are done (see `SETUP.md`).
|
||||
|
||||
### Blocking the knowledge base (customer conversation)
|
||||
@@ -32,8 +32,8 @@ Server is delivered; base setup + NVIDIA driver are done (see `SETUP.md`).
|
||||
- [ ] TLS: currently plain HTTP on `chat.phytron.local`; retrofit an internal CA
|
||||
certificate (AD passwords travel in clear text until then)
|
||||
- [ ] Hostname mismatch: server reports `phy-srv-gpu01`, repo/inventory/label use
|
||||
`phy-z-srv-gpu01` — align
|
||||
- [ ] Finish base hardening in `SETUP.md` (ssh, updates) via `just run phy_z_srv_gpu01`
|
||||
`phy-srv-gpu01` — align
|
||||
- [ ] Finish base hardening in `SETUP.md` (ssh, updates) via `just run phy_srv_gpu01`
|
||||
|
||||
### Done
|
||||
|
||||
|
||||
+2
-2
@@ -10,5 +10,5 @@ phy-z-srv-git ansible_host=192.168.66.67
|
||||
[phy_z_dmz_sftp01]
|
||||
phy-z-dmz-sftp01 ansible_host=192.168.99.68
|
||||
|
||||
[phy_z_srv_gpu01]
|
||||
phy-z-srv-gpu01 ansible_host=192.168.66.69
|
||||
[phy_srv_gpu01]
|
||||
phy-srv-gpu01 ansible_host=192.168.66.69
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
# Automates server/phy-z-srv-gpu01/manuals/20260714-nvidia-driver-install.md
|
||||
# Automates server/phy-srv-gpu01/manuals/20260714-nvidia-driver-install.md
|
||||
# Requires Docker (geerlingguy.docker) for the container toolkit part.
|
||||
|
||||
- name: Check Secure Boot state
|
||||
|
||||
+1
-1
@@ -69,7 +69,7 @@
|
||||
tags: base
|
||||
|
||||
|
||||
- hosts: phy_z_srv_gpu01
|
||||
- hosts: phy_srv_gpu01
|
||||
become: yes
|
||||
vars_files:
|
||||
- "group_vars/secrets.yml"
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
# phy-z-srv-gpu01
|
||||
# phy-srv-gpu01
|
||||
|
||||
GPU server for AI/ML workloads. Hardware is ordered/assessed; OS setup and configuration are the next step.
|
||||
|
||||
| | |
|
||||
| - | - |
|
||||
| IP | 192.168.66.69 (planned) |
|
||||
| Ansible group | `phy_z_srv_gpu01` |
|
||||
| Ansible group | `phy_srv_gpu01` |
|
||||
| Status | planned — not yet configured |
|
||||
|
||||
## Hardware
|
||||
+4
-4
@@ -1,4 +1,4 @@
|
||||
# Projektplan — Setup phy-z-srv-gpu01 (LLM-Server mit SMB-Wissensbasis)
|
||||
# Projektplan — Setup phy-srv-gpu01 (LLM-Server mit SMB-Wissensbasis)
|
||||
|
||||
Date: 2026-07-10 (Stand-Update: 2026-09-03)
|
||||
Status: **In Umsetzung** — Server geliefert, Basis-Setup + GPU-Treiber fertig
|
||||
@@ -144,7 +144,7 @@ und Indexgröße (§2.6).
|
||||
|
||||
### 2.4 Ansible vorbereiten (im Repo, testbar ohne GPU)
|
||||
|
||||
- `run.yml`: Play für `phy_z_srv_gpu01` ergänzen
|
||||
- `run.yml`: Play für `phy_srv_gpu01` ergänzen
|
||||
- Rollen-Skelett unter `ansible/roles/` (Details erst bei Umsetzung):
|
||||
- `nvidia_gpu` — Treiber (≥ 580), Container Toolkit, optional MIG, DCGM
|
||||
- `cifs_mounts` — ro-Mounts, Credentials aus `group_vars/secrets.yml`
|
||||
@@ -152,7 +152,7 @@ und Indexgröße (§2.6).
|
||||
Embedding-Server, Open WebUI + pgvector, Reverse Proxy (TLS), oikb-Timer
|
||||
- Bestehendes nachnutzen: `geerlingguy.security` (Basis-Härtung),
|
||||
`geerlingguy.docker`
|
||||
- `group_vars/phy_z_srv_gpu01.yml`: nur Overrides (Modellname, VRAM-Quote,
|
||||
- `group_vars/phy_srv_gpu01.yml`: nur Overrides (Modellname, VRAM-Quote,
|
||||
Share-Liste, LDAP-Parameter)
|
||||
- Alles außer GPU-Rolle ist vorab in einer Wegwerf-VM testbar (`just run`-Pfad)
|
||||
|
||||
@@ -210,7 +210,7 @@ wenn der Vollindex schlechte Treffer liefert.
|
||||
|
||||
| Phase | Inhalt | Ergebnis/Abnahme |
|
||||
|---|---|---|
|
||||
| **1. Basis** (Woche 1) | Rack/Strom (600-W-GPU!), iLO, Firmware, RAID, Ubuntu 24.04 LTS, Eintrag in Ansible-Basis-Setup (Security, Pakete, Nutzer) | `just run phy_z_srv_gpu01` läuft grün |
|
||||
| **1. Basis** (Woche 1) | Rack/Strom (600-W-GPU!), iLO, Firmware, RAID, Ubuntu 24.04 LTS, Eintrag in Ansible-Basis-Setup (Security, Pakete, Nutzer) | `just run phy_srv_gpu01` läuft grün |
|
||||
| **2. GPU-Stack** (Woche 1–2) | Rolle `nvidia_gpu`: Treiber, Container Toolkit, DCGM; **Burn-in unter Dauerlast** (SM120-Risiken, Deep Dive §1.6); MIG erst mal **aus** | `nvidia-smi` ok, 48 h-Lasttest ohne Reset |
|
||||
| **3. Inference** (Woche 2) | vLLM-Container (NVIDIA-Build) mit gewähltem Modell, feste VRAM-Quote, Embedding-Server daneben; Benchmark mit Eval-Set | deutsche Antworten ok, Ziel-Parallelität erreicht |
|
||||
| **4. UI + Auth** (Woche 2–3) | Open WebUI + pgvector, LDAP-Login, Reverse Proxy + TLS, vorkonfigurierter „Phytron-Assistent" | Login mit AD-Konto, Chat läuft |
|
||||
+36
-9
@@ -16,8 +16,20 @@
|
||||
.\list-shares.ps1
|
||||
|
||||
.EXAMPLE
|
||||
# remote, and with size per share (slow on large shares)
|
||||
.\list-shares.ps1 -ComputerName Z-FILESERVER -WithSize
|
||||
# with size per share — run this ON the file server, it then walks the local
|
||||
# paths instead of the UNC paths (far faster). Still minutes to hours on
|
||||
# millions of files; progress is printed every 50k files.
|
||||
.\list-shares.ps1 -WithSize
|
||||
|
||||
.NOTES
|
||||
Faster alternative when all shares live under one drive (as on Z-FILESERVER,
|
||||
where everything sits under D:): run share-analysis.ps1 once with a matching
|
||||
folder depth instead of measuring every share separately, e.g.
|
||||
|
||||
.\share-analysis.ps1 -Paths "D:" -FolderDepth 2
|
||||
|
||||
That produces size/count per D:\Abteilungen\<Share> in a single pass over
|
||||
the disk, which is what the share list here maps onto.
|
||||
#>
|
||||
param(
|
||||
[string]$ComputerName = $env:COMPUTERNAME,
|
||||
@@ -55,13 +67,28 @@ $result = foreach ($sh in $shares) {
|
||||
|
||||
$sizeGB = $null; $fileCount = $null
|
||||
if ($WithSize) {
|
||||
Write-Host " measuring size ..." -ForegroundColor DarkGray
|
||||
try {
|
||||
$m = Get-ChildItem -LiteralPath "\\$ComputerName\$($sh.Name)" -Recurse -File -Force -ErrorAction SilentlyContinue |
|
||||
Measure-Object -Property Length -Sum
|
||||
$sizeGB = [Math]::Round($m.Sum / 1GB, 2)
|
||||
$fileCount = $m.Count
|
||||
} catch { }
|
||||
# Walk the LOCAL path when we are on the server itself — going through
|
||||
# the UNC path (\\server\share) routes every single file through the SMB
|
||||
# stack and is slower by orders of magnitude on large shares.
|
||||
$scanPath = if ($ComputerName -eq $env:COMPUTERNAME -and $sh.Path) {
|
||||
$sh.Path
|
||||
} else {
|
||||
"\\$ComputerName\$($sh.Name)"
|
||||
}
|
||||
|
||||
$sw = [System.Diagnostics.Stopwatch]::StartNew()
|
||||
$n = 0L; $bytes = 0L
|
||||
Get-ChildItem -LiteralPath $scanPath -Recurse -File -Force -ErrorAction SilentlyContinue |
|
||||
ForEach-Object {
|
||||
$n++; $bytes += $_.Length
|
||||
if ($n % 50000 -eq 0) {
|
||||
Write-Host (" ... {0:N0} files, {1:N1} GB ({2:N0}s)" -f $n, ($bytes / 1GB), $sw.Elapsed.TotalSeconds) -ForegroundColor DarkGray
|
||||
}
|
||||
}
|
||||
$sw.Stop()
|
||||
$sizeGB = [Math]::Round($bytes / 1GB, 2)
|
||||
$fileCount = $n
|
||||
Write-Host (" {0:N0} files, {1:N1} GB in {2:N0}s" -f $n, ($bytes / 1GB), $sw.Elapsed.TotalSeconds) -ForegroundColor DarkGray
|
||||
}
|
||||
|
||||
[PSCustomObject]@{
|
||||
+23
-3
@@ -26,7 +26,11 @@ param(
|
||||
[string]$OutDir = (Join-Path (Get-Location) ("share-analysis-" + (Get-Date -Format "yyyyMMdd-HHmmss"))),
|
||||
|
||||
# PDFs sampled per share for the text-layer check
|
||||
[int]$PdfSampleSize = 200
|
||||
[int]$PdfSampleSize = 200,
|
||||
|
||||
# How many folder levels to aggregate in the *-toplevel-folders.csv.
|
||||
# 1 = D:\Abteilungen, 2 = D:\Abteilungen\AA (matches the share layout here).
|
||||
[int]$FolderDepth = 1
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Continue'
|
||||
@@ -80,7 +84,17 @@ function Test-PdfTextLayer {
|
||||
# --- per-share pass ----------------------------------------------------------
|
||||
|
||||
foreach ($root in $Paths) {
|
||||
# Label used for the output filenames. Must not contain characters that are
|
||||
# illegal in a filename — e.g. -Paths "D:" would otherwise produce
|
||||
# "D:-file-types.csv" and Export-Csv fails with "path format not supported".
|
||||
$shareName = ($root.TrimEnd('\') -split '[\\/]')[-1]
|
||||
if (-not $shareName) { $shareName = 'root' }
|
||||
foreach ($c in [System.IO.Path]::GetInvalidFileNameChars()) {
|
||||
$shareName = $shareName.Replace($c, '_')
|
||||
}
|
||||
$shareName = $shareName.TrimEnd('_', '.', ' ')
|
||||
if (-not $shareName) { $shareName = 'root' }
|
||||
|
||||
Write-Host "=== Analyzing '$root' ..." -ForegroundColor Cyan
|
||||
|
||||
if (-not (Test-Path -LiteralPath $root)) {
|
||||
@@ -118,9 +132,15 @@ foreach ($root in $Paths) {
|
||||
$extStats[$ext].Count++
|
||||
$extStats[$ext].Bytes += $_.Length
|
||||
|
||||
# top-level folder relative to the share root
|
||||
# folder relative to the share root, aggregated at -FolderDepth levels
|
||||
$rel = $_.FullName.Substring($rootLen).TrimStart('\')
|
||||
$top = if ($rel.Contains('\')) { $rel.Split('\')[0] } else { '(root)' }
|
||||
$parts = $rel.Split('\')
|
||||
if ($parts.Count -le 1) {
|
||||
$top = '(root)'
|
||||
} else {
|
||||
$n = [Math]::Min($FolderDepth, $parts.Count - 1)
|
||||
$top = ($parts[0..($n - 1)] -join '\')
|
||||
}
|
||||
if (-not $topStats.ContainsKey($top)) { $topStats[$top] = @{ Count = 0L; Bytes = 0L } }
|
||||
$topStats[$top].Count++
|
||||
$topStats[$top].Bytes += $_.Length
|
||||
Reference in New Issue
Block a user