System Monitoring & Plotting

System Monitoring & Plotting#

orchestrant.monitoring records CPU, memory and GPU (NVIDIA and AMD) metrics while Python code runs, summarises them and plots them.

  • Metrics: CPU and RAM usage; GPU utilization, memory, temperature and power draw (NVIDIA everywhere, AMD on Linux or via ADL PMLog).

  • Logging: configurable sampling intervals, structured metric storage, statistical summaries.

  • Plots (matplotlib): CPU and memory, GPU utilization, GPU memory and GPU temperature, alone or as one dashboard; interactive, or exported to PNG/PDF.

Installation#

The dependencies come with the package:

pip install -e .

  • psutil - CPU and memory monitoring

  • nvidia-ml-py - NVIDIA GPU monitoring (optional, NVIDIA hosts only)

  • matplotlib - Visualization

AMD needs no extra package: on Windows the monitor uses the ADL library that ships with the graphics driver, on Linux the amdgpu sysfs counters. The gpu-rocm extra installs AMD SMI (amdsmi), which is optional and only resolves the card’s marketing name.

Quick start#

from orchestrant import SystemMonitor, MetricsPlotter, quick_plot

monitor = SystemMonitor(interval=0.5)
monitor.start()

for i in range(10):
    # Do some work
    monitor.record()  # Take a snapshot

monitor.stop()
monitor.print_summary()

plotter = MetricsPlotter(monitor.get_metrics())
plotter.plot_all()
plotter.save_figure("output/metrics.png")
plotter.show()  # Display interactively

# or in one line:
quick_plot(monitor.get_metrics(), output_path="metrics.png", show=True)

The same start/record/stop pattern profiles a training run, a long-running batch job (a larger interval, record() per batch) or two implementations side by side (one monitor each, then print_summary() on both).

python examples/monitoring.py runs three examples: basic monitoring, monitoring with visualization, and continuous background monitoring through different workload phases.

API#

SystemMonitor#

SystemMonitor(interval=1.0, gpu_index=0)

  • interval: Time between measurements (seconds)

  • gpu_index: GPU device index to monitor

Methods: start() starts a session, record() records a snapshot, stop() stops the session, get_metrics() returns every snapshot, print_summary() prints the statistics.

SystemMetrics#

One snapshot:

Attribute

Meaning

timestamp

Unix timestamp

cpu_percent

CPU usage (%)

memory_percent

Memory usage (%)

memory_used_mb

Used memory (MB)

memory_available_mb

Available memory (MB)

gpu_utilization

GPU usage (%), optional

gpu_memory_used_mb

GPU memory used (MB), optional

gpu_memory_total_mb

Total GPU memory (MB), optional

gpu_temperature

GPU temperature (°C), optional

MetricsPlotter#

MetricsPlotter(metrics: Sequence[SystemMetrics])

  • plot_cpu_memory(ax=None), plot_gpu_utilization(ax=None), plot_gpu_memory(ax=None), plot_gpu_temperature(ax=None) - one plot each

  • plot_all(figsize=(14, 10)) - the whole dashboard

  • save_figure(filepath, dpi=150) - save to a file

  • show() - display interactively

quick_plot()#

quick_plot(metrics, output_path=None, *, show=True)

GPU support#

The monitor probes NVIDIA first (NVML, the nvidia-ml-py extra), then AMD, and exposes the same metrics either way:

  • NVIDIA: any host where nvidia-ml-py imports; requires the driver.

  • AMD on Windows: the AMD Display Library (atiadlxx.dll) installed by the graphics driver — no Python package. Utilization, temperature and power come from the PMLog sensors; VRAM from the adapter memory APIs. AMD adapters are ordered with the largest dedicated VRAM first, so gpu_index=0 is the discrete GPU on an APU+dGPU machine.

  • AMD on Linux: the amdgpu sysfs counters under /sys/class/drm/card*/device — no package, no privileges.

If no backend answers, monitoring runs without GPU metrics. GPUProbe.vendor names the vendor that answered ("nvidia", "amd" or "none") and GPUProbe.backend the mechanism (nvml, adl, sysfs).

Troubleshooting#

GPU not detected#

NVIDIA:

  1. The driver is installed:

    nvidia-smi

  2. nvidia-ml-py is installed:

    pip list | grep nvidia-ml-py

  3. NVML answers:

    import pynvml

pynvml.nvmlInit()
print(pynvml.nvmlDeviceGetCount())

AMD:

  1. Linux: the kernel driver is loaded and the counters exist:

    ls /sys/class/drm/card*/device/gpu_busy_percent

    Monitoring needs no root and no ROCm installation; amdsmi (uv sync --extra gpu-rocm) only improves the reported card name.

  2. Windows: the graphics driver is installed (atiadlxx.dll is part of it) and another monitoring tool can see the card. The monitor probes it lazily; when in doubt, check GPUProbe().vendor and GPUProbe().backend:

    from orchestrant.monitoring.gpu import GPUProbe

probe = GPUProbe()
print(probe.vendor, probe.backend, probe.gpu_name)
print(probe.read())

High memory usage#

The monitor keeps every snapshot in memory. For a long session use a larger sampling interval, save and clear the metrics periodically, or stream them to a database.