Windows network probe with Blackbox Exporter and vmagent

Purpose

This solution deploys a lightweight network diagnostic probe to Windows servers or endpoints.

It performs local network tests such as:

ICMP reachability and RTT to a gateway, internal devices, and public IPs
TCP connect tests to ports such as TCP/443
DNS lookup tests against a specific resolver
Optional HTTP/HTTPS checks

blackbox_exporter performs the probes locally. vmagent schedules the probes and sends the metrics to VictoriaMetrics.

The key requirement is local buffering: if the remote VictoriaMetrics endpoint is unavailable, vmagent continues collecting metrics and stores unsent data on local disk. Once connectivity returns, it replays the buffered samples with their original timestamps.

Tested component versions

The successful test deployment used:

Component	Version/package
Blackbox Exporter	`blackbox_exporter-0.28.0.windows-amd64.zip`
vmagent	`vmutils-windows-amd64-v1.146.0.zip`
vmagent binary	`vmagent-windows-amd64-prod.exe`
Service wrapper	WinSW x64

Important: download the vmutils Windows package for vmagent.

Do not use the normal VictoriaMetrics single-server package when the aim is to run vmagent. That package contains the VictoriaMetrics database binary, not the agent.

Architecture

Windows endpoint / server
│
├─ Blackbox Exporter
│  └─ Local listener: 127.0.0.1:9115
│
├─ vmagent
│  ├─ Scrapes Blackbox Exporter every 10 seconds
│  ├─ Local listener: 127.0.0.1:8429
│  └─ Persistent queue: C:\NetProbe\queue
│
└─ HTTPS remote-write
   └─ Cloudflare Tunnel public hostname
      └─ Internal VictoriaMetrics server

External clients do not need direct access to the VictoriaMetrics server. The endpoint sends outbound HTTPS to the Cloudflare Tunnel hostname.

VictoriaMetrics ingress through Cloudflare Tunnel

A Cloudflare Tunnel is hosted internally, for example from a DMZ server.

Example public hostname:

https://metrics.example.com/api/v1/write

The Cloudflare public-hostname route should forward only this exact path:

^/api/v1/write$

to the internal VictoriaMetrics remote-write endpoint:

http://<victoriametrics-server>:8428/api/v1/write

Do not expose a catch-all route for this hostname. VMUI and other VictoriaMetrics endpoints should not be published through this ingress hostname.

For a production/shared implementation, protect this endpoint with Cloudflare Access service tokens or another machine-to-machine authentication method.

Package layout

The deployment package is structured as follows:

package\
  Install-NetworkProbe.ps1
  Uninstall-NetworkProbe.ps1
 
  blackbox\
    blackbox_exporter.exe
    blackbox.yml
    blackbox-service.exe
    blackbox-service.xml
 
  vmagent\
    vmagent-windows-amd64-prod.exe
    promscrape.yml
    vmagent-service.exe
    vmagent-service.xml

blackbox-service.exe and vmagent-service.exe are renamed copies of the WinSW executable.

The installed layout is:

C:\NetProbe\
  blackbox\
  vmagent\
  logs\
    blackbox\
    vmagent\
  queue\

Blackbox Exporter configuration

Save as:

blackbox\blackbox.yml

Recommended baseline template:

modules:
  icmp:
    prober: icmp
    timeout: 5s
    icmp:
      preferred_ip_protocol: ip4
 
  tcp_connect:
    prober: tcp
    timeout: 5s
    tcp:
      preferred_ip_protocol: ip4
 
  http_2xx:
    prober: http
    timeout: 10s
    http:
      preferred_ip_protocol: ip4
      valid_status_codes: [200, 204]
      follow_redirects: true
 
  dns_public_cloudflare:
    prober: dns
    timeout: 5s
    dns:
      preferred_ip_protocol: ip4
      transport_protocol: udp
      query_name: "cloudflare.com"
      query_type: "A"
      valid_rcodes: ["NOERROR"]
 
  dns_internal_service:
    prober: dns
    timeout: 5s
    dns:
      preferred_ip_protocol: ip4
      transport_protocol: udp
      query_name: "service.example.internal"
      query_type: "A"
      valid_rcodes: ["NOERROR"]

DNS module behaviour

A DNS module defines what record is queried.

For example:

query_name: "service.example.internal"
query_type: "A"

means Blackbox asks the target DNS server:

What is the A record for service.example.internal?

The DNS resolver IP is configured in promscrape.yml.

A single DNS module uses one configured query name. For several records, create separate modules, for example:

dns_internal_service
dns_internal_ad
dns_public_cloudflare

vmagent scrape configuration

Save as:

vmagent\promscrape.yml

Template:

global:
  scrape_interval: 10s
  scrape_timeout: 8s
 
  external_labels:
    customer: CUSTOMER_NAME
    site: SITE_NAME
    probe_agent: DEVICE_NAME
    environment: prod
    probe_profile: network-diagnostics
 
scrape_configs:
  - job_name: blackbox_icmp
    metrics_path: /probe
    params:
      module: [icmp]
 
    static_configs:
      - targets:
          - 192.168.1.1
        labels:
          target_role: default_gateway
 
      - targets:
          - 10.0.0.53
        labels:
          target_role: internal_dns
 
      - targets:
          - 1.1.1.1
          - 8.8.8.8
        labels:
          target_role: external_ip
 
      - targets:
          - google.co.uk
          - bbc.co.uk
        labels:
          target_role: external_dns
 
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
 
      - source_labels: [__param_target]
        target_label: instance
 
      - target_label: __address__
        replacement: 127.0.0.1:9115
 
  - job_name: blackbox_tcp
    metrics_path: /probe
    params:
      module: [tcp_connect]
 
    static_configs:
      - targets:
          - google.co.uk:443
          - bbc.co.uk:443
        labels:
          target_role: external_tcp
          service: https
 
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
 
      - source_labels: [__param_target]
        target_label: instance
 
      - target_label: __address__
        replacement: 127.0.0.1:9115
 
  - job_name: blackbox_dns_internal
    metrics_path: /probe
    params:
      module: [dns_internal_service]
 
    static_configs:
      - targets:
          - 10.0.0.53
        labels:
          target_role: internal_dns
          dns_query: service.example.internal
          query_type: A
 
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
 
      - source_labels: [__param_target]
        target_label: instance
 
      - target_label: __address__
        replacement: 127.0.0.1:9115

Labels

Use stable labels only.

Recommended vmagent-wide labels:

customer: CUSTOMER_NAME
site: SITE_NAME
probe_agent: DEVICE_NAME
environment: prod
probe_profile: network-diagnostics

Recommended per-target labels:

target_role: default_gateway
target_role: internal_dns
target_role: external_ip
target_role: external_tcp
service: https
dns_query: service.example.internal

Avoid labels that change frequently, such as timestamps, user names, dynamically discovered Wi-Fi details, or ticket-update values.

Avoid duplicate targets

A target must not appear in more than one static_configs block for the same job unless that is intentional.

For example, this creates two time series because target_role differs:

- targets:
    - 10.0.0.20
  labels:
    target_role: Switches
 
- targets:
    - 10.0.0.20
  labels:
    target_role: VMHOSTS

Keep each target in only its correct role group.

WinSW service configuration

Blackbox service XML

Save as:

blackbox\blackbox-service.xml

<service>
  <id>NetworkProbeBlackbox</id>
  <name>Network Probe - Blackbox Exporter</name>
  <description>Local ICMP, TCP, DNS and HTTP probe engine.</description>
 
  <executable>%BASE%\blackbox_exporter.exe</executable>
  <arguments>--config.file="%BASE%\blackbox.yml" --web.listen-address=127.0.0.1:9115</arguments>
 
  <startmode>Automatic</startmode>
  <hidewindow>true</hidewindow>
 
  <logpath>C:\NetProbe\logs\blackbox</logpath>
  <log mode="roll"/>
 
  <onfailure action="restart" delay="10 sec"/>
</service>

vmagent service XML

Save as:

vmagent\vmagent-service.xml

<service>
  <id>NetworkProbeVMAgent</id>
  <name>Network Probe - vmagent</name>
  <description>Schedules local probes and forwards buffered metrics to VictoriaMetrics.</description>
 
  <executable>%BASE%\vmagent-windows-amd64-prod.exe</executable>
  <arguments>
    -promscrape.config="%BASE%\promscrape.yml"
    -remoteWrite.url="https://metrics.example.com/api/v1/write"
    -remoteWrite.tmpDataPath="C:\NetProbe\queue"
    -remoteWrite.maxDiskUsagePerURL=1GB
    -httpListenAddr=127.0.0.1:8429
  </arguments>
 
  <depend>NetworkProbeBlackbox</depend>
 
  <startmode>Automatic</startmode>
  <hidewindow>true</hidewindow>
 
  <logpath>C:\NetProbe\logs\vmagent</logpath>
  <log mode="roll"/>
 
  <onfailure action="restart" delay="10 sec"/>
</service>

The vmagent service depends on Blackbox, so the local probe engine starts first.

Installation

Explicit install script

Save the following as:

package\Install-NetworkProbe.ps1

This script expects the package structure shown earlier. It can be run repeatedly: it stops and removes an existing deployment, replaces the program folders, preserves C:\NetProbe\queue, then installs and starts fresh services.

#requires -RunAsAdministrator
[CmdletBinding()]
param()
 
Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
 
$InstallRoot = 'C:\NetProbe'
$PackageRoot = Split-Path -Parent $PSCommandPath
 
$BlackboxSource = Join-Path $PackageRoot 'blackbox'
$VMAgentSource  = Join-Path $PackageRoot 'vmagent'
 
$BlackboxTarget = Join-Path $InstallRoot 'blackbox'
$VMAgentTarget  = Join-Path $InstallRoot 'vmagent'
 
$BlackboxService = 'NetworkProbeBlackbox'
$VMAgentService  = 'NetworkProbeVMAgent'
 
function Stop-AndRemoveService {
    param(
        [Parameter(Mandatory)]
        [string]$Name,
 
        [Parameter(Mandatory)]
        [string]$WrapperPath
    )
 
    $service = Get-Service -Name $Name -ErrorAction SilentlyContinue
 
    if ($service) {
        Write-Host "Stopping $Name..."
        Stop-Service -Name $Name -Force -ErrorAction SilentlyContinue
 
        if (Test-Path $WrapperPath) {
            Write-Host "Uninstalling $Name via WinSW..."
            & $WrapperPath uninstall
        }
        else {
            Write-Warning "WinSW wrapper not found. Removing $Name using sc.exe."
            & sc.exe delete $Name | Out-Null
        }
 
        Start-Sleep -Seconds 2
    }
}
 
if (-not (Test-Path $BlackboxSource)) {
    throw "Package folder not found: $BlackboxSource"
}
 
if (-not (Test-Path $VMAgentSource)) {
    throw "Package folder not found: $VMAgentSource"
}
 
New-Item -ItemType Directory -Force -Path $InstallRoot | Out-Null
New-Item -ItemType Directory -Force -Path (Join-Path $InstallRoot 'logs\blackbox') | Out-Null
New-Item -ItemType Directory -Force -Path (Join-Path $InstallRoot 'logs\vmagent') | Out-Null
New-Item -ItemType Directory -Force -Path (Join-Path $InstallRoot 'queue') | Out-Null
 
Stop-AndRemoveService `
    -Name $VMAgentService `
    -WrapperPath (Join-Path $VMAgentTarget 'vmagent-service.exe')
 
Stop-AndRemoveService `
    -Name $BlackboxService `
    -WrapperPath (Join-Path $BlackboxTarget 'blackbox-service.exe')
 
Remove-Item -Path $BlackboxTarget -Recurse -Force -ErrorAction SilentlyContinue
Remove-Item -Path $VMAgentTarget -Recurse -Force -ErrorAction SilentlyContinue
 
Copy-Item -Path $BlackboxSource -Destination $BlackboxTarget -Recurse -Force
Copy-Item -Path $VMAgentSource  -Destination $VMAgentTarget  -Recurse -Force
 
$BlackboxWrapper = Join-Path $BlackboxTarget 'blackbox-service.exe'
$VMAgentWrapper  = Join-Path $VMAgentTarget 'vmagent-service.exe'
 
if (-not (Test-Path $BlackboxWrapper)) {
    throw "Blackbox WinSW wrapper not found: $BlackboxWrapper"
}
 
if (-not (Test-Path $VMAgentWrapper)) {
    throw "vmagent WinSW wrapper not found: $VMAgentWrapper"
}
 
Write-Host 'Installing Blackbox Exporter service...'
& $BlackboxWrapper install
 
Write-Host 'Installing vmagent service...'
& $VMAgentWrapper install
 
Start-Service -Name $BlackboxService
Start-Service -Name $VMAgentService
 
Start-Sleep -Seconds 3
 
$services = Get-Service -Name $BlackboxService, $VMAgentService
$services | Format-Table Name, Status, StartType -AutoSize
 
$failed = $services | Where-Object Status -ne 'Running'
if ($failed) {
    throw "One or more services did not start. Check C:\NetProbe\logs."
}
 
Write-Host 'Network Probe installation completed successfully.'

Explicit uninstall script

Save the following as:

package\Uninstall-NetworkProbe.ps1

By default it removes the Windows services and program folders but retains C:\NetProbe\logs and C:\NetProbe\queue for investigation. Use -RemoveData only when the logs and queued metrics can be discarded.

#requires -RunAsAdministrator
[CmdletBinding()]
param(
    [switch]$RemoveData
)
 
Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
 
$InstallRoot = 'C:\NetProbe'
$BlackboxTarget = Join-Path $InstallRoot 'blackbox'
$VMAgentTarget  = Join-Path $InstallRoot 'vmagent'
 
$BlackboxService = 'NetworkProbeBlackbox'
$VMAgentService  = 'NetworkProbeVMAgent'
 
function Stop-AndRemoveService {
    param(
        [Parameter(Mandatory)]
        [string]$Name,
 
        [Parameter(Mandatory)]
        [string]$WrapperPath
    )
 
    $service = Get-Service -Name $Name -ErrorAction SilentlyContinue
 
    if (-not $service) {
        Write-Host "$Name is not installed."
        return
    }
 
    Write-Host "Stopping $Name..."
    Stop-Service -Name $Name -Force -ErrorAction SilentlyContinue
 
    if (Test-Path $WrapperPath) {
        Write-Host "Uninstalling $Name via WinSW..."
        & $WrapperPath uninstall
    }
    else {
        Write-Warning "WinSW wrapper not found. Removing $Name using sc.exe."
        & sc.exe delete $Name | Out-Null
    }
 
    Start-Sleep -Seconds 2
}
 
Stop-AndRemoveService `
    -Name $VMAgentService `
    -WrapperPath (Join-Path $VMAgentTarget 'vmagent-service.exe')
 
Stop-AndRemoveService `
    -Name $BlackboxService `
    -WrapperPath (Join-Path $BlackboxTarget 'blackbox-service.exe')
 
Remove-Item -Path $BlackboxTarget -Recurse -Force -ErrorAction SilentlyContinue
Remove-Item -Path $VMAgentTarget -Recurse -Force -ErrorAction SilentlyContinue
 
if ($RemoveData) {
    Write-Warning "Removing all Network Probe data, including logs and queued metrics."
    Remove-Item -Path $InstallRoot -Recurse -Force -ErrorAction SilentlyContinue
}
else {
    Write-Host 'Logs and queue data have been retained under C:\NetProbe.'
    Write-Host 'Run .\Uninstall-NetworkProbe.ps1 -RemoveData to remove them as well.'
}
 
Get-Service -Name $BlackboxService, $VMAgentService -ErrorAction SilentlyContinue |
    Format-Table Name, Status, StartType -AutoSize
 
Write-Host 'Network Probe uninstall completed.'

Running the scripts

Run from an elevated PowerShell window:

.\Install-NetworkProbe.ps1

Remove services and binaries while retaining logs/queue:

.\Uninstall-NetworkProbe.ps1

Remove everything, including persisted queue data:

.\Uninstall-NetworkProbe.ps1 -RemoveData

PowerShell execution policy limitation

Some managed customer devices enforce PowerShell AllSigned through Group Policy.

Example:

UserPolicy       AllSigned
Process          Bypass

In this situation, the policy overrides:

Set-ExecutionPolicy -Scope Process Bypass

and also overrides:

powershell.exe -ExecutionPolicy Bypass -File .\Install-NetworkProbe.ps1

Do not attempt to bypass the customer policy. The supported deployment options are:

Code-sign the PowerShell scripts with a certificate trusted by the customer endpoint.
Deploy using customer-approved Intune/RMM/software-distribution tooling.
Package the solution as a signed MSI.
For a one-off approved test, manually run the reviewed WinSW commands below.

Manual installation commands

Run PowerShell as Administrator.

Copy the prepared blackbox and vmagent folders to:

C:\NetProbe\

Create folders:

New-Item -ItemType Directory -Force C:\NetProbe\logs\blackbox
New-Item -ItemType Directory -Force C:\NetProbe\logs\vmagent
New-Item -ItemType Directory -Force C:\NetProbe\queue

Install Blackbox:

cd C:\NetProbe\blackbox
.\blackbox-service.exe install
Start-Service NetworkProbeBlackbox

Install vmagent:

cd C:\NetProbe\vmagent
.\vmagent-service.exe install
Start-Service NetworkProbeVMAgent

Validate service state:

Get-Service NetworkProbeBlackbox, NetworkProbeVMAgent

Both should show:

Running

Validation and troubleshooting

Check Blackbox locally

Invoke-WebRequest "http://127.0.0.1:9115/metrics" -UseBasicParsing

Expected result:

StatusCode : 200

Check vmagent targets

Open locally:

http://127.0.0.1:8429/targets

All configured targets should show UP.

Validate the vmagent YAML before starting

.\vmagent-windows-amd64-prod.exe `
  '-promscrape.config=.\promscrape.yml' `
  '-promscrape.config.dryRun'

Expected result:

-promscrape.config is ok; exiting with 0 status code

Useful VictoriaMetrics / VMUI queries

Probe state:

probe_success{
  customer="CUSTOMER_NAME",
  site="SITE_NAME",
  probe_agent="DEVICE_NAME"
}

Generic total probe duration in milliseconds:

probe_duration_seconds{
  customer="CUSTOMER_NAME",
  site="SITE_NAME",
  probe_agent="DEVICE_NAME"
} * 1000

ICMP RTT in milliseconds:

probe_icmp_duration_seconds{
  customer="CUSTOMER_NAME",
  site="SITE_NAME",
  probe_agent="DEVICE_NAME",
  phase="rtt"
} * 1000

TCP connection duration in milliseconds:

probe_tcp_connect_duration_seconds{
  customer="CUSTOMER_NAME",
  site="SITE_NAME",
  probe_agent="DEVICE_NAME"
} * 1000

DNS timing in milliseconds:

probe_dns_lookup_time_seconds{
  customer="CUSTOMER_NAME",
  site="SITE_NAME",
  probe_agent="DEVICE_NAME"
} * 1000

Verify buffering

To confirm offline persistence:

Stop or block access to the remote-write endpoint temporarily.
Confirm vmagent remains running and probes remain UP locally.
Check that C:\NetProbe\queue receives data.
Restore connectivity to the remote-write endpoint.
Confirm historical samples appear in VictoriaMetrics/VMUI.

Grafana dashboard variables

Recommended dashboard variables:

Customer → Site → Agent → Job → Instance

The variable dependencies make dropdowns context-aware:

Selecting a customer limits Site values to that customer.
Selecting a site limits Agent values to that customer/site.
Selecting an agent limits Job values to that agent.
Selecting a job limits Instance values to matching targets.

Variables do not automatically filter graphs. Each panel query must include the selected variables.

Example panel query:

probe_duration_seconds{
  customer=~"${Customer:regex}",
  site=~"${Site:regex}",
  probe_agent=~"${Agent:regex}",
  job=~"${Job:regex}",
  instance=~"${Instance:regex}"
} * 1000

Use =~ with ${Variable:regex} so that multi-select and All values work correctly.

Removal

Use the explicit Uninstall-NetworkProbe.ps1 script documented above.

Default uninstall:

.\Uninstall-NetworkProbe.ps1

Full removal, including logs and the persisted vmagent queue:

.\Uninstall-NetworkProbe.ps1 -RemoveData

Future improvements

Build a signed MSI installer for managed customer endpoints.
Use an approved code-signing certificate for scripts/installers.
Protect Cloudflare ingress with Cloudflare Access service tokens.
Store the package and configuration templates in Git.
Build GitHub workflows for package releases and Cloudflared container deployment.
Add endpoint-health monitoring for the probe services themselves.

tags	observability, monitoring, windows-network-probe
aliases	Windows Network Probe, Windows network probe with Blackbox Exporter and vmagent

HighSecLab Wiki

Explorer

Windows network probe with Blackbox Exporter and vmagent

Windows network probe with Blackbox Exporter and vmagent

Purpose

Tested component versions

Architecture

VictoriaMetrics ingress through Cloudflare Tunnel

Package layout

Blackbox Exporter configuration

DNS module behaviour

vmagent scrape configuration

Labels

Avoid duplicate targets

WinSW service configuration

Blackbox service XML

vmagent service XML

Installation

Explicit install script

Explicit uninstall script

Running the scripts

PowerShell execution policy limitation

Manual installation commands

Validation and troubleshooting

Check Blackbox locally

Check vmagent targets

Validate the vmagent YAML before starting

Useful VictoriaMetrics / VMUI queries

Verify buffering

Grafana dashboard variables

Removal

Future improvements

Graph View

Table of Contents

Backlinks