Skip to main content

BDR Offline & Connectivity

Audience: T1 / T2
Version: 1.3
Last Updated: August 2026

Overview

This guide addresses scenarios where the BDR appliance itself is unreachable — cannot ping, RDP, or access the Veeam console.

Common symptoms:

  • Cannot ping BDR IP address
  • RDP connection refused or times out
  • NinjaRMM shows BDR as offline
  • Veeam console on BDR is inaccessible
  • All backup jobs fail or don't start

⚠️ CRITICAL: If the BDR is offline, all backups are failing. This is a severity 1 issue requiring immediate response.

Tier Definitions

Tier Scope Expected Resolution
T1 IPMI entry-point recovery, site scope triage, environmental rule-outs, basic connectivity 10-20 minutes
T2 Remote diagnostics, service recovery; escalate to T3 team if hardware intervention needed 30-60 minutes

T1 — First Response

1. Remote Power Control via IPMI

When to use this: This is your first move on any BDR-offline alert. Most of the time, especially when other machines at the site are still online, IPMI is the fastest path to recovery.

Background — what you're actually doing here

When a BDR won't respond, your first instinct might be to call the client and ask them to physically reboot it. Before you do that, try this.

Most BDR appliances have a hidden management connection built into the hardware. It has its own login page and stays powered on even when the BDR appears totally dead in NinjaOne.

Think of it like this: the BDR is a car that won't start. IPMI is a hidden panel under the hood that lets you crank the engine remotely, even if the dashboard is completely dark.

The steps below walk you through finding that hidden connection and using it to reboot the BDR without anyone touching it physically.

Step 1 — Remote into a machine at the client site

Use NinjaOne to remote into any online machine at the client's site. The server is preferred, but any workstation will work. You need to be on their network to find the BDR's hidden management connection.

Step 2 — Download and run Advanced IP Scanner

On the remote machine, open a browser and Google "Advanced IP Scanner." Click the first result and download it. You don't need to install it. Just open it and run it.

Step 3 — Scan the network

When Advanced IP Scanner opens, you'll see an IP range already filled in (something like 192.168.0.1-254 or 192.168.1.1-254). Leave it as-is and click Scan. Wait for it to finish.

Step 4 — Find the IPMI entry

Look for entries in the scan results with any of these labels:

  • IPMI
  • Supermicro
  • HTTP, Megarac SP
  • An IP with an HTTP entry expanded underneath it (this means IPMI is on the same IP as the BDR itself, not a separate one)

These entries often have no device name. Just an IP address and the label above.

Any of these is the IPMI entry you're after. If none appear, move to Step 2: Verify Site Scope below.

Step 5 — Open the login page

Right-click the IPMI entry and select Tools > HTTP. A browser window will open on the remote machine showing a login page.

Step 6 — Log in

Open 1Password and find the entry named BDR Appliance - IPMI. This one entry contains the default credentials used across all BDR IPMI consoles.

Username is case sensitive. Try admin (all lowercase) first. If that fails, try ADMIN (all uppercase). The password is the same either way and it's in the 1Password entry.

These are not the same as the Windows login. IPMI has its own separate username and password.

Step 7 — Reboot the BDR

Once logged in, find the Power Control section. What you see depends on the hardware:

  • Megarac interface (ASRock/Equus hardware): Navigate to Power Control and click the reboot option.
  • Supermicro interface: Power Control is on the right-hand side of the main System page you land on after login. Click Reset.

In both cases, confirm when prompted.

If Power Status already shows ON: the BDR is likely hung or frozen. This is common. Click Reset first. If Reset doesn't bring it back within 5 to 10 minutes, use Power Cycle (off then on) to force a full power-off and boot.

You may also see a Remote Console Preview. It's a live screenshot of the BDR's screen. Useful to check if Windows is at a login screen, crashed, or mid-boot.

Step 8 — Wait and watch

Go back to NinjaOne and watch the BDR device. It will take 5 to 10 minutes to fully boot back up. Do not try anything else until it shows back online.

2. Verify Site Scope — BDR Only, or Whole Site?

If IPMI didn't work (couldn't reach it, no matching entry, no recovery after reboot), figure out what else at the site is affected. This tells you whether the problem is local to the BDR or environmental.

Check other agents at the site in NinjaOne:

  • Filter NinjaOne to the client's organization or site
  • Are other machines showing online? Servers, workstations, network devices?
  • If yes: only the BDR is affected. Skip to Step 4: Basic Connectivity below.
  • If no: whole site appears down. Continue to Step 3: Environmental Checks.

Optional: ping another device at the site. If you're already remoted into a machine there or can reach one, ping a workstation or server IP to confirm the site is reachable.

3. Environmental Checks — Whole Site Down

When the whole site appears offline, don't assume it's a BDR problem. Rule out environmental causes first.

Call the office.

  • If someone answers, phone lines are up, which usually means power is on
  • Ask if their computers are working, if the internet is up, if they've had any electrical issues
  • This gives you eyes on the ground with no ambiguity

Check the client's local power utility for outages.

  • Most DTC Maryland clients are served by BGE. Their outage map is at bge.com/outages
  • For non-MD clients, check the utility that serves their area
  • If you're not sure which utility, search "[client's zip code] power outage"

Check the client's ISP status page.

  • Comcast, Verizon Fios, and other major ISPs publish status pages showing regional outages
  • Search "[ISP name] outage" or "[ISP name] status" to find the right page
  • Also useful: downdetector.com for the ISP or region

If any of these turn up an active outage, document it in the Halo ticket, notify the client, and monitor for restoration. No further action needed on your end until services come back up.

4. Basic Connectivity — BDR Only

If IPMI failed but other machines at the site are online, run through basic connectivity checks on the BDR itself.

Ping the BDR.

From your workstation or another server on the same network:

Test-Connection -ComputerName [BDR_IP] -Count 4
  • Ping succeeds: BDR is online but services may be down → escalate to T2 for service recovery
  • Ping fails: BDR is off, network-disconnected, or its IP has changed

Try RDP.

mstsc /v:[BDR_IP]
  • RDP prompt appears: Windows is running and network is working → escalate to T2 for service-level diagnostics
  • RDP fails: Windows isn't fully booted or the network path is broken

Check ARP for DHCP lease change (edge case).

Only relevant if the BDR isn't on a static IP. From a workstation on the same network:

arp -a | Select-String [BDR_PARTIAL_IP]

If the MAC is present but tied to a different IP, the BDR got a new DHCP lease. Update your bookmarks and DNS with the new IP. Long-term fix: configure a static IP per DTC standard.

If nothing here recovers the BDR: escalate to T2 with a summary of everything you tried at T1.

T2 — Remote Diagnostics & Service Recovery

1. Check NinjaRMM Status & Run Diagnostics

If NinjaRMM agent is installed on the BDR:

  1. Log into NinjaRMM dashboard
  2. Search for the BDR device
  3. Check last seen time
  4. If recently online: Run remote command to check services

NinjaRMM remote PowerShell:

Get-Service Veeam* | Select Name, Status, StartType
Get-EventLog -LogName System -Newest 50 | Where-Object {$_.EntryType -eq 'Error'} | Select TimeGenerated, Source, Message

2. Attempt WinRM / Remote PowerShell

From a Windows server on the same network (if WinRM is enabled on BDR):

Enter-PSSession -ComputerName [BDR_IP] -Credential (Get-Credential)

# Once connected:
Get-Service Veeam* | Select Name, Status
Get-EventLog -LogName System -Newest 20

If WinRM fails: Services may be stopped or Windows is unresponsive → T3

3. Check for Windows Update or Reboot Pending

If you get connected via RDP or WinRM:

# Check for pending reboot
Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired" -ErrorAction SilentlyContinue

# Check last boot time
(Get-CimInstance Win32_OperatingSystem).LastBootUpTime

# Check if Windows Update is running
Get-Process wuauclt, TrustedInstaller -ErrorAction SilentlyContinue

If pending reboot or update in progress:

  • Schedule maintenance window with client
  • Reboot BDR
  • Wait 10-15 minutes for full boot and Veeam services

4. Restart Veeam Services

If BDR is online but Veeam console won't open:

# Restart all Veeam services
Get-Service Veeam* | Restart-Service -Force

# Verify status
Get-Service Veeam* | Select Name, Status

Expected services:

  • VeeamBackupSvc - Veeam Backup Service
  • VeeamBrokerSvc - Veeam Broker Service
  • VeeamCatalogSvc - Veeam Catalog Service
  • VeeamCloudSvc - Veeam Cloud Connect Service (if enabled)
  • VeeamDeploymentService - Veeam Deployment Service
  • VeeamMountSvc - Veeam Mount Service
  • VeeamNFSSvc - Veeam NFS Service
  • VeeamRESTSvc - Veeam REST API Service
  • VeeamTransportSvc - Veeam Transport Service

All should be Running. If any fail to start, check Event Viewer → Application logs for Veeam errors.

5. Check Disk Space on BDR

Critical: If BDR C: drive is full, Windows and Veeam may become unresponsive.

Get-WmiObject Win32_LogicalDisk | Select DeviceID, @{N='FreeGB';E={[math]::Round($_.FreeSpace/1GB,2)}}, @{N='TotalGB';E={[math]::Round($_.Size/1GB,2)}}

If C: drive < 1 GB free:

  • Clear temp files: Remove-Item C:\Windows\Temp\* -Recurse -Force
  • Check for massive log files: Get-ChildItem C:\ProgramData\Veeam\Backup\*.log | Sort Length -Descending | Select -First 10
  • See BDR Storage Alerts & Capacity Issues for full cleanup procedures

Prevention & Monitoring

1. Enable UPS Monitoring

Ensure BDR is on UPS with monitoring:

  • Configure UPS software to shut down BDR gracefully on low battery
  • NinjaRMM can monitor UPS status if agent is installed

2. Configure Email Alerts for BDR Downtime

In Veeam console:

  1. Main Menu → Notifications
  2. Enable Email notifications
  3. Add recipients
  4. Set alerts for:
    • Backup job failures
    • Repository warnings
    • Service failures

In NinjaRMM:

  • Set up device offline alert for BDR (5-minute threshold)

3. Regular Configuration Backups

DTC Standard: Veeam configuration should be backed up weekly.

In Veeam console:

  1. Main Menu → Configuration Backup
  2. Enable scheduled backups
  3. Save to:
    • Network share (not on BDR itself)
    • USB drive (rotated weekly)
    • Cloud storage (Backblaze B2, OneDrive, etc.)

4. Document BDR Build & Recovery Procedures

In HALO or BookStack:

  • BDR hardware specs (model, CPU, RAM, storage config)
  • Windows Server version and license
  • Veeam version and license
  • Network configuration (static IP, gateway, DNS)
  • Repository paths and retention settings

Use for: Quick rebuild if BDR fails catastrophically.

Quick Reference — BDR Offline Scenarios

Symptom Likely Cause Immediate Action
Cannot ping, cannot RDP Powered off, hung, network unplugged, or IP changed Try IPMI reboot first; verify site scope via NinjaOne
Whole site offline Power outage, ISP outage, or environmental issue Call office; check BGE (or local utility); check ISP status
Ping works, RDP fails Windows Firewall blocking or RDP service stopped Check services (T2); reboot via IPMI if needed
RDP works, Veeam console won't open Veeam services crashed or stopped Restart Veeam services via PowerShell (T2)
Intermittent offline/online Network flapping, bad cable, or switch issue Replace cable, check switch logs
Offline after Windows Update Update failed or reboot pending IPMI console access; boot Safe Mode if needed
Blue screen/crash on boot Hardware failure (disk, RAM, PSU) Escalate to T3 team for onsite diagnostics

Escalation & Client Communication

If BDR is offline and cannot be recovered remotely:

  1. Notify client immediately:

    • BDR is offline
    • All backups are currently not running
    • Estimated time to resolution

  2. Document in HALO:

    • Symptoms observed
    • Troubleshooting steps performed
    • Client notification timestamp
    • Onsite visit scheduled (if applicable)

  3. Escalate to team lead if:

    • Onsite visit required but no tech available
    • Hardware replacement needed
    • Client needs same-day resolution

Document History:

Version Date Author Changes
1.0 April 2026 DTC Initial creation - BDR offline troubleshooting
1.1 May 2026 Zachary Ireland Added T1 Step 5: Remote Power Control via IPMI — covers Megarac and Supermicro interfaces, Advanced IP Scanner process, and 1Password credential lookup
1.2 August 2026 Zachary Ireland T1 Fast Path callout for automated BDR-offline alerts; removed NinjaOne IP lookup step (Advanced IP Scanner labels are the primary anchor); expanded IPMI entry labels (IPMI, Supermicro, Megarac SP); updated 1Password guidance to single "BDR Appliance - IPMI" entry with case-sensitive username note; corrected Supermicro Power Control location (right-hand side); added hung/frozen troubleshooting (Reset, then Power Cycle)
1.3 August 2026 Zachary Ireland Full T1 restructure. IPMI is now Step 1 (fastest entry point). New Step 2 Verify Site Scope (BDR only vs whole site). New Step 3 Environmental Checks (call office, BGE/local utility, ISP status). New Step 4 Basic Connectivity (ping/RDP/ARP). Removed old Steps 1-4 duplication (Check Physical Status overlapped IPMI). Removed Fast Path callout (redundant with IPMI as Step 1). Removed T3 Physical Intervention section (T3 team owns their own runbook). Audience narrowed to T1/T2. Updated Tier Definitions and Quick Reference tables to match.
Back to top