Skip to main content

DRAFT — Failed BDR: Interim Backup Coverage & Replacement Runbook

Document Type: Runbook (DRAFT v0.1) | Audience: T2/T3 | Last Updated: September 2026 | Status: Draft — not yet approved

⚠️ DRAFT. Written from a live incident (Sept 2026) because no replacement procedure existed for a totally failed BDR. Sections marked [VERIFY] need confirmation before this becomes an approved SOP. Reviewer: Nate Smith.

Use this page when a BDR appliance has failed completely — hardware dead, storage array failed, or otherwise unrecoverable — and the client is running with no local backup while a replacement is sourced. This covers the full arc: emergency interim coverage → replacement → reattaching offsite → verification → decommissioning the interim backups.

This is not for a BDR that is merely offline or out of space. See BDR Offline & Connectivity and BDR Storage Alerts & Capacity Issues first.

Principles

Principle Why
No client goes without backups. Interim coverage goes up the same day the BDR is declared dead. A dental practice with no backup is an existential risk to them and to DTC.
The existing S3 bucket is reused, never replaced. The bucket holds the client's existing restore points. A new bucket orphans backup history and restarts the full seed over WAN.
Interim backups are not torn down until Veeam is proven working. "Veeam looks configured" is not the same as "Veeam has completed and verified a job."
One ticket owns the whole arc. The gap between interim and permanent is exactly where clients get dropped.

Phase 0 — Declare the BDR Failed

Before starting, confirm this is a total failure and not something recoverable.

  1. Confirm the appliance is unreachable and not recoverable — see BDR Offline & Connectivity
  2. Confirm with T3 / Nate that the unit is being replaced, not repaired
  3. Record in the ticket: date/time of last successful backup job, last successful S3 copy job, and current S3 bucket name (NinjaOne custom field Veeam S3 Bucket Name)
  4. Note whether the offsite S3 copy is still intact — in most failure modes it is, and it is the client's only remaining restore path until interim coverage is live

Ticket handling: open a service ticket, assigned to the responding tech, and keep it open until the replacement BDR is in and verified. Do not close it when interim backups go live — the interim teardown is part of this ticket.

Phase 1 — Interim Coverage (NinjaOne Backup)

Goal: same-day protection of anything that would end a practice — patient data, imaging, and the domain.

1.1 Scope

Target Backup type Notes
Domain controller(s) Image Full image, per DTC standard for DCs
Practice management / imaging application data File-level DB dump first — see 1.2
File shares (docs, scans, signed forms) File-level Enumerate rather than guess — see 1.3

Reference: NinjaOne Image Backup Plan Configuration Standard.

1.2 Databases must dump before the file job runs

NinjaOne file-level backup uses VSS, which gives a crash-consistent copy of live database files — not application-consistent. A live MDF/LDF captured that way may or may not restore, and you find out at restore time. Every database in scope needs a native dump to disk first, with the file-level job scheduled after it completes.

Sequence:

  1. Scheduled task dumps the database to a local folder (e.g. D:\SQLBackups)
  2. Time the dump so you know the real window
  3. NinjaOne file-level job starts after that window, and includes the dump folder

SQL Server example (single database, no compression so it works on Express):

New-Item -ItemType Directory -Force -Path D:\SQLBackups
sqlcmd -S .\INSTANCENAME -E -Q "BACKUP DATABASE [DBNAME] TO DISK = N'D:\SQLBackups\DBNAME.bak' WITH INIT, STATS = 10"

Find the instance and its data file locations first:

Get-Service | ? Name -like 'MSSQL*' | select Name,DisplayName,Status
sqlcmd -S .\INSTANCENAME -E -Q "SELECT DB_NAME(database_id) AS db, name, physical_name FROM sys.master_files ORDER BY db" -W -s"|"
sqlcmd -S .\INSTANCENAME -E -Q "SELECT SERVERPROPERTY('Edition') AS ed, SERVERPROPERTY('ProductVersion') AS ver" -W -s"|"

Application-specific database locations and backup considerations:

Cloud PMS note: if the practice is on a cloud platform (Curve, etc.), there may be no local PM database at all — but imaging (DEXIS, CS Imaging, Sidexis) is still local and still needs this treatment. Confirm what actually runs on the server rather than assuming from the PMS name.

1.3 Enumerate file share scope

Get-SmbShare | ? Name -notlike "*$" | select Name,Path
Get-SmbOpenFile | select Path -Unique

Run the second command midday — it shows what staff are actually using, which catches shares that matter and skips shares that merely exist. Also check for: scanner drop folders, signed-forms/consent PDF stores, redirected folders or home directories, VoIP recordings stored locally, and license/cert files for line-of-business apps.

1.4 Verify the interim backups actually ran

Do not consider Phase 1 complete on configuration alone.

Confirm specifically that the .bak files are present, non-zero, and timestamped before the file job's start time.

Phase 2 — Replacement BDR Deployment

Deploy per the standard SOP: Veeam BDR Deployment SOP. Job configuration per Veeam Backup and Replication Standards. Firewall per Veeam BDR — Firewall Rules.

Replacement-specific items the deployment SOP does not cover:

Item Action
Old appliance in NinjaOne Do not delete until the new BDR is verified — the S3 bucket name custom field lives on it [VERIFY: whether the field carries to the new device automatically or must be set by hand]
Old appliance in Veeam licensing Release/reassign the license from the dead unit [VERIFY: current process with Nate]
Protected computers Re-add servers and workstations per Adding & Replacing Computers in Veeam BDR
Job schedule Stagger against the interim NinjaOne jobs while both run — do not have both hitting the same server at once

Phase 3 — Reattach Offsite to the Existing S3 Bucket

This is the step most likely to be done wrong. The default path creates a new bucket; that orphans the client's existing restore points and forces a full re-seed over their WAN.

  1. Retrieve the original bucket name recorded in Phase 0 (NinjaOne custom field Veeam S3 Bucket Name)
  2. Point the new BDR's Backblaze S3 repository at that existing bucket — do not let the provisioning script mint a new one [VERIFY: whether "Veeam Configure Backblaze S3 Repo" accepts an existing bucket name as a parameter, or whether the repo must be added manually in the Veeam console for a replacement]
  3. Run "Veeam Configure S3 Copy Job" (NinjaOne script) to create the copy job against that repository
  4. Confirm the copy job appears in the Veeam console
  5. Confirm the first S3 copy job completes — not just starts
  6. Confirm NinjaOne custom fields update: Veeam S3 Bucket Name populated, Veeam S3 Copy Job Missing = False

Script sequence and field verification follow the same pattern as the Veeam S3 Remediation Project — the repo must exist before the copy job can reference it.

[VERIFY: whether old restore points in the reused bucket remain visible/usable to the new BDR, or whether they are only recoverable by importing the backup files. This determines what we can promise the client about their pre-failure restore history.]

Standards reference: Cloud Backup Architecture Standards.

Phase 4 — Soak & Verify Before Teardown

Both backup systems run in parallel through this phase. Do not disable the NinjaOne interim backups yet.

Minimum bar before teardown — all must be true:

  • Local Veeam job has completed successfully on [VERIFY: how many consecutive nights — suggest 3]
  • S3 copy job has completed successfully at least once, ideally matching the local job count
  • Application-aware processing / VSS is enabled and reporting success on database servers
  • A test restore has been performed — file-level at minimum; image-level for a DC if feasible
  • Backup health check / integrity verification passes per Veeam Backup Daily Operations & Verification SOP
  • NinjaOne fields green; no open backup alerts for the site

A test restore is not optional. A job reporting success is evidence the job ran, not evidence the data is recoverable.

Phase 5 — Decommission Interim Coverage

Only after every box in Phase 4 is checked.

  1. Disable (do not immediately delete) the NinjaOne interim backup plans
  2. Leave the existing interim restore points in place for [VERIFY: retention period — suggest 30 days] as a fallback
  3. Disable or remove the interim database dump scheduled task [VERIFY: or keep it — a nightly native dump alongside Veeam is arguably good practice for dental DBs; ask Nate whether this becomes standard]
  4. Remove the dead BDR from NinjaOne and Veeam infrastructure
  5. Update IT Glue / client documentation with the new appliance details
  6. Confirm billing reflects the correct backup product if interim coverage was billed differently — see Backup Coverage, Veeam and Backblaze B2 (Windmill)
  7. Close the ticket with a summary: failure date, interim coverage window, replacement date, verification evidence

Client Communication

The practice should hear three things, in plain language:

  1. What happened and what it means right now — their backup appliance failed; their offsite copy is intact; they are not currently protected locally until interim coverage is running.
  2. What we did today — cloud backups of their patient data, imaging, and servers, running while the replacement is sourced.
  3. What changes when the new unit arrives — back to normal fast local restores; nothing for them to do.

Avoid implying zero exposure. There is a window between failure and interim coverage, and the client's decision-maker should know it existed.

Open Items for This Draft

# Item
1 All [VERIFY] markers above — chiefly the S3 bucket reuse mechanics in Phase 3
2 Whether interim coverage should be NinjaOne Backup in all cases, or Restic for T3-only scenarios — see Restic — Emergency Interim Backup: Deployment & Restore (T3 Only)
3 Define the soak period and test-restore requirements as hard numbers, not suggestions
4 Decide whether nightly native DB dumps become permanent standard alongside Veeam
5 Add a "storage failed but appliance alive" variant — partial failure has a different path than total failure
Back to top