Skip to main content

Veeam v13 Fleet Upgrade — Runbook & Milestone Tracker

Book: Backup Operations | Audience: T3 / Code Commander (T2 with supervision) Platform: Veeam Backup & Replication 12.x → 13.0.2.29 on DTC BDR appliances Ticket: HALO 1179664 (project tracking) — engineering history under HALO 1146283 Author: Z. Boogher | Status: Active project

Purpose

End-to-end procedure for taking a DTC BDR appliance from any 12.x/13.0.1.x build to 13.0.2.29, proving the upgrade did no harm, applying DTC's scheduling policy, and confirming the site is clean afterwards.

Any T3 can pick up any device mid-sequence. Find the device on the tracker, see which milestone reads PENDING, and follow that section below.

No field work is required for any milestone. Every step is remote. The only reason to dispatch is hardware failure — a dead disk, a BDR that will not power on, or a box that cannot be brought back online remotely.

The two scripts

Script Repo path NinjaOne Purpose
Veeam v13 Upgrade msp-script-library/bdr-veeam/Invoke-VeeamV13Upgrade.ps1 Script 203 M1 + M2
Veeam Schedule Window (Standard) msp-script-library/bdr-veeam/Set-VeeamScheduleWindow.ps1 — M3

Both Run As SYSTEM. Both work on 12.x and 13.x unmodified.

READ THE LOG, NOT THE BADGE

Both scripts use exit code 2 for expected mid-states — hop complete awaiting reboot, offsite copy-watch holding, a preflight gate halt, changes pending in report mode. NinjaOne renders exit 2 as FAILURE.

A red result is very often a device progressing normally. Always open the activity log and read the text.

Exit Meaning
0 Converged / clean
2 Expected mid-state, or a validation/gate/copy condition needing attention
1 Genuine script failure

Milestone model

A device is only COMPLETE at M7. Being on build 13.0.2.29 is M1 — roughly one seventh of the work.

M1 — Upgraded

Installed build is exactly 13.0.2.29.

Run script 203. It is convergent and idempotent: every run resolves to VALIDATE, OFFSITE COPY WATCH, UPGRADE, or NOTHING based on what it finds. Safe to re-run at any time.

Hop logic (Veeam KB4763):

  • at or above 13.0.2.29 → converged, no-op
  • at or above 12.3.1.1139 → single direct hop to v13
  • below 12.3.1.1139 → hop via the 12.3.2 ISO first

Both hops chain in one run; the script reboots itself between them.

Proof: log line Post-upgrade build: 13.0.2.29, or live NinjaOne inventory.

M2 — Validated

The upgrade was proven non-destructive on that specific device. Jobs, repositories, object storage, S3 credentials, copy jobs and restore points are diffed against a baseline captured immediately before the hop, and validation fails closed.

Automatic on the run after the hop reboot. Nothing to do by hand.

Validation cannot be done retroactively. It is a diff against a pre-hop baseline. A device that was already at 13.0.2.29 before this project can never earn M2 — mark it N/A and rely on M4 instead.

Restore-point checks tolerate one missed hourly cycle during the reboot. A newest point moving backwards beyond 26 h, or a count drop beyond 10%, still fails and holds the device.

Proof: log line VALIDATION PASSED.

M3 — Schedule Applied

DTC scheduling policy written to the jobs so a local backup cannot hold a restore-point lock while the offsite copy is reading it.

Job type Policy
Server / image (periodic) permitted 06:00–20:59, Mon–Sat
Workstation (daily) start 21:00, Mon–Fri
Cloud / S3 copy left at its 22:00 daily start

Run intervals are preserved — an hourly job stays hourly at the same offset. The script verifies this by reading Kind / FullPeriod / HourlyOffset back after each write and failing the job if any moved.

Procedure:

  1. mode=report (default) — read-only. Confirm what it intends to change.
  2. mode=apply + applyChanges=1 + serverDays=Mon,Tue,Wed,Thu,Fri,Sat
  3. Confirm in the console.

Proof: APPLIED ... interval ... (unchanged) for each job, and the console Time Periods grid showing green Mon–Sat 06:00–20:59 with Sunday empty.

Rollback: the script saves original state to C:\ProgramData\DTC\Logs\VeeamSchedule\schedule-original_<timestamp>.json before the first write and logs the exact undo command.

M4 — Jobs Verified

The jobs actually run inside the new window and still succeed. This is the milestone that proves the schedule change did no harm.

Wait one full cycle after M3, then confirm:

  • no server session outside 06:00–20:59
  • the workstation job fired at 21:00
  • the offsite copy fired at 22:00 and succeeded

A copy job that was already failing before the upgrade is a pre-existing fault. Flag it; do not count it against M4.

M5 — Agents Current

Veeam Agents do not update themselves. A site can be fully on v13 with every endpoint still running a 6.x agent. This is the most commonly missed milestone.

Console → Inventory → Protection Group → Agent Version column. Upgrade from the console, or via Install-VBRDiscoveredComputerAgent.

Endpoints reporting RebootRequired=True cannot finish the agent upgrade until that workstation reboots. Reboot it, then retry — otherwise the attempt simply times out.

Script 203 also upgrades any agent the v13 installer names as a blocker (event id=106), so some sites clear M5 as a side effect of M1.

Proof: no agent below 7.x remains, or every remaining one is recorded with a reason.

M6 — Cleanup Done

Stale and orphaned backup data removed: offsite restore points for machines that no longer exist, disabled or abandoned jobs, registrations for decommissioned endpoints.

List offsite restore points per VM and compare against the live job configuration. Anything with no matching job is orphaned.

Confirm with the client or AM before deleting any backup data. A stale restore point can also mean a broken agent or a disabled backup account rather than a dead machine — both have been seen on this fleet.

Removing an agent registration does not remove its backup files; those stay on the repository and remain restorable.

M7 — Complete

M1–M6 all DONE. Roll-up only.

Known failure modes

Every one of these was found on a live device during the fleet waves. The scripts handle them; this section is for reading logs and for anything done by hand.

Installer

The v13 setup engine refuses LocalSystem (event id=103). Start-Process -Credential cannot fix it — that yields a UAC-filtered token and fails identically. The installer must run via a one-shot Scheduled Task as the LAPS local admin at RunLevel Highest, and the task account must be COMPUTERNAME\user (a .\user form fails SID resolution).

A rejected credential fails at launch, silently. The task registers, Windows rejects the logon, the task never enters Running, and LastTaskResult stays 0 from a prior state — which reads as a successful install that never happened. Look for Security event 4625, substatus 0xC000006A (bad password). Cause is almost always a stale lapsPassword custom field. The current script validates the credential before the download and fails with a named reason.

Exit 3010 with an unchanged build is not a failure. It means a prerequisite was installed and needs a reboot (event id=012, Microsoft Visual C++ redistributable). Reboot and re-run; the hop completes on the next pass.

Exit 1603 with the build at target is a partial success. An ancillary component failed while the core product upgraded fine. Check the component logs; the reboot usually clears it.

The patch engine cannot run from read-only media — running setup from a mounted ISO fails with "The media is write protected". Setup must be copied to a writable local folder first.

Service handling

Never force-kill VeeamBackupSvc. taskkill wedged a BDR badly enough that the service would not restart and orphaned workers held port 9392 for hours. Only a reboot cleared it. Graceful stop only.

The installer's 5-minute stop budget is the core problem. Setup restarts VeeamBackupSvc to analyse the config DB, and each service start spawns a batch of maintenance jobs (STARTRESYNC, STARTDBMAINTENANCE, STARTCATCLEANUP and others). The service cannot complete a stop while those run; setup gives up at 300 s with event id=113 and rolls back. Pre-stopping does not help — the blocking jobs are created after setup restarts the service. The script runs a watchdog during the install window that terminates those workers mid-install.

A service stuck in StopPending never resolves on its own and will halt every subsequent run. Reboot clears it.

MSSQL boot race: VeeamBackupSvc has no SQL dependency in SCM, so at boot it can start before MSSQL is ready, fail, exhaust its retries and stay down along with its dependents.

PowerShell

Veeam PowerShell v13 requires PowerShell 7 — v13 is built on .NET Core. NinjaOne invokes scripts under Windows PowerShell 5.1, so on a v13 box every Veeam cmdlet fails at Import-Module.

The v12 module is the mirror image: .NET Framework, and it fails under pwsh with "The type initializer for Veeam.Backup.Common.SSslOptions threw an exception".

Working around it by hand:

# on a v13 box
& pwsh -NoProfile -Command {
  Import-Module Veeam.Backup.PowerShell -DisableNameChecking -WarningAction SilentlyContinue
  Get-VBRJob -WarningAction SilentlyContinue | Select-Object Name, JobType
}

PowerShell 7 is a DTC build standard and is present on the BDR fleet. The scripts select the runtime per call from the installed build.

Console

The v13 console must be launched AS ADMINISTRATOR or it fails with "Access to the registry key ...\Plugins is denied". The key grants Administrators, and a non-elevated process carries that as a deny-only SID under UAC.

Disk

Setup needs roughly 29.3 GB free on C: regardless of where the ISO is staged. Several fleet BDRs have 109–117 GB system drives beside multi-TB repositories, so the script stages the ISO on the largest volume but still gates C: separately.

Veeam's own log tree is usually the culprit when C: is short. One fleet device held 58 GB across 12,522 files in C:\ProgramData\Veeam\Backup dating back to 2024. The folder is hidden, so a plain directory scan misses it. The script prunes it before the space gate, reading the path from Veeam's LogDirectory value and abandoning the prune entirely if it resolves inside a repository path.

Configuration backup

Not a gate, deliberately. A large share of the fleet has a broken configuration-backup job — six devices share a single dangling repository GUID inherited from a provisioning template. It is not repairable by cmdlet: all four *ConfigurationBackup* cmdlets were enumerated on both 12.2 and 13.0, neither exposes a settings-level setter, and the GUID is stored binary so it appears nowhere as text in the config DB. Only the console dialog writes the configuration fresh.

Halting those upgrades protected a rollback point the devices did not have. Repair is a separate workstream.

Scheduler XML format (reference)

Not documented by Veeam. Reverse-engineered on a live device; recorded here because any future schedule automation depends on it.

The permitted-hours window for a periodic job is an XML string at OptionsPeriodically.Schedule:

<scheduler><Sunday>1,1,1,...</Sunday><Monday>1,1,1,1,1,1,0,0,...,0,1,1,1</Monday>...</scheduler>

Seven day elements, 24 comma-separated values each, hour 0–23.

POLARITY IS INVERTED: 0 = PERMITTED, 1 = DENIED. Proven against live session data — a job whose XML carried 1 at hours 0–4 and 20–23 had observed start hours of 5 through 21, and its all-1 Sunday row had never produced a Sunday run.

Other gotchas:

  • OptionsBackupWindow is not the control. IsEnabled was False and BackupWindow a zero-length string on every job tested. Do not use it.
  • FullPeriod is in seconds despite Kind reading "Hours". A job at FullPeriod 3600 / HourlyOffset 15 runs at :15 every hour.
  • The daily day-set property is DaysSrv, not DayOfWeek, and setting it requires Kind = 'SelectedDays'. Read it with GetDays().
  • A daily job has one start time and no window. Only periodic jobs have a Time Periods grid.

Copy-job periodic conversion — do not use

Converting a SimpleBackupCopyPolicy from Daily to Periodic works via PowerShell and reads back correctly, but the Veeam console cannot render the result — opening the job throws "Sequence contains no matching element".

The capability is retained in the script behind convertCopyToPeriodic, defaulted off. Reverting the schedule clears the console error completely.

A daily 22:00 copy start already runs uninterrupted into Monday morning, which was the substantive goal.

Reference devices

The process is proven end to end on real hardware, not in theory.

DTCBSURE-4557 — the reference device for the full sequence. Upgraded 12.2.0.334 → 12.3.2.4465 → 13.0.2.29 and passed validation against baseline (4 jobs, 3 repositories, 1 object storage repo, 5 S3 credentials, 3 backup objects, no restore-point regression). Schedule policy applied 2026-09-01 with both jobs verified by independent read-back and the hourly interval preserved. Jobs confirmed running and successful in the new window. M1–M4 complete.

Every installer failure mode in this page was found and fixed on this device.

DTCBSURE-GODWIN — the reference device for the schedule work. The scheduler XML format and its inverted polarity were reverse-engineered here, and the copy-job console limitation was found here. M3 applied and verified.

Tracker

Per-device milestone tracking lives on the workbook attached to HALO 1179664:

  • Milestones — definitions and completion criteria (mirrors this page)
  • Fleet Status — one row per BDR, M1–M7 per device
  • Summary — progress by milestone, build and track
  • Blockers — the open worklist with owners
  • Change Log — what was done, when, by whom

Change record

Date Change By
2026-09-01 Page created. Runbook, milestone model M1–M7, scheduler XML format, known failure modes. Z. Boogher
Back to top