Cloud Backup Architecture Standards
This page defines the standard architecture for how DTC provisions cloud object storage (Backblaze B2, S3-compatible) for backup workloads. It applies to all cloud-backed backup systems regardless of source platform — Veeam BDR, TrueNAS Cloud Sync, MSP360, or anything new that lands in the stack. The principles are workload-agnostic. The naming scheme is consistent. The credential pattern is uniform. If a new backup workload is being designed, it must conform to this page.
For platform-specific provisioning procedures (Veeam BDR, TrueNAS, etc.), see the SOPs in the Deployment chapter.
Tenancy Models
DTC operates two distinct cloud bucket architectures depending on how the workload handles multi-tenancy:
- Mode A — Per-(client, location, workload) buckets (this page). DTC enforces tenant boundaries at the storage layer by giving each tenant its own bucket. Used for backup workloads that are single-tenant by assumption (Veeam BDR, TrueNAS Cloud Sync). Most of DTC's backup fleet runs this way.
- Mode B — Workload-instance buckets (Workload-Instance Bucket Architecture). The application enforces tenant boundaries internally; DTC provisions one bucket per workload instance. Used for backup workloads that are multi-tenant by design (MSP360) and for DTC-built applications that manage their own per-client data segregation.
Default to Mode A when uncertain. Mode B is only correct when the application's tenant-aware UI, internal mapping, and credential model are all sufficient to make per-tenant bucket scoping unnecessary. The rest of this page describes Mode A in full.
Core Principles
1. Bucket per (client, location, workload)
Every cloud backup bucket is scoped to exactly one combination of client organization, physical location, and workload. A dental practice with two offices and both a Veeam BDR and a TrueNAS box at each location gets four buckets: one for (ClientCo, Downtown, veeam), one for (ClientCo, Downtown, truenas), one for (ClientCo, Uptown, veeam), one for (ClientCo, Uptown, truenas).
This is a deliberate single-tenant design. The reasons are safety-driven:
- Backup applications are single-tenant by assumption. Veeam Backup & Replication, TrueNAS Cloud Sync, MSP360, and similar tools assume they own the storage endpoint they're pointed at. They do not enforce tenant boundaries inside a shared bucket, and their device-facing UIs do not show which tenant a given folder belongs to. Pointing two clients' devices at the same bucket means nothing at the app layer prevents cross-tenant writes or reads.
- Credentials live on the client's devices. TrueNAS stores the B2 application key locally to run Cloud Sync Tasks. Veeam stores repository credentials locally. Those credentials are on hardware physically at the client's site, accessible by anyone with local access to the device. If a credential can reach other tenants' data in the same bucket, that is a multi-tenant boundary violation by design — and the app cannot be made to stop using the credential it was handed.
- Technician selection safety. In a shared bucket, a technician performing a restore or picking a source folder sees every tenant's data in the object browser. The workflow of "find this client's device among all the others and don't pick the wrong one" is exactly the workflow that causes tech-error data incidents. Per-(client, location, workload) buckets make that class of mistake impossible at the storage layer — the wrong data simply is not in the bucket the tech has open.
- Per-location lifecycle flexibility. Different locations of the same client may legitimately need different lifecycle or retention policies — a regulated-data office, a client-mandated legal hold, a seasonal or temporary location with a different RPO. Bucket-per-location lets each location's policy be bucket-wide and bucket-scoped without affecting its sibling locations.
- Per-workload lifecycle simplicity. Veeam and TrueNAS need very different versioning behavior (see Bucket Baseline Settings below). Splitting workloads into separate buckets lets each lifecycle rule be bucket-wide and trivial to reason about, instead of managing prefix-based rules per device.
- Blast radius is per-(client, location, workload). A compromised credential, a rogue script, or an accidental deletion is contained to one slice of one client's backup data. Never touches another client, never touches another location, never touches another workload.
Per-location isolation also gives bucket-level cost attribution for free — B2's bucket-level usage reports expose per-location cost data directly without any additional tagging work.
How this maps to the fields: the bucket-name/folder fields are set at the organization (client) level and inherit down, so a client's default is one bucket per (client, workload). A location that genuinely needs its own bucket — separate lifecycle, isolation, or cost line — gets one by overriding the field at the location level. Per-(client, location, workload) isolation is therefore available on demand rather than mandatory for every location; use an override wherever a location warrants a distinct bucket, and keep the shared org bucket where it does not.
2. Device GUID is the only identity stored on the device
The only DTC-managed identifier we persist on the device itself is its device GUID. Not the org UUID, not the location UUID, not the bucket name. Just the device GUID.
The reason is simple: devices move. BDRs and TrueNAS boxes get reassigned between organizations and locations over their lifetime — bench swaps, site relocations, client transitions, asset returns. If the device carries stale org/location metadata, every move creates a drift bug waiting to happen. The device GUID is the one thing about a device that never changes once assigned.
Everything else is ambient context that lives in NinjaOne as inherited custom fields (Organization → Location → Device). The bucket name, folder, and backup policy are held at the org/location level and inherit down to the device; a provisioning script reads them from the device's effective (inherited) values at runtime. When a device moves to a new location, it inherits that location's values automatically — there is no stale local state to reconcile, because the device never authored those values in the first place.
Principle: identity lives on the device, context lives in NinjaOne (as inherited org/location fields), and the device reads whatever context it currently inherits.
3. NinjaOne is the source of truth for context
The bucket name, folder, backup policy (RPO/RTO), schedule, and retention all live as custom fields in NinjaOne, set at the Organization (client) level and, where needed, overridden at Location or Device. They are not authored on the device, not in hardcoded config files, not in an operator's memory. NinjaOne holds the authoritative mapping and resolves the effective value for each device through field inheritance; the provisioning/configuration scripts read whatever NinjaOne says is true right now.
Two scope rules follow from Principle #1 (bucket per client, location, workload):
- Bucket name and folder are set at the Organization (client) level — the inheritance root — and flow down to location and device. The org value is the default bucket for the client. A location overrides it with its own
veeam-{uuidv7}wherever Principle #1 warrants a separate bucket (a location that needs its own lifecycle, isolation, or cost attribution); a device-level override exists but is rarely used. - Policy fields inherit client → location → device. RPO/RTO, schedule, retention, and immutability are set once at the client level and overridden per location or per device only when a site or box genuinely differs.
Corollary: if NinjaOne is wrong, the backup architecture is wrong. Keep device placements clean, keep the location and org records populated, keep the inherited fields accurate. The entire cloud backup architecture depends on NinjaOne being the single point of truth.
4. Scoped credentials, vault-stored
Every bucket gets its own scoped B2 application key. The key is restricted to the bucket it provisions and does not have account-wide permissions. Per-client scoped credentials are stored in the client's IT Glue record. DTC-internal credentials (B2 admin/master key) are stored in 1Password. Credentials are never stored in plaintext on the device, never committed to git, never pasted into ticket notes.
Naming Scheme
The Operational UUID (OUID)
The Operational UUID (OUID) is DTC's universal entity identifier. Every managed entity — org, location, device, user, workload — has its own OUID. It is the canonical identity across all DTC-managed systems and the shared key that internally developed applications use to bridge NinjaOne, Backblaze, TrueNAS, Veeam, and other platforms.
For the full OUID specification, generation instructions, and design principles, see Operational UUID (OUID) Standard in Pillars of Technology.
NinjaOne is the authoritative source for OUIDs at the org and location level. The custom fields Org UUID and Location UUID hold the OUIDs for those entities. Devices also have OUIDs — the device GUID referenced throughout this page is the device's OUID. Same concept, same pattern, applied at every level of the hierarchy.
The bucket name is not derived from any of these OUIDs. It is an opaque, standalone identifier (see below). The OUIDs remain the join keys for reporting and lookup in NinjaOne — you go from a bucket back to its client/location by reading the NinjaOne mapping, never by parsing the bucket name.
Bucket name
Bucket name
{workload}-{uuidv7}
| Component | Format | Source |
|---|---|---|
workload |
Lowercase token, controlled vocabulary | Per-workload constant (see table below) |
uuidv7 |
Lowercase UUIDv7 with dashes | Generated once at provisioning and stored in NinjaOne (see below) |
The bucket name references nothing. It only has to be globally unique. It is pure metadata: a random UUIDv7 with a workload prefix, carrying no org, location, or device information in the string itself. The (client, location, workload) → bucket mapping lives entirely in NinjaOne custom fields, not in the name. This is deliberate — an opaque name never goes stale when a device moves or a location is bought or sold, because there is nothing encoded in it to become wrong. The mapping is a field lookup, and a field is trivial to update; a name baked into thousands of stored objects is not.
Where the name comes from — generated once, then inherited:
- Windmill generates it at client onboarding. When a client is created with a recurring invoice line item of
BDR-MSP, Windmill mints a{workload}-{uuidv7}bucket name, provisions the bucket, and writes the name into the NinjaOne organization (client) custom field (veeamS3Bucketfor Veeam; the equivalent per-workload field otherwise). Folder is written the same way. Where a location needs its own bucket, that location's field is overridden with its own generated name. - It is stable for the life of the client/location. The device never generates or writes it — the device inherits the value at runtime. If a BDR is replaced, the new box inherits the same bucket and folder and reclaims the existing data. No recompute, no rename, no data migration.
UUIDv7 (not v4) so the identifiers are time-ordered and sort roughly by creation — the same OUID format used everywhere else at DTC. B2 bucket names are globally unique across all accounts; a full UUIDv7 makes a collision effectively impossible.
Workload tokens (controlled vocabulary):
| Workload | Token | Example bucket length |
|---|---|---|
| Veeam Backup & Replication | veeam |
42 chars |
| TrueNAS Cloud Sync | truenas |
44 chars |
| Future | TBD | add here when introduced |
Keep workload tokens short, singular, and hyphen-free. The 50-char B2 bucket-name limit is the constraint; {workload}- plus a 36-char dashed UUIDv7 leaves room for a workload token up to 13 chars. Do not add new workload tokens without updating this page AND adding a per-workload lifecycle policy (see Bucket Baseline Settings below).
Example bucket names:
veeam-0190a1b2-c3d4-7e5f-8091-a2b3c4d5e6f7
truenas-0190a1b2-c3d4-7e5f-8091-a2b3c4d5e6f7
Two workloads at the same location get two unrelated UUIDv7 bucket names — the fact that they share a location is recorded in NinjaOne, not inferable from the strings.
Folder (object prefix inside the bucket)
Folder (object prefix inside the bucket)
{folder}/{optional-sub-path}/
The top-level folder is a stored, inherited value, not the device GUID. Like the bucket name, it is written into the NinjaOne location custom field (veeamS3Folder for Veeam) at onboarding and inherited by whatever device serves that location. It is stable across device replacement — swap the BDR and the new box inherits the same folder and lands on the existing data. This is the change from the earlier device-GUID-as-folder scheme: a device-GUID folder would strand the old data the moment the box was replaced, because the replacement gets a new GUID.
Grep-ability across systems is preserved through NinjaOne, the ZFS com.dtctoday:device-guid property, and log lines — the device GUID still appears in all of those; it just is not baked into the object path, so the path can outlive any single device.
The workload token is already part of the bucket name, so it does not repeat inside the folder path. A device's data in a Veeam bucket is already Veeam data by virtue of which bucket it lives in.
Optional sub-path
Below the folder, the {optional-sub-path} has two possible meanings depending on how the workload behaves:
1. The app manages its own internal layout. If the backup app creates and organizes files inside the folder you give it, there is no sub-path from DTC — the app owns everything below the folder. Veeam Backup & Replication is the canonical example: point its repository at the folder and let Veeam do whatever it does internally. One repository root per location bucket, and the folder is that repository.
2. The app doesn't organize further — we pick a root label. If the app just syncs a source into a destination and has no concept of further structure, the sub-path is a stable, human-readable label for the source: the dataset root name, the pool name, or the source root path converted to a path-safe form. TrueNAS Cloud Sync is the canonical example — each Cloud Sync task maps one source dataset or root path to one sub-path under the folder.
Examples:
Veeam (app manages structure — no sub-path, the folder is the repository root):
Veeam/
TrueNAS (multiple datasets, one sub-path per source):
data/tank/
data/nvme-fast/
data/archive/
One Cloud Sync task per sub-path, each pointing at one source dataset or root path on the NAS. Name the sub-path to match the source: if the source is the ZFS dataset tank, the sub-path is tank/; if the source is the root path /mnt/archive, the sub-path is archive/. Stable and grep-able is the goal.
Full S3 path example (TrueNAS, one dataset):
truenas-0190a1b2-c3d4-7e5f-8091-a2b3c4d5e6f7/data/tank/
Adopting an existing bucket (migration)
Not every BDR was born under this scheme — many already back up to a bucket created the old way. Do not mint a fresh veeam-{uuidv7} for a device that already has a working bucket. Adopt the existing one:
- Inventory discovers it.
veeam-inventoryreads the bucket the device's live S3 copy job actually targets and writes it toveeamS3Bucketat the device level — but only when no value is already inherited from org/location, so it seeds un-migrated boxes without shadowing already-adopted clients. - Windmill promotes it. A Windmill automation sweeps BDRs that carry a device-level
veeamS3Bucketand copies that value up to the organization (client) field when the client has none yet. The existing bucket becomes the client's inherited default — no new bucket is created. - Clear the device override. Once the value lives at org, clear the device-level
veeamS3Bucketso the device inherits cleanly. A lingering device override permanently shadows the org value.
The result: legacy buckets are adopted in place, and only a genuinely new client (a BDR-MSP line item with no existing bucket anywhere in its hierarchy) gets a freshly minted veeam-{uuidv7}.
Bucket Baseline Settings
Every cloud backup bucket must be provisioned with these settings. Do not deviate without a documented exception.
| Setting | Value | Reason |
|---|---|---|
| Object Lock | Enabled (compliance mode) | Immutability — protects backups from ransomware, insider threat, accidental deletion |
| Default retention period | 14 days (minimum) | Baseline floor; per-workload retention can be longer via individual object stamps |
| Server-side encryption | SSE-B2 (AES-256) enabled | Data-at-rest encryption managed by Backblaze |
| Versioning | Default on (implicit with object lock) | Required for object lock; hidden-version cleanup governed by the bucket's lifecycle rule |
| Lifecycle rule | Single bucket-wide rule, per workload (see below) | Each bucket holds exactly one workload, so one bucket-wide rule is sufficient. No prefix management, no per-device sprawl. |
| Bucket visibility | allPrivate | Never expose backup buckets to public or authenticated anonymous access |
| CORS | Not configured | No browser access required |
Lifecycle rule details
Principle: the object storage layer controls versioning and retention when the backup application does not. Some apps (Veeam) manage their own versioning and retention entirely at the application layer — Veeam stores its own block versions as blobs in object storage and does not rely on B2's native versioning for recovery points. The moment a version becomes hidden at the B2 layer, it is already stale by definition and should be deleted as fast as B2 allows. Other apps (TrueNAS Cloud Sync) have no version concept of their own and just sync current filesystem state, so the bucket must provide the versioning window or historical file recovery is impossible.
Because each bucket is scoped to exactly one workload, one bucket-wide lifecycle rule per bucket is sufficient.
Per-workload policy:
| Workload token | Hidden-version retention | Rationale |
|---|---|---|
veeam |
Delete immediately — daysFromHidingToDeleting: 1 (B2 minimum) |
Veeam stores its own block versions as blobs and manages retention at the application layer. A hidden version at the B2 layer is already stale. Intent is "delete ASAP"; B2's floor of 1 day is the closest the platform allows. |
truenas |
Delete hidden versions after 14 days | TrueNAS Cloud Sync has no native version concept. B2 versioning provides a 14-day historical recovery window for accidental deletion, ransomware encryption, or bad edits. |
Example rule (Veeam bucket):
[{
"fileNamePrefix": "",
"daysFromUploadingToHiding": null,
"daysFromHidingToDeleting": 1
}]
daysFromHidingToDeleting: 1 is the B2 minimum. The documented intent is immediate deletion as soon as a version becomes stale; B2's 1-day floor is the platform implementation of that intent.
Example rule (TrueNAS bucket):
[{
"fileNamePrefix": "",
"daysFromUploadingToHiding": null,
"daysFromHidingToDeleting": 14
}]
Interaction with object lock: any lifecycle deletion is a no-op against objects still under compliance lock. The 14-day object lock retention floor always wins — the lifecycle sweep can only act on hidden versions once their lock has expired. For the Veeam bucket, this means the effective delete window for a hidden object is max(1 day, remaining_lock_time). The policy documents intent; B2 + object-lock mechanics are the implementation floor.
Per-location lifecycle overrides: the above are defaults. A specific client location may legitimately need a different lifecycle (regulated-data office, client-mandated legal hold, seasonal or temporary location with a different RPO). Because buckets are per-(client, location, workload), a per-location override is a single edit to that one bucket's lifecycle rule — no prefix management, no risk of bleed-over to other locations or other workloads. Document any deviation in the client's documentation.
Adding a new workload: define its lifecycle policy here at the same time the workload token is added to the controlled vocabulary. A workload without a documented lifecycle policy does not get onboarded.
Why compliance mode for object lock
Compliance mode means no one — including the root account — can shorten or remove retention before the lock expires. Governance mode allows privileged override. For backup workloads, compliance mode is the correct default: the scenario we're defending against includes compromised administrative credentials, and governance mode would allow an attacker with admin creds to bypass the lock and delete backups. Compliance is non-negotiable.
Credential Management
Scoped application key
Every bucket gets a dedicated B2 application key with these properties:
| Property | Value |
|---|---|
| Capabilities | listBuckets, readBuckets, writeBuckets, deleteBuckets, listFiles, readFiles, writeFiles, deleteFiles, readBucketEncryption, writeBucketEncryption, readBucketRetentions, writeBucketRetentions, readFileLegalHolds, writeFileLegalHolds, readFileRetentions, writeFileRetentions, bypassGovernance |
| Bucket restriction | Restricted to the provisioned bucket only |
| Name prefix | Not restricted (full bucket access, but only to this bucket) |
| Duration | Unlimited (no expiration) |
The capability list is intentionally broad within the bucket because Veeam and TrueNAS both need to read, write, delete, and manipulate retention metadata for normal operation. The isolation comes from the bucket restriction, not from capability trimming.
B2 does not support folder-prefix restriction on application keys. A compromised key has access to the entire bucket. This is bounded because a bucket is one client's data at one location for one workload, so the worst case is that single slice of backup data being exposed — not the client's other locations, not their other workloads, not the whole MSP. This is one of the structural reasons for per-(client, location, workload) bucket scoping.
Credential storage
- Primary storage: client's IT Glue record, tagged with the bucket name and workload. Per-client scoped B2 keys are client credentials and live in the client's IT Glue record, per the Approved Credential Storage standard. The DTC B2 admin/master key (used to provision new buckets and keys) lives in 1Password as a DTC-internal credential.
- Device-local storage: only required for the workload to function (e.g. TrueNAS Cloud Sync credential storage, Veeam repository credential storage). The device holds the minimum necessary to run.
Credentials rotate on any of these events:
- Key suspected of compromise
- Technician with access departs
- Annual scheduled rotation (tracked as a recurring HaloPSA ticket)
Device Identity
The device GUID
Every DTC-managed backup device carries a Device GUID — the device's OUID, a lowercase UUIDv7 with dashes, generated once at first provisioning and stored persistently on the device. This is the sole DTC-managed identifier on the device itself. See the OUID Standard for the canonical format, generation, and storage rules; this section mirrors them for the backup context.
Format example: 9babab19-acc0-4587-aa05-ccd8103a1148
Generated how: UUIDv7 per the OUID Standard. On Windows with .NET 9+, [System.Guid]::CreateVersion7().ToString().ToLower(); otherwise npx uuid v7 or Python 3.14+ uuid.uuid7(). Do not use uuidgen (no v7 support). On Windows, rmm-ninja/ninja-ensure-device-guid.ps1 mints and persists it, with a dependency-free UUIDv7 fallback for older runtimes.
Legacy note: devices provisioned before the UUIDv7 standard may carry a UUIDv4 device GUID. These remain valid and are not regenerated — an OUID is assigned once and never changes. Only newly provisioned devices get v7.
Generated when: at first provisioning. If the provisioning script runs on a device with no existing device GUID, it generates one and writes it. On every subsequent run, the script reads the existing GUID and uses it unchanged. The GUID is the device's stable cross-system key (NinjaOne, ZFS property, logs); storage-path stability across reboots, rebuilds, and device replacement comes from the location-inherited bucket/folder fields, not from the GUID.
Stored where:
| Platform | Storage mechanism |
|---|---|
| Windows (Veeam BDR, etc.) | Registry HKLM\SOFTWARE\DTC\DeviceGuid and the NinjaOne device custom field |
| TrueNAS (CORE or SCALE) | ZFS user property com.dtctoday:device-guid on the root pool |
| Proxmox / Linux with ZFS | ZFS user property com.dtctoday:device-guid on the root pool |
| Linux without ZFS | /etc/dtctoday/device-guid |
| Any device | Additionally mirrored in NinjaOne as a device custom field for cross-system lookup |
The com.dtctoday namespace
All DTC-managed metadata stored on a host via ZFS user properties uses the com.dtctoday: namespace. This matches the ZFS convention of using a reverse-DNS name you actually own, and dtctoday.com is the authoritative DTC domain.
Properties in scope:
| Property | Meaning | Required? |
|---|---|---|
com.dtctoday:device-guid |
The device GUID | Yes, always |
Additional properties are not stored on the device — context lives in NinjaOne. Do not add com.dtctoday:org-uuid, com.dtctoday:location-uuid, or com.dtctoday:bucket-name as ZFS properties. They are held in NinjaOne as inherited org/location fields and read at runtime; persisting them on the device would create drift when the device moves.
NAS discovery via NinjaOne NMS
NAS devices (TrueNAS, Synology, QNAP, etc.) are added to NinjaOne via the NMS (Network Monitoring System) subsystem, not as standard agented endpoints. The exception is UniFi gear, which has its own management surface and does not need NMS.
Every NAS added to NinjaOne gets a device GUID generated and stored — the GUID is what relates the NAS record back to its buckets and backup state.
Standard NMS credentials for NAS onboarding:
| Protocol | Settings |
|---|---|
| SNMP version | SNMPv3 only |
| Authentication | Random username, random password |
| Encryption | Random encryption secret |
| Auth protocol | SHA-256 (or higher) |
| Priv protocol | AES-128 (or higher) |
SNMPv1 and SNMPv2c are forbidden. Credentials are randomly generated per-device (not shared across fleet) and stored in the client's IT Glue record.
Device Moves and Context Drift
Device Moves and Context Drift
When a device moves in NinjaOne, it inherits whatever bucket/folder its new placement resolves to — automatically, because those values are inherited org/location fields and the device never held its own copy. The device GUID itself stays identical. A move within the same client keeps the org-level bucket (unless the destination location carries an override); a move to a different client picks up the new client's org bucket.
Note the asymmetry the design buys you: if a location is reassigned to a different client (bought, sold, transferred) and that location had its own bucket override, the override moves with the location record, so its bucket does not change and its data stays put — only the org association in NinjaOne updates. A pure org relabel with no physical relocation requires no re-provisioning. Only a move that changes the resolved bucket triggers reconfiguration.
Handling a move:
- Move the device record in NinjaOne (standard asset transfer process).
- Re-run the configuration script on the device. It reads the device's now-inherited location values (new bucket, new folder, new policy) and reconfigures the workload's repository/task to point at the new bucket.
- If the new location has no bucket yet (e.g. a location that never carried this workload), Windmill provisions one and writes the location field first; the device then inherits it.
- The old bucket's data remains available under its original object-lock retention. It ages out per lifecycle policy or is explicitly deleted when the old location's offboarding workflow runs.
Do not: attempt to rename buckets, copy data between buckets, or reuse a bucket that belongs to a different (client, location, workload). Each bucket is scoped to its location and stays there for its entire lifecycle.
Do: use the backup inventory script to detect devices whose live repository/task bucket no longer matches the bucket they currently inherit from their location (i.e. devices that moved but weren't reconfigured). Flag for review.
Teardown Workflows
Single device decommission, location remains
A BDR or TrueNAS box is being retired but the location it served is still active:
- Remove the device's repository/task from the backup software.
- Leave the bucket in place. Other devices at the same location for the same workload may share the bucket if they were provisioned into it.
- The device's folder under the bucket ages out naturally under lifecycle, or can be explicitly pruned once object lock on its objects has expired.
- Clear the device GUID from the device before repurposing it (delete the ZFS property or NinjaOne custom field).
Location closure
The location is being decommissioned entirely:
- Remove all backup tasks/repositories pointing at the buckets at that location (one per workload).
- Wait for object lock retention to expire on the last backup in each bucket, or accept the delay. Compliance mode will not allow early deletion.
- Delete each of the location's buckets via
b2_delete_bucket. One bucket per workload at the location; each delete is a single atomic operation with a bounded blast radius. - Remove the NinjaOne location record.
Safety check before bucket deletion: query NinjaOne for any remaining devices at that location. If any device still has a live backup job pointing at a bucket, bail. Do not delete a bucket while a live job is writing to it.
Whole client offboarding
The client is leaving DTC entirely:
- Iterate all locations for the org in NinjaOne.
- For each location, run the location-closure workflow above.
- After all buckets are deleted, remove the NinjaOne org record.
- Confirm in Backblaze that no orphaned buckets remain. Because nothing about the client is encoded in the bucket name, you cannot derive the bucket set — enumerate the org's locations in NinjaOne, read each location's stored bucket-name field(s), and verify every one of those buckets is deleted.
More buckets to delete than a one-bucket-per-client scheme would have, but each deletion is atomic and bounded. Script it and batch it. Safer than a single massive prefix walk and safer than any shared-bucket alternative.
Why This Architecture
The short version, for the next tech or AI that reads this and wonders why it's not simpler:
- Per-(client, location, workload) buckets: because every backup application we integrate with is single-tenant by assumption, stores credentials on devices physically at the client site, and has a UI surface where a technician picks a device from whatever's in the bucket. A multi-tenant bucket makes wrong-device data incidents trivially easy to cause. Tight bucket scoping makes them impossible at the storage layer. Splitting on location additionally gives per-location lifecycle flexibility (not every location has the same retention requirements) and per-location cost attribution for free. Splitting on workload makes lifecycle rules bucket-wide and trivial to reason about. The operational cost of more buckets is far cheaper than the consequence of a single cross-tenant restore or a misrouted retention policy.
- Opaque
{workload}-{uuidv7}bucket names: because a name that encodes org/location/device is a name that goes stale the moment any of those change. A random UUIDv7 references nothing, so it can never be wrong; the(client, location, workload) → bucketmapping lives in NinjaOne where it is cheap to update, instead of being baked into the identity of thousands of stored objects. - Device GUID is device-only state: because devices move between orgs and locations over their lifetime. Everything else is ambient context — bucket, folder, policy — that lives in NinjaOne as inherited org/location fields and would drift if pinned to the device.
- NinjaOne as source of truth (via inherited fields): because we already maintain NinjaOne as the authoritative asset registry. The client sets policy once and it inherits to every location and device; per-site exceptions are a single override. Duplicating that state into device-local files creates two sources of truth, which always diverge.
- Bucket-wide lifecycle rules, not prefix rules: because each bucket holds exactly one workload, so one rule per bucket is enough. No prefix management, no per-device rule sprawl.
- Object storage controls versioning only when the app doesn't: Veeam stores its own block versions as blobs and manages retention at the app layer, so B2 drops hidden versions ASAP (subject to the 1-day B2 floor). TrueNAS Cloud Sync has no version concept, so B2 provides a 14-day window. The storage layer fills the gap; it does not duplicate the app's logic.
- Scoped application keys per bucket: because the blast radius of a compromised key should be one (client, location, workload) slice, not the entire fleet.
- Compliance-mode object lock: because the ransomware threat model includes compromised administrative credentials, and governance mode does not defend against that.
Every piece of this architecture is answering a specific failure mode. When changing any of it, ask: what failure mode was this defending against, and is the change still safe against that failure?
Related
- Workload-Instance Bucket Architecture — Mode B companion (app-managed multi-tenancy, MSP360, DTC apps)
- Deployment chapter: per-workload provisioning SOPs (Veeam BDR Deployment SOP, TrueNAS Cloud Sync Provisioning SOP)
- Pillars of Technology → Backup & Data Protection Standards (for RPO/RTO targets, retention philosophy, testing requirements)
- DevOps book (if present): scripting conventions used by provisioning scripts
msp-script-libraryrepo →bdr-veeam/and futurenas-truenas/directories