Backup Methodology & Direction
Status: direction, not yet standard. This page records where DTC's data protection model is heading, why, and how it would be built. The authoritative, in-force policy remains Backup & Data Protection Standards. Nothing here changes a client configuration until it is ratified into that page. Decision points still open are marked OPEN and listed at the end.
This document exists because "how we do backups" is currently spread across three tier definitions, two platform standards, and forty troubleshooting pages — and none of them say what we are actually optimizing for. That gap is why the same argument gets re-litigated every time a site's internet cannot carry an image.
Where We Are Today
DTC's protection model is image-first and hybrid. Every managed server, and the pan/imaging workstations that hold clinical data, gets a full block-level image. That image is written to local storage (NAS or BDR appliance) and replicated to cloud. Tiers 1 through 3 differ in how fast we can bring the client back, not in what we capture.
This has real virtues and they should not be waved away:
- One capture mechanism covers every workload. No per-application engineering.
- Bare-metal recovery to dissimilar hardware is possible because the whole disk is there.
- Nothing depends on knowing what the line-of-business application keeps or where.
- The cloud copy is immutable (S3 Object Lock), which is a compliance answer we can give in one sentence.
Why Image-First Is Not Enough
The problems with a pure image model are structural, not bugs to be fixed:
The image is sized by the disk, not by the data. A practice with 400 GB of Windows, applications, and page file and 40 GB of actual patient data ships an image sized by the former. Every night. The 90% we do not care about sets the transfer cost.
It couples RPO to the client's ISP. This is the failure we keep hitting. A site with thin or unreliable upstream cannot push a nightly image inside the overnight window, so the cloud leg fails — and it fails silently in aggregate, because the local leg stays green, the dashboard shows a partial success, and the offsite copy quietly ages.
Restore granularity is wrong for the actual failure modes. Bare-metal disasters are rare. What actually happens: a database gets corrupted, a staff member deletes a folder, an imaging series goes missing, ransomware hits a share. Every one of those is a logical recovery, and every one currently means mounting an image to get one thing out of it.
Crash-consistent is not application-consistent. VSS gets us close, but a block-level snapshot of a running database is a crash-consistent copy that happens to usually work. We already have a page documenting the case where it did not (MariaDB Crash-Consistent Backup — Missing InnoDB Tablespace Files). "Usually works" is not a recovery posture.
Restore requires the platform that made the backup. An image in a proprietary container needs the vendor's console, agent, or boot media to open. That is a dependency on a live vendor relationship at the exact moment you are least able to tolerate one.
And the deepest one: a nightly image is periodic physical protection standing in for continuous logical protection. It is a photograph of a disk taken once a day, used as a substitute for knowing that the data itself is safe somewhere else right now. Those are not the same guarantee, and at scale the difference is what bites.
The Model: Protect State by Class, Across Fault Domains
The image model asks one question — is this machine captured? This model asks two:
- What class of state is this?
- How many independent fault domains does it survive?
Everything else follows from those two answers.
Naming — proposal
Working name: Structural Data Protection (SDP). The per-client test it produces is "is this application structurally safe?" — which is answerable by a technician, at a site, without a spreadsheet.
Recommended because it names the actual claim: safety is a property of how the environment is structured — where state lives, how many failure boundaries separate its copies, what the production storage itself guarantees — not a property of a backup schedule. It scales in both directions, from a single-server dental office to a containerized workload on object storage, without changing vocabulary.
Alternates considered: Fault-Domain Data Protection (precise, names what 3-2-1 fails to count, but dry); Data-Class Protection Model (emphasizes the classification step, understates the fault-domain math); 3-2-1 per Class (positions it as an extension of something everyone already knows, at the cost of implying it is a minor tweak). OPEN — the name is Nate's call.
Fault domains, defined
A fault domain is a boundary such that a failure inside it does not propagate outside it. Naming them explicitly is what makes the model usable, because "offsite" hides most of the interesting distinctions:
| Fault domain | What failing inside it looks like |
|---|---|
| The production server / VM | Disk failure, OS corruption, bad patch, database corruption |
| The site LAN | Ransomware reaching every writable SMB share, including the NAS |
| The site | Power, flood, fire, theft, circuit outage |
| The storage system | NAS controller failure, array loss, filesystem corruption |
| DTC cloud / object storage | Our own platform failure, our own credential compromise |
| The application vendor's cloud | Vendor outage, vendor data loss, account termination |
| Region | Provider regional outage |
Note what this exposes about the current model. A hybrid client's local NAS leg is a writable SMB share on the site LAN. It is a different device, but it is not a different fault domain from the thing most likely to kill them — ransomware with domain credentials. The Standards page already says the local NAS leg is not immutable and should be treated as a fast-restore convenience. The fault-domain table is why that sentence matters more than it reads.
How this differs from 3-2-1
3-2-1 counts copies: three copies, two media, one offsite. It is a good heuristic for one machine and it is silent on everything that matters at scale:
- It does not distinguish classes of data, so it prescribes the same treatment for a 2 GB database and a 4 TB imaging archive.
- It counts copies, not failure boundaries — three copies inside one blast radius satisfies the arithmetic and none of the intent.
- It treats protection as something done to production, never something built into it. It has no way to express "this storage layer is already versioned, snapshotted, and immutable."
- It has no notion of recovery granularity — restoring one file and restoring a server are the same event to 3-2-1.
SDP keeps 3-2-1 as a floor and replaces the counting rule: for each data class, enumerate the fault domains its copies survive, and require that count to match the criticality of that class. 3-2-1 is what you tell a client. This is what you build.
The Two Classes
Class A — Application state: the app and its database, together
Do not try to pull these apart. The application and its database are a single stateful unit and they can and should live together on the compute's storage. Attempting to architecturally separate a line-of-business application from its database at a dental practice buys nothing, breaks vendor support, and is not on offer anyway.
Class A is protected by extracting its logical state to another fault domain — a dump, an export, an API pull, or a continuously shipped transaction log. Not by relocating it.
Characteristics that drive this: small relative to the disk, changes constantly, transactionally structured, catastrophic if lost, and — critically — cheap to move once you have it in logical form. A practice-management database that is 8 GB on disk is a small nightly transfer as a compressed dump, and a trivial one as a stream of transaction log records.
Class B — Files, blobs, objects
This is the class that must be separated out. Imaging studies, scanned documents, attachments, exports, archives. Characteristics: large, grows without bound, individual items are immutable once written, and the recovery request is almost always "this one thing" rather than "all of it."
Class B is protected by making the production storage itself fault tolerant in place — versioning, snapshots, and immutability across fault domains — rather than by copying it somewhere every night. Backup is part of the Class B solution, not the whole of it.
The reason to separate Class B is not tidiness. It is that in-place resilience actually works for immutable, ever-growing, individually-addressable data, and does not work for a hot transactional database. Different physics, different mechanism.
Class B — How To Make File and Object Data Fault Tolerant In Place
Target architecture
Object storage for blobs. Container for the application. Database on disk.
- The application runs in a container. Stateless, replaceable, rebuilt from an image and a config. Nothing to back up because nothing irreplaceable is in it.
- The database sits on disk — a real block volume on the compute host. Class A rules apply to it.
- Blobs live in S3-compatible object storage, addressed by the application directly, never through a file share.
This is the arrangement to aim for in anything DTC builds or hosts, and the arrangement to prefer when evaluating a vendor application.
What the object layer must be configured to do
The protection lives in the storage configuration. Enable all of it — these are not optional extras, they are the reason this architecture is chosen:
- Versioning on. Every overwrite and every delete creates a version rather than destroying data. This is what makes ransomware and accidental deletion survivable without a restore: the prior version is already there, addressable, in place.
- Object Lock / immutability, in compliance or governance mode per the client's requirement. A version under lock cannot be deleted by us, by the client, or by an attacker holding our credentials.
- Snapshots at bucket or dataset level, for point-in-time recovery of the namespace rather than of individual objects.
- Lifecycle rules governing version expiry and storage-class transition, so versioning does not become an unbounded cost. This is the control that makes the above affordable.
- Cross-region or cross-provider replication where the client's requirement justifies a second geographic fault domain. Optional, deliberate, and priced — not default.
- Access separation. The credential the application uses to write objects must not be able to delete versions or alter lock settings. A single compromised application credential should not be able to unwind the protection.
The result: recovery of a deleted or encrypted file is a version-list call and a copy, measured in seconds, with no backup job in the path at all. Backup then exists to cover the failure modes versioning does not — the bucket itself, the account, the platform.
When the vendor application will not cooperate
Most dental software writes blobs to a local path or a UNC share and cannot be told to use object storage. That constraint is real and it is not going away by wishing.
Options, in order of preference:
- Front it with a sync to object storage — a one-way, append-oriented replication from the share into a versioned, locked bucket. Individual items in this class are immutable once written, which is exactly what makes one-way sync safe here: there are no legitimate overwrites to reconcile.
- Put the share on storage that provides its own versioning and snapshots (ZFS/TrueNAS snapshots, for instance) so at least the storage-system fault domain is covered in place, and treat the object copy as the cross-domain layer.
- Accept that it is backup-only, document that the recovery path for these files is a restore rather than a version fetch, and set the client's expectation accordingly.
Option 3 is a legitimate answer. Writing it down as a deliberate choice is what separates it from the status quo, which is the same posture arrived at by accident.
Class A — How To Get Application State Off The Box
This is the priority. For a single-server office it is the most important thing in this entire document, and it is the thing DTC currently does not do at all.
The priority inversion
Today: the image is the primary protection, and a logical dump either does not exist or exists as an afterthought inside the image.
Target: the logical state of the database is extracted to a different fault domain first, on its own schedule and by its own mechanism. Traditional backup then picks it up as a file, using whatever is already deployed.
The dump lands on disk somewhere the existing backup already sweeps, so Ninja file-level, Ninja image, Veeam, or anything else captures it for free with no new configuration. The dump also goes directly to a second fault domain — another endpoint on site, or DTC cloud object storage — so its safety does not depend on the backup job succeeding.
That ordering is the whole change. The dump existing off-box is the guarantee. The backup catching it is a bonus that costs nothing.
Extraction mechanisms, by engine class
| Engine class | Nightly logical extract | Continuous / point-in-time |
|---|---|---|
| Microsoft SQL Server | Native BACKUP DATABASE to file |
Transaction log backups on a short interval, shipped |
| MySQL / MariaDB | mysqldump, or mariabackup for larger sets |
Binary log (binlog) shipping |
| PostgreSQL | pg_dump |
WAL archiving to a remote target |
| Sybase SQL Anywhere | Vendor dbbackup utility |
Transaction log truncation and copy |
| Embedded / proprietary (Faircom c-tree and similar) | Vendor-provided export or backup utility only; no generic path | Usually unavailable — nightly export is the ceiling |
| API-only / vendor-cloud | Vendor API export | Incremental API pull on a schedule |
Verify the engine per product and per version — do not assume from this table. Mapping each dental application to its actual engine, version, and supported extraction method is the first real work item of the suite below, and the reason it is work rather than a lookup is that the answer differs by product and by release. Where a vendor provides an API, prefer it: it is supported, it survives schema changes, and it does not require touching the database directly.
Where the extract goes
Both, always:
- Local, on the compute host's own storage (or storage it has access to). Cheap, instant, and it means every existing backup mechanism captures it with no new configuration. This is also the fast-restore copy.
- A second fault domain, directly and independently: another endpoint at the site, or DTC cloud object storage. Cloud is strongly preferred, because another endpoint on the same LAN shares the ransomware fault domain with the server.
Realtime versus nightly
A nightly dump gives a 24-hour RPO on the data that matters most. Continuous transaction log or WAL shipping gives minutes, plus true point-in-time recovery — the ability to restore to just before the corruption rather than to the last midnight.
For a single-server office, nightly is the floor and it is a large improvement over nothing. Continuous log shipping is the goal where the engine supports it and the site can carry a trickle of log traffic — which almost any site can, because log volume is a fraction of a percent of image volume. This is the case where a site with genuinely bad internet can still hold a near-realtime offsite copy of the only data that is irreplaceable.
The structural safety bar
An application is structurally safe when all four hold:
- Its logical state exists in current form outside the production server's fault domain.
- That extract is versioned — not a single file overwritten nightly, which is one bad night away from being worthless.
- The extract is inside a retained backup (file-level or image, any platform).
- A restore of the extract has been tested, per the Standards page rule that an untested backup is not a backup.
Three of the four say nothing about images, appliances, or tiers. That is the point: structural safety is a property of the data's position across fault domains, not of the product that captured it.
The Application Backup Suite — Target State
The end state is that DTC owns this capability rather than assembling it per site out of scripts.
What it is: a purpose-built application backup service, deployable in DTC cloud or on-premises, that understands line-of-business applications rather than disks.
How it works:
- Per-application connectors. Each supported application gets a connector that extracts its logical state via the vendor API where one exists, or via direct database access where it does not. Starting with the full set of dental software DTC supports, because that is where the fleet is.
- Scheduled outside business hours, per client, across the whole fleet — the same operational posture as backup today, but producing logical exports rather than block images.
- Versioned retention. Every run is a retained version. Logical exports are small enough that retention far longer than the 30-day image window is affordable, which is a client-visible improvement that costs almost nothing.
- Dual destination, always. DTC's own object storage — versioned, immutable, and designed around this service rather than borrowed from a backup vendor — and back onto the compute host storage it came from, so existing Ninja and Veeam jobs sweep it up with no per-site configuration.
- Verification built in. Hash each export on write and re-verify on read. This delivers most of the "future state hash-based verification" already on the Standards page roadmap, as a property of the design rather than a separate project.
Why build rather than buy: no backup vendor is going to write a Dentrix or Eaglesoft connector for a fleet of dental practices. The applications are the value and the applications are what nobody else covers.
Why it matters commercially: application-level recovery of a practice-management database in minutes, with long version history and a tested restore path, is a materially different product from a nightly image with a one-week RTO. It is also defensible in the cyber-insurance and compliance conversations where "we have backups" is starting to stop being an answer.
Sequencing note. The suite is the destination, not the starting move. Scripted nightly dumps landing in two fault domains — done by hand, per site, with no new software — deliver most of the risk reduction and can start now. The suite makes that repeatable, monitored, and sellable. Do not let the second block the first.
What Determines Which Model a Site Gets
Recording the criteria rather than a list of client names, because the client list goes stale and the reasoning does not.
Class A extraction is not optional and is not bandwidth-gated. A compressed nightly dump, and certainly a stream of transaction log records, fits on any circuit DTC would tolerate for management traffic. There is no site where "the internet is too slow" justifies having no offsite copy of the database. That excuse only ever applied to images.
Everything else is bound by overnight upstream throughput versus nightly change volume. Compute it, do not estimate it:
- Measure the site's sustained overnight upstream throughput — not the sold speed. Apply the 80% after-hours throttle ceiling from the Standards page.
- Measure the actual nightly change delta, separately for the image and for Class A and B data.
- Divide by the usable overnight window.
| Condition | Model |
|---|---|
| Image delta fits the overnight window with headroom | Standard hybrid image plus Class A extraction. Class A is additive, not a replacement. |
| Image delta does not fit, but Class A and B data does | Split the legs. Image local only; Class A and B to cloud. Client must be told they no longer hold an offsite bare-metal image. |
| Neither fits | Bandwidth problem, not a backup problem. Escalate as a circuit upgrade — but ship Class A regardless, because it fits on anything. |
| Bare-metal RTO is contractually or clinically required | Image stays, locally, regardless of the above. |
Corollaries worth stating outright:
- A failing cloud leg is a capacity signal, not a job to keep retrying. When the cloud leg fails repeatedly at a site whose local leg is green, the answer is the table above, not another retry schedule. Track these in the Open Backup Failure Tracker.
- Never silently drop the offsite copy to make the numbers work. Reducing what goes offsite changes the client's recovery position. It gets written down and agreed to.
- Never trade immutability for bandwidth. If it does not fit immutably, it does not fit.
- The image protects the machine. Class A and B protect the data. They change at different rates, are needed at different times, and should not share a schedule or a transport. Machine state is near-static between patch cycles; data state changes every hour the practice is open.
What This Changes Operationally
- Deployment grows an application-discovery step: what is the LOB application, what engine and version backs it, where does it live, what is the supported extraction method, where do the blobs live. Today the image assumes all of this away.
- Monitoring becomes per-layer. "Last successful backup" as a single value stops being meaningful when a site runs continuous log shipping, a nightly logical export, a file sync, and a weekly image. Each layer needs its own clock and its own threshold — and the Class A clock is the one that should page someone.
- Verification gets cheaper, not harder. Hashing and restore-testing a dump is achievable per-client per-month. Restore-testing an image is not, which is why it does not happen today.
- RTO/RPO get derived per class. A site may hold a 15-minute data RPO and a 7-day machine RPO simultaneously. That is coherent and defensible, and the current tier table cannot express it.
- Per-application engineering is real, recurring work. Each application needs a documented, tested extraction and restore procedure. The image model's great virtue was skipping this. Adopting SDP means paying it, once per application, forever.
- Storage design moves upstream of backup design. Choosing versioned, immutable object storage for Class B at deployment time is what makes the protection nearly free later. Choosing a plain SMB share and planning to back it up is the expensive path, and it is the current default.
OPEN — Not Yet Decided
- The name. Structural Data Protection is the proposal above; alternates listed. Nate's call.
- Does the tier model change, or do tiers gain class profiles? Protecting by data class does not map onto Tier 1/2/3 as written. Rewriting three tiers into a matrix touches agreements and sales collateral more than it touches engineering.
- What performs Class A extraction in the interim, before the suite exists — RMM-orchestrated scripts, native application scheduling, or a thin agent. This gates the near-term win and should be answered first.
- Which object storage backs the suite. Own it, or build on a provider with Object Lock. Affects cost model, compliance posture, and how honestly we can call it a separate fault domain from the client's other copies.
- Image cadence once it is no longer primary — weekly, monthly, or triggered on change detection.
- Retention for logical extracts. The 14-daily-plus-weekly-to-30-days default was sized around images. Logical exports are small enough that a year is affordable, which may be worth more to clients than a tier upgrade they are not buying.
- Whether split-leg becomes standard or stays a per-site exception driven by the bandwidth table above.
- Whether MoCo is the pilot. MoCo is already bandwidth-bound with a healthy local leg and a failing cloud leg — the exact shape this model is built for, and a site where Class A extraction would restore an offsite copy of the data that matters without touching the circuit. Pilots at a live client need scope and a rollback.