How to Create an Irrefutable Reason for Backup?

How can you determine how IT can present an appropriate business continuity design with the input of business-focused personnel? For this purpose, you need to explore the capabilities of technologies you can use. After learning this, you can estimate the costs. You can then return this analysis to business groups to create a final plan that balances what your organization wants for disaster recovery and its willingness to pay for it.

15 Dec 2023 SintelSedat Akfidan 16 min read
How to Create an Irrefutable Reason for Backup?

How to Create an Irrefutable Business Case for Backup

By understanding the capabilities of technologies that can provide a suitable business continuity design, you can determine how your workforce-oriented personnel can input. To achieve this goal, you need to explore the capabilities of the technologies you can use.

After learning this, you can estimate costs. You can return this analysis to business groups to create a final plan that balances what your organization wants for disaster recovery and the willingness to pay for it.

Mapping out your backup requirements helps you plan software subscription plans to meet your needs. Hornetsecurity is aware of the need for multiple backup solutions and therefore provides data backup and recovery services for all critical Microsoft 365 services (Exchange mailboxes, SharePoint, OneDrive, Teams, etc.) and also provides virtual machine backup for your data backup and recovery service.

Exploring the Technological Capabilities of Data Protection Systems

At this point, you have a high-level list of items. There are few backup solutions that target LOB applications. Therefore, you need to divide this list into items that the backup and replication programs can understand.

To attract the widest customer base, manufacturers specify most of the services and products that most organizations use. Common protections include:

  • Windows Server and Windows desktops;
  • UNIX/Linux systems;
  • Database servers;
  • Mail servers;
  • Virtual machines;
  • Cloud-based resources;
  • Physical hardware configurations.

You need to create a map from prioritized business-level items down to basic technologies. To make sure you do not miss anything, bring in technical experts. Collect input on what is needed to recover various systems used in your organization.

Most of them require more effort than a simple backup and restore procedure. Some examples include:

  • Active Directory;
  • Daily-based SQL recovery;
  • Mail servers;
  • Multi-tier systems;
  • Cluster nodes.

In addition to application specialists, get information from server and infrastructure experts. Investigate the experiences of those who have faced a recovery situation with your most trusted systems. You may find surprising exceptions or special procedures.

First Line of Defense: Fault-Tolerant Systems

Ideally, you should never have to implement a recovery plan. Although you cannot completely eliminate this possibility, you can reduce it with fault-tolerant systems. "Fault tolerance" refers to the ability to continue operating with a faulty component.

Most fault-tolerant systems operate at a low level in the internal components of computers. They usually use hardware-level data replication methods to protect.

In the event of a failure, they use the mirrored copy to continue providing the expected functionality. Examples include multiple power supplies, disks, Network Interface Cards (NICs), etc.

However, until someone replaces the faulty part, the system does not provide redundancy. More errors can cause downtime and potentially data loss.

QR Codes – The Criminal's New Best Friend

Storage technologies form most of fault-tolerant systems. It's no coincidence, they also have the highest failure rates. You can protect both short-term storage (main system memory) and long-term storage (spinning and solid-state disks).

Memory Error Tolerance

To provide full error tolerance, memory controllers allow you to match memory modules. Every write to a module creates an identical copy on another. If one fails, the other continues to operate by itself.

If the computer supports memory replacement while running and there's a way for technicians to access the inside without removing anything, the new one can be installed without stopping the system.

Of course, system memory remains one of the more expensive components, and each system has a limited number of slots. Therefore, you should reduce your general workload by half to use fault-tolerant memory.

Doubling the number of hosts will cost more than most organizations are willing to pay. Fortunately, the total failure rate of memory modules is low. The probability of encountering temporary issues that can be addressed with cheaper solutions is much higher.

Server-class computer systems usually support ECC memory modules. ECC modules include technologies that allow the detection and correction of memory errors.

Some vendors provide special technologies to defend against issues.

In most cases, you choose ECC memory instead of fully fault-tolerant schemes. ECC cannot defend against module failures, but such failures are rare enough to be of little concern. ECC is more expensive than non-ECC memory, but it's still significantly cheaper than doubling your main computer purchase.

Hard Drive Error Tolerance

Hard drives, especially traditional spinning types, have a high failure rate. They require the most protection because nearly all live data is stored there. Due to the prevalence of the problem, the industry has produced many fault-tolerant solutions for hard drives.

RAID (Redundant Array of Independent Disks) systems form most of the fault-tolerant designs for hard drives. These industry-standard designs use a combination of the following technologies to protect data:

Mirroring

Every bit written to one disk is written to the same location on at least one other disk. If a disk fails, the array uses the mirror(s).

Mirroring

Striping

Every bit written to one disk is written to the same location on at least one other disk. If a disk fails, the array uses the mirror(s).

Striping

Parity

Parity also uses a striping pattern, with a major difference. One or more blocks in each stripe hold parity data instead of live data. The operating system or array controller calculates parity data from the live data as it writes the stripe.

If any disk in the array fails, it can use the parity data in place of the live data. A parity array can continue to function with the loss of one disk per parity block per stripe.

Parity

If you want to use RAID, you can choose from a series of "levels". Each RAID level provides its own balance of redundancy, speed, and capacity. Except for RAID-0 (pure striping for performance with no redundancy), all RAID levels require you to sacrifice some of the usable space for protection.

Disks offer a relatively low cost compared to system memory and provide many expansion options beyond the basic capacity of a system case. Therefore, RAID provides a higher cost per stored bit than single-disk systems but is usually not prohibitive.

When it comes to RAID, you have several options. Many levels have fallen out due to insufficient protection and some are too space-inefficient for cost-effectiveness. You usually encounter the following types:

  • RAID-1 – A simple mirror of two disks. Slightly slower than normal write speeds, faster than normal read speeds, and provides sufficient protection with a 50% capacity loss.

RAID-1

  • RAID-5 – A striped array with a single parity block. Requires at least three disks. Each stripe alternates which disk holds the parity data, so only 1/n of the stripes need parity calculations during a failure scenario. Can withstand the loss of one disk, provides sufficient protection with normal write speeds and faster than normal read speeds, but with a capacity loss of 1/n. Not recommended for arrays using large disks due to the higher likelihood of disk failure during rebuilds and the increased probability of errors during scheduled reads.

RAID-5

  • RAID-6 – Similar to RAID-5, but with two parity blocks per stripe. Requires at least four disks. More secure than RAID-5, but similar concerns about large disks apply. Slower than RAID-5 and has a capacity loss of 2/n.
  • RAID-10 – Disks are first mirrored, then a striped write is performed without parity on one side of the mirror set and this stripe is copied to the corresponding mirror disk. Can operate with a single disk loss per mirror but cannot lose two disks in the same mirror. Provides better performance and higher security than parity schemes, but with a capacity loss of up to 50% of the total drive capacity.

RAID-10

Due to the prevalence of drive failures and the low performance of standardized redundancy schemes, many vendors offer special solutions that aim to address the shortcomings of RAID.

While RAID works on bit and block levels, most vendor-specific systems add techniques at a meta-data level to enhance protection or performance.

You have many options when it comes to fault-tolerant disk storage, so keep a few touchpoints in mind:

  • Storage vendors naturally want you to purchase their most expensive equipment. Use planning tools to estimate your capacity and performance requirements before starting the purchasing process. Businesses often overestimate area and performance requirements.
  • You can usually expand your storage after the first implementation. Unlike system memory, you don't need to limit yourself to the capacity of a single case.
  • Solid-state drives have significantly lower failure rates than spinning disks. You can take advantage of hybrid systems that include both to achieve an acceptable balance of performance, redundancy, and cost.
  • The most important point: downtime costs money. Storage redundancy directly reduces the likelihood of an unplanned outage.

Advanced Storage Error Tolerance

The emergence of affordable, truly high-speed networks (ten gigabits and above) has brought exciting new options for storage protection. Today's network speeds exceed even high-end storage equipment.

What was once the exclusive domain of expensive, high-end storage area networks (SAN) devices can now be achieved with commodity-level and even data center-level storage redundancy.

These technologies rely on real-time or synchronous replication. In the simplest design, two storage units mirror each other.

Systems that rely on these can connect to a virtual endpoint that allows failover or operate in an active/passive configuration simultaneously. In more complex designs, control systems distribute data across multiple storage units and dynamically provide access.

Real-time replication is discussed in more detail in the article titled "How to Use Replication to Easily Achieve Business Continuity."

The most advanced examples of these technologies emerge in relatively new hyper-converged solutions. These use software to merge the compute layer with storage layers in standard server-class computing hardware.

In most cases, they include a hypervisor to control the software layer and special software to manage storage.

Although the costs of distributed storage and hyper-converged systems have dropped significantly, they still remain in the upper end of the cost spectrum.

Unlike traditional discrete systems, you will need significant infrastructure and technical expertise to support them properly. You can consider these replicated data as "hot copies." They are instantly updated and can be failed over quickly.

Some synchronous replication systems even allow transparent failover or active/active usage.

Application and Operating System Error Tolerance

At the top level, you have the ability to mirror an operating system instance to another physical system. For this to work, you need to run the cloud server under a hypervisor that can mirror operations.

This is a complex configuration with many limitations. Few hypervisors offer this feature, it doesn't work universally, it can't handle all problems, and performance improvements may not be worth it for the applications you want to protect the most.

At a more accessible level, some applications allow for a certain degree of error tolerance through layering. For example, you can usually run a web front-end for a database. Load balancers can instantly transfer client connections from one web server to another in case of failure.

Some database servers allow multiple synchronous instances that can direct connections to a working node. These technologies offer more functionality and feasibility than operating system error tolerance.

In most cases, when an application offers its own built-in redundancy options (such as Exchange Server Database Availability Groups or SQL Server Always On availability groups), they are always preferred over general operating system or Hyper-V high availability options, see below.

Error Tolerance Warnings

When exploring error tolerance options, you will quickly realize that they come with a significant cost. Almost all technologies require you to purchase at least two of everything. Most of them will also require additional infrastructure.

All of them depend on expertise for installation, configuration, and maintenance. These costs must always be considered in relation to the cost of downtime.

The primary goal of error tolerance is to rely on copies to continue working during a failure. However, this has a negative side effect: your fault-tolerant solution might copy something you didn't want.

For example, if ransomware attacks your storage system, having a RAID or geographically redundant SAN won't help you at all. Even without the presence of malicious actors, redundant systems can happily copy data corruption or accidentally delete all examples of a critical email.

While error tolerance can benefit your organization, it cannot stand alone. You will always need a backup solution for asynchronous data replication. However, there are options between error tolerance and backup. These technologies fall under the high availability category.

Second Line of Defense: High Availability

You can't use error tolerance for everything. Some systems simply have no way to apply it. Some are excessively expensive. Instead, you can distribute high availability solutions. High availability has a more ambiguous definition than error tolerance. It applies less to actual technologies and more to results.

Whereas error tolerance means operating without interruption during a failure, high availability measures the actual uptime against expected uptime.

For example, if your organization wants a system that is always operational, it might set a target of 99.99% annual availability. To achieve this, you must ensure that the system experiences no more than a few minutes of total downtime in a year.

365 days multiplied by 99.99% equals 364,9635 days of uptime, allowing for just a little over 48 minutes. That's an aggressive target.

When setting high availability targets, be sure to distinguish whether your measurements include planned downtime. Including these can significantly reduce your tolerance for failures.

If a system that is expected to achieve 99.99% uptime requires five minutes per month for backup systems during patch cycles and you include that in your measurement, it would violate the expected uptime by 12 minutes per year even without unexpected outages.

In addition to adjusting for planned maintenance, you can also adjust the availability scope. For example, you can maintain a 99.99% target but specify that it only applies to weekdays from 06:00 to 18:00. You can exclude company holidays.

Be careful with two critical steps:

  • Clearly state any exceptions that are not open. If you set an expectation of 99.99% in large font and carefully list the conditions below, you end up with someone feeling deceived and betrayed. Avoid this at all costs from the start.
  • Define a clear standard for "uptime." While it's better to prioritize user experience in your results, you should also have something objective that can be measured. For example, "a customer can successfully place an order on the customer website" works well as an abstract target, but how do you measure it? If a system failure prevents a customer from placing an order but no customer attempts to do so, is that considered downtime? How do you determine whether a system is faulty if a customer's order fails?

Technologically, any tool that helps improve uptime falls under the high availability umbrella. All error-tolerant technologies are suitable. However, some of them allow for a bit more downtime in exchange for lower cost, broader applicability, and simpler operation. Clustering is usually the most common among these.

High Availability with Clustering

Clustering typically involves using multiple computers or resource nodes in an active/passive configuration to host a single instance of a resource. Some examples related to Microsoft's failover clustering technology include:

Microsoft SQL

A clustered Microsoft SQL database runs on one of many nodes. During a planned failover, the active node is stopped while a passive node is started, making the database unavailable for several seconds. If the active node fails, the passive node starts, and the database is unavailable for a few seconds. Active processes may fail during an unplanned failover.

Hyper-V

A clustered virtual machine can quickly move to another node either online (Dynamic Migration) or offline (Quick Migration) during a planned failover. If the active node fails, the virtual machine crashes but can be quickly restarted on another node.

File Server

A standard clustered Microsoft file server is hosted via an active node that allows both planned and unplanned failovers to happen quickly. Microsoft also offers a more fault-tolerant scale-out file server.

Storage Spaces Direct

Typically referred to as "S2D," Storage Spaces Direct is Microsoft's distributed file system offering. It runs on Windows Server for basic storage needs. Azure Stack HCI also implements it to provide a complete hyper-converged infrastructure solution.

You'll find clustering technologies in other operating systems, hypervisors, and physical devices. Remember that they differ from error tolerance by allowing a certain amount of downtime. However, they significantly reduce the risk of downtime compared to standalone systems.

High Availability with Clustering

Clustering Warnings

Clustering provides a copy of the process layer. It ensures that a clustered workload has somewhere to run. It doesn't create any copies of the data. Without additional technology, a critical storage failure could bring down the entire cluster.

Due to hardware replication requirements, clustering is at least twice as expensive as running without a cluster. You may need to purchase additional software features to enable a clustered configuration. Clustering requires personnel who know how to configure, manage, and maintain it.

You should also ensure that your chosen backup solution can properly protect your clustered resources. Solutions like Hornetsecurity's VM Backup protect virtual machine clusters. Sometimes you can successfully use a backup solution that doesn't work with your high availability solution, but this requires significantly more management effort.

High Availability with Asynchronous Replication

You can use technologies that regularly copy data from one storage unit to another. Asynchronous replication uses a snapshot technique to maintain full file system consistency. Some replication applications use a simple file copy mechanism that works well for basic file sharing but not for applications.

Some applications have built-in asynchronous replication. Microsoft's Active Directory automatically sends updates between domain controllers. Most SQL servers have a variety of replication options. Microsoft Hyper-V can create, manage, and monitor virtual machine replications.

Data created by asynchronous replication can be considered as "hot copies." They require some process to bring them online after a failure, but they can be brought back online quickly.

Asynchronous Replication Warnings

Unlike clustering, asynchronous replication requires some human interaction to switch to a copy after a failure. Clustering technologies use a type of control technique to prevent split-brain scenarios where two copies are active and operating at the same time. Most replication systems don't have a built-in way to do this. Therefore, when you choose replication, be sure to plan accordingly.

Replication shares the same disadvantages as clustering: duplicated hardware, special software, and expertise required. Additionally, it doesn't protect against data corruption, including [ransomware](https://