Zmanda Team Blog

A well-known challenge of new technologies such as cloud backup is that there are no set standards. Take pricing, what would you expect to pay for storing 10 GB of your data on cloud today? Given that the answer can be anything from zero to a few hundred dollars, how do you know that you are not paying more than you really should for your requirements? The question worth asking essentially is - since businesses are different and have different backup needs, why shouldn’t they be allowed to control how much they want to pay for cloud backup?

We broached this question in our recent Zmanda Cloud Backup (ZCB) webinar titled “How to get the maximum out of ZCB” (recording available here) and looked at ways to optimize ZCB costs for one’s requirements. While exploring different options, we realized something interesting – ZCB’s flexibility not only makes it very versatile, but when combined with its pay-as-you-go pricing model, it also allows great leeway in optimizing backup costs. In this post, I will try to explore the options available in ZCB to do just that.

Before we begin, allow me to clarify – while the bulk of this post focuses on cost optimization options with ZCB, the intent is to provide a systematic way of thinking about cloud backup costs. If you are a ZCB user, you can use these options directly. And if you are not a ZCB user, you can map some of these options to your backup solution (and for the benefit of all of us, please do remember to post your results in a comment below!) and see how better (or worse) it fares.

First, a look at the ZCB pricing model

ZCB’s pricing model has two components:

• Fixed monthly license fee: $4.95 per month
• Usage based fee:
  • Storage: $0.15 per GB-month
  • Upload to cloud: $0.15 per GB
  • Data download from cloud: Free

Admittedly, this does look more complicated than a fixed monthly cost, but its complexity really emerges from its flexibility, which does leave a lot of room for optimizing costs. Let’s see how.

Step 1: “Divide and conquer” the monthly license fee!

Got multiple machines to back up? Congratulations! Unlike most other backup services, which charge a fee per machine, ZCB allows a single ZCB license to be used to protect an unlimited number of systems. So if you have, say, 5 or 10 machines to be backed up, the fixed monthly cost per backup system becomes non-significant. (However, just be aware that machines that share a ZCB license can potentially access the backup data of each other – although use of encryption can alleviate the potential privacy issue).

Step 2: Optimize the usage based fee!

The usage based fee with ZCB simply means you pay for data storage and data uploads. Thus, optimizing this fee can involve two steps:

Step 2.1: Optimize your total backup size

Let’s first try to see how much data you really need to backup and how to shrink the size of backup media to store the backed up data. ZCB offers following options here:

• Carefully choose what data needs to be backed up: While backing up applications such as Exchange with ZCB, you can select specific datastores instead of all datastores. For file system backups, you entirely control what gets backed up (ZCB does NO automatic selection of *.mp3, *.jpg files etc.) and you can also specify an “exclude list” to skip backing up large user files by mentioning patterns such as *.mp3 or *.mov. This point may look obvious, but doing this is not easily allowed by many cloud backup applications which attempt to maximize your backup data size, for obvious reasons ;).

  

  Figure 1: Exclude list

• Use backup levels: Incremental and differential backups contain only the data which changed since a last backup and hence reduces backup size. Use incremental and differential backups judiciously to reduce data size while still adhering to your backup strategy.

  

  Figure 2: Differential backups – backup changed data since last full backup

  

  Figure 3: Incremental backups – backup changed data since any last backup

• Choose backup frequency: How often you backup directly impacts your total backup data size. So you need to choose the right backup frequency which fits your backup requirements but also keeps your total data size manageable. With ZCB you can choose to do only manual backups (when you want) or choose from powerful scheduling options to perform backups every 15 mins to a certain date in every year.

  

  Figure 4: Choose backup frequency

• Enable compression: Depending on your data type (document, text files are more compression friendly), enabling compression may help you shrink your storage requirement by about 10-50%.Here is a figure which summarizes all of these ZCB options:

  

  Figure 5: Summary of all options to optimize total backup size

Step 2.2: Optimize how much cloud storage is used to store backup data

Now that you have optimized the total backup data size, let’s see how you can reduce the storage required on the cloud for keeping this backup data. Here are your options with ZCB:

• Blend cloud storage with local storage: ZCB allows you to store all or some of your backup data to your local or network storage. e.g. you may choose to store only certain full backups on cloud storage while using your local/network disk storage for your primary/frequent backups. Below is an example:

  

  Figure 6: A sample backup strategy to minimize cloud storage (only monthly backups go to cloud, rest all backups go to local/network storage)

• Judiciously choose the cloud data retention policy: ZCB allows complete control over the retention period for your backup data. So you can choose to adopt as aggressive retention policy as your backup policy allows, such as “retain full backups for 2 weeks and retain incremental backups for 2 days”.
• Monitor, Monitor and Monitor: Monitor your cloud usage regularly and purge old backup runs which you don’t require. For monitoring, you can use Amazon bills, ZCB Global Dashboard and jets3t tool. And for purging old backup data which is not required, you can click on File > Purge Backup Runs Before and select a historic date all the backup runs before which will be deleted by ZCB. Do note that deleting any data which is required by subsequent backup runs (such as deleting full backups while retaining incremental/differential backups) may make your dependent backups useless for any restoration requirement in future.

  

  Figure 7: Purging old backup data which is no longer required

• Exploit the ZCB free tier: ZCB offers 5 GB free cloud storage and uploads for each of the 5 Amazon S3 regions, making it possible to use up to 25 GB free cloud storage across all 5 regions completely free! You can scatter your data across all the 5 AWS regions to fully exploit this free tier. (With two more AWS regions supported in ZCB 4.1, this free tier will soon become 35 GB free tier and hence this option becomes even more effective!).

  

  Figure 8: The ZCB free tier

Quite a handful ways to optimize costs, isn’t it? And perhaps the best part is - since ZCB as well as the pricing model is super-flexible, the above is not even an exhaustive list!

Are you a ZCB user? If yes, do consider these steps and let us know if/how they worked for you. And if you are not a ZCB user, I’m very curious to know how you are optimizing your costs with your current solution?

Have a “cost effective” new year!

-Nik

Okay first let me say this: I love Dropbox and like many of you depend on it each day to seamlessly access my important files from office/home/shared computers and from my cell phone. Also ever since Dropbox released the developer APIs, an increasing number of innovative applications (see here and here for a few examples) are coming to the fore that extend Dropbox beyond its “native” features of syncing, sharing and collaboration.

This is great but creates a potential problem. With all this excitement it is easy to get carried away and think of using Dropbox to solve a problem which it was never designed to solve – a robust cloud backup. Even at a conceptual level, classic data backup technology based tools such as Zmanda Cloud Backup (ZCB) and sync-sharing tools such as Dropbox solve very different needs of businesses. To most of backup administrators it would seem outlandish to even suggest that one can be used in place of the other (a silicon valley based system administrator, I tossed this idea to, frowned upon it and found the comparison so illogical that he spent a few seconds thinking about where to begin his explanation from!).

But yet over the last few months, since the same time since Dropbox started gaining mass acceptance, we’ve been seeing this confusion pop up in the heads of some of our prospective users. Thanks to the (well-deserved) widespread attention which Dropbox has gathered in recent times, such users would begin comparing ZCB with Dropbox for solving their data backup problems. And so far, to clear up the matter, we largely just tried to remind them about the fundamentals of disaster recovery and how Dropbox is an excellent tool to share and synchronize data but a very primitive tool to perform data backups.  I can’t tell how far we’ve succeeded in conveying this, but I know some of them indeed saw our point (they became our customers!).

But this post became unavoidable, since the plot seems to have thickened with the recent introduction of Dropbox for teams. With this latest offering, Dropbox now consciously targets businesses by offering them huge shared storage (1 TB) along with some administrative tools to manage the service. Not a bad idea really. The problem however is that to sell SMBs this much storage, Dropbox now seems to be telling them to use this storage for data backups, something it never claimed to do well so far.

So let’s scratch the surface a bit here to see what Dropbox is and what it can or can’t backup.

At the outset, let’s try to see what problems Dropbox has been designed to solve and how data backup was not one of those problems. This is how the Wikipedia defines Dropbox:

Dropbox is a Web-based file hosting service operated by Dropbox, Inc. that uses cloud computing to enable users to store and share files and folders with others across the Internet using file synchronization.

This is what it really is. You give Dropbox some files which you want to share and it laps them up, stores them on its cloud storage and shares them among multiple Dropbox clients:

Source: http://www.dropbox.com/static/images/install_graphic.gif

And when any of your files change from any shared machine, the changes are instantaneously replicated across all the shared devices. So what’s the secret sauce? Well the steadfast decision process to keep things simple for syncing and sharing the user files. See such an instance of decision making on this page.

On the other hand, a true backup solution, such as ZCB, exists to ensure that all your data gets backed up regularly and you can go back to any of the backed-up states of your machine when the sky comes crumbling down. This may sound similar, so let’s see why this goal is not achievable with Dropbox:

1. Completeness: At a higher level, the data in your computer can be classified in following categories:
   1. User files: These are independent files like documents, presentations, spreadsheets which are created by users for their official or personal work.
   2. File system/Interlinked files: These can be your entire directory structure such as D:\, a particular special directory such as “My Documents” or a set of some files which are inter-linked – for e.g. a bunch of website files or a spreadsheet with embedded images or macros.
   3. Application data: The data created and used by your business applications such as SQL Server or Outlook. These can be databases, configuration files, temporary files, etc. and are generally created in the installation directory of these applications. Also these files are “open” when the application is running.
   4. Applications: Binaries and configuration files of applications which are installed such as Microsoft office and Adobe PDF suite.
   5. Operating system and system configuration: The installed operating system, its configuration (“System State” in Windows) and other system information such as partition table, etc.

   Now looking at the above, it is obvious that Dropbox can only be considered for data in the first and second categories. And even in second category, some special folders (e.g. C:\Program Files) can’t even be put inside your Dropbox folder. And those which can be, you are likely to have problems during restores. With many interlinked files, how are you going to find a logically consistent set of interlinked files as it existed at a particular historic point in time?

   A true backup solution such as ZCB, on the other hand, backs up almost all the above categories of data (ZCB backups Windows system state though not the operating system and boot loader/partitioning information), and the backup archives represent logical and consistent states at particular points in time.

2. Modification/Deletion of original copy of data: A true backup solution never modifies the original copy of data, let alone deleting it. In fact even changing a file’s meta-data (archival bit, modification time etc) has been considered unacceptable by many backup administrators, since that may interfere with some other installed applications.

   But since the primary goal of Dropbox is to “synchronize” data across multiple machines, it will do all which is necessary to accomplish this goal. So if a file gets accidently deleted or corrupted on one system, Dropbox will gleefully and promptly propagate that accident to all the shared machines.This is obviously a serious problem and hence in its paid versions, Dropbox offers an “unlimited undo history” feature to allow you to undelete files. Though this surely helps, but from Disaster Recovery standpoint it still is a risky situation, since this would mean that you have lost all your local copies and now have only one remaining copy of your original data. What’s worse - it is only available on the cloud, so if you need it when you have no or poor internet connectivity, you are out of luck.

   On the other hand, a true cloud backup solution such as ZCB supports smart redundancy options where you can keep backup data on local as well as cloud storage. Since you will have 3 copies of your data (original + 2 separate copies), even if you accidently delete your original copy of file you still have two redundant copies to restore from.

3. Security: The tricky thing about security is that it’s like insurance – you may not care for it in steady state but it can be catastrophic when something goes wrong. And security has been the number one reason why Dropbox is still unwelcome in many enterprises today. Some issues:
   1. True data privacy: Dropbox encrypts your data on the Amazon S3 cloud using an encryption key which is unique to your Dropbox account. Also note that this encryption key is known to Dropbox. This means two things. First, your data is not truly private as Dropbox personnel can potentially see your data (Of course, we believe that this is unlikely). Second, it means you can’t have any data privacy between two of your users sharing the same Dropbox account.

      The only way out here would be to use a separate file/volume level encryption tool on top of Dropbox (such as TrueCrypt). But in addition to burdening your users with new workflows related to encryption/decryption, this would most probably also make the Dropbox synchronizations inefficient, thus defeating the whole purpose of using Dropbox in first place. I recommend checking out the experiences of the commenters on this blog for the gory details of such problems if you are indeed thinking of going down this path.

      In comparison, a true backup solution like ZCB offers asymmetric encryption with the user generated certificates, making it virtually impossible for anyone else to see your encrypted data.

   2. The disadvantage of being a public “data sharing” service: Dropbox was designed to support data exchanges among multiple devices and multiple users over the internet. You can imagine that such a service needs to have somewhat relaxed rules when it comes to authentication, access rules, open ports, etc. Dropbox has already had its share of such issues – see this page and this page for examples.

   Again, in contrast, a true backup application such as ZCB has much more tighter security mechanisms. It can securely encrypt your data with user-generated keys as soon it is backed up, can send the data over a SSL tunnel to the cloud which is protected by multiple layers of authentication for gaining access. This ensures that your backup data is safe and secure; irrespective of its location - on local disk or cloud.

4. Flexibility in choosing data retention policy: Choosing retention policy is a very important decision variable for your Disaster Recovery plan as it decides the oldest historic time you can restore to and has direct implication on your storage costs.But since Dropbox has the “unlimited undo history” feature, why should one even worry about this? My doubts about the long term sustainability of a truly “unlimited” deleted file history notwithstanding, there are at least two reasons why data retention policy still is an issue with Dropbox:
   1. There is no automatic management of your storage quota – so you need to manually delete the older files manually to free up space for newer data. With multiple users working on your shared data, won’t it be challenging to identify what data is too old and delete it manually? Until of course you buy a storage quota which is multiple times of your actual storage requirement, so you never have to delete anything!
   2. In addition, many organizations need to abide with the data storage laws which stipulate which geographical location to store data and even the maximum time customer data can be retained by a business. You don’t have any such control with Dropbox.
5. Scheduling uploads for making them efficient and unobtrusive: One key issue for many businesses while considering cloud backup is the lack of adequate internet bandwidth. During normal business hours there is only so much bandwidth which you can devote for data backups. This is why many administrators like to schedule the backup uploads to run during the idle times such as weekends.

   Telling Dropbox when to sync is not possible, and even if it is made possible, it surely defeats the whole purpose of using such a sync tool. Yet another problem (feature!) with Dropbox is that it immediately syncs every change of your data. So if you make frequent changes to your files during the day, each of them will be synced across all your devices thus wasting your bandwidth, even though you may have just wanted to make a copy of your file at the end of the day. Again, for syncing and sharing this “churn” is the necessity and one of the core benefits of Dropbox but for backups, it is nothing but “noise” which is wasteful and disruptive for your normal business network traffic.

As you can see, the above list is by no means an exhaustive one. As you go deeper into this, more such differences pop up. But the question is – is that surprising? Given that Dropbox was conceived, designed and implemented to solve the need of syncing and sharing and not robust cloud backup, isn’t trying to do the latter is more of a “hack” than a true solution?

And did I mention that we have a webinar coming up on Dec 7th, 2011 in which we will be discussing how to get the maximum out of your ZCB installation and will also be taking some of the above issues for discussion? Please register for this webinar here. Hope to see you then!

-Nikunj

Today, we announced the immediate availability of Zmanda Cloud Backup (ZCB) 4, our comprehensive cloud backup offering for Windows servers, desktops and laptops. ZCB 4 had been under a limited beta program for past few weeks and was extensively tested by many end users and resellers. We received some great feedback which led to many improvements and bug fixes. Thank you all who participated!

ZCB 4 is a huge step forward for the idea of cloud backup. And after working with many ZCB users for a while, we say this with a lot of conviction. We took a hard look at what users wanted to achieve with cloud backup and compared that with the available solutions on the market. We identified various gaps, which are addressed in ZCB4:

1. Flexibility of choosing where to store backup data: Users of cloud backup have different needs. Some are embracing it as an extra line of defense and hence want to backup to both on-premises disks and on the cloud. On the other hand, some users are looking at cloud storage as their primary and only backup location and hence are looking for a solution which backs up data only to the cloud.ZCB had this covered from day 1. The data was always first backed up to disk and then to cloud. The data on disk could then be deleted or retained, depending on which of the above two use cases you wanted to deploy. But we realized that we could improve it further by offering a “backup to cloud” operation which would back up your data to cloud directly without using any temporary local storage. So, if you are short on disk space or don’t want backup on disk, then this operation would be handy.

   
   

2. Improving transfer speeds: Users who have either a lot of data to backup or less Internet bandwidth to use, are hit by a basic problem – how to transfer data to and from cloud within required time limits? We also discovered cases where users had the bandwidth available and met their part of the bargain, but the backup solution was either not capable to use the bandwidth completely or the backup provider imposed limits to the upload/download speeds for the users.ZCB never imposed any transfer limits, whatsoever, on upload/download speeds and always tried hard to maximize the throughput. In ZCB 4, we made it even better by adding support for multithreaded uploads/downloads. This feature makes ZCB use multiple concurrent connections to Amazon S3 cloud and hence unlocks the bandwidth you always had available. And true to our promise of providing flexibility to users, we have made this feature entirely configurable:

   

   So by default, we use 3 concurrent connections for data transfer. If you wish you can tweak this value to experiment and find out what works best in your work environment. Higher thread count may be beneficial if you have spare bandwidth and CPU/memory resources to push or pull data.

3. Manageability and usability: We believe usability is core to the idea of cloud backup, as it involves critical decisions about when/where/how you backup your systems. Users need to be given full freedom to make these decisions and yet it shouldn’t be hard to configure and monitor the solution. Though our users and experts always rated us high in this area, we realized that the user interface  needed to be ready to handle our planned rapid growth, both in terms of product features as well as customer use cases. So in ZCB 4, we redesigned our user interface, made many workflow improvements and made it much more intuitive and easier to use. To see it in action, you can view the new ZCB screenshots.

In addition to the above, ZCB 4 also offers:

• Backup/Restore of selective databases in SQL Server and Exchange server
• Differential backup of SharePoint server
• Parallel operations across multiple backup sets
• Extensive Reporting across multiple backup sets
• Hundreds of other improvements

ZCB 4 is also available in German and Japanese languages. For more details on ZCB 4, please refer to the release notes page.

ZCB 4 brings to the market a comprehensive, flexible and practical cloud backup solution. Also, as we gain scale, we are also working on the pricing (in case you didn’t notice, we recently announced the 25 GB free tier, perhaps a first in the industry) to make it affordable to a bigger set of users.

We are already working very aggressively on our next releases and will soon be making some exciting announcements. If you have a suggestion for us, please do drop me a line at nikunj@zmanda.com.