Archive for the ‘Cloud Backup’ Category

Introducing ZCB 4 – Next Generation Cloud Backup!

Monday, August 29th, 2011

Today, we announced the immediate availability of Zmanda Cloud Backup (ZCB) 4, our comprehensive cloud backup offering for Windows servers, desktops and laptops. ZCB 4 had been under a limited beta program for past few weeks and was extensively tested by many end users and resellers. We received some great feedback which led to many improvements and bug fixes. Thank you all who participated!

ZCB 4 is a huge step forward for the idea of cloud backup. And after working with many ZCB users for a while, we say this with a lot of conviction. We took a hard look at what users wanted to achieve with cloud backup and compared that with the available solutions on the market. We identified various gaps, which are addressed in ZCB4:

  1. Flexibility of choosing where to store backup data: Users of cloud backup have different needs. Some are embracing it as an extra line of defense and hence want to backup to both on-premises disks and on the cloud. On the other hand, some users are looking at cloud storage as their primary and only backup location and hence are looking for a solution which backs up data only to the cloud.ZCB had this covered from day 1. The data was always first backed up to disk and then to cloud. The data on disk could then be deleted or retained, depending on which of the above two use cases you wanted to deploy. But we realized that we could improve it further by offering a “backup to cloud” operation which would back up your data to cloud directly without using any temporary local storage. So, if you are short on disk space or don’t want backup on disk, then this operation would be handy.


    Cloud Backup new operation

  2. Improving transfer speeds: Users who have either a lot of data to backup or less Internet bandwidth to use, are hit by a basic problem – how to transfer data to and from cloud within required time limits? We also discovered cases where users had the bandwidth available and met their part of the bargain, but the backup solution was either not capable to use the bandwidth completely or the backup provider imposed limits to the upload/download speeds for the users.ZCB never imposed any transfer limits, whatsoever, on upload/download speeds and always tried hard to maximize the throughput. In ZCB 4, we made it even better by adding support for multithreaded uploads/downloads. This feature makes ZCB use multiple concurrent connections to Amazon S3 cloud and hence unlocks the bandwidth you always had available. And true to our promise of providing flexibility to users, we have made this feature entirely configurable:

    Cloud Backup multithreading

    So by default, we use 3 concurrent connections for data transfer. If you wish you can tweak this value to experiment and find out what works best in your work environment. Higher thread count may be beneficial if you have spare bandwidth and CPU/memory resources to push or pull data.

  3. Manageability and usability: We believe usability is core to the idea of cloud backup, as it involves critical decisions about when/where/how you backup your systems. Users need to be given full freedom to make these decisions and yet it shouldn’t be hard to configure and monitor the solution. Though our users and experts always rated us high in this area, we realized that the user interface  needed to be ready to handle our planned rapid growth, both in terms of product features as well as customer use cases. So in ZCB 4, we redesigned our user interface, made many workflow improvements and made it much more intuitive and easier to use. To see it in action, you can view the new ZCB screenshots.

In addition to the above, ZCB 4 also offers:

  • Backup/Restore of selective databases in SQL Server and Exchange server
  • Differential backup of SharePoint server
  • Parallel operations across multiple backup sets
  • Extensive Reporting across multiple backup sets
  • Hundreds of other improvements

ZCB 4 is also available in German and Japanese languages. For more details on ZCB 4, please refer to the release notes page.

ZCB 4 brings to the market a comprehensive, flexible and practical cloud backup solution. Also, as we gain scale, we are also working on the pricing (in case you didn’t notice, we recently announced the 25 GB free tier, perhaps a first in the industry) to make it affordable to a bigger set of users.

We are already working very aggressively on our next releases and will soon be making some exciting announcements. If you have a suggestion for us, please do drop me a line at nikunj@zmanda.com.

Zmanda Cloud Backup adds Tokyo as its latest cloud storage location

Wednesday, March 16th, 2011

We are adding support for Asia Pacific (Tokyo) Region in Zmanda Cloud Backup (ZCB). This is the fifth worldwide location supported by ZCB.

This support provides faster uploads for ZCB users in Japan. Throughput will be significantly higher because of less hops along the way and very high bandwidth connections typically available in Japan. Overall processing will be faster because of lower latency (expected to be single digit millisecond latency for most end users in Japan).

Cloud Backup to Three Continents Now Includes Japan

Cloud Backup to Three Continents Now Includes Japan

This support enables users to ensure that their data does not leave Japan, e.g. if required for compliance reasons.

In summary, users in Japan now have an effective and scalable solution to backup their Windows Filesystems, Microsoft Applications and Databases (MySQL, SQL Server, Oracle) to a robust storage cloud

As an introductory offer to our customers in Japan, we are waiving all transfer and storage charges to the Tokyo location until April 30th, 2011. You only pay for the initial setup fees ($4.95) and pro-rated monthly fees ($4.95 per month). After April 30th, our regular charges will apply at par with all other supported regions.

There is more on the horizon for our Japanese customers. We are soon going to offer a fully localized Japanese version of ZCB (Current shipping version has already been tested with Japanese file and folder names). Watch this space for an announcement on that within few weeks.

Zmanda Cloud Backup with Japanese Files/Folders

MySQL Backup Webinar Series: Scalable backup of live databases

Thursday, October 14th, 2010

mysql logo

Setting up of a good backup and recovery strategy is crucial for any serious MySQL implementation. This strategy can vary from site to site based on various factors including size of the database, rate of change, security needs, retention and other compliance policy etc. In general, it is also required from MySQL DBAs to have least possible impact on usability and performance of the database at the time of backup - i.e MySQL and its dependent applications should remain hot during backup.

Join MySQL backup experts from Zmanda for two webinars dedicated to hot backup of MySQL:

MySQL Backup Essentials: In this webinar, we will go over best practices for backing up live MySQL databases. We will also cover Zmanda Recovery Manager (ZRM) for MySQL product in detail, including a walk through the configuration and management processes. We will discuss various features of ZRM including full backups using snapshots, point-in-time recovery, monitoring and reporting.

Register for MySQL Backup Essentials Webinar on November 23rd at 10:00AM PT

MySQL Backup to Cloud: In this webinar, we will focus on backing up MySQL databases running on Windows to the cloud. Cloud Storage provides an excellent alternative to backing up to removable media and shipping it to remote secure site. We will provide live demonstration of the Zmanda Cloud Backup (ZCB) product backing up MySQL to Amazon S3 storage. ZCB is an easy-to-use cloud backup solution which supports all Windows platforms. We will also discuss recovering MySQL database in the cloud, creating a radically low cost disaster recovery solution for MySQL.

Register for MySQL Backup to Cloud Webinar on November 30th at 10:00AM PT

Go Tapeless - Use Zmanda Cloud Backup for backup and disaster recovery

Wednesday, June 23rd, 2010

If you are in charge of ensuring backup and disaster recovery of critical servers for your business, you have undoubtedly grappled with unwieldy tapes. In this age of digital everything, writing to tapes and then shipping them to a remote location seems like a relic from another era. Advances in Cloud based services, e.g. those offered by Amazon Web Services, provide an excellent alternative to tapes for backup and disaster recovery.

We have been offering Amazon S3 based cloud backup solution for about three years now. Today we are announcing the third generation of our Zmanda Cloud Backup product. Particularly exciting for me is the support for the Asia Pacific Region.

Cloud Backup to Three Continents

Cloud Backup to Three Continents

For many of the same reasons that Amazon picked Singapore as their first Asia Pacific Region, Singapore is a great destination to preserve your valuable assets. Performance and robustness provided by Singapore’s Internet connectivity is a major plus for backup and disaster recovery needs.

Backing up your data to the cloud requires several steps. You need to (1) Plan what do you want to backup and when; (2) Extract data out of your live applications, e.g. SQL Server or Exchange; (2) Stage this backup image to transfer to the cloud; (3) Monitor the transfer for any Internet hiccups and take corrective actions; and (4) Delete backup images which have expired per your retention policy. Zmanda Cloud Backup automates these steps through an easy GUI based backup configuration and management. ZCB integrates with S3’s REST API to coordinate transfer of on-premises data to the storage cloud.

In third-gen ZCB we also added support for international character sets. So, ZCB is friendly with files and folders named in e.g. Chinese (Simplified or Traditional), Japanese or Korean.

Backup What screenshot - Chinese filenames

Backup What screenshot - Chinese filenames

Backup What screenshot - Japanese filenames

Backup What screenshot - Japanese filenames

While a lot of Zmanda’s customers backup to local disks or tapes, Cloud Backup is fastest growing part of our business. In many environments, customers are backing up some backup sets to local media and other backup sets to the Cloud - with plans to move entire backup to the storage on the Cloud in a few years. We have seen this adoption across the board, including in the traditionally conservative financial industry. So, it appears more and more IT managers are daring to go tapeless when it comes to their backup operations!

Disaster Recovery in the Cloud

Monday, June 21st, 2010

Most small and medium sized business do not have a formal Disaster Recovery (DR) plan and implementation because of its cumbersome and costly nature. Various factors make DR complex, including: (1) Allocation and administration of remote compute and storage resources; (2) Data transport mechanism - e.g. tape shipment or data replication; and (3) Application environment synchronization. To makes matter worse, regular testing of a DR implementation tends to be complicated, and in many cases not practical.

Cloud Computing provides an excellent means to radically simplify the DR process. This is achieved by backing up your critical applications to a Storage Cloud (e.g. Amazon S3), and making preparation to quickly recover in the nearby Compute Cloud (e.g. Amazon EC2).

We have two solutions for backup and DR in the cloud: Amanda Enterprise (with the Amazon S3 Option) and Zmanda Cloud Backup (ZCB). Amanda Enterprise is meant for environments with heterogeneous systems, whereas ZCB is targeted at small businesses with a handful of Windows servers and desktops.

Amanda Enterpise DR in the Cloud

Setup of Amanda Enterprise for Cloud Based DR

 

Zmanda Cloud Backup DR in the Cloud

Setup of Zmanda Cloud Backup for Cloud Based DR

 

The process of setting up DR in the cloud is as follows:

  1. Set up backup process to Amazon S3.
  2. Complete first backup of applications on primary site to S3.
  3. Configure standby VMs on EC2 to match the OS (and patch level) of the corresponding systems on your primary site. For all data storage, use Elastic Block Storage, so you have persistent data across reboots.
  4. Install Zmanda backup software on these standby VMs.
  5. Install the same S3 certificate that is used in step #1 on the standby VMs.
  6. In case of Amanda Enterprise setup the AE-DR option to replicate backup catalog and configuration to the standby VM running the AE server.
  7. Perform full recovery from S3 to standby VMs.
  8. Take a snapshot of the standby VMs.
  9. Shutdown standby VMs.
  10. Optionally start standby VMs periodically to perform steps #6-#8. This will help in reducing the time to recover after a disaster and also tests your DR process.

If you are considering the Cloud for your DR needs, come join us tomorrow (June 22nd) for a webinar: Noted Storage Analyst, Lauren Whitehouse from Enterprise Strategy Group, will be joining me: Leveraging the Cloud for Radically Simple and Cost-Effective Disaster Recovery

Taking a Snapshot of a Thousand Dancing Dolphins

Monday, April 12th, 2010

An increasing number of large MySQL applications, e.g. social networking and SaaS back-ends, use a distributed MySQL architecture. MySQL data is distributed logically or heuristically on multiple, and in some cases thousands of, real or virtual servers. Backing up such large and dynamic environments presents its own complexities.

In this blog, we will use the cluster terminology - but we do not imply that NDB Cluster storage engine is being used for MySQL. Most implementations use InnoDB for data and MyISAM for dictionary. Typical architecture for such applications uses Database Sharding - i.e. shared-nothing partitioning of data across similarly configured nodes.

In most sharded environments, high availability is built-in - i.e. the cluster can continue to answer the queries and commit the transactions of all users in face of a node failure. This is typically accomplished either by database level replication or by designing the application so that each row is mirrored on two or more nodes. If MySQL Replication is being used, then slaves can be used for load-balancing as well - as long as it is ok that some clients may not get the latest data on the master node. E.g. a profile update by a user may not be visible to all her friends right away.

But built-in high availability does not do away with the need for setting up a backup and recovery process. Just like RAID does not replace backup, Sharding with redundancy does not replace backup either. The inherent complexity of large scale distributed database environments makes errors (human, system, environmental) more probable. Also, the implied availability of these environments increases the stress during the recovery process.

Here are the backup and recovery needs for such environments, some of the needs conflict with each other:

  • Application managers desire a point-in-time restore which is coordinated across multiple servers.
  • IT managers want to have as identical configuration as possible across all nodes - so process of replacing nodes becomes simple.
  • Depending on the application, retention policy could be several years.
  • Overall application should be able to recover from multiple node failures, human errors or sabotage, and geographic problems (disaster, connectivity etc.)

Zmanda Recovery Manager for MySQL is designed to meet these challenging needs. It uses various backup methods for backing up individual shards, and manages backup and recovery of the overall MySQL environment.

For point-in-time restore capability, ZRM uses MySQL binary logs. In very high update-oriented environments - size of these binary logs can become very big. In such environments, if the organization’s Recovery Point Objective (RPO) requires to be able to recover to any point within the past few weeks, it may not be possible to store these binary logs on the MySQL node itself. In any case, in order to be able to recover in the face of complete node failures, these logs need to be stored outside of the node. So, a storage environment which is physically or logically shared among the nodes is typically a requirement for storing the backup images. This shared secondary storage does not violate the shared-nothing principles of sharding, because it is not in the path of actual application. It is out-of-band storage being accessed and managed by the backup software. Also note that ZRM can automatically remove the binary logs from the MySQL node once they have been copied over to their archive location.

Taking a Snapshot of a multiple MySQL databases

ZRM can use two techniques to allow for point-in-time recovery of distributed MySQL environments: Coordinated Backups or Coordinated Restores:

Coordinated backup provides a backup image of all nodes consistent to a specific event. E.g. all rows are backed up until a specific Global Sequence Number (GSN) - assuming a GSN exists in the application. Another option is to create a checkpoint event specifically for backup purposes. Of course, having a GSN or a checkpoint event may create periodic brief hiccups which may or may not be acceptable for the business needs. But this process creates the cleanest backup images for the whole application.

Coordinated restore allows for each individual node to be backed up completely independent of each other. This eliminates the need for a backup checkpoint event. However at the time of recovery more processing is required to make sure all nodes are recovered to a point which is logically acceptable to the higher level application. ZRM can be scripted to identify this point in the backed up binary logs for every shard. Also, the visual log analyzer feature of ZRM helps DBAs to efficiently search for these points. Note that it is possible that all shards are not recovered to their state as it existed at exact same time, however they should be recovered to a state which is acceptable for the overall application. Having the clocks of nodes synchronized will also help the DBAs to identify points-of-recovery across nodes - by being able to correlate events easily.

Being able to backup a smaller shard instead of the whole dataset provides some opportunities both from technical and logical perspective. Since the size of each shard may be relatively small, a particular backup method may be acceptable even though it would not have been acceptable if the whole dataset was in one monolithic database. If data was distributed among shards using some external criteria (e.g. users of each zip code go to a particular shard), then backup images of each shard may be individually usable by an application. ZRM creates portable backup images - a key need for backing up shards - so backups from one node can be restored on another.

If recovery from a site wide disaster is also an objective, then suitable backup images need to be securely transported to the remote site. This can be done via the new Disaster Recovery Option now available for ZRM. This option replicates backup images, backup catalog and configuration data to the remote site - enabling full disaster recovery on an as-needed basis. Individual nodes need not be replicated, saving huge hassle and cost.

If your show is backed by a pod of dancing dolphins, a well implemented and documented backup and disaster recovery process is a good investment.

Fast Backups of MySQL Running on Amazon EC2

Saturday, January 23rd, 2010

If you are running your MySQL databases on the Amazon EC2 compute cloud, Zmanda Recovery Manager (ZRM) for MySQL can perform fast full backups of these databases by using Elastic Block Store (EBS) Snapshots. ZRM takes only a momentary read lock on the MySQL database during the creation of the snapshot, in order to ensure consistency of the backed up database archive. MySQL Backups using Amazon EBS snapshots are differential backups, meaning that only the blocks that have changed since your last full backup (via EBS snapshot) will be saved. For example, if you have a database with 100 GBs of data, but only 5 GBs of data has changed since your last snapshot, only the 5 additional GBs of snapshot data will be stored back to Amazon S3 during the current full backup run.

EC2 to S3 mysql backup diagram

ZRM automatically deletes EBS snapshots (containing full backups of MySQL) according to the configured retention policy. Just like other snapshot based full backups, ZRM intelligently correlates EBS Snapshots with incremental backups using MySQL logs, enabling you to recover your MySQL instances running on EC2 to any point in time.

Backups made using EBS snapshots can be recovered on the original EC2 instance or on a new EC2 instance. This also provides a quick and convenient mechanism to instantiate new MySQL database servers based on the database state from a desired point-in-time.

ZRM can run on the same EC2 instance as the MySQL database. On the other hand, if you have multiple EC2 instances with MySQL databases, you can run ZRM on one centralized EC2 instance dedicated for backup purposes. In this case, backup configuration and management for all MySQL databases is performed via Zmanda Management Console from this centralized backup server.

We have created an Amazon Machine Image (AMI) with ZRM pre-configured. This makes implementation of a MySQL backup solution on the cloud even simpler. We have used the “EC2 Small Instance” - which is powerful enough to backup most MySQL workloads in the cloud. This also makes it a very cost-effective option. This AMI is available to all ZRM customers, as part of the ZRM Enterprise subscription. You will need to create your own Amazon EC2 account, and pay standard per hour price to Amazon to run an instance based on this AMI. Note that you can configure your backup server instance to run only during the backup window. So, if you are backing up your databases once a week, and your backups takes less than an hour, then you can have this instance up only during that hour. EC2 pricing is per instance-hour consumed from the time an instance is launched until it is terminated. Each partial instance-hour consumed will be billed as a full hour. In addition to the EC2 compute capacity, you will pay standard storage charges for Amazon S3 (to store EBS Snapshots).

Join us on January 28th for a webinar on MySQL Backups (hosted by Sun/MySQL). Along with an introduction to Zmanda Recovery Manager, we will also discuss backing up MySQL applications on the cloud, and demonstrate the new ZRM AMI.

Windows XP -> Cloud -> Windows 7

Monday, November 23rd, 2009

We recently added support for Windows 7 to both Zmanda Cloud Backup and Amanda Enterprise. Zmanda Cloud Backup stores its backup archives on the Amazon S3 Storage Cloud. Amanda Enterprise has the option to do so. Users can backup both the Windows file systems and system state, as well as various Microsoft applications, Oracle and MySQL databases. Now we support all Windows versions supported by Microsoft, including Windows 7.

To upgrade from Windows XP to Windows 7, Microsoft recommends users to backup their Windows XP to external hard disk and then install Windows 7. Backup to (and Restore from) Cloud offers another alternative, which we have tested in our labs.

xp to win7 diagram

If you have your backups (either created by ZCB or Amanda Enterprise) in the cloud, you can upgrade to Windows 7 and restore file system after Windows 7 installation. The system state backup includes application state that can be restored to an alternate location and restored selectively to Windows 7. As an added benefit, your data will be preserved in an off-site secure location until the time you are sure your new environment is stable.

Transitioning to the Cloud: Implications for Reliability, Redundancy and Recoverability

Wednesday, July 29th, 2009

Last week we had a lively panel discussion moderated by Dave Nielsen, founder of CloudCamp, with leading experts in cloud computing on the panel: Chander Kant (CEO, Zmanda) & Michael Crandell (CEO, RightScale). Chander and Michael shared their insights into how their customers are using cloud computing and achieving new levels of reliability and recoverability. Here is a video archive of the panel

This panel discusses how you can migrate your apps and data to the cloud in a way that’s affordable and reliable and how to integrate the cloud into your IT strategy. Also hear about real-world examples of companies using the cloud for data backup and recovery and understand the hurdles to moving to the cloud, and how you can overcome them.

 

Sun Heats Up Cloud Storage

Wednesday, March 18th, 2009

Today Sun announced the Sun Cloud, including its Sun Cloud Storage Service at CommunityOne East in New York. We announced our partnership with Sun on integrating all our three products: Amanda Enterprise, Zmanda Recovery Manager (ZRM) for MySQL and Zmanda Cloud Backup with Sun Cloud. In near future, our customers will be able to chose between Sun Cloud Storage or Amazon S3 as the destination of their backup archives. Also, users running critical application on the Sun Cloud Compute Service will be able to protect their data using Zmanda’s products.

Sun CloudOf particular interest to us is Sun’s approach to use open and well defined APIs for their cloud services. We got access to Sun Cloud Storage Service (SCSS) few weeks ago. We are able to use either an S3 Compatible API or a WebDAV API for integration. We chose to use S3 APIs for integrating with Amanda, and WebDAV API for integrating with ZRM. In the short-term, these APIs made it easy for us to do the integration. In the long-term, this open architecture will result in rapid innovation. As an industry, open APIs enable us to stand on each other’s shoulders (rather than step on each other’s toes by developing closed solutions from scratch which have overlapping components).

This marks key next step in Zmanda’s cloud backup strategy. We are integrating with storage clouds using an open Cloud API, which is extension of our Device API. We will be the link between on-premises data and any storage cloud of customer’s choice.

Cloud backup space is certainly heating up, and Sun added its warmth to the space today.