Archive for the ‘Chander Kant’ Category

Backup Zen

Monday, April 20th, 2009

ZenOne of the questions that comes up often in the Backup world is “Why can’t I just write a script to do this myself!?”. Well, as a do-it-yourselfer myself, the answer is “Absolutely, a customized backup script can be written, and in fact, the first version of it won’t be that complicated to develop either”. However, a home-grown backup script can quickly become tedious to enhance and maintain.

Lets look at the progression of events following a decision made by a system administrator or a DBA to develop their own backup scripts. Lets take the example of Joe, a MySQL Database Administrator at an e-commerce company SuperWidgets:

1. January: SuperWidgets has been around for one year, and sales of their super widgets have been increasing. The CEO of SuperWidgets, Mary, has faced serious consequences of losing customer data before, and she tells Joe to make sure their MySQL database, which powers their web store, is being backed up.

2. Joe starts looking at ways to backup the web store database (running on Red Hat Linux), and discovers various tools that came with his MySQL installation: mysqldump, mysqlhotcopy and MySQL replication. After spending a couple of days of doing research on various tools, he decides to use the mysqlhotcopy utility to make a quick raw backup of the database.

3. Joe begins digging into the syntax for mysqlhotcopy, and in a day has a script running under cronjob which performs a backup of the web store database at midnight every day.

4. February: SuperWidgets uses Windows as the development platform for their web applications. One morning, Tom the database developer finds that a filesystem level corruption had taken out his database. He used mirroring, but since the corruption was logical rather than physical, both copies are damaged beyond repair. This causes a downtime of two days for the development team. Mary instructs Joe to make sure that all three databases on the development platform are also backed up.

5. Joe discovers that mysqlhotcopy doesn’t work on Windows. So, after doing further research, he writes a custom script to use mysqldump for backing up the development databases nightly via Windows Task Scheduler.

6. Since MySQL database backup has become a hot button for Mary, Joe starts monitoring the status of each of his MySQL backup scripts. He periodically logs onto each of the five systems where MySQL instances are being backed up by his scripts and makes sure that archives were successfully created the previous night.

7. March: SuperWidgets decides to use Alfresco as the Content Management System (CMS) for an internal project, with MySQL as the underlying database. Tom is in charge of the Alfresco implementation. The data stored in this CMS is sensitive and important. Mary gives Tom the responsibility of backing up and, when needed, restoring the CMS data.

8. Joe ports one of his backup scripts to the system running CMS, and trains Tom on nuances of feeding and caring for his script.

9. April: Tom upgrades his MySQL database and discovers that one of the options for mysqldump has changed, causing the backup scripts to fail. He fixes the script to work with the new mysqldump syntax.

9. May: The business of SuperWidgets has gone through the roof. But, one afternoon the webstore is brought to its knees because of an application error causing database to have inconsistent data. Fortunately, Joe’s script worked and he is able to recover the database using an archive from the previous night. Unfortunately, this meant that the transaction data of hundreds of customers since last night is lost. This forced Mary and rest of the management team to take several actions to manage reputation of now well known SuperWidgets.

10. As a result, Joe is instructed to ensure that MySQL can be recovered to any point-in-time, rather than just to the previous night’s status. Also, he has been instructed to send a high-level summary of MySQL backups to the management on a weekly basis. He has also been asked to look at reducing the amount of time the MySQL database is locked up while backups were being done. To add to Joe’s woes, Tom has decided to leave the organization. Joe must now takeover the backup of the CMS. Joe discovers that Tom has modified the original backup scripts for CMS without providing any documentation.

Joe’s situation is not atypical and shows how a Backup solution involves more than just putting a simple script around a utility which makes copy of data. For workloads of even moderate importance, any organization will find the need for cataloging of backup archives, monitoring and reporting to be vital. A common user interface across various backup methods, which is easy for new personnel to learn, has a huge long-term value as well. This is precisely where our backup solutions come in. Specifically for MySQL, our Zmanda Recovery Manager offers a great solution to Joe’s woes, by providing:

- An intelligent MySQL backup solution which figures out the best way to backup a particular MySQL database
- A common user interface across all platforms, whether they are Linux, Solaris or Windows
- A common user interface across all backup methods, whether they are raw backups, logical backups or snapshot based backups
- Integration between backup methods (e.g. snapshots) and MySQL logs to be able to recover MySQL to any point in time
- Role based access control, enabling management and DBAs to have control over who has access to what data
- A centralized backup solution, enabling a quick and automated health check of the entire backup infrastructure
- A customizable Reporting module, enabling automated reporting for desired levels of details

Zen InnovationsOne of Zmanda’s customers, Zen Innovations, initially backed up their data using scripts and manual backup procedures, but soon found that this was not scalable, and opted for Zmanda’s backup solutions. According to Sergio Laberer, Managing Director of Zen Innovations: “Zmanda’s ability to manage multiple platforms over a web based GUI was exactly what we were looking for. Our initial manual processes, scripts and cron jobs quickly started to get complicated as we grew our infrastructure. We needed to do backups regularly and be in a position to recover quickly without too much manual intervention. Our initial approach was neither scalable nor suitable to work efficiently.”

Sun Heats Up Cloud Storage

Wednesday, March 18th, 2009

Today Sun announced the Sun Cloud, including its Sun Cloud Storage Service at CommunityOne East in New York. We announced our partnership with Sun on integrating all our three products: Amanda Enterprise, Zmanda Recovery Manager (ZRM) for MySQL and Zmanda Cloud Backup with Sun Cloud. In near future, our customers will be able to chose between Sun Cloud Storage or Amazon S3 as the destination of their backup archives. Also, users running critical application on the Sun Cloud Compute Service will be able to protect their data using Zmanda’s products.

Sun CloudOf particular interest to us is Sun’s approach to use open and well defined APIs for their cloud services. We got access to Sun Cloud Storage Service (SCSS) few weeks ago. We are able to use either an S3 Compatible API or a WebDAV API for integration. We chose to use S3 APIs for integrating with Amanda, and WebDAV API for integrating with ZRM. In the short-term, these APIs made it easy for us to do the integration. In the long-term, this open architecture will result in rapid innovation. As an industry, open APIs enable us to stand on each other’s shoulders (rather than step on each other’s toes by developing closed solutions from scratch which have overlapping components).

This marks key next step in Zmanda’s cloud backup strategy. We are integrating with storage clouds using an open Cloud API, which is extension of our Device API. We will be the link between on-premises data and any storage cloud of customer’s choice.

Cloud backup space is certainly heating up, and Sun added its warmth to the space today.

Cloud Backup II

Tuesday, February 24th, 2009

In my previous blog on Cloud Backup, I wrote about the solutions we offer to backup to the Storage Cloud (e.g. Amazon S3). In this blog I will talk about backup of cloud, i.e. backup of your applications running on a Compute Cloud (e.g. Amazon EC2).

Let’s say you are migrating some on-premises applications (e.g. a customer facing enterprise app), which are currently being backed up to a tape library, to the cloud (fig 1).
Applications migrating to the cloud

Clouds don’t have a notion of a local tape library. So, your current backup solution will likely not work after this migration.

Backup of Apps on Clouds?So, where do you backup? Note that Compute Cloud vendors do not offer automatic backups. While they may offer storage redundancy features, e.g. replication and snapshots, these are not replacement for a complete backup solution.

You still need backup archives and a backup catalog for those archives to be able to recover from user and application errors. Just like RAID is not a backup solution for on-premises data, storage redundancy features offered by cloud vendors don’t provide a backup solution either. In summary, an automated backup solution is a must-have regardless of where the applications are running: on-premises or in the cloud. Well this is where Zmanda comes into play. We offer three choices as destination for backup of data residing in applications on the cloud(fig 3):

Backup of Apps on Clouds1. Backup to a local Storage Cloud - e.g. Amazon S3 if your applications are running on EC2. This is great option from a cost or performance perspective, but not so great from spreading-your-risk perspective.

2. Backup to a remote cloud. This requires having relationships with two different cloud vendors, but reduces your risks the most.

3. Backup to disks on your premises. This requires local infrastructure, but gives you complete control of your backup archives.

All three of our products: Amanda Enterprise, ZRM for MySQL, and Zmanda Cloud Backup are tested and supported on virtual machines running on the cloud.

So, today you can buy these products and run them on a virtual machine of your choice (as long as it runs an operating system from our compatibility matrix). In near future, we will be shipping virtual appliances which can be run either in your data center or in the cloud. So, lets say you bought and run few VMs in the cloud to run a set of your applications. In order to backup these applications, you will simply buy a Zmanda Backup Appliance (which will be another VM in the same cloud), and quickly configure this appliance to know about your application VMs and the destination for your archives. You don’t have to worry about dependencies, installation issues or optimizing the OS for backup purposes.

Currently we are working actively with Novell and VMware to build our first backup appliance. Novell today announced its partnership with VMware to build SUSE Linux Enterprise Appliances on VMware ESX. We are very excited about this development. SLES is already in Tier 1 of operating systems we test and support. We have hundreds of customers running Amanda Enterprise on SLES (e.g. Zen Innovations). Several of our customers protect guest OS’es running on ESX. We are now polishing up this combination to create an appliance-like experience, presented on a browser via Zmanda Management Console.

Reasons to migrate your applications to cloud are to increase efficiency and to dramatically reduce your IT costs. Our virtual backup appliances will help you with both of these goals.

Cloud Backup

Tuesday, February 17th, 2009

Today we made two major announcements: Amanda Enterprise 3.0 and GA of Zmanda Cloud Backup. We announced the release of several cloud backup features and optimizations as part of these product offerings.

When we say “Cloud Backup”, we mean two things:

1. Backup to Cloud: Backup of your on-premises data to a Storage Cloud (e.g. Amazon S3)
2. Backup of Cloud: Backup of your applications running on a Compute Cloud (e.g. Amazon EC2)

We offer solutions for both of above. In this blog, I will focus on Backup to Cloud.

Storage Clouds offer an excellent alternative to tapes or optical media for off-site backup storage. Your data is stored off-site and yet is on-line. You can initiate a recovery anytime and from anywhere, as long as you have internet access and your access credentials. You pay only for how much storage and bandwidth you use, and don’t have to invest in local tape infrastructure. Also, even as a small business you get have a profile of a much larger vendor (Tier 1 Storage Cloud vendors spend millions of dollars to ensure robustness of their infrastructure).

We have two different products to securely transfer your backup data to a storage cloud: Integration of Amanda Enterprise with Amazon S3 and Zmanda Cloud Backup.

These two products meet different needs:

Amanda Enterprise network diagram

If you would like to consolidate backup of a heterogeneous set of systems, including those running Solaris, Linux, Windows and OS-X, and then ship this consolidated backup to the storage cloud, our Amanda Enterprise (AE) 3.0 release is optimized for this. We added some key cloud optimizations in AE 3.0:

  • Cloud Vaulting: We allow you to backup to your local media and then vault to the storage cloud on its own independent schedule.
  • Transfer Block Size control: We allow you to control the size of your block transferred to Amazon S3. So far we had it pegged to 10MB. But now depending on your uplink speed and quality, you can pick a transfer block size of anywhere between 1KB and 100MB. So, if you have a fat and reliable pipe and you are not too many hops away from your storage cloud, you can use a much bigger block to get better performance. On the other hand, if you are stuck behind an unreliable pipe you can use a small block size to avoid retries.
  • Location control: We now enable an Amanda administrator to chose between storage cloud in US or Europe as destination for their storage. So, if you are in Europe and would like to backup to a US data center for disaster recovery, you can instruct AE to do so. Or if your country doesn’t allow certain kind of data to leave EU boundaries for compliance purposes, you can restrict the backup location to Europe.
  • Built-in Integrity Check: We do an integrity check on each block transferred to S3. So, any data corruption on transit will be caught right away.
  • Optional Encryption: Until now encryption during data transfer was hard-coded, i.e. you didn’t have a choice but to have your data encrypted during transfer to the storage cloud. We had done so to address the data privacy concern while transferring backup data over the Internet. Well, we got feedback that in many conditions the data being transferred was public data (e.g. postings on a forum). It was important data for business owner for protection, but there was no privacy concern about this data. In this case, the administrator wanted the fastest possible transfer and not incur the overhead of encryption. So, we decided to make encryption while transfer optional. Note that this is independent of encryption of data at rest - which is a configurable parameter on its own.

Zmanda Cloud Backup DiagramZmanda Cloud Backup (ZCB) is our offering designed for small businesses with a handful of Windows servers. It creates a direct link between a single Windows system and Storage Cloud. There is no backup server required. ZCB provides a very simple to use interface and configured automated backup of Windows file systems and applications. It is integrated with Exchange, SQL Server and Oracle using Volume Shadow Copy Service - so, it can backup these applications while they are running. You only pay for actual data transfered and stored (in addition to a small one-time software fees). So, if your data gets compressed 70%, you only pay for 30% of data size being backed up. This is in contrast to other online backup services such as Mozy Pro which charge on the basis of amount of data being backed up.

Also, in contrast to most other backup services, ZCB allows you to set up a retention policy of your choice. So, you can have a retention policy of a month, a year, or seven years (or any other timeframe). You can base this retention policy based on business needs, including any compliance needs.

In addition to file systems and applications, ZCB also backs up and recovers Windows System State (including Active Directory). This enables you to do a bare metal recovery, in case you lose your entire system.

Some leading journalists and bloggers today covered our Cloud Backup news:

BTW, we continue to adhere to our open standards and open formats philosophy. ZCB stores data on the cloud in ZIP64 format. So, any utility which understands this format can extract your files from the archive.

Thailand Metropolitan Waterworks Authority Protects Emails Using Zmanda

Tuesday, December 23rd, 2008

Metropolitan Waterworks Authority Logo Metropolitan Water Authority (MWA) is a public water utility providing water supply to residences and businesses in key parts of Thailand including Bangkok. With more than four thousand employees and approximately two million customers, MWA generates a lot of email on a daily basis.

For this challenge, MWA turned to CommuniGate Pro for their email server, and to Amanda Enterprise for protecting the email data. MWA chose Solaris 10 to run the Amanda Server.

Congratulations to Veerachart Charassirikulchai (I just call him Vee!) and his team at SoftDebut for working with MWA and installing Zmanda’s solution in their environment.

Data Protection for Today’s Economy: Amber Road and Zmanda

Monday, November 10th, 2008

Today Sun announced the new 7000 line (aka Amber Road) of open storage appliances. Amber Road runs OpenSolaris and ZFS on industry-standard x86 hardware and includes innovative management software developed by Sun’s FISHworks (Fully Integrated Software and Hardware) group.

Our engineers worked along with Sun’s technologists on Amber Road in Sun’s labs for past few weeks, and today we are announcing support for Amber Road with both of our products Amanda Enterprise and Zmanda Recovery Manager for MySQL.

Amber Road is another example of innovation and value created by combination of open source and open systems. Combine  Amber Road with Zmanda’s open source backup products and you can deploy an extremely scalable and blazing fast backup solution at 1/5th the cost of products from proprietary vendors. We are also offering a free 60 days trial to all Sun Storage Systems customers (please contact your Sun or Zmanda sales rep for this).

Amanda Backup to Tape Library

Wednesday, October 1st, 2008

Over the years Amanda has added lots of media choices to store backup archives: SAN, NAS, Amazon S3 etc. But tapes remain one of the most popular destination. Gavin, who blogs under a to-the-point title “I Do Linux”, has written a detailed how-to on installing and configuring Amanda with Tape Libraries. He has two setups: One setup (which backs up over a terabyte of Maildir for 3000 users) has Amanda backing up to a Spectra Logic T50 LTO2 tape library, containing 5 magazines/terapacks and 50 tape slots. Second setup (which backs up 4TB of research data for over 100 users) has Amanda backup archives going to a Dell PowerVault TL2000 LTO4 tape library, containing 2 magazines and 23 tape slots.

Vyatta: From Homegrown Backup Scripts to Amanda Enterprise

Monday, September 29th, 2008

VyattaVyatta has rocked the networking world by developing the first commercially supported, open source router and firewall solution. Ambitions of changing the rules of the game of a huge industry, come with very critical data - which doubles in size every year! They also come with a very heterogeneous environment: Fedora, Debian, Red Hat, and Ubuntu - well, as long as it is a Linux distro :)

Vyatta recognized the need for a formal backup solution to replace the homegrown backup scripts, and chose Amanda Enterprise. Amanda backs up terabytes of Vyatta’s data spread across web sites, e-mail server, source-code repository, databases, and user files. Here is the full story.

Interview with Ray Zhu (Senior IT Manager at Vyatta) about his experience with Zmanda was published  in Inc. Magazine in December. Here are some interesting excerpts:

“We did not have a commercial solution before. We were a startup and, in the beginning, someone developed a script for backing up that supported Linux. As we grew, we realized we needed to have something more scalable. Before, every time we had to add a new server, we had to have a developer write a new script. We needed to have something more manageable. “

“We went with Zmanda because it met the criteria. It supports all the flavors of Linux. It’s got a beautiful GUI interface. And it’s able to recover in a few hours. The results have been beautiful. We have nothing to complain about. What I need to do is once in a while look at the dashboard. If it is red, then something is missing. If it’s green, then it’s working. Now when we add a new machine, I can just go the Web-based GUI and type the server host name, which directory we need to back up on and which schedule. The interface is designed in an intuitive way based on what we want to back up and to where and when we want to back up. We have done quite a few tests and we’ve recovered all the machines each time. “

PowerReviews: A Zmanda Success Story

Monday, September 22nd, 2008

If you are shopping for a backpack for an upcoming trip to Yosemite this fall, chances are you may head over to REI.COM or to compare various options and buy the backpack of your choice. On these websites you will also run into extensive reviews to help guide your decision. These reviews are powered by PowerReviews - a fast growing company which provides a platform for online retailers to quickly add consumer reviews functionality to their websites.

When PowerReviews started looking for a backup solution to protect rapidly growing critical data in their environment, they turned to Amanda Enterprise. Today Amanda Enterprise protects 30 Windows and 16 Mac OS X Clients in their environment. Here is the full story.

Zmanda Recovery Manager for MySQL - Detailed Review by Linux Journal

Friday, September 12th, 2008

Zmanda on Linux Journal Cover

This month’s (September 2008) edition of Linux Journal has an in-depth review of Zmanda Recovery Manager for MySQL (written by Alolita Sharma a veteran of Open Source). Oh, and we made it on the cover as well! If you are browsing Magazines section of your local bookstore, do look us up..