Archive for the ‘MySQL Backup and Recovery’ Category

ZRM Community Edition 3.0 (Beta) with Parallel Logical Backup Support for MySQL

Tuesday, April 23rd, 2013

We are pleased to announce the release of Zmanda Recovery Manager (ZRM) Community Edition 3.0 (Beta). This release features support for parallel logical backups as an additional full backup method, which is made possible by integrating with the mydumper open source project.  This backup method represents a faster, scalable way to backup large databases.  The mysqldump (single threaded logical backup) support is still available for backing up stored procedures/routines.  ZRM Community Edition allows you to create multiple backup sets with different backup methods and policies; so, now you can do MySQL database backups with mydumper, as well as mysqldump in the same server.

We have also made many additional improvements and bug fixes since our earlier 2.2 release. We currently plan to release a final version of ZRM Community Edition 3.0 later this quarter, and in the meantime, we look forward to your feedback on the Zmanda forums.

Zmanda Recovery Manager for MySQL - What’s New in Version 3.5

Wednesday, March 20th, 2013

As we continue to see MySQL being implemented in bigger and more challenging environments, we are working to ensure Zmanda Recovery Manager for MySQL (ZRM) matches this growth and provides a comprehensive, scalable backup management solution for MySQL that can easily integrate into any network backup infrastructure.

The latest release of ZRM for MySQL is a significant next step, bringing disk space and network usage optimization and enhanced backup reporting, along with simplified management to help configure backups quickly and intelligently.   Additionally, ZRM for MySQL 3.5 now supports backup of MySQL Enterprise Edition, MySQL Community Edition, SkySQL, MariaDB, and MySQL databases running on latest versions of Red Hat Enterprise Linux, CentOS, Debian, Ubuntu and Windows – giving you an open choice for your MySQL infrastructure, now and in future, with confidence that your backup solution will continue to work.

Here is a look at the key updates in ZRM:

Optimization of Disk Space: We’ve implemented streaming for various backup methods so that you don’t need to provide additional disk space on the systems running MySQL servers. This will allow you to do hot backup of your MySQL databases without having to allocate additional space on the system running MySQL. Backup data will get directly stored on the ZRM server.

Optimization of Network Usage: We have implemented client-side compression for various backup methods so you can choose to compress backup data even before it is sent to the ZRM server. Of course, you also have the choice to compress on the backup server; for example, if you don’t want to burden the MySQL server with backup compression operation.

Enhanced Backup Reporting: Backup is often where IT meets compliance. ZRM allows you to generate backup reports for all of the MySQL databases in your environment. With the latest version, now you can generate unified backup reports across backup sets too.

Simplified Management: One of the key features of ZRM is that it hides nuances of particular types of backup method for MySQL behind an easy-to-use GUI, the Zmanda Management Console (ZMC). With the new release, ZMC brings new features for applicable backup methods, such as parallelism, throttling, etc. You will also find several tool tips to help you configure your backups quickly and intelligently, without having to dig through documentation on specific backup methods.

Broad Platform Coverage: MySQL gets implemented in various shapes and forms on various operating systems. We continue to port and test all variants of MySQL on all major operating system platforms. ZRM 3.5 supports backup of MySQL Enterprise Edition, MySQL Community Edition, SkySQL and MariaDB. Backup of MySQL databases running on latest versions of Red Hat Enterprise Linux, CentOS, Debian, Ubuntu and Windows is supported.

Seamless Integration with Backup Infrastructure: ZRM is architected for the MySQL DBAs. In order for DBAs to integrate and comply with the overall backup methodology of their corporate environment, we have made sure that ZRM can integrate well into any of the network backup infrastructures being used. While ZRM is already known to work well with almost all network backup environments, we have completed specific integration and testing of ZRM 3.5 with Amanda Enterprise, Symantec NetBackup, and Tivoli Storage Manager.

If you are putting together a new MySQL based environment, or looking to add a well managed backup solution to your existing MySQL infrastructure, our MySQL backup solutions team is ready to help: zsales@zmanda.com

MySQL Backup Updated

Tuesday, April 10th, 2012

As MySQL continues to grow (as a technology and as an ecosystem) the need and importance of creating and deploying robust MySQL backup solutions grows as well. In many circles Zmanda is known as “The MySQL Backup Company”. While we provide backup of a wide variety of environments, we gladly take the label of backing up the most popular open source database in the world, especially as we kick off our presence at the 2012 MySQL Conference.

Here are some of the updates to our MySQL backup technologies that we are announcing at the conference:

Announcing Zmanda Recovery Manager 3.4

We have updated the popular Zmanda Recovery Manager (ZRM) for MySQL product for scalability. Our customers continue to deploy ZRM to backup ever larger MySQL environments. Some of the scalability features include: Better support for hundreds of backup sets within one ZRM installation, support for more aggressive backup schedules, better support for site-wide templates, and deeper integration with NetApp’s snapshot mechanisms. We have also added support for the latest versions of XtraBackup and MySQL Enterprise Backup. We have also added experimental support for backing up Drizzle (via XtraBackup). If you are deploying Drizzle in your environment, we are looking for beta customers.

Many of our customers store their MySQL databases on NetApp storage. ZRM can be used in conjunction with NetApp Snapshot and SnapVault products to create database consistent backups without moving the data out of NetApp storage. ZRM creates snapshots of MySQL database volumes, which it can then move to another Netapp storage using Netapp SnapVault. SnapVault moves the data efficiently between various NetApp filers. This provides customers a way to protect the backups without impacting their corporate LAN. ZRM uses SnapRestore functionality to quickly restore the databases in case of a failure.

Announcing MySQL Backup Agent for Symantec NetBackup

If you have Symantec NetBackup deployed in your environment, and you would like to consolidate your MySQL backups within the umbrella of NetBackup based backup infrastructure, now you have a well integrated solution. We have released MySQL backup Agent, which is deeply integrated with Symantec NetBackup. This agent allows you do perform live backups of your MySQL databases directly from your MySQL servers to your NetBackup server.

NetBackup MySQL Agent


Backup of your MySQL databases to the Cloud

Public or Private Cloud Storage is a great choice for offsite store for backup archives. You can also use compute clouds as inexpensive DR site for your MySQL databases. For MySQL databases running on Windows, our Zmanda Cloud Backup product provides a very easy and inexpensive way to backup to Amazon S3 or Google Cloud Storage.

If you have MySQL databases running on Linux or heterogeneous environments, you have two choices for backing up to the cloud: You can use our Amanda Enterprise product with Amazon S3 or Google Cloud Storage option to move backup images created by ZRM to the cloud. Second option is to use the recently released Amazon Storage Gateway in conjunction with ZRM.

ZRM Backing Up To AWS Gateway Storage

We have published an integration report (available on Zmanda Network under the MySQL Backup section - free registration required) to show how you can deploy AWS Gateway to asynchronously upload backup files created by ZRM to Amazon S3.

As you can see, we have been busy updating our MySQL backup solutions. All of above improvements and feature additions have been done based on feedback provided by MySQL DBAs. If you are visiting the MySQL user conference this week, please do visit us at our booth - we would love to understand and discuss your MySQL backup challenges.

Zmanda Cloud Backup adds Tokyo as its latest cloud storage location

Wednesday, March 16th, 2011

We are adding support for Asia Pacific (Tokyo) Region in Zmanda Cloud Backup (ZCB). This is the fifth worldwide location supported by ZCB.

This support provides faster uploads for ZCB users in Japan. Throughput will be significantly higher because of less hops along the way and very high bandwidth connections typically available in Japan. Overall processing will be faster because of lower latency (expected to be single digit millisecond latency for most end users in Japan).

Cloud Backup to Three Continents Now Includes Japan

Cloud Backup to Three Continents Now Includes Japan

This support enables users to ensure that their data does not leave Japan, e.g. if required for compliance reasons.

In summary, users in Japan now have an effective and scalable solution to backup their Windows Filesystems, Microsoft Applications and Databases (MySQL, SQL Server, Oracle) to a robust storage cloud

As an introductory offer to our customers in Japan, we are waiving all transfer and storage charges to the Tokyo location until April 30th, 2011. You only pay for the initial setup fees ($4.95) and pro-rated monthly fees ($4.95 per month). After April 30th, our regular charges will apply at par with all other supported regions.

There is more on the horizon for our Japanese customers. We are soon going to offer a fully localized Japanese version of ZCB (Current shipping version has already been tested with Japanese file and folder names). Watch this space for an announcement on that within few weeks.

Zmanda Cloud Backup with Japanese Files/Folders

MySQL Backup Webinar Series: Scalable backup of live databases

Thursday, October 14th, 2010

mysql logo

Setting up of a good backup and recovery strategy is crucial for any serious MySQL implementation. This strategy can vary from site to site based on various factors including size of the database, rate of change, security needs, retention and other compliance policy etc. In general, it is also required from MySQL DBAs to have least possible impact on usability and performance of the database at the time of backup - i.e MySQL and its dependent applications should remain hot during backup.

Join MySQL backup experts from Zmanda for two webinars dedicated to hot backup of MySQL:

MySQL Backup Essentials: In this webinar, we will go over best practices for backing up live MySQL databases. We will also cover Zmanda Recovery Manager (ZRM) for MySQL product in detail, including a walk through the configuration and management processes. We will discuss various features of ZRM including full backups using snapshots, point-in-time recovery, monitoring and reporting.

Register for MySQL Backup Essentials Webinar on November 23rd at 10:00AM PT

MySQL Backup to Cloud: In this webinar, we will focus on backing up MySQL databases running on Windows to the cloud. Cloud Storage provides an excellent alternative to backing up to removable media and shipping it to remote secure site. We will provide live demonstration of the Zmanda Cloud Backup (ZCB) product backing up MySQL to Amazon S3 storage. ZCB is an easy-to-use cloud backup solution which supports all Windows platforms. We will also discuss recovering MySQL database in the cloud, creating a radically low cost disaster recovery solution for MySQL.

Register for MySQL Backup to Cloud Webinar on November 30th at 10:00AM PT

Zmanda @ Oracle OpenWorld 2010

Tuesday, September 7th, 2010

Oracle Open World

If you are coming to this year’s Oracle OpenWorld 2010, please do visit us at Booth #3824.

We will have our backup solution experts at the show to discuss any of your database or infrastructure backup needs.

When it comes to backing up various products offered by Oracle, we have several solutions:

We hope to see you at the show!

Taking a Snapshot of a Thousand Dancing Dolphins

Monday, April 12th, 2010

An increasing number of large MySQL applications, e.g. social networking and SaaS back-ends, use a distributed MySQL architecture. MySQL data is distributed logically or heuristically on multiple, and in some cases thousands of, real or virtual servers. Backing up such large and dynamic environments presents its own complexities.

In this blog, we will use the cluster terminology - but we do not imply that NDB Cluster storage engine is being used for MySQL. Most implementations use InnoDB for data and MyISAM for dictionary. Typical architecture for such applications uses Database Sharding - i.e. shared-nothing partitioning of data across similarly configured nodes.

In most sharded environments, high availability is built-in - i.e. the cluster can continue to answer the queries and commit the transactions of all users in face of a node failure. This is typically accomplished either by database level replication or by designing the application so that each row is mirrored on two or more nodes. If MySQL Replication is being used, then slaves can be used for load-balancing as well - as long as it is ok that some clients may not get the latest data on the master node. E.g. a profile update by a user may not be visible to all her friends right away.

But built-in high availability does not do away with the need for setting up a backup and recovery process. Just like RAID does not replace backup, Sharding with redundancy does not replace backup either. The inherent complexity of large scale distributed database environments makes errors (human, system, environmental) more probable. Also, the implied availability of these environments increases the stress during the recovery process.

Here are the backup and recovery needs for such environments, some of the needs conflict with each other:

  • Application managers desire a point-in-time restore which is coordinated across multiple servers.
  • IT managers want to have as identical configuration as possible across all nodes - so process of replacing nodes becomes simple.
  • Depending on the application, retention policy could be several years.
  • Overall application should be able to recover from multiple node failures, human errors or sabotage, and geographic problems (disaster, connectivity etc.)

Zmanda Recovery Manager for MySQL is designed to meet these challenging needs. It uses various backup methods for backing up individual shards, and manages backup and recovery of the overall MySQL environment.

For point-in-time restore capability, ZRM uses MySQL binary logs. In very high update-oriented environments - size of these binary logs can become very big. In such environments, if the organization’s Recovery Point Objective (RPO) requires to be able to recover to any point within the past few weeks, it may not be possible to store these binary logs on the MySQL node itself. In any case, in order to be able to recover in the face of complete node failures, these logs need to be stored outside of the node. So, a storage environment which is physically or logically shared among the nodes is typically a requirement for storing the backup images. This shared secondary storage does not violate the shared-nothing principles of sharding, because it is not in the path of actual application. It is out-of-band storage being accessed and managed by the backup software. Also note that ZRM can automatically remove the binary logs from the MySQL node once they have been copied over to their archive location.

Taking a Snapshot of a multiple MySQL databases

ZRM can use two techniques to allow for point-in-time recovery of distributed MySQL environments: Coordinated Backups or Coordinated Restores:

Coordinated backup provides a backup image of all nodes consistent to a specific event. E.g. all rows are backed up until a specific Global Sequence Number (GSN) - assuming a GSN exists in the application. Another option is to create a checkpoint event specifically for backup purposes. Of course, having a GSN or a checkpoint event may create periodic brief hiccups which may or may not be acceptable for the business needs. But this process creates the cleanest backup images for the whole application.

Coordinated restore allows for each individual node to be backed up completely independent of each other. This eliminates the need for a backup checkpoint event. However at the time of recovery more processing is required to make sure all nodes are recovered to a point which is logically acceptable to the higher level application. ZRM can be scripted to identify this point in the backed up binary logs for every shard. Also, the visual log analyzer feature of ZRM helps DBAs to efficiently search for these points. Note that it is possible that all shards are not recovered to their state as it existed at exact same time, however they should be recovered to a state which is acceptable for the overall application. Having the clocks of nodes synchronized will also help the DBAs to identify points-of-recovery across nodes - by being able to correlate events easily.

Being able to backup a smaller shard instead of the whole dataset provides some opportunities both from technical and logical perspective. Since the size of each shard may be relatively small, a particular backup method may be acceptable even though it would not have been acceptable if the whole dataset was in one monolithic database. If data was distributed among shards using some external criteria (e.g. users of each zip code go to a particular shard), then backup images of each shard may be individually usable by an application. ZRM creates portable backup images - a key need for backing up shards - so backups from one node can be restored on another.

If recovery from a site wide disaster is also an objective, then suitable backup images need to be securely transported to the remote site. This can be done via the new Disaster Recovery Option now available for ZRM. This option replicates backup images, backup catalog and configuration data to the remote site - enabling full disaster recovery on an as-needed basis. Individual nodes need not be replicated, saving huge hassle and cost.

If your show is backed by a pod of dancing dolphins, a well implemented and documented backup and disaster recovery process is a good investment.

Fast Backups of MySQL Running on Amazon EC2

Saturday, January 23rd, 2010

If you are running your MySQL databases on the Amazon EC2 compute cloud, Zmanda Recovery Manager (ZRM) for MySQL can perform fast full backups of these databases by using Elastic Block Store (EBS) Snapshots. ZRM takes only a momentary read lock on the MySQL database during the creation of the snapshot, in order to ensure consistency of the backed up database archive. MySQL Backups using Amazon EBS snapshots are differential backups, meaning that only the blocks that have changed since your last full backup (via EBS snapshot) will be saved. For example, if you have a database with 100 GBs of data, but only 5 GBs of data has changed since your last snapshot, only the 5 additional GBs of snapshot data will be stored back to Amazon S3 during the current full backup run.

EC2 to S3 mysql backup diagram

ZRM automatically deletes EBS snapshots (containing full backups of MySQL) according to the configured retention policy. Just like other snapshot based full backups, ZRM intelligently correlates EBS Snapshots with incremental backups using MySQL logs, enabling you to recover your MySQL instances running on EC2 to any point in time.

Backups made using EBS snapshots can be recovered on the original EC2 instance or on a new EC2 instance. This also provides a quick and convenient mechanism to instantiate new MySQL database servers based on the database state from a desired point-in-time.

ZRM can run on the same EC2 instance as the MySQL database. On the other hand, if you have multiple EC2 instances with MySQL databases, you can run ZRM on one centralized EC2 instance dedicated for backup purposes. In this case, backup configuration and management for all MySQL databases is performed via Zmanda Management Console from this centralized backup server.

We have created an Amazon Machine Image (AMI) with ZRM pre-configured. This makes implementation of a MySQL backup solution on the cloud even simpler. We have used the “EC2 Small Instance” - which is powerful enough to backup most MySQL workloads in the cloud. This also makes it a very cost-effective option. This AMI is available to all ZRM customers, as part of the ZRM Enterprise subscription. You will need to create your own Amazon EC2 account, and pay standard per hour price to Amazon to run an instance based on this AMI. Note that you can configure your backup server instance to run only during the backup window. So, if you are backing up your databases once a week, and your backups takes less than an hour, then you can have this instance up only during that hour. EC2 pricing is per instance-hour consumed from the time an instance is launched until it is terminated. Each partial instance-hour consumed will be billed as a full hour. In addition to the EC2 compute capacity, you will pay standard storage charges for Amazon S3 (to store EBS Snapshots).

Join us on January 28th for a webinar on MySQL Backups (hosted by Sun/MySQL). Along with an introduction to Zmanda Recovery Manager, we will also discuss backing up MySQL applications on the cloud, and demonstrate the new ZRM AMI.

Transitioning to the Cloud: Implications for Reliability, Redundancy and Recoverability

Wednesday, July 29th, 2009

Last week we had a lively panel discussion moderated by Dave Nielsen, founder of CloudCamp, with leading experts in cloud computing on the panel: Chander Kant (CEO, Zmanda) & Michael Crandell (CEO, RightScale). Chander and Michael shared their insights into how their customers are using cloud computing and achieving new levels of reliability and recoverability. Here is a video archive of the panel

This panel discusses how you can migrate your apps and data to the cloud in a way that’s affordable and reliable and how to integrate the cloud into your IT strategy. Also hear about real-world examples of companies using the cloud for data backup and recovery and understand the hurdles to moving to the cloud, and how you can overcome them.

 

Our price increased today. Now we are one-tenth the cost of Symantec.

Friday, July 10th, 2009

Today we increased price for the Amanda Enterprise Backup Server. The new price for our Standard subscription level is $500 per year. Our online store is a place to quickly checkout prices for all our products on a single page. This price increase was done in conjunction with release of Amanda Enterprise 3.0, which represents several man years of R&D on the backup server, including advanced media management such as D2D2T. Our subscription provides access to software and enterprise-class support.

Amanda Enterprise is used by businesses of all shapes and sizes. But a typical scenario is the following:

  • Backup Server on Linux
  • One tape library with one or two tape drives. Or VTL on a NAS device.
  • A mix of Linux & Windows servers and desktops to be protected
  • A mix of applications (e.g. Exchange) and databases (e.g. MySQL or Postgres) to be protected
  • Encryption on the server to protect data at rest

In above scenario, customers often consider NetBackup from Symantec as a potential product. Lets compare the new Amanda Enterprise pricing with NetBackup pricing.

First of all, finding prices for NetBackup for a particular configuration is a harrowing experience. There is no place on Symantec website which provides prices for all NetBackup options and features in one consolidated location. Rumor has it that the internal licensing guide for NetBackup is more than 40 pages long!

The least expensive way to buy NetBackup is one of the “Starter Packs”. Their 5 client starter pack with 1 NetBackup server and 1 tape drive license costs $3995. This price does not include any support. Maintenance is priced separately: $720 per year (similar support level to our Standard subscription). This restricts NetBackup server to “Tier 1 and Tier 2″ systems. Tiering is one of the several confusing aspects of Symantec pricing. If your backup server has four or more CPUs, you are out of luck on the Starter Packs. A la carte pricing for NetBackup server and clients is significantly more expensive. A standard NetBackup server for a Tier 3 Linux server lists at $3200 + maintenance contract. Amanda Enterprise Servers or Clients have no tiered pricing. You can choose as hefty a server as your requirements dictate and pay us the same standard price.

Encryption on the backup server is a desired option for IT managers. This protects critical data at rest (or in transit - e.g. when a backup tape is being transferred to a remote location) from unauthorized access. By encrypting on the backup server, you relieve the CPUs on production clients from the burden of encryption. With Netbackup you need to buy the Media Server Encryption Option - list price for which is $10K+ (and this does not include the maintenance cost). Encryption is a built-in feature of the Amanda Enterprise Server.

Per Library Charge: NetBackup’s pricing options for Tape Drives and Tape Libraries is at best confusing. The starter pack above gives access to one tape drive. If you want to use another tape drive in a tape library, you need to buy a Tape Library Option for $3K list price. (And again, that does not include maintenance.). You can drive as many Tape Drives and Libraries you want from a single Amanda Enterprise server.

Want to use a VTL? You need to buy NetBackup Standard Disk or Enterprise Disk option. Standard Disk option is $995 for up to 1TB of data protected. Symantec doesn’t even pass the savings of data compression to you. (On top of that, you guessed it right, that does not include maintenance.). Amanda Enterprise has a built-in capability of transforming disk into virtual tapes. You can also use a VTL of your choice, with no additional cost.

This recent blog gives more color on NetBackup Licensing (Note: We don’t have any affiliation with this blog or its author).

In summary, a $500 Standard subscription for the Amanda Enterprise gives you a backup server which runs on any Tier server, including a Linux or a Solaris server, which can backup to as many tape drives and libraries you want - including VTLs, with server side encryption support, unlimited disk based backup, and vaulting (D2D2T) support. You will not get anything close to that for $5K from Symantec.

How are we able to do this? Our open source development, marketing and distribution model allows us to innovate aggressively with a huge community of developers and users providing extensive feedback on a regular basis. Proprietary software companies spend 60% to 80% of their budget on sales and marketing (Source: Time for a New Software Model), and these costs are passed on to the customers. We have a different equation. Our freely downloadable community editions represent the bulk of our marketing budget. This marketing budget is spent in the form of R&D to innovate and add features and usability to both our community and enterprise editions. So, instead of paying for sales and marketing overhead of proprietary backup software companies, you only pay for the R&D which provides direct benefit to you.

Symantec’s excessive use of tiering, options, different licensing models, components, and packs can be mind numbing for a hapless IT manager just looking to protect their systems and applications. This Doonesbury strip captures Symantec’s nickeling-and-diming pricing strategy for their backup software: