MySQL Enterprise Backup User's Guide (Version 4.1.5)

MySQL Enterprise Backup User's Guide (Version

4.1.4)

Abstract

This is the user manual for MySQL Enterprise Backup, a commercially licensed backup utility for MySQL databases.

It explains the different kinds of backup and restore that can be performed with MySQL Enterprise Backup, and

describes the commands for performing them. Strategies for optimizing backup and restore operations are also

discussed.

For notes detailing the changes in each release, see the MySQL Enterprise Backup 4.1 Release Notes.

For legal information, including licensing information, see the Preface and Legal Notices.

For help with using MySQL, please visit the MySQL Forums, where you can discuss your issues with other MySQL

users.

Document generated on: 2021-01-29 (revision: 8273)

Table of Contents

Preface and Legal Notices ................................................................................................................. ix

I Getting Started with MySQL Enterprise Backup ................................................................................. 1

1 Introduction to MySQL Enterprise Backup ................................................................................ 5

1.1 The mysqlbackup Client ................................................................................................ 5

1.2 Overview of Backup Types ........................................................................................... 5

1.3 Files that Are Backed Up .............................................................................................. 7

1.3.1 Types of Files Contained in a Backup ................................................................ 7

1.3.2 Files Backed up for InnoDB Data ..................................................................... 19

1.3.3 Files Backed up for Data Stored with MyISAM and Other Storage Engines .......... 20

1.3.4 Files Generated by mysqlbackup ...................................................................... 20

2 Installing MySQL Enterprise Backup ....................................................................................... 21

3 What's New in MySQL Enterprise Backup 4.1? ....................................................................... 23

II Using MySQL Enterprise Backup ................................................................................................... 25

4 Backing Up a Database Server .............................................................................................. 29

4.1 Before the First Backup .............................................................................................. 29

4.1.1 Collect Database Information ............................................................................ 29

4.1.2 Grant MySQL Privileges to Backup Administrator .............................................. 31

4.1.3 Designate a Location for the Backup Directory .................................................. 32

4.2 The Typical Backup / Verify / Restore Cycle ................................................................. 32

4.2.1 OS User for Running mysqlbackup ................................................................... 33

4.2.2 Backing Up an Entire MySQL Instance ............................................................. 33

4.2.3 Verifying a Backup ........................................................................................... 36

4.2.4 Restoring a Database ...................................................................................... 37

4.3 Backup Scenarios and Examples ................................................................................ 40

4.3.1 Making a Single-File Backup ............................................................................ 40

4.3.2 Making a Full Backup ...................................................................................... 45

4.3.3 Making a Differential or Incremental Backup ...................................................... 46

4.3.4 Making a Compressed Backup ......................................................................... 51

4.3.5 Making a Partial Backup .................................................................................. 52

4.3.6 Making an Optimistic Backup ........................................................................... 56

4.3.7 Making a Back Up of In-Memory Database Data ............................................... 59

4.3.8 Making Scheduled Backups ............................................................................. 59

4.4 Making Backups with a Distributed File System (DFS) or Storage Access Network

(SAN) ............................................................................................................................... 60

5 Recovering or Restoring a Database ...................................................................................... 61

5.1 Performing a Restore Operation .................................................................................. 61

5.1.1 Restoring a Compressed Backup ...................................................................... 62

5.1.2 Restoring an Encrypted Backup Image ............................................................. 63

5.1.3 Restoring an Incremental Backup ..................................................................... 63

5.1.4 Restoring Backups Created with the --use-tts Option .................................... 64

5.1.5 Restoring a Backup from Cloud Storage to a MySQL Server .............................. 65

5.1.6 Restoring External InnoDB Tablespaces to Different Locations ........................... 66

5.1.7 Advanced: Preparing and Restoring a Directory Backup ..................................... 66

5.2 Point-in-Time Recovery ............................................................................................... 67

5.3 Restoring a Backup with a Database Upgrade or Downgrade ........................................ 69

6 Working with Encrypted InnoDB Tables .................................................................................. 71

7 Using MySQL Enterprise Backup with Replication ................................................................... 75

7.1 Setting Up a New Replica ........................................................................................... 75

7.2 Backing up and Restoring a Replica Database ............................................................. 77

7.3 Restoring a Source Database ..................................................................................... 78

8 Using MySQL Enterprise Backup with Group Replication ......................................................... 81

iii

MySQL Enterprise Backup User's Guide (Version 4.1.4)

9 Encryption for Backups .......................................................................................................... 83

10 Using MySQL Enterprise Backup with Media Management Software (MMS) Products .............. 85

10.1 Backing Up to Tape with Oracle Secure Backup ........................................................ 85

11 Performance Considerations for MySQL Enterprise Backup ................................................... 89

11.1 Optimizing Backup Performance ................................................................................ 89

11.2 Optimizing Restore Performance ............................................................................... 92

12 Monitoring Backups with MySQL Enterprise Monitor .............................................................. 95

13 Troubleshooting for MySQL Enterprise Backup ..................................................................... 97

13.1 Exit codes of MySQL Enterprise Backup .................................................................... 97

13.2 Working Around Corruption Problems ........................................................................ 98

13.3 Using the MySQL Enterprise Backup Logs ................................................................. 99

13.4 Using the MySQL Enterprise Backup Manifest .......................................................... 101

III mysqlbackup Command Reference ........................................................................................... 103

14 mysqlbackup ................................................................................................................... 107

15 mysqlbackup commands ................................................................................................. 109

15.1 Backup Operations ................................................................................................. 109

15.2 Update Operations .................................................................................................. 110

15.3 Restore Operations ................................................................................................. 111

15.4 Validation Operations .............................................................................................. 114

15.5 Other Single-File Backup Operations ....................................................................... 115

15.6 Other Operations .................................................................................................... 117

16 mysqlbackup Command-Line Options .............................................................................. 119

16.1 Standard Options .................................................................................................... 125

16.2 Connection Options ................................................................................................. 127

16.3 Server Repository Options ....................................................................................... 129

16.4 Backup Repository Options ..................................................................................... 132

16.5 Metadata Options .................................................................................................... 137

16.6 Compression Options .............................................................................................. 137

16.7 Incremental Backup Options .................................................................................... 139

16.8 Partial Backup and Restore Options ........................................................................ 141

16.9 Single-File Backup Options ..................................................................................... 148

16.10 Performance / Scalability / Capacity Options ........................................................... 150

16.11 Message Logging Options ..................................................................................... 158

16.12 Progress Report Options ....................................................................................... 159

16.13 Encryption Options ................................................................................................ 163

16.14 Options for Working with Encrypted InnoDB Tablespaces ........................................ 163

16.15 Cloud Storage Options .......................................................................................... 165

16.16 Options for Special Backup Types ......................................................................... 168

16.17 Other Options ....................................................................................................... 170

17 Configuration Files and Parameters .................................................................................... 173

IV Appendixes ................................................................................................................................ 175

A Frequently Asked Questions for MySQL Enterprise Backup ................................................... 179

B Limitations of MySQL Enterprise Backup .............................................................................. 181

C Compatibility Information for MySQL Enterprise Backup ........................................................ 183

C.1 Supported Platforms ................................................................................................. 183

C.2 Cross-Platform Compatibility ..................................................................................... 183

C.3 Compatibility with MySQL Versions ........................................................................... 183

C.4 Compatibility with Older MySQL Enterprise Backup .................................................... 183

D Backup History Table Update .............................................................................................. 185

MySQL Enterprise Backup Glossary ........................................................................................ 187

Index .............................................................................................................................................. 201

List of Tables

1.1 Types of Files in a Backup ........................................................................................................... 7

4.1 Information Needed to Back Up a Database ................................................................................ 29

13.1 MySQL Enterprise Backup Exit Codes and Messages ................................................................ 97

16.1 List of All Options ................................................................................................................... 119

List of Examples

4.1 Single-File Backup to Absolute Path ............................................................................................ 41

4.2 Single-File Backup to Relative Path ............................................................................................. 41

4.3 Single-File Backup to Standard Output ........................................................................................ 41

4.4 Convert Existing Backup Directory to Single Image ...................................................................... 41

4.5 Extract Existing Image to Backup Directory .................................................................................. 41

4.6 List Single-File Backup Contents ................................................................................................. 41

4.7 Validate a Single-File Backup ..................................................................................................... 41

4.8 Extract Single-File Backup into Current Directory ......................................................................... 41

4.9 Extract Single-File Backup into a Backup Directory ...................................................................... 42

4.10 Selective Extract of Single File .................................................................................................. 42

4.11 Selective Extract of Single Directory .......................................................................................... 42

4.12 Dealing with Absolute Path Names ............................................................................................ 42

4.13 Single-File Backup to a Remote Host ........................................................................................ 43

4.14 Single-file Backup to a Remote MySQL Server .......................................................................... 43

4.15 Stream a Backup Directory to a Remote MySQL Server ............................................................. 43

4.16 Creating a Cloud Backup on Oracle Cloud Infrastructure Object Storage Classic .......................... 44

4.17 Creating a Cloud Backup on Oracle Cloud Infrastructure Object Storage ..................................... 44

4.18 Creating a Cloud Incremental Backup on Oracle Cloud Infrastructure Object Storage .................... 44

4.19 Creating a Cloud Backup on an OpenStack Object Storage ........................................................ 44

4.20 Creating a Cloud Backup on Amazon S3 ................................................................................... 45

4.21 Extract an Existing Image from an Oracle Cloud Infrastructure Object Storage Classic Container

to a Backup Directory ....................................................................................................................... 45

4.22 Extract an Existing Image from Amazon S3 Cloud Storage to a Backup Directory ......................... 45

4.23 Making an Uncompressed Partial Backup of InnoDB Tables ....................................................... 55

4.24 Making a Compressed Partial Backup ....................................................................................... 55

4.25 Optimistic Backup Using the Option optimistic-time=YYMMDDHHMMSS .................................. 57

4.26 Optimistic Backup Using the Option optimistic-time=now .................................................... 57

4.27 Optimistic Backup Using the optimistic-busy-tables Option .............................................. 57

4.28 Optimistic and Partial Backup Using both the optimistic-busy-tables and optimistic-

time Options ................................................................................................................................... 57

5.1 Restoring a Database ................................................................................................................. 61

5.2 Restoring a Compressed Backup ................................................................................................ 62

5.3 Restoring a Compressed Directory Backup .................................................................................. 63

5.4 Restoring a Compressed and Prepared Directory Backup ............................................................. 63

5.5 Restoring an Encrypted Backup Image ........................................................................................ 63

5.6 Restoring an Incremental Backup Image ..................................................................................... 63

5.7 Restoring Selected Tables from a TTS Image Backup .................................................................. 65

5.8 Restoring and Renaming a Table from a TTS Backup .................................................................. 65

5.9 Restoring a Single-file Backup from an Oracle Cloud Infrastructure (OCI) Object Storage Classic

Container to a MySQL Server ........................................................................................................... 65

5.10 Restoring a Single-file Backup from an Oracle Cloud Infrastructure (OCI) Object Storage to a

MySQL Server .................................................................................................................................. 65

5.11 Restoring a Cloud Incremental Backup from an Oracle Cloud Infrastructure (OCI) Object Storage

Service to a MySQL Server .............................................................................................................. 66

5.12 Restoring a Single-file Backup from an OpenStack Object Storage to a MySQL Server ................. 66

5.13 Restoring a Single-file Backup from Amazon S3 to a MySQL Server ........................................... 66

5.14 Restoring a Backup Directory using copy-back-and-apply-log ............................................ 67

5.15 Applying the Log to a Backup ................................................................................................... 67

5.16 Applying the Log to a Compressed Backup ............................................................................... 67

10.1 Sample mysqlbackup Commands Using MySQL Enterprise Backup with Oracle Secure

Backup ............................................................................................................................................. 87

vii

MySQL Enterprise Backup User's Guide (Version 4.1.4)

15.1 Apply Log to Full Backup ........................................................................................................ 111

17.1 Example backup-my.cnf file ................................................................................................. 173

viii

Preface and Legal Notices

This is the user manual for the MySQL Enterprise Backup product.

Licensing information. This product may include third-party software, used under license. See the

MySQL Enterprise Backup 4.1 License Information User Manual for licensing information, including

licensing information relating to third-party software that may be included in this MySQL Enterprise Backup

release.

Legal Notices

This software and related documentation are provided under a license agreement containing restrictions

on use and disclosure and are protected by intellectual property laws. Except as expressly permitted

in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast,

modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any

means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for

interoperability, is prohibited.

The information contained herein is subject to change without notice and is not warranted to be error-free.

If you find any errors, please report them to us in writing.

If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it

on behalf of the U.S. Government, then the following notice is applicable:

U.S. GOVERNMENT END USERS: Oracle programs (including any operating system, integrated

software, any programs embedded, installed or activated on delivered hardware, and modifications of

such programs) and Oracle computer documentation or other Oracle data delivered to or accessed by

U.S. Government end users are "commercial computer software" or "commercial computer software

documentation" pursuant to the applicable Federal Acquisition Regulation and agency-specific

supplemental regulations. As such, the use, reproduction, duplication, release, display, disclosure,

modification, preparation of derivative works, and/or adaptation of i) Oracle programs (including any

operating system, integrated software, any programs embedded, installed or activated on delivered

hardware, and modifications of such programs), ii) Oracle computer documentation and/or iii) other Oracle

data, is subject to the rights and limitations specified in the license contained in the applicable contract.

The terms governing the U.S. Government's use of Oracle cloud services are defined by the applicable

contract for such services. No other rights are granted to the U.S. Government.

This software or hardware is developed for general use in a variety of information management

applications. It is not developed or intended for use in any inherently dangerous applications, including

applications that may create a risk of personal injury. If you use this software or hardware in dangerous

applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other

measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages

caused by use of this software or hardware in dangerous applications.

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks

of their respective owners.

Intel and Intel Inside are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks

are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD,

Epyc, and the AMD logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a

registered trademark of The Open Group.

This software or hardware and documentation may provide access to or information about content,

products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and

Documentation Accessibility

expressly disclaim all warranties of any kind with respect to third-party content, products, and services

unless otherwise set forth in an applicable agreement between you and Oracle. Oracle Corporation and its

affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of

third-party content, products, or services, except as set forth in an applicable agreement between you and

Oracle.

This documentation is NOT distributed under a GPL license. Use of this documentation is subject to the

following terms:

You may create a printed copy of this documentation solely for your own personal use. Conversion to other

formats is allowed as long as the actual content is not altered or edited in any way. You shall not publish

or distribute this documentation in any form or on any media, except if you distribute the documentation in

a manner similar to how Oracle disseminates it (that is, electronically for download on a Web site with the

software) or on a CD-ROM or similar medium, provided however that the documentation is disseminated

together with the software on the same medium. Any other use, such as any dissemination of printed

copies or use of this documentation, in whole or in part, in another publication, requires the prior written

consent from an authorized representative of Oracle. Oracle and/or its affiliates reserve any and all rights

to this documentation not expressly granted above.

Documentation Accessibility

For information about Oracle's commitment to accessibility, visit the Oracle Accessibility Program website

https://www.oracle.com/corporate/accessibility/.

Access to Oracle Support for Accessibility

Oracle customers that have purchased support have access to electronic support through My Oracle

Support. For information, visit

https://www.oracle.com/corporate/accessibility/learning-support.html#support-tab.

Part I Getting Started with

MySQL Enterprise Backup

Table of Contents

1 Introduction to MySQL Enterprise Backup ........................................................................................ 5

1.1 The mysqlbackup Client ....................................................................................................... 5

1.2 Overview of Backup Types ................................................................................................... 5

1.3 Files that Are Backed Up ...................................................................................................... 7

1.3.1 Types of Files Contained in a Backup ........................................................................ 7

1.3.2 Files Backed up for InnoDB Data ............................................................................. 19

1.3.3 Files Backed up for Data Stored with MyISAM and Other Storage Engines .................. 20

1.3.4 Files Generated by mysqlbackup .............................................................................. 20

2 Installing MySQL Enterprise Backup .............................................................................................. 21

3 What's New in MySQL Enterprise Backup 4.1? .............................................................................. 23

Chapter 1 Introduction to MySQL Enterprise Backup

Table of Contents

1.1 The mysqlbackup Client ............................................................................................................... 5

1.2 Overview of Backup Types ........................................................................................................... 5

1.3 Files that Are Backed Up .............................................................................................................. 7

1.3.1 Types of Files Contained in a Backup ................................................................................ 7

1.3.2 Files Backed up for InnoDB Data ..................................................................................... 19

1.3.3 Files Backed up for Data Stored with MyISAM and Other Storage Engines .......................... 20

1.3.4 Files Generated by mysqlbackup ...................................................................................... 20

MySQL Enterprise Backup is a backup utility for MySQL servers. It is a multi-platform, high-performance

tool, offering rich features like “hot” (online) backup, incremental and differential backup, selective backup

and restore, support for direct cloud storage backup, backup encryption and compression, and many other

valuable features.

While optimized for use with InnoDB tables, MySQL Enterprise Backup is capable of backing up and

restoring all kinds of tables created by any kinds of storage engines supported by MySQL. The parallelism

of its read and write processes (performed in independent, multiple threads) and its block-level parallelism

(different threads can read, process, or write different chunks within a single file) allow backup and restore

processes to be completed with great speed, and often with a significant performance gain when compared

to a logical backup using tools like mysqldump.

MySQL Enterprise Backup is a valuable tool for maintaining and safeguarding your MySQL data, and for

quick and reliable recovery when accidents or disasters strike. It is part of the MySQL Enterprise Edition,

available to subscribers under a commercial license.

Among other things, this manual explains:

• How to install MySQL Enterprise Backup.

• The different kinds of backups that can be performed with MySQL Enterprise Backup, how to perform

them, and some tips on choosing the right kind of backups for your system.

• How to restore backups created by MySQL Enterprise Backup.

• How to use MySQL Enterprise Backup in special situations or for special purposes (for example, setting

up replication, using Media Management Software (MMS) products, or using a Distributed File System

(DFS)).

• The mysqlbackup client, its commands and command options.

1.1 The mysqlbackup Client

All MySQL Enterprise Backup functions are executed with the mysqlbackup client. It is used for

performing different types of backup and restore operations, as well as other related tasks like backup

compression, decompression, validation, and so on.

Use of the mysqlbackup client and the related commands are explained and illustrated throughout this

manual. For detailed information on the mysqlbackup commands and command options, see Part III,

“mysqlbackup Command Reference”.

1.2 Overview of Backup Types

Kinds of backups according to the level of service disruption

When it comes to formulating your backup strategy, performance and storage space are the key

considerations. You want the backup to complete quickly, with as little CPU overhead on the database

server as possible. You also want the backup data to be compact, so you can keep multiple backups on

hand to restore at a moment's notice. Transferring the backup data to a different system should be quick

and convenient. Under such considerations, various strategies for backing up your database often give

you different advantages, for the different trade-offs you make when choosing a particular strategy. To

choose the strategy that best fits your needs, you have to understand the nature of each kind of backups

that MySQL Enterprise Backup can perform, for which this section is giving a brief overview.

Kinds of backups according to the level of service disruption

Depending on how the database operations would be disrupted during a backup, the backup is classified

as “hot,”“warm,” or “cold”:

• Very Low to Low Level of Disruption: A hot backup is a backup performed while the database is running.

This type of backups does not block normal database operations. It captures even changes that occur

while the backup is happening. Comparing to the other backup types, it causes the least disruption to

the database server, and it is a desirable backup option when you want to avoid taking your application,

web site, or web service offline. However, before a hot backup can be restored, there needs to be an

extra process of preparing the backup to make it consistent (i.e., correctly reflecting the state of the state

of the database at the time the backup was completed). See Section 5.1.7, “Advanced: Preparing and

Restoring a Directory Backup” for more explanations.

When connected to a running MySQL server, MySQL Enterprise Backup performs hot backup for

InnoDB tables.

• Medium to High Level of Disruption: A warm backup is a backup performed with the database put under

a read-only state. This type of backups blocks any write operations to the tables during the backup

process, but still allow tables to be read.

When connected to a running MySQL server, MySQL Enterprise Backup backs up all MyISAM and other

non-InnoDB tables using the warm backup technique after all InnoDB tables have already been backed

up with the hot backup method. Therefore, to back up as much data as possible during the hot backup

phase, you should designate InnoDB as the default storage engine for new tables (which has been the

default setting since MySQL 5.5), or convert existing tables to use the InnoDB storage engine.

• High to Very High Level of Disruption: A cold backup is a backup created while the database is stopped.

Not only is it most disruptive to the database service, it also renders a lot of MySQL Enterprise Backup

features unusable (for example, the ability to retrieve information about the structure of the database

automatically through a database connection, so they do not have to be supplied to MySQL Enterprise

Backup by a configuration file or command-line options).

To avoid service disruption, you would typically perform a cold backup on a replica, which can be

stopped without taking down the entire application or web site.

Kinds of backups according to whether all data, or recent changes only are

backed up

According to whether you want to include all data into your backup or only the recent changes, and

according to recent changes since when, you can perform either a full backup, a differential backup, or an

incremental backup. The three types of backups have different levels of requirements for CPU overhead

and disk space, thus are suitable for different situations:

• A full backup includes the complete data from the database (except in cases where some tables are

excluded with the partial backup options).

Compressed versus uncompressed backups

• A differential backup includes all changes to the data since the last full backup. It is faster than a full

backup, saves storage space on the database server, and saves on network traffic when the backup is

being transferred to a different server. However, it requires additional processing to make the backup

ready for restore, which you can perform on a different system to minimize CPU overhead on the

database server.

• An Incremental backup includes all changes to the data since the last backup. It offers similar

advantages over a full backup as a differential backup does, and often to a even greater extent by further

decreasing the backup size. But it might also require more preparations on a longer series of backups,

before a restore can be performed.

Compressed versus uncompressed backups

Backup compression saves you storage space and network traffic to transfer the backup data onto a

different server. Compression does add some CPU overhead, but the overhead is algorithm dependent

and it is fairly low for the default algorithm used by MySQL Enterprise Backup. Also, compression often

reduces the IO overhead by a great deal, which might shorten the restore time, especially for slower IO

devices. However, during the restore process, you need time for decompression and also storage space

for both the compressed and decompressed data at the same time. So, take into account the additional

storage space and the extra time needed during a restore when considering whether to create compressed

backups.

When streaming backup data to another server, you might want to compress the backup either on the

original server or the destination server, depending on which server has more spare CPU capacity and

how much network traffic the compression could save.

For more on techniques and tradeoffs involving backup and restore performance, see Chapter 11,

Performance Considerations for MySQL Enterprise Backup.

1.3 Files that Are Backed Up

This section explains the various types of files contained in a backup.

1.3.1 Types of Files Contained in a Backup

The following table shows the different types of files that are included in a single-file backup image or a

directory backup. In the case of a single-file backup, unpack the file into a backup directory structure using

the extract or the image-to-backup-dir command to view the files.

Table 1.1 Types of Files in a Backup

File Name, Pattern, or Extension Relation to Original Data Files Notes

ibdata* The InnoDB system tablespace,

containing multiple InnoDB tables

and associated indexes.

Because the original files might

change while the backup is in

progress, the apply-log step

applies the same changes to the

corresponding backup files.

*.ibd An InnoDB tablespace, which can

be (a) a file-per-table tablespace,

containing a single InnoDB

table and associated indexes,

or (b) a file-per-table external

tablespace located outside

of the server's data directory,

containing a single InnoDB table

Because the original files might

change while the backup is in

progress, the apply-log step

applies the same changes to the

corresponding backup files.

Types of Files Contained in a Backup

File Name, Pattern, or Extension Relation to Original Data Files Notes

and associated indexes, or (c) a

general tablespace, containing

one or more tables and their

indexes.

*.ibz Compressed form of InnoDB

data files from the MySQL data

directory.

Produced instead of .ibd files

in a compressed backup. The

ibdata* files representing

the InnoDB system tablespace

also receive this extension in a

compressed backup.

The .ibz files are uncompressed

during the apply-log, copy-

back, or copy-back-and-

apply-log step.

*.frm Hold metadata about all MySQL

tables.

The database is put into a read-

only state while these files are

copied. These files are copied

unmodified.

*.MYD MyISAM table data. The database is put into a read-

only state while these files are

copied. These files are copied

unmodified.

*.MYI MyISAM index data. The database is put into a read-

only state while these files are

copied. These files are copied

unmodified.

*.CSM Metadata for CSV tables. These files are copied unmodified.

The backup_history and

backup_progress tables

created by mysqlbackup use

the CSV format, so the backup

always includes some files with

this extension.

*.CSV Data for CSV tables. These files are copied unmodified.

The backup_history and

backup_progress tables

created by mysqlbackup use

the CSV format, so the backup

always includes some files with

this extension.

*.MRG MERGE storage engine

references to other tables.

The database is put into a read-

only state while these files are

copied. These files are copied

unmodified.

*.TRG Trigger parameters. The database is put into a read-

only state while these files are

copied. These files are copied

unmodified.

Types of Files Contained in a Backup

File Name, Pattern, or Extension Relation to Original Data Files Notes

*.TRN Trigger namespace information. The database is put into a read-

only state while these files are

copied. These files are copied

unmodified.

*.opt Database configuration

information.

The database is put into a read-

only state while these files are

copied. These files are copied

unmodified.

*.par Definitions for partitioned tables. The database is put into a read-

only state while these files are

copied. These files are copied

unmodified.

*.ARM ARCHIVE storage engine table

metadata.

The database is put into a read-

only state while these files are

copied. These files are copied

unmodified.

*.ARZ ARCHIVE storage engine table

data.

The database is put into a read-

only state while these files are

copied. These files are copied

unmodified.

backup-my.cnf Records the configuration

parameters that specify the layout

and other important information

about the MySQL data files.

The file is created during

a backup, and it contains

crucial parameters describing

the backed-up data like

innodb_data_file_path,

innodb_log_file_size,

innodb_log_files_in_group,

and so on. It might also contain

other InnoDB parameters like

innodb_data_home_dir and

innodb_undo_directory if

some of the backup repository

options were used during the

backup. mysqlbackup uses the

parameters stored in this file to

understand the structure of the

backup and to perform various

operations. You might need to

supply some of these parameters

to mysqlbackup during a restore

and to mysqld when you start the

target server if the target server

and the backup are configured

differently. See the discussion

in Section 4.2.4, “Restoring a

Database” for details.

ibbackup_ibd_files Records names of the .ibd files

and their space IDs during an

incremental backup.

This file is created during an

incremental backup. During a

restore, the information in the

file is used to delete the tables

Types of Files Contained in a Backup

File Name, Pattern, or Extension Relation to Original Data Files Notes

from the full backup that has been

removed between the time of the

full backup and the time of the

incremental backup.

ibbackup_logfile A condensed version of the

ib_logfile* files from the

MySQL data directory.

The InnoDB log files

(ib_logfile*) are fixed-size

files that are continuously updated

during the database's operation.

For backup purposes, only the

changes that are committed

while the backup is in progress

are needed. These changes are

recorded in ibbackup_logfile,

and used to re-create the

ib_logfile* files during the

apply-log phase.

ibbackup_redo_log_only Created instead of the

ibbackup_logfile for

incremental backups taken with

the --incremental-with-

redo-log-only option.

ib_logfile* Created in the backup directory by

mysqlbackup during the apply-

log phase after the initial backup.

These files are not copied from

the original data directory, but

rather re-created in the backup

directory during the apply-log

phase after the initial backup,

using the changes recorded in the

ibbackup_logfile file.

*.bl Renamed version of each .isl file

from the backed-up server.

A .isl file is created when

you specify the location of an

InnoDB table using the syntax

CREATE TABLE ... DATA

DIRECTORY = ... (see Creating

Tables Externally for details) or

when a table is put into a general

tabelspace (only for MySQL 5.7

and after; see general tablespace

for details). The .isl file acts

like a symbolic link pointing to

the tablespace file. The .bl files

might or might not be turned

back into .isl files during the

copy-back or copy-back-

and-apply-log operation. If the

specified directory does not exist

on the server where the backup is

restored, mysqlbackup attempts

to create it. If the directory cannot

be created, the restore operation

fails. Thus, if you would like to use

the DATA DIRECTORY clause to

Types of Files Contained in a Backup

File Name, Pattern, or Extension Relation to Original Data Files Notes

put tables at different locations

or to restore to a server with

a different file structure where

the corresponding directories

cannot be created, edit the .bl

files before restoring to point to

directories that do exist on the

destination server.

When restoring a backup created

with transportable tablespaces

(TTS), if the directory on the target

server pointed to by a .bl file

already contains .ibd files, the --

force option is required when

you restore the backup.

Timestamped directory, such as

2011-05-26_13-42-02

Created by the --with-

timestamp option. All the backup

files go inside this subdirectory.

Use the --with-timestamp

option to easily keep more than

one set of backup data under the

same main backup directory.

datadir directory A subdirectory that stores the data

files and database subdirectories

from the original MySQL instance.

Created under the backup

directory by mysqlbackup.

binary log files Binary log files from the server,

which are included in a backup by

default (except when the backup

is created with the --use-tts

option). They allow a snapshot

of the server to be taken, so a

server can be cloned to its exact

state. Using a full backup as a

basis, the binary log files that

are included with an incremental

backup can be used for a point-

in-time recovery (PITR), which

restores a database to its state

at a certain point in time after the

last full backup. See Section 5.2,

“Point-in-Time Recovery” for

details.

Saved under the datadir

directory inside the backup. A

copy of the index file on the

MySQL server that lists all the

used binary log files, with the

locations of the binary log files

properly updated to point to the

files' locations in the backup, is

included into the backup as well

also under the datadir directory.

Use the --skip-binlog option

to exclude the binary log from the

backup.

For offline backups, use the --

log-bin-index option to specify

the absolute path of the index file

on the MySQL server that lists

all the used binary log files, if it is

different from the default value of

the option, for mysqlbackup to

find the binary log files and include

them in the backups.

For release 4.1.2 and later: By

default, the binary log files and

the index file are restored to

the same locations they were

Types of Files Contained in a Backup

File Name, Pattern, or Extension Relation to Original Data Files Notes

found on the backed-up server.

Use the --log-bin option to

specify a different target location

for the binary log. Use the --

skip-binlog option to skip the

restoring of the binary log.

For release 4.1.1 and earlier: The

binary log files and the index file

are restored to the data directory

of the restored server. Use the --

skip-binlog option to skip the

restoring of the binary log.

The binary log files are

compressed and saved with

the .bz extension when being

included in a compressed backup.

Notes

• If any

binary

log files

are

missing

on the

server

you are

backing

up, you

should

use

the --

skip-

binlog

option

to avoid

mysqlbackup

throwing

an error

for the

missing

files.

• No

binary

log files

are

copied

into the

incremental

backup

Types of Files Contained in a Backup

File Name, Pattern, or Extension Relation to Original Data Files Notes

if the --

use-

tts

option or

the --

start-

lsn

option is

used. To

include

binary

log files

for the

period

covered

by the

incremental

backup,

do not

use the

--use-

tts

option

and,

instead

of --

start-

lsn, use

the --

incremental-

base

option,

which

provides

the

necessary

information

for

mysqlbackup

ensure

that

no gap

exists

between

binary

log data

included

in the

backup

and the

Types of Files Contained in a Backup

File Name, Pattern, or Extension Relation to Original Data Files Notes

current

incremental

backup.

relay log files Relay log files from a replica

server, which are included in a

backup of a replica server by

default (except when the backup

is created with the --use-tts

option). Their inclusion saves the

time and resources required for

fetching the relay logs from the

source when the replica is being

restored.

Saved under the datadir

directory under the backup

directory. A copy of the index

file on the replica server that

lists all the used relay log files,

with the locations of the relay

log files properly updated to

point to the files' locations in the

backup directory, is included

into the backup as well, under

the datadir directory. Use the

--skip-relaylog option to

exclude the relay log from the

backup.

For offline backup, use the --

relay-log-index option to

specify the absolute path of the

index file on the MySQL server

that lists all the used relay log

files, if it is different from the

default value of the option, for

mysqlbackup to find the relay

log files and include them in the

backups.

For release 4.1.2 and later: By

default, the relay log files and

the index file are restored to the

same locations they were found

on the backed-up replica server.

Use the --relay-log option to

specify a different target location

for the relay log. Use the --

skip-relaylog option to skip

the restoring of the relay log.

For release 4.1.1 and earlier: The

relay log files and the index file

are restored to the data directory

of the restored server. Use the --

skip-relaylog option to skip

the restoring of the relay log.

The relay log files are compressed

and saved with the .bz extension

when being included in a

compressed backup.

Types of Files Contained in a Backup

File Name, Pattern, or Extension Relation to Original Data Files Notes

*.bz Compressed binary log or relay

log files.

The binary log and relay log files

are compressed and saved with

the .bz extension when being

included in a compressed backup.

They are decompressed during a

restore.

*.bkt (for release 4.1.0, or release

4.1.1 and later working with

MySQL 5.7.20 and earlier)

Transfer file created for an

encrypted InnoDB table during

backup .

It contains the reencrypted

tablespace key and other

information related to the

encryption. See Chapter 6,

Working with Encrypted InnoDB

Tables for detail.

encrypted keyring data file (for

release 4.1.1 and later working

with MySLQ 5.7.21 and later)

For a server using the

keyring_encrypted_file

plugin, the file specified by the

keyring_encrypted_file_data

option on the server is copied over

into the backup with its original

name under the meta folder.

For a server using the a

keyring plugin other than

keyring_encrypted_file,

the file is named keyring_kef,

saved under the meta folder.

An encrypted file containing the

master key for InnoDB table

encryption . See Chapter 6,

Working with Encrypted InnoDB

Tables for detail.

replication metadata repository

files

Usually named master.info

and relay-log.info, they are

included by default in a backup of

a replica database in a replication

setup. See Replication Metadata

Repositories, for details.

Saved under the datadir

directory under the backup

directory. For an offline backup,

use the--master-info-file

and --relaylog-info-file

options to specify the absolute

paths of the information files,

if they are different from the

default values of the options, for

mysqlbackup to find those files

and include them in the backups.

The copying of these files are

skipped during a backup or

a restore when the --skip-

relay-log option is used.

Backup image file A single-file backup produced by

the backup-to-image option,

with a name specified by the --

backup-image option.

You can move the image file

without losing or damaging the

contents inside it, then unpack

it with mysqlbackup using

the extract command and

specifying the same image name

with the --backup-image

option. Although some extra files

such as backup-my.cnf and the

meta subdirectory are present in

Types of Files Contained in a Backup

File Name, Pattern, or Extension Relation to Original Data Files Notes

the backup directory, these files

are also included in the image

file and do not need to be moved

along with it.

Any other files in subdirectories

under the datadir directory

(that is, under backup-

dir/datadir/subdir)

Copied from the database

subdirectories under the MySQL

data directory.

By default, any unrecognized

files in subdirectories under the

MySQL data directory are copied

to the backup. To omit such files,

specify the --only-known-

file-types option.

Note

Some

limitations

apply

to this

behavior.

See the

discussion

here in

Appendix B,

Limitations

of MySQL

Enterprise

Backup.

meta directory A subdirectory that stores files

with metadata about the backup.

Created under the backup

directory by mysqlbackup. All

files listed below go inside the

meta subdirectory.

backup_variables.txt Holds important information

about the backup. For use by

mysqlbackup only.

mysqlbackup consults and

possibly updates this file during

operations after the initial backup,

such as the apply-log phase or the

restore phase.

image_files.xml Contains the list of all the files

(except itself) that are present in

the single-file backup produced

by the backup-to-image or

backup-dir-to-image options.

For details about this file, see

Section 13.4, “Using the MySQL

Enterprise Backup Manifest”.

This file is not modified at any

stage once generated.

backup_create.xml Lists the command line arguments

and environment in which the

backup was created. For details

about this file, see Section 13.4,

“Using the MySQL Enterprise

Backup Manifest”.

This file is not modified once

it is created. You can prevent

this file from being generated

by specifying the --disable-

manifest option.

Types of Files Contained in a Backup

File Name, Pattern, or Extension Relation to Original Data Files Notes

backup_content.xml Essential metadata for the files

and database definitions of the

backup data. It also contains

details of all the plugins defined

on the backed-up server, by which

users should make sure the same

plugins are defined in the same

manner on the target server for

restoration. For details about

this file, see Section 13.4, “Using

the MySQL Enterprise Backup

Manifest”.

This file is not modified once

created. You can prevent this

file from being generated by

specifying the --disable-

manifest option.

comments.txt Produced by the --comments or

--comments-file option.

The comments are specified by

you to document the purpose or

special considerations for this

backup job.

backup_gtid_executed.sql Signifies the backup came from a

server with GTIDs enabled.

GTIDs are a replication feature

in MySQL 5.6 and higher.

See Replication with Global

Transaction Identifiers for details.

When you back up a server

with GTIDs enabled using

mysqlbackup, the file named

backup_gtid_executed.sql

is created in the meta folder

under the backup directory. Edit

and execute this file after restoring

the backup data on a replica

server; see Section 7.1, “Setting

Up a New Replica” for details.

server-my.cnf Contains values of the backed-up

server's global variables that are

set to non-default values. Use this

file or server-all.cnf to start

the target server for restoration.

During a copy-back or copy-

back-and-apply-log

operation, the server repository

options values (e.g., --datadir,

--innodb_data_home_dir,

etc.) in the file are modified if

the command makes changes

to them through the command

options. However, during an

apply-incremental-backup

operation, the values already

saved in the file take precedence

and they are not modified by the

option values supplied through the

command.

Warning

When

using

the file

Types of Files Contained in a Backup

File Name, Pattern, or Extension Relation to Original Data Files Notes

to restart

the target

server,

change

parameters

like --

tmpdir,

general-

log, etc.,

and any

global

variable

that

uses an

absolute

filepath to

avoid the

accidental

usage

of the

wrong file

locations

by the

target

server.

server-all.cnf Contains values of all the global

variables of the backed-up server.

Use this file or server-my.cnf

to start the target server for

restoration.

During a copy-back or copy-

back-and-apply-log

operation, the server repository

options values (e.g., --datadir,

--innodb_data_home_dir,

etc.) in the file are modified if

the command makes changes

to them through the command

options. However, during an

apply-incremental-backup

operation, the values already

saved in the file take precedence

and they are not modified by the

option values supplied through the

command.

Warning

When

using

the file

to restart

the target

server,

change

parameters

Files Backed up for InnoDB Data

File Name, Pattern, or Extension Relation to Original Data Files Notes

like --

tmpdir,

general-

log, etc.,

and any

global

variable

that

uses an

absolute

filepath to

avoid the

accidental

usage

of the

wrong file

locations

by the

target

server.

ib_buffer_pool The file produced

on the server when

innodb_buffer_pool_dump_at_shutdown

(enabled by default on

MySQL 5.7.7 and after) or

innodb_buffer_pool_dump_now

is enabled. It holds the list of

tablespace IDs and page IDs of

the server's buffer pool.

The actual file name might be

different, as it can be configured

by the server's system variable

innodb_buffer_pool_filename"

With the default setting on

MySQL server 5.7.7 and after

(innodb_buffer_pool_load_at_startup=ON),

the target server, during start up,

is going to restore the buffer pool

state of the backed up server

using this file. See Saving and

Restoring the Buffer Pool State for

details.

1.3.2 Files Backed up for InnoDB Data

The InnoDB-related data files that are backed up include the ibdata* files (which represent the system

tablespace and possibly the data for some user tables), any .ibd files (which contains data from user tables

created with the file-per-table setting enabled), and the data extracted from the ib_logfile* files (the redo log

information representing changes that occur while the backup is running), which is stored in a new backup

file ibbackup_logfile.

If you use the compressed backup feature, the .ibd files are renamed in their compressed form to .ibz

files.

The files, as they are originally copied, form a raw backup that requires further processing before it is ready

to be restored. You then run the apply step (either as part of a copy-back-and-apply-log command

or a backup-and-apply-log command, or as a separate apply-log command), which updates

the backup files based on the changes recorded in the ibbackup_logfile file, producing a prepared

backup. At this point, the backup data corresponds to a single point in time. The files are now ready to be

restored, or for some other uses such as testing, reporting, or deployment as a replica.

Files Backed up for Data Stored with MyISAM and Other Storage Engines

To restore InnoDB tables to their original state, you must also have the corresponding .frm files along with

the backup data. Otherwise, the table definitions could be missing or outdated if someone has run ALTER

TABLE or DROP TABLE statements since the backup. By default, mysqlbackup automatically copies the

.frm files during a backup operation and restores the files during a restore operation.

Note

To avoid concurrency issues during backups of busy databases, you can use the

--only-innodb or --only-innodb-with-frm option to back up only InnoDB

tables and the associated data.

1.3.3 Files Backed up for Data Stored with MyISAM and Other Storage Engines

mysqlbackup also backs up the .MYD files, .MYI files, and the .frm files associated with the MyISAM

tables. Files with other extensions that are backed up are shown in Table 1.1, “Types of Files in a Backup”.

Note

While MySQL Enterprise Backup can back up non-InnoDB data (like MYISAM

tables), the MySQL server to be backed up must support InnoDB (i.e., the backup

process will fail if the server was started up with the --innodb=OFF or --skip-

innodb option), and the server must contain at least one InnoDB table.

MyISAM tables and these other types of files cannot be backed up in the same non-blocking way as

InnoDB tables can be. They can be backed up using the warm backup technique: changes to these tables

are prevented while they are being backed up, possibly making the database unresponsive for a time, but

no shutdown is required during the backup.

1.3.4 Files Generated by mysqlbackup

Inside the image backup file created by the backup-to-image command of mysqlbackup are some

new files that are produced during the backup process. These files are used to control later tasks such as

verifying and restoring the backup data. The files generated during the backup process include:

• meta/backup_create.xml: Lists the command line arguments and environment in which the backup

was created.

• meta/backup_content.xml: Essential metadata for the files and database definitions of the backup

data.

• backup-my.cnf: Records the crucial configuration parameters that apply to the backup. These

configuration parameters are read by mysqlbackup during operations like apply-log to determine

how the backup data is structured. These parameters are also checked during a restore operation for

their compatibility with your target server's configuration.

• server-my.cnf: Contains values of the backed-up server's global variables that are set to non-default

values.

• server-all.cnf: Contains values of all the global variables of the backed-up server.

• *.bkt: Transfer file created for an encrypted InnoDB table during backup. It contains the reencrypted

tablespace key and other information related to the encryption. See Chapter 6, Working with Encrypted

InnoDB Tables for detail.

For details about these are other files contained in the backup, see Table 1.1, “Types of Files in a Backup”.

Chapter 2 Installing MySQL Enterprise Backup

Install MySQL Enterprise Backup on each database server whose contents you intend to back up.

Typically, you perform all backup and restore operations locally, by running mysqlbackup on the same

server as the MySQL instance.

MySQL Enterprise Backup is packaged as either an archive file (.tgz, archived with tar and compressed

with gzip) or as a platform-specific installer.

Installing on Unix and Linux Systems

For all Linux and Unix systems, the product is available as a .tgz file. Unpack this file as follows:

tar xvzf package.tgz

mysqlbackup is unpacked into a subdirectory. You can either copy them into a system directory

(preserving their execute permission bits), or add to your $PATH setting the directory where you unpacked

it.

For certain Linux distributions, the product is also available as an RPM archive. When you install the RPM

using the command sudo rpm -i package_name.rpm, the mysqlbackup client is installed in the

directory /opt/mysql/meb-4.1. You must add this directory to your $PATH setting.

Installation packages for Debian and Ubuntu platforms are also available. Install the package with the

following command sudo dpkg -i package_name.deb.

Installing on Windows Systems

The product can be installed together with other MySQL products with the MySQL Installer for Windows. It

can also be installed separately with either an individual .msi installer or .zip file.

When installing with a .msi installer, specify the installation location, preferably under the same directory

where other MySQL products have been installed. Choose the option Include directory in Windows

PATH, so that you can run mysqlbackup from any directory.

When installing with a .zip file, simply unzip the file and put mysqlbackup.exe at the desired

installation location. You can add that location to the %PATH% variable, so that you can run the

mysqlbackup client from any directory.

Verify the installation by selecting the menu item Start > Programs > MySQL Enterprise Backup 4.1 >

MySQL Enterprise Backup Command Line. The menu item displays version information and opens a

command prompt for running the mysqlbackup command.

Chapter 3 What's New in MySQL Enterprise Backup 4.1?

This chapter highlights the new features in MySQL Enterprise Backup 4.1, as well as any significant

changes made to MySQL Enterprise Backup with the release of this series.

• MySQL Enterprise Backup now supports optimistic incremental backup, in which mysqlbackup scans

only those InnoDB data files that have been modified since the last backup for changed pages and then

saves them into the incremental backup. It potentially makes incremental backups faster. See Full-scan

versus Optimistic Incremental Backup for details.

• A full set of exit codes have now been implemented for MySQL Enterprise Backup. Also, a new

mysqlbackup command, print-message, returns an exit message for any given exit code supplied

with the new option --error-code. See Section 13.1, “Exit codes of MySQL Enterprise Backup” for

details.

• Apply-log operations can now be performed with multiple worker threads in parallel, which can improve

performance for the operations. The number of threads to be used can be specified with the --

process-threads option.

• MySQL Enterprise Backup now supports the --ssl-mode option, which enables you to specify the

security state of the connection to the server. It replaces the client side --ssl and --ssl-verify-

server-cert options, which are now deprecated. See the description of the --ssl-mode option in

MySQL 5.7 Reference Manual for details.

• A number of measures have been implemented to increase the performance for hot backups by

decreasing the duration of the final phase of hot backups in which the server is locked. See the MySQL

Enterprise Backup 4.1 Release Notes for details.

• A new option, --skip-final-rescan, makes mysqlbackup skip the final rescan for InnoDB tables

that are modified by DDL operations after the database has been locked near the end of a backup

operation. See the description for --skip-final-rescan for details.

• The output by mysqlbackup, which goes to the stderr stream and the message log, has now been

improved to include the timestamps and thread IDs for all steps taken by mysqlbackup, in order to

provide more information for debugging purposes.

• The backup_history table now includes the following new columns:

• start_time_utc

• end_time_utc

• consistency_time_utc

• meb_version

• server_uuid (for release 4.1.2 and later)

• For MySQL Enterprise Backup 4.1.1 and later working with MySQL Server 5.7.21 and later: Servers' use

of the keyring_encrypted_file and keyring_aws plugins is now supported by MySQL Enterprise Backup.

See Chapter 6, Working with Encrypted InnoDB Tables for details.

• For MySQL Enterprise Backup 4.1.1 and later: HTTP Basic Authentication and non-chunked transfer are

now supported for backup and restore using OpenStack Swift-compatible object storage services. See

Section 16.15, “Cloud Storage Options” for details.

• For MySQL Enterprise Backup 4.1.2 and later: OAuth is now supported for Oracle Cloud Infrastructure

Object Storage client authentication. Two new options, --cloud-storage-url and --cloud-

oauth-token, have been introduced for the purpose. See Section 16.15, “Cloud Storage Options” for

details.

• For MySQL Enterprise Backup 4.1.2 and later: The binary log for a backed-up server, instead of being

restored always to the data directory on the target server, is now restored by default to the same location

it was found on the backed-up server. It can also be restored to a different location specified with the

new --log-bin option.

• For MySQL Enterprise Backup 4.1.2 and later: The relay log for a backed-up replica server, instead of

being restored always to the data directory on the target replica server, is now restored by default to the

same location it was found on the backed-up replica server. It can also be restored to a different location

specified with the new --relay-log option.

• For MySQL Enterprise Backup 4.1.2 and later: When working with a Group Replication setup,

mysqlbackup now makes the backup history available to all members of the server group by making

sure that the backup_history table is updated on a primary node after each mysqlbackup operation.

See Chapter 8, Using MySQL Enterprise Backup with Group Replication for details, including the

resulting new user privilege requirement for mysqlbackup to connect to a server, regardless of whether

the server belongs to a Group Replication setup.

• For MySQL Enterprise Backup 4.1.2 and later: The storage engine of the mysql.backup_history

table on a backed-up server has switched from CSV to InnoDB. See here for the special user privileges

required by mysqlbackup for the mandatory table migration to take place.

• For MySQL Enterprise Backup 4.1.3 and later: In addition to the requirement that the target data

directory for a restore specified by --datadir must be non-existent or empty, mysqlbackup now

enforces the same rule for the --innodb_data_home_dir, --innodb_log_group_home_dir, and

--innodb_undo_directory options (the --force option cannot be used to override the requirement

on the three options).

• For MySQL Enterprise Backup 4.1.4 and later:

• A new option, --lock-wait-retry-count, can now be used to specify the maximum number of

retries to be attempted by mysqlbackup after the FLUSH TABLES WITH READ LOCK statement,

issued during the final stage of a backup to temporarily put the database into a read-only state, fails

due to a timeout. See the description of the option for details.

• The --uncompress option is now supported for the extract operation, so that files from a

compressed single-file backup can now be extracted and uncompressed with a single command.

Part II Using MySQL Enterprise Backup

Table of Contents

4 Backing Up a Database Server ...................................................................................................... 29

4.1 Before the First Backup ...................................................................................................... 29

4.1.1 Collect Database Information .................................................................................... 29

4.1.2 Grant MySQL Privileges to Backup Administrator ...................................................... 31

4.1.3 Designate a Location for the Backup Directory .......................................................... 32

4.2 The Typical Backup / Verify / Restore Cycle ......................................................................... 32

4.2.1 OS User for Running mysqlbackup ........................................................................... 33

4.2.2 Backing Up an Entire MySQL Instance ..................................................................... 33

4.2.3 Verifying a Backup ................................................................................................... 36

4.2.4 Restoring a Database .............................................................................................. 37

4.3 Backup Scenarios and Examples ........................................................................................ 40

4.3.1 Making a Single-File Backup .................................................................................... 40

4.3.2 Making a Full Backup .............................................................................................. 45

4.3.3 Making a Differential or Incremental Backup .............................................................. 46

4.3.4 Making a Compressed Backup ................................................................................. 51

4.3.5 Making a Partial Backup .......................................................................................... 52

4.3.6 Making an Optimistic Backup ................................................................................... 56

4.3.7 Making a Back Up of In-Memory Database Data ....................................................... 59

4.3.8 Making Scheduled Backups ..................................................................................... 59

4.4 Making Backups with a Distributed File System (DFS) or Storage Access Network (SAN) ........ 60

5 Recovering or Restoring a Database .............................................................................................. 61

5.1 Performing a Restore Operation .......................................................................................... 61

5.1.1 Restoring a Compressed Backup ............................................................................. 62

5.1.2 Restoring an Encrypted Backup Image ..................................................................... 63

5.1.3 Restoring an Incremental Backup ............................................................................. 63

5.1.4 Restoring Backups Created with the --use-tts Option ............................................ 64

5.1.5 Restoring a Backup from Cloud Storage to a MySQL Server ...................................... 65

5.1.6 Restoring External InnoDB Tablespaces to Different Locations ................................... 66

5.1.7 Advanced: Preparing and Restoring a Directory Backup ............................................ 66

5.2 Point-in-Time Recovery ....................................................................................................... 67

5.3 Restoring a Backup with a Database Upgrade or Downgrade ............................................... 69

6 Working with Encrypted InnoDB Tables .......................................................................................... 71

7 Using MySQL Enterprise Backup with Replication ........................................................................... 75

7.1 Setting Up a New Replica ................................................................................................... 75

7.2 Backing up and Restoring a Replica Database .................................................................... 77

7.3 Restoring a Source Database ............................................................................................. 78

8 Using MySQL Enterprise Backup with Group Replication ................................................................ 81

9 Encryption for Backups .................................................................................................................. 83

10 Using MySQL Enterprise Backup with Media Management Software (MMS) Products ..................... 85

10.1 Backing Up to Tape with Oracle Secure Backup ................................................................ 85

11 Performance Considerations for MySQL Enterprise Backup ........................................................... 89

11.1 Optimizing Backup Performance ........................................................................................ 89

11.2 Optimizing Restore Performance ....................................................................................... 92

12 Monitoring Backups with MySQL Enterprise Monitor ...................................................................... 95

13 Troubleshooting for MySQL Enterprise Backup ............................................................................. 97

13.1 Exit codes of MySQL Enterprise Backup ............................................................................ 97

13.2 Working Around Corruption Problems ................................................................................ 98

13.3 Using the MySQL Enterprise Backup Logs ......................................................................... 99

13.4 Using the MySQL Enterprise Backup Manifest ................................................................. 101

Chapter 4 Backing Up a Database Server

Table of Contents

4.1 Before the First Backup .............................................................................................................. 29

4.1.1 Collect Database Information ............................................................................................ 29

4.1.2 Grant MySQL Privileges to Backup Administrator .............................................................. 31

4.1.3 Designate a Location for the Backup Directory .................................................................. 32

4.2 The Typical Backup / Verify / Restore Cycle ................................................................................ 32

4.2.1 OS User for Running mysqlbackup ................................................................................... 33

4.2.2 Backing Up an Entire MySQL Instance ............................................................................. 33

4.2.3 Verifying a Backup ........................................................................................................... 36

4.2.4 Restoring a Database ...................................................................................................... 37

4.3 Backup Scenarios and Examples ................................................................................................ 40

4.3.1 Making a Single-File Backup ............................................................................................ 40

4.3.2 Making a Full Backup ...................................................................................................... 45

4.3.3 Making a Differential or Incremental Backup ..................................................................... 46

4.3.4 Making a Compressed Backup ......................................................................................... 51

4.3.5 Making a Partial Backup .................................................................................................. 52

4.3.6 Making an Optimistic Backup ........................................................................................... 56

4.3.7 Making a Back Up of In-Memory Database Data ............................................................... 59

4.3.8 Making Scheduled Backups ............................................................................................. 59

4.4 Making Backups with a Distributed File System (DFS) or Storage Access Network (SAN) ............... 60

This section explains the preparations you need for creating backups with MySQL Enterprise Backup,

the typical backup-verify-restore cycle, and the different backup scenarios for using MySQL Enterprise

Backup. It also includes sample commands and outputs, showing you how to use the mysqlbackup client

in different situations.

4.1 Before the First Backup

This section outlines some of the preparations needed before you can start working with MySQL Enterprise

Backup.

4.1.1 Collect Database Information

Before backing up a particular database server for the first time, gather some information and use it to

make some planning decisions, as outlined in the following table.

Table 4.1 Information Needed to Back Up a Database

Information to Gather Where to Find It How to Use It

Path to MySQL configuration file Default system locations,

hardcoded application default

locations, or from the --

defaults-file option in the

mysqld startup script.

The preferred way to convey

database configuration information

to mysqlbackup is to use the

--defaults-file option.

When connection and data layout

information is available from the

configuration file, you no longer

need to supply separately most of

the information listed below.

Collect Database Information

Information to Gather Where to Find It How to Use It

MySQL port MySQL configuration file or

mysqld startup script.

Used to connect to the database

instance during backup operations.

Specified via the --port option of

mysqlbackup. The specification

is not needed if the information

is available from the MySQL

configuration file. Also not needed

when doing a cold (offline) backup,

which works directly on the files

using OS-level file operations.

Path to MySQL data directory MySQL configuration file or

mysqld startup script.

Used to retrieve files from the

database instance during backup

operations, and to copy files back

to the database instance during

restore operations. Automatically

retrieved from database connection

for hot and warm backups,

and taken from the MySQL

configuration file for cold backups.

ID and password of privileged

MySQL user

You record this during

installation of your own

databases, or get it from the

DBA when backing up databases

you do not own. Not needed

when doing an offline (cold)

backup, which works directly

on the files using OS-level file

permissions. For cold backups,

you log in as an administrative

user.

Specified via the --password

option of the mysqlbackup.

Prompted at the terminal if the --

password option is present without

the password argument.

Path under which to store backup

data or metadata, temporarily or

permanently

You choose this. See

Section 4.1.3, “Designate

a Location for the Backup

Directory” for details.

In general, this directory must be

empty for mysqlbackup to write

data into it.

Owner and permission information

for backed-up files (for Linux,

Unix, and OS X systems)

In the MySQL data directory. If you perform the backup and

restore using a different OS user

than the one who runs the server,

this information might become

important. See Section 4.2.1, “OS

User for Running mysqlbackup” for

details.

Size of InnoDB redo log files Calculated from the values of the

innodb_log_file_size and

innodb_log_files_in_group

configuration variables. Use the

technique explained for the --

incremental-with-redo-

log-only option.

Only needed if you perform

incremental backups using the --

incremental-with-redo-log-

only option rather than the --

incremental option. The size

of the InnoDB redo log and the

rate of generation for redo data

dictate how often you must perform

incremental backups.

Grant MySQL Privileges to Backup Administrator

Information to Gather Where to Find It How to Use It

Rate at which redo data is

generated

Calculated from the values of the

InnoDB logical sequence number

at different points in time. Use

the technique explained for the

--incremental-with-redo-

log-only option.

Only needed if you perform

incremental backups using the --

incremental-with-redo-log-

only option rather than the --

incremental option. The size

of the InnoDB redo log and the

rate of generation for redo data

dictate how often you must perform

incremental backups.

4.1.2 Grant MySQL Privileges to Backup Administrator

For most backup operations, the mysqlbackup command connects to the MySQL server using the

credentials supplied with the --user and --password options. The specified user needs certain

privileges. You can either create a new user with a minimal set of privileges, or use an administrative

account such as root. Here are the privileges required by mysqlbackup:

• The minimum privileges for the MySQL user with which mysqlbackup connects to the server are:

• RELOAD on all databases and tables.

• CREATE, INSERT, DROP, and UPDATE on the tables mysql.backup_progress and

mysql.backup_history, and also SELECT and ALTER on mysql.backup_history.

• SUPER, to enable and disable logging, and to optimize locking in order to minimize disruption to

database processing.

• REPLICATION CLIENT, to retrieve the binary log position, which is stored with the backup.

• PROCESS, to process DDL statements with the ALGORITHM = INPLACE clause.

• SELECT on performance_schema.replication_group_members, to know whether the server

instance is part of a Group Replication setup and, if so, to gather information on the group members

(required by release 4.1.2 and later).

To create a MySQL user (mysqlbackup in this example) and set the above-mentioned privileges for the

user to connect from localhost, issue statements like the following from the mysql client program:

CREATE USER 'mysqlbackup'@'localhost' IDENTIFIED BY 'password';

GRANT RELOAD, SUPER, PROCESS ON *.* TO 'mysqlbackup'@'localhost';

GRANT CREATE, INSERT, DROP, UPDATE ON mysql.backup_progress TO 'mysqlbackup'@'localhost';

GRANT CREATE, INSERT, SELECT, DROP, UPDATE, ALTER ON mysql.backup_history

TO 'mysqlbackup'@'localhost';

GRANT REPLICATION CLIENT ON *.* TO 'mysqlbackup'@'localhost';

GRANT SELECT ON performance_schema.replication_group_members TO 'mysqlbackup'@'localhost';

• The following additional privileges are required when using MySQL Enterprise Backup 4.1.2 or later

for the first time on a MySQL Server that has been upgraded from 5.7.22 or earlier and has been

backed up by MySQL Enterprise Backup before:

• CREATE, INSERT, and DROP on mysql.backup_history_old.

• CREATE, INSERT, DROP, and ALTER on mysql.backup_history_new.

Grant these privileges by issuing these sample statements at the mysql client:

Designate a Location for the Backup Directory

GRANT CREATE, INSERT, DROP ON mysql.backup_history_old TO 'mysqlbackup'@'localhost';

GRANT CREATE, INSERT, DROP, ALTER ON mysql.backup_history_new TO 'mysqlbackup'@'localhost';

Note

If you are working with a multiprimary Group Replication setting, make sure these

privileges are granted on all primary nodes; see also Chapter 8, Using MySQL

Enterprise Backup with Group Replication.

These privileges are for the attempt to migrate the mysql.backup_history table to a newer format

(see Appendix D, Backup History Table Update for details), and they are no longer needed after the first

backup operation by MySQL Enterprise Backup 4.1.2 or later has taken place on the server, by which

point they can be revoked.

• The following additional privileges are required for using specific features of MySQL Enterprise Backup:

• For using transportable tablespaces (TTS) to back up and restore InnoDB tables:

• LOCK TABLES and SELECT for backing up tables. CREATE for restoring tables.

• DROP for dropping tables if the restore fails for some reasons.

• FILE for restoring tables in external tablespaces outside of the server's data directory.

• For creating tape backups using the System Backup to Tape (SBT) API :

• CREATE, INSERT, DROP, and UPDATE on the mysql.backup_sbt_history table

Set those additional privileges if you are using the features that require them. To set all of them, issue a

statement like the following from the mysql client program:

GRANT LOCK TABLES, SELECT, CREATE, DROP, FILE ON *.* TO 'mysqlbackup'@'localhost';

GRANT CREATE, INSERT, DROP, UPDATE ON mysql.backup_sbt_history TO 'mysqlbackup'@'localhost';

• For privileges required for using MySQL Enterprise Backup with a Group Replication setting, see

Chapter 8, Using MySQL Enterprise Backup with Group Replication.

4.1.3 Designate a Location for the Backup Directory

Most mysqlbackup operations, including those on single-file backups, write data or metadata to a

designated directory referred to as the “backup directory” in this manual. See the description for --

backup-dir for details on its usage for different operations.

Choose in advance for this directory a location on a file system with sufficient storage; it could even be

remotely mounted from a different server. You specify the path to this directory with the --backup-dir

option for many mysqlbackup commands.

If you use the backup directory as a location to store your backups, it is preferable to keep each backup

within a timestamped subdirectory underneath the main backup directory. To make mysqlbackup

create these subdirectories automatically, specify the --with-timestamp option each time you run

mysqlbackup.

4.2 The Typical Backup / Verify / Restore Cycle

To illustrate the basic steps in creating and making use of a backup, the following example shows how to

perform a full backup, verify it, and then restore it to a server.

OS User for Running mysqlbackup

4.2.1 OS User for Running mysqlbackup

For Linux and other Unix-like platforms: mysqlbackup does not record file ownership or permissions

of the files that are backed up. To ensure no file permission issues prevent a server to be backed up,

restored, and restarted successfully, it is highly recommended that you run mysqlbackup with the same

OS user who runs the MySQL server (typically mysql). If that is not possible, pay attention to the following

guidelines:

• For backups, mysqlbackup should be run by a user that can read all the server files and directories

and can execute all server directories. To satisfy that requirement, the OS user that runs mysqlbackup

should, for example, have the group owner of the server files and directories (typically mysql) as its

primary group or secondary group.

• For restores, unless mysqlbackup is run by the same user that runs the server, it can be very hard to

ensure that the server has access to all the restored server files and folders, especially in the case of an

online restore, where the server must be able to access the files immediately after they are restored. For

an offline restore, you might need to, for example, set a umask to the user before the restore and adjust

the permissions of the restored files and folders using a series of chmod and chown commands, so that

the original permissions for the backed-up files and folders are reproduced.

4.2.2 Backing Up an Entire MySQL Instance

In the following example, we back up an entire MySQL instance to a single file using the backup-to-

image command, which appears at the end of the sample command. We specify some of the connection

information for the database using the --user and --host options (and, with the --password option, tell

the MySQL server to prompt for a user password). The location and filename for the single-file backup is

specified using the --backup-image option, and the location for an empty folder to store temporary files

is supplied with the --backup-dir option.

The output echoes all the parameters used by the backup operation, including several that are retrieved

automatically using the database connection. The unique ID for this backup job is recorded in special

tables that mysqlbackup creates inside the MySQL instance, allowing you to monitor long-running

backups and view information on previous backups. The final output section repeats the location of the

backup data and provides the LSN values that you might use when you perform an incremental backup

next time over the full backup that has just been made.

$ ./mysqlbackup --user=root --password --host=127.0.0.1 --backup-image=/home/admin/backups/my.mbi \

--backup-dir=/home/admin/backup-tmp backup-to-image

MySQL Enterprise Backup version 4.1.0 Linux-2.6.39-400.215.10.el5uek-x86_64 [2017/02/13]

170214 17:38:59 MAIN INFO: A thread created with Id '140286817412928'

170214 17:38:59 MAIN INFO: Starting with following command line ...

./mysqlbackup --user=root --password --host=127.0.0.1

--backup-image=/home/admin/backups/my.mbi

--backup-dir=/home/admin/backup-tmp backup-to-image

170214 17:38:59 MAIN INFO:

Enter password:

170214 17:39:07 MAIN INFO: MySQL server version is '5.7.17'

170214 17:39:07 MAIN INFO: MySQL server compile os version is 'Linux'

170214 17:39:07 MAIN INFO: Got some server configuration information from running server.

170214 17:39:07 MAIN INFO: Server system variable 'old_alter_table' was set to '0'. Setting it to '1'.

IMPORTANT: Please check that mysqlbackup run completes successfully.

At the end of a successful 'backup-to-image' run mysqlbackup

prints "mysqlbackup completed OK!".

170214 17:39:07 MAIN INFO: MEB logfile created at /home/admin/backup-tmp/meta/MEB_2017-02-14.17-39-07_image_backup.log

Backing Up an Entire MySQL Instance

--------------------------------------------------------------------

Server Repository Options:

--------------------------------------------------------------------

datadir = /var/lib/mysql/

innodb_data_home_dir =

innodb_data_file_path = ibdata1:12M:autoextend

innodb_log_group_home_dir = /var/lib/mysql/

innodb_log_files_in_group = 2

innodb_log_file_size = 50331648

innodb_undo_directory = /var/lib/mysql/

innodb_undo_tablespaces = 0

innodb_undo_logs = 128

innodb_buffer_pool_filename = ib_buffer_pool

innodb_page_size = 16384

innodb_checksum_algorithm = crc32

--------------------------------------------------------------------

Backup Config Options:

--------------------------------------------------------------------

datadir = /home/admin/backup-tmp/datadir

innodb_data_home_dir = /home/admin/backup-tmp/datadir

innodb_data_file_path = ibdata1:12M:autoextend

innodb_log_group_home_dir = /home/admin/backup-tmp/datadir

innodb_log_files_in_group = 2

innodb_log_file_size = 50331648

innodb_undo_directory = /home/admin/backup-tmp/datadir

innodb_undo_tablespaces = 0

innodb_undo_logs = 128

innodb_buffer_pool_filename = ib_buffer_pool

innodb_page_size = 16384

innodb_checksum_algorithm = crc32

Backup Image Path = /home/admin/backups/my.mbi

170214 17:39:07 MAIN INFO: Unique generated backup id for this is 14871119474450201

170214 17:39:07 MAIN INFO: Creating 14 buffers each of size 16777216.

170214 17:39:07 MAIN INFO: Full Image Backup operation starts with following threads

1 read-threads 6 process-threads 1 write-threads

170214 17:39:07 MAIN INFO: Found checkpoint at lsn 2543123.

170214 17:39:07 MAIN INFO: Starting log scan from lsn = 2543104 at offset = 2536448 and checkpoint = 2543123 in file /var/lib/mysql/ib_logfile0.

170214 17:39:07 RDR1 INFO: A thread created with Id '140286535427840'

170214 17:39:07 RDR1 INFO: Copying meta file /home/admin/backup-tmp/backup-my.cnf.

170214 17:39:07 PCR1 INFO: A thread created with Id '140286527035136'

170214 17:39:07 PCR2 INFO: A thread created with Id '140286518642432'

170214 17:39:07 RDR1 INFO: Copying meta file /home/admin/backup-tmp/meta/backup_create.xml.

170214 17:39:07 RDR1 INFO: System tablespace file format is Antelope.

170214 17:39:07 RDR1 INFO: Starting to copy all innodb files...

170214 17:39:07 PCR3 INFO: A thread created with Id '140286510249728'

170214 17:39:07 PCR4 INFO: A thread created with Id '140286382569216'

170214 17:39:07 PCR5 INFO: A thread created with Id '140286501857024'

170214 17:39:07 PCR6 INFO: A thread created with Id '140286493464320'

170214 17:39:07 WTR1 INFO: A thread created with Id '140286485071616'

170214 17:39:07 RLR1 INFO: A thread created with Id '140286476678912'

170214 17:39:07 RLW1 INFO: A thread created with Id '140286468286208'

170214 17:39:07 RLP1 INFO: A thread created with Id '140286459893504'

170214 17:39:07 RDR1 INFO: Copying /var/lib/mysql/ibdata1 (Barracuda file format).

170214 17:39:08 RDR1 INFO: Copying /var/lib/mysql/mysql/engine_cost.ibd (Barracuda file format).

170214 17:39:08 RDR1 INFO: Copying /var/lib/mysql/mysql/gtid_executed.ibd (Barracuda file format).

170214 17:39:08 RDR1 INFO: Copying /var/lib/mysql/mysql/help_category.ibd (Barracuda file format).

170214 17:39:08 RDR1 INFO: Copying /var/lib/mysql/mysql/help_keyword.ibd (Barracuda file format).

170214 17:39:08 RDR1 INFO: Copying /var/lib/mysql/mysql/help_relation.ibd (Barracuda file format).

170214 17:39:08 RDR1 INFO: Copying /var/lib/mysql/mysql/help_topic.ibd (Barracuda file format).

170214 17:39:09 RDR1 INFO: Copying /var/lib/mysql/mysql/innodb_index_stats.ibd (Barracuda file format).

170214 17:39:10 RDR1 INFO: Copying /var/lib/mysql/mysql/innodb_table_stats.ibd (Barracuda file format).

170214 17:39:10 RDR1 INFO: Copying /var/lib/mysql/mysql/plugin.ibd (Barracuda file format).

170214 17:39:10 RDR1 INFO: Copying /var/lib/mysql/mysql/server_cost.ibd (Barracuda file format).

170214 17:39:10 RDR1 INFO: Copying /var/lib/mysql/mysql/servers.ibd (Barracuda file format).

Backing Up an Entire MySQL Instance

170214 17:39:10 RDR1 INFO: Copying /var/lib/mysql/mysql/slave_master_info.ibd (Barracuda file format).

170214 17:39:10 RDR1 INFO: Copying /var/lib/mysql/mysql/slave_relay_log_info.ibd (Barracuda file format).

170214 17:39:10 RDR1 INFO: Copying /var/lib/mysql/mysql/slave_worker_info.ibd (Barracuda file format).

170214 17:39:10 RDR1 INFO: Copying /var/lib/mysql/mysql/time_zone.ibd (Barracuda file format).

170214 17:39:10 RDR1 INFO: Copying /var/lib/mysql/mysql/time_zone_leap_second.ibd (Barracuda file format).

170214 17:39:10 RDR1 INFO: Copying /var/lib/mysql/mysql/time_zone_name.ibd (Barracuda file format).

170214 17:39:10 RDR1 INFO: Copying /var/lib/mysql/mysql/time_zone_transition.ibd (Barracuda file format).

170214 17:39:10 RDR1 INFO: Copying /var/lib/mysql/mysql/time_zone_transition_type.ibd (Barracuda file format).

170214 17:39:10 RDR1 INFO: Copying /var/lib/mysql/pets/cats.ibd (Barracuda file format).

170214 17:39:10 RDR1 INFO: Copying /var/lib/mysql/sys/sys_config.ibd (Barracuda file format).

170214 17:39:10 RDR1 INFO: Completing the copy of innodb files.

170214 17:39:10 RDR1 INFO: Requesting a dump of the InnoDB buffer pool

170214 17:39:10 RDR1 INFO: Waiting for the dump of the InnoDB buffer pool to complete

170214 17:39:11 RDR1 INFO: The dump of the InnoDB buffer pool completed

170214 17:39:11 RDR1 INFO: Could not find binlog index file. binlogs will not be copied for this backup.

Point-In-Time-Recovery will not be possible.

If this is online backup then server may not have started with --log-bin.

You may specify its location with --log-bin-index option.

170214 17:39:11 RDR1 INFO: Preparing to lock tables: Connected to mysqld server.

170214 17:39:11 RDR1 INFO: Starting to lock all the tables...

170214 17:39:11 RDR1 INFO: Flushing the default MyISAM key cache.

170214 17:39:11 RDR1 INFO: All tables are locked and flushed to disk

170214 17:39:11 RDR1 INFO: Completed the copy of binlog files...

170214 17:39:11 RDR1 INFO: Opening backup source directory '/var/lib/mysql'

170214 17:39:11 RDR1 INFO: Starting to backup all non-innodb files in

subdirectories of '/var/lib/mysql'

170214 17:39:11 WTR1 INFO: Adding database directory: datadir

170214 17:39:11 RDR1 INFO: Completing the copy of all non-innodb files.

170214 17:39:11 RDR1 INFO: Requesting completion of redo log copy. Rapid: 0

170214 17:39:11 RLR1 INFO: Signaled '1' to complete log copy.

170214 17:39:11 RLR1 INFO: Redo log reader waited = 1270.00 ms for logs to generate.

170214 17:39:11 RLW1 INFO: A copied database page was modified at 2543123.

(This is the highest lsn found on page)

Scanned log up to lsn 2543132.

Was able to parse the log up to lsn 2543132.

170214 17:39:13 RDR1 INFO: All tables unlocked

170214 17:39:13 RDR1 INFO: All MySQL tables were locked for 1.367 seconds.

170214 17:39:13 RDR1 INFO: Setting server system variable 'old_alter_table' back to '0'.

170214 17:39:13 RDR1 INFO: Reading all global variables from the server.

170214 17:39:13 RDR1 INFO: Completed reading of all global variables from the server.

170214 17:39:13 RDR1 INFO: Writing config file for server '5.7.17'.

170214 17:39:13 RDR1 INFO: Creating server config files server-my.cnf and server-all.cnf in /home/admin/backup-tmp

170214 17:39:13 RDR1 INFO: Copying meta file /home/admin/backup-tmp/meta/backup_variables.txt.

170214 17:39:13 RDR1 INFO: Copying meta file /home/admin/backup-tmp/datadir/ibbackup_logfile.

170214 17:39:13 RDR1 INFO: Copying meta file /home/admin/backup-tmp/server-all.cnf.

170214 17:39:13 RDR1 INFO: Copying meta file /home/admin/backup-tmp/server-my.cnf.

170214 17:39:13 RDR1 INFO: Copying meta file /home/admin/backup-tmp/meta/backup_content.xml.

170214 17:39:13 RDR1 INFO: Copying meta file /home/admin/backup-tmp/meta/image_files.xml.

170214 17:39:13 MAIN INFO: Full Image Backup operation completed successfully.

170214 17:39:13 MAIN INFO: Backup image created successfully.

170214 17:39:13 MAIN INFO: Image Path = /home/admin/backups/my.mbi

-------------------------------------------------------------

Parameters Summary

-------------------------------------------------------------

Start LSN : 2543104

End LSN : 2543132

-------------------------------------------------------------

mysqlbackup completed OK!

Verifying a Backup

4.2.3 Verifying a Backup

You can check the integrity of your backup using the validate command. The following is a sample

command for validating a backup image and the output for the successful validation:

$ ./mysqlbackup --backup-image=/home/admin/backups/my.mbi validate

MySQL Enterprise Backup version 4.1.0 Linux-2.6.39-400.215.10.el5uek-x86_64 [2017/02/13]

170214 17:46:09 MAIN INFO: A thread created with Id '139996701841216'

170214 17:46:09 MAIN INFO: Starting with following command line ...

./mysqlbackup --backup-image=/home/admin/backups/my.mbi validate

170214 17:46:09 MAIN INFO:

IMPORTANT: Please check that mysqlbackup run completes successfully.

At the end of a successful 'validate' run mysqlbackup

prints "mysqlbackup completed OK!".

170214 17:46:09 MAIN INFO: Backup Image MEB version string: 4.1.0 [2017/02/13]

170214 17:46:09 MAIN INFO: MySQL server version is '5.7.17'

170214 17:46:09 MAIN INFO: Creating 14 buffers each of size 16777216.

170214 17:46:09 MAIN INFO: Validate operation starts with following threads

1 read-threads 6 process-threads

170214 17:46:09 MAIN INFO: Validating image ... /home/admin/backups/my.mbi

170214 17:46:09 PCR1 INFO: A thread created with Id '139996431771392'

170214 17:46:09 PCR2 INFO: A thread created with Id '139996423378688'

170214 17:46:09 PCR3 INFO: A thread created with Id '139996414985984'

170214 17:46:09 PCR4 INFO: A thread created with Id '139996406593280'

170214 17:46:09 PCR5 INFO: A thread created with Id '139996263982848'

170214 17:46:09 PCR6 INFO: A thread created with Id '139996398200576'

170214 17:46:09 RDR1 INFO: A thread created with Id '139996440164096'

170214 17:46:09 PCR1 INFO: Validate: [Dir]: meta

170214 17:46:10 PCR4 INFO: Validate: [Dir]: datadir/mysql

170214 17:46:10 PCR2 INFO: Validate: [Dir]: datadir/performance_schema

170214 17:46:10 PCR2 INFO: Validate: [Dir]: datadir/pets

170214 17:46:10 PCR2 INFO: Validate: [Dir]: datadir/sys

170214 17:46:10 PCR2 INFO: Validate: [Dir]: datadir/mysql

170214 17:46:10 PCR2 INFO: Validate: [Dir]: datadir/performance_schema

170214 17:46:10 PCR2 INFO: Validate: [Dir]: datadir/pets

170214 17:46:10 PCR2 INFO: Validate: [Dir]: datadir/sys

170214 17:46:10 MAIN INFO: Total files as specified in image: 297

170214 17:46:10 MAIN INFO: datadir/ibdata1 validated.

170214 17:46:10 MAIN INFO: datadir/mysql/engine_cost.ibd validated.

170214 17:46:10 MAIN INFO: datadir/mysql/gtid_executed.ibd validated.

170214 17:46:10 MAIN INFO: datadir/mysql/help_category.ibd validated.

170214 17:46:10 MAIN INFO: datadir/mysql/help_keyword.ibd validated.

170214 17:46:10 MAIN INFO: datadir/mysql/help_relation.ibd validated.

170214 17:46:10 MAIN INFO: datadir/mysql/help_topic.ibd validated.

170214 17:46:10 MAIN INFO: datadir/mysql/innodb_index_stats.ibd validated.

170214 17:46:10 MAIN INFO: datadir/mysql/innodb_table_stats.ibd validated.

170214 17:46:10 MAIN INFO: datadir/mysql/plugin.ibd validated.

170214 17:46:10 MAIN INFO: datadir/mysql/server_cost.ibd validated.

170214 17:46:10 MAIN INFO: datadir/mysql/servers.ibd validated.

170214 17:46:10 MAIN INFO: datadir/mysql/slave_master_info.ibd validated.

170214 17:46:10 MAIN INFO: datadir/mysql/slave_relay_log_info.ibd validated.

170214 17:46:10 MAIN INFO: datadir/mysql/slave_worker_info.ibd validated.

170214 17:46:10 MAIN INFO: datadir/mysql/time_zone.ibd validated.

170214 17:46:10 MAIN INFO: datadir/mysql/time_zone_leap_second.ibd validated.

170214 17:46:10 MAIN INFO: datadir/mysql/time_zone_name.ibd validated.

170214 17:46:10 MAIN INFO: datadir/mysql/time_zone_transition.ibd validated.

170214 17:46:10 MAIN INFO: datadir/mysql/time_zone_transition_type.ibd validated.

170214 17:46:10 MAIN INFO: datadir/pets/cats.ibd validated.

170214 17:46:10 MAIN INFO: datadir/sys/sys_config.ibd validated.

170214 17:46:10 MAIN INFO: Validate operation completed successfully.

170214 17:46:10 MAIN INFO: Backup Image validation successful.

170214 17:46:10 MAIN INFO: Source Image Path = /home/admin/backups/my.mbi

Restoring a Database

mysqlbackup completed OK!

Furthermore, you can also verify that your backup has been successful by restoring the backup data on a

different server and run the MySQL daemon (mysqld) on the new data directory. You can then execute

SHOW statements to verify the database and table structures, and execute queries to verify further details

of the database. See Section 4.2.4, “Restoring a Database” for the basic steps for restoring a backup, and

see Chapter 5, Recovering or Restoring a Database for more detailed instructions.

Warning

Do not try to verify a backup by using the backup directory as a data directory

to start a MySQL server. This will crash the server, and might also corrupt your

backup. See Appendix A, Frequently Asked Questions for MySQL Enterprise

Backup for details.

4.2.4 Restoring a Database

To restore a MySQL instance from a backup to a database server:

• Shut down the database server.

• Delete all files inside the server's data directory. Also delete all files inside the directories specified by the

--innodb_data_home_dir, --innodb_log_group_home_dir, and --innodb_undo_directory

options for restore, if the directories are different from the data directory.

• Use, for example, the copy-back-and-apply-log command, which converts the raw backup into

a prepared backup by updating it to a consistent state, and then copies the tables, indexes, metadata,

and any other required files onto a target server. For the various options that you can specify for this

operation, see Section 15.3, “Restore Operations”.

In the example below, the single-file backup created in the example given in Section 4.2.2, “Backing Up an

Entire MySQL Instance” is restored using the copy-back-and-apply-log command. Besides the usual

connection parameters, the following options are used:

• --defaults-file supplies the configuration for restoring the data. It must be the first option to appear

in a mysqlbackup command, if ever used. In most cases, you can supply to mysqlbackup with this

option the configuration file for the target server to which you are restoring the data. However, when the

following InnoDB settings for the backup are different from those on the target server, it is important to

supply the values for the backup to mysqlbackup during restore and to mysqld when you start the

restored server (otherwise, the restore might fail, and you might have problem starting the restored

server):

• innodb_data_file_path

• innodb_log_file_size

• innodb_log_files_in_group

• innodb_page_size

• innodb_checksum_algorithm

• innodb_undo_tablespaces

• innodb_buffer_pool_filename

Restoring a Database

If you are not sure about those settings for your backup, they are stored in the backup-my.cnf file

during the backup—you can find the file either in the temporary directory you specified with --backup-

dir when you created the single-image backup, or in a backup directory you can create by unpacking

the backup image using the extract command. If the values of these options differ from those on the

target server, add them to the configuration file you are supplying to mysqlbackup and also to the

configuration file you are going to use to start the server afterwards; alternatively, you can also supply

them as command line options to mysqlbackup and mysqld.

For some of the options listed above (namely, innodb_data_file_path, innodb_log_file_size,

innodb_log_files_in_group, and innodb_undo_tablespaces), mysqlbackup checks the

values you supply for them to ensure that you will be able to start the target server afterwards with those

values: it throws an error if any of them does not match with the actual values for the backup. Warnings

are given if those values are not specified for mysqlbackup in either the configuration file or on the

command line (which is the case in the example below).

• --datadir supplies the location of the data directory for restoring the data. You must specify this option

for any restore operation.

• --backup-image provides the path of the single-file backup.

• --backup-dir provides the location of an empty folder to store some temporary files created during

the restore procedure.

$ ./mysqlbackup --defaults-file=/etc/mysql/my.cnf --datadir=/var/lib/mysql \

--backup-image=/home/admin/backups/my.mbi --backup-dir=/home/admin/restore-tmp copy-back-and-apply-log

MySQL Enterprise Backup version 4.1.0 Linux-2.6.39-400.215.10.el5uek-x86_64 [2017/02/13]

170214 22:11:38 MAIN INFO: A thread created with Id '140027243398976'

170214 22:11:38 MAIN INFO: Starting with following command line ...

./mysqlbackup --defaults-file=/etc/mysql/my.cnf --datadir=/var/lib/mysql

--backup-image=/home/admin/backups/my.mbi

--backup-dir=/home/admin/restore-tmp copy-back-and-apply-log

170214 22:11:38 MAIN INFO:

IMPORTANT: Please check that mysqlbackup run completes successfully.

At the end of a successful 'copy-back-and-apply-log' run mysqlbackup

prints "mysqlbackup completed OK!".

170214 22:11:38 MAIN INFO: Backup Image MEB version string: 4.1.0 [2017/02/13]

170214 22:11:38 MAIN INFO: MySQL server version is '5.7.17'

170214 22:11:38 MAIN WARNING: If you restore to a server of a different version, the innodb_data_file_path parameter might have a different default. In

that case you need to add 'innodb_data_file_path=ibdata1:12M:autoextend' to the target server configuration.

170214 22:11:38 MAIN WARNING: If you restore to a server of a different version, the innodb_log_files_in_group parameter might have a different default.

In that case you need to add 'innodb_log_files_in_group=2' to the target server configuration.

170214 22:11:38 MAIN WARNING: If you restore to a server of a different version, the innodb_log_file_size parameter might have a different default. In th

at case you need to add 'innodb_log_file_size=50331648' to the target server configuration.

170214 22:11:38 MAIN INFO: Keyring service not initialized. MEB cannot validate the encryption keys.

170214 22:11:38 MAIN INFO: Valid keyring to be specified on server start to access the encrypted tables.

170214 22:11:38 MAIN INFO: MEB logfile created at /home/admin/restore-tmp/meta/MEB_2017-02-14.22-11-38_copy_back_img_to_datadir.log

--------------------------------------------------------------------

Server Repository Options:

--------------------------------------------------------------------

datadir = /var/lib/mysql

innodb_data_home_dir = /var/lib/mysql

innodb_data_file_path = ibdata1:12M:autoextend

innodb_log_group_home_dir = /var/lib/mysql

innodb_log_files_in_group = 2

innodb_log_file_size = 50331648

innodb_buffer_pool_filename = ib_buffer_pool

Restoring a Database

innodb_page_size = Null

innodb_checksum_algorithm = crc32

--------------------------------------------------------------------

Backup Config Options:

--------------------------------------------------------------------

datadir = /home/admin/restore-tmp/datadir

innodb_data_home_dir = /home/admin/restore-tmp/datadir

innodb_data_file_path = ibdata1:12M:autoextend

innodb_log_group_home_dir = /home/admin/restore-tmp/datadir

innodb_log_files_in_group = 2

innodb_log_file_size = 50331648

innodb_buffer_pool_filename = ib_buffer_pool

innodb_page_size = 16384

innodb_checksum_algorithm = crc32

170214 22:11:38 MAIN INFO: Creating 14 buffers each of size 16777216.

170214 22:11:38 MAIN INFO: Copy-back-and-apply-log operation starts with following threads

1 read-threads 6 process-threads 1 write-threads

170214 22:11:38 RDR1 INFO: A thread created with Id '140026981721856'

170214 22:11:38 PCR1 INFO: A thread created with Id '140026973329152'

170214 22:11:38 PCR1 INFO: Copying database directory: meta

170214 22:11:38 PCR2 INFO: A thread created with Id '140026964936448'

170214 22:11:38 PCR3 INFO: A thread created with Id '140026956543744'

170214 22:11:38 PCR4 INFO: A thread created with Id '140026948151040'

170214 22:11:38 PCR5 INFO: A thread created with Id '140026939758336'

170214 22:11:38 PCR6 INFO: A thread created with Id '140026931365632'

170214 22:11:38 WTR1 INFO: A thread created with Id '140026922972928'

170214 22:11:38 RDR1 INFO: Copying ibdata1.

170214 22:11:38 RDR1 INFO: Copying mysql/engine_cost.ibd.

170214 22:11:38 RDR1 INFO: Copying mysql/gtid_executed.ibd.

170214 22:11:38 RDR1 INFO: Copying mysql/help_category.ibd.

170214 22:11:38 RDR1 INFO: Copying mysql/help_keyword.ibd.

170214 22:11:38 RDR1 INFO: Copying mysql/help_relation.ibd.

170214 22:11:38 RDR1 INFO: Copying mysql/help_topic.ibd.

170214 22:11:38 PCR2 INFO: Copying database directory: mysql

170214 22:11:39 RDR1 INFO: Copying mysql/innodb_index_stats.ibd.

170214 22:11:39 RDR1 INFO: Copying mysql/innodb_table_stats.ibd.

170214 22:11:39 RDR1 INFO: Copying mysql/plugin.ibd.

170214 22:11:39 RDR1 INFO: Copying mysql/server_cost.ibd.

170214 22:11:39 RDR1 INFO: Copying mysql/servers.ibd.

170214 22:11:39 RDR1 INFO: Copying mysql/slave_master_info.ibd.

170214 22:11:39 RDR1 INFO: Copying mysql/slave_relay_log_info.ibd.

170214 22:11:39 RDR1 INFO: Copying mysql/slave_worker_info.ibd.

170214 22:11:39 RDR1 INFO: Copying mysql/time_zone.ibd.

170214 22:11:39 RDR1 INFO: Copying mysql/time_zone_leap_second.ibd.

170214 22:11:39 RDR1 INFO: Copying mysql/time_zone_name.ibd.

170214 22:11:39 RDR1 INFO: Copying mysql/time_zone_transition.ibd.

170214 22:11:39 RDR1 INFO: Copying mysql/time_zone_transition_type.ibd.

170214 22:11:40 PCR5 INFO: Copying database directory: performance_schema

170214 22:11:40 PCR5 INFO: Copying database directory: pets

170214 22:11:40 RDR1 INFO: Copying pets/cats.ibd.

170214 22:11:40 PCR5 INFO: Copying database directory: sys

170214 22:11:40 RDR1 INFO: Copying sys/sys_config.ibd.

170214 22:11:40 PCR5 INFO: Copying database directory: mysql

170214 22:11:40 PCR5 INFO: Copying database directory: performance_schema

170214 22:11:40 PCR5 INFO: Copying database directory: pets

170214 22:11:40 PCR5 INFO: Copying database directory: sys

170214 22:11:40 MAIN INFO: Total files as specified in image: 297

170214 22:11:40 MAIN INFO: MySQL server version is '5.7.17'

170214 22:11:40 MAIN INFO: MySQL server compile os version is 'Linux'

170214 22:11:40 MAIN INFO: Writing config file for server '5.7.17'.

170214 22:11:40 MAIN INFO: Creating server config files server-my.cnf and server-all.cnf in /var/lib/mysql

[Some lines of output not shown]

170214 22:11:40 MAIN INFO: Copy-back operation completed successfully.

Backup Scenarios and Examples

170214 22:11:40 MAIN INFO: Source Image Path = /home/admin/backups/my.mbi

170214 22:11:40 MAIN INFO: MySQL server version is '5.7.17'

170214 22:11:40 MAIN INFO: Restoring ...5.7.17 version

170214 22:11:40 MAIN INFO: Creating 14 buffers each of size 65536.

170214 22:11:40 MAIN INFO: Apply-log operation starts with following threads

1 read-threads 1 process-threads 6 apply-threads

170214 22:11:40 MAIN INFO: Using up to 100 MB of memory.

170214 22:11:40 MAIN INFO: ibbackup_logfile's creation parameters:

start lsn 2543104, end lsn 2543132,

start checkpoint 2543123.

170214 22:11:40 ALW1 INFO: A thread created with Id '140026922972928'

170214 22:11:40 ALW6 INFO: A thread created with Id '140027204466432'

170214 22:11:40 ALW5 INFO: A thread created with Id '140027212859136'

170214 22:11:40 ALW4 INFO: A thread created with Id '140026948151040'

170214 22:11:40 ALW3 INFO: A thread created with Id '140026939758336'

170214 22:11:40 ALW2 INFO: A thread created with Id '140026931365632'

170214 22:11:40 RDR1 INFO: A thread created with Id '140027196073728'

170214 22:11:40 PCR1 INFO: A thread created with Id '140027187681024'

170214 22:11:40 PCR1 INFO: InnoDB: Doing recovery: scanned up to log sequence number 2543132.

170214 22:11:40 PCR1 INFO: InnoDB: Starting an apply batch of log records to the database...

InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43

44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93

94 95 96 97 98 99

170214 22:11:40 PCR1 INFO: InnoDB: Setting log file size to 50331648.

170214 22:11:45 PCR1 INFO: InnoDB: Setting log file size to 50331648.

170214 22:11:50 PCR1 INFO: We were able to parse ibbackup_logfile up to

lsn 2543132.

170214 22:11:50 PCR1 INFO: The first data file is '/var/lib/mysql/ibdata1'

and the new created log files are at '/var/lib/mysql'

170214 22:11:50 MAIN INFO: MySQL server version is '5.7.17'

170214 22:11:50 MAIN INFO: Restoring ...5.7.17 version

170214 22:11:50 MAIN INFO: Apply-log operation completed successfully.

170214 22:11:50 MAIN INFO: Full Backup has been restored successfully.

mysqlbackup completed OK! with 3 warnings

Now the original database directory is restored from the backup. Depending on how you are going to start

the restored server, you might need to adjust the ownership of the restored data directory. For example,

if the server is going to be started by the user mysql, use the following command to change the owner

attribute of the data directory and the files under it to the mysql user, and the group attribute to the mysql

group.

$ chown -R mysql:mysql /path/to/datadir

You are now ready to start the restored database server. For more discussions on how to perform different

kinds of restores, see Section 5.1, “Performing a Restore Operation”.

4.3 Backup Scenarios and Examples

4.3.1 Making a Single-File Backup

To avoid having a large number of backup files to keep track, store, and transport, mysqlbackup

conveniently creates backups in a single-file format. It can also pack an existing backup directory into a

single file, unpack the single file back to a backup directory, list the contents of a single-file backup, verify

the contents of a single-file backup against embedded checksums, or extract a single file into a directory

tree. For the syntax of the relevant mysqlbackup options, see Section 16.9, “Single-File Backup Options”.

Advanced: While mysqlbackup can also create a directory backup (see description for the backup

command for details) instead of a single-file backup, the single-file format is preferable in most cases: a

Making a Single-File Backup

single-file backup is easier to handle and store, and certain functions of mysqlbackup are not supported

for directory backups—for example, backup to cloud and backup to tape using the System Backup to

Tape (SBT) API. Throughout the manual, directory backup is mostly treated as an advanced topic, and

information and examples for directory backups are marked with the Advanced tag, like this paragraph.

Because the single-file backup can be streamed or piped to another process such as a tape backup or a

command, you can use the technique to put the backup onto another storage device or server and avoid

significant storage overhead on the original database server.

To create a single-file backup, use the backup-to-image command. The following examples illustrate

how to perform a single-file backup and other related operations.

Example 4.1 Single-File Backup to Absolute Path

This command creates a single backup image on the given absolute path. It still requires --backup-dir,

which is used to hold temporary output, status, and metadata files.

mysqlbackup --backup-image=/backups/sales.mbi --backup-dir=/backup-tmp backup-to-image

Example 4.2 Single-File Backup to Relative Path

When a relative path instead of an absolute path was supplied with the --backup-image option, the path

is taken to be relative to the backup directory. Therefore, in this example, the resulting single-file backup is

created as /backups/sales.mbi.

mysqlbackup --backup-image=sales.mbi --backup-dir=/backups backup-to-image

Example 4.3 Single-File Backup to Standard Output

The following command dumps the backup output to standard output. Again, the folder specified with the

--backup-dir option is used as a temporary directory.

mysqlbackup --backup-dir=/backups --backup-image=- backup-to-image > /backup/mybackup.mbi

Example 4.4 Convert Existing Backup Directory to Single Image

The backup-dir directory is bundled into the /backup/my.mbi file.

mysqlbackup --backup-image=/backup/my.mbi --backup-dir=/var/mysql/backup backup-dir-to-image

Example 4.5 Extract Existing Image to Backup Directory

The image contents are unpacked into backup-dir.

mysqlbackup --backup-dir=/var/backup --backup-image=/backup/my.mbi image-to-backup-dir

Example 4.6 List Single-File Backup Contents

The image contents are listed, with each line indicating a file or directory entry.

mysqlbackup --backup-image=/backup/my.mbi list-image

Example 4.7 Validate a Single-File Backup

The following command verifies that the single-file backup is not corrupted, truncated, or damaged by

validating the checksum value for each data page in the backup.

mysqlbackup --backup-image=/logs/fullimage.mi validate

Example 4.8 Extract Single-File Backup into Current Directory

The following command extracts all contents from a single-file backup into the current working directory.

Making a Single-File Backup

mysqlbackup --backup-image=/var/my.mbi extract

Example 4.9 Extract Single-File Backup into a Backup Directory

This command behaves like the image-to-backup-dir option by extracting all contents of a single-file

backup into the directory specified with the --backup-dir option.

mysqlbackup --backup-image=/var/my.mbi --backup-dir=/var/backup extract

Example 4.10 Selective Extract of Single File

The following command extracts the single file meta/comments.txt from the backup image my.mbi into

the local path./meta/comments.txt.

mysqlbackup --backup-image=/var/my.mbi \

--src-entry=meta/comments.txt extract

The following command extracts the meta/comments.txt file from the backup image my.mbi into a

specified path /tmp/mycomments.txt by using the --dst-entry option.

mysqlbackup --backup-image=/var/my.mbi \

--src-entry=meta/comments.txt \

--dst-entry=/tmp/mycomments.txt extract

The following command dumps the contents of meta/comments.txt (which is inside the single-file

backup my.mbi) to standard output.

mysqlbackup --backup-image=/var/my.mbi --src-entry=meta/comments.txt --dst-entry=- extract

Example 4.11 Selective Extract of Single Directory

The following command extracts a single directory meta from the backup image my.mbi into a local file

system path ./meta. All contents in the meta directory are extracted, including any subdirectories.

mysqlbackup --backup-image=/backup/my.mbi --src-entry=meta extract

The following command extracts all contents of the meta directory, including all its files and subdirectories,

into the directory /tmp/my-meta.

mysqlbackup --backup-image=/backup/my.mbi --src-entry=meta \

--dst-entry=/tmp/my-meta extract

Example 4.12 Dealing with Absolute Path Names

Since absolute pathnames are extracted to the same paths in local system, it could be a problem if you do

not have write permission for that path. You can remap absolute paths as follows:

mysqlbackup --backup-image=/backup/my.mbi --src-entry=/ --dst-entry=/myroot extract

mysqlbackup --backup-image=/backup/my.mbi --src-entry=. extract

The first command extracts all absolute paths to /myroot directory in the local system. The second

command extracts all relative paths to the current directory.

4.3.1.1 Streaming the Backup Data to Another Device or Server

To limit the storage overhead on the database server, you can transfer the backup data to a different

server without ever storing it locally. You can achieve that with a single-file backup. To send the single-

file backup to standard output, use the mysqlbackup command backup-to-image without specifying

the --backup-image option. (You can also specify --backup-image=- to make it obvious that the

data is sent to stdout.) To stream the data, you use the single-file backup in combination with operating

system features such as pipes, ssh, and so on, which take the input from standard output and create

Making a Single-File Backup

an equivalent file on a remote system. You can either store the single-file backup directly on the remote

system, or invoke mysqlbackup with the copy-back-and-apply-log command on the other end to

restore the backup to a remote MySQL server.

Example 4.13 Single-File Backup to a Remote Host

The following command streams the backup as a single-file output to a remote host to be saved under the

file name my_backup.img (--backup-dir=/tmp designates the directory for storing temporary files

rather than the final output file):

mysqlbackup --defaults-file=~/my_backup.cnf --backup-image=- --backup-dir=/tmp backup-to-image | \

ssh <user name>@<remote host name> 'cat > ~/backups/my_backup.img'

For simplicity, all the connection and other necessary options are assumed to be specified in the default

configuration file. ssh can be substituted with another communication protocol like ftp, and cat can be

substituted with another command (for example, dd or tar for normal archiving).

Example 4.14 Single-file Backup to a Remote MySQL Server

The following command streams the backup as a single backup file to be restored on a remote MySQL

server:

mysqlbackup --backup-dir=backup --backup-image=- --compress backup-to-image | \

ssh <user name>@<remote host name> 'mysqlbackup --backup-dir=backup_tmp --datadir=/data \

--innodb_log_group_home_dir=. \

--innodb_log_files_in_group=<innodb_log_files_in_group_of_backedup_server> \

--innodb_log_file_size=<innodb_log_file_size_of_backedup_server> \

--innodb_data_file_path=<innodb_data_file_path_of_backedup_server> \

--uncompress --backup-image=- copy-back-and-apply-log'

Example 4.15 Stream a Backup Directory to a Remote MySQL Server

The following command streams a backup directory as a single backup file to be restored on a remote

MySQL server:

mysqlbackup --backup-image=- --backup-dir=/path/to/my/backup backup-dir-to-image | \

ssh <user name>@<remote host name> 'mysqlbackup --backup-dir=backup_tmp --datadir=/data --backup-image=- copy-back-and-apply-log'

4.3.1.2 Backing Up to Tape

Tape drives are affordable, high-capacity storage devices for backup data. MySQL Enterprise Backup can

interface with media management software (MMS) such as Oracle Secure Backup (OSB) to drive MySQL

backup and restore jobs. The media management software must support Version 2 or higher of the System

Backup to Tape (SBT) interface.

For information about doing tape backups in combination with MMS products such as Oracle Secure

Backup, see Chapter 10, Using MySQL Enterprise Backup with Media Management Software (MMS)

Products.

4.3.1.3 Backing Up to Cloud Storage

MySQL Enterprise Backup supports cloud backups. Only single-file backups can be created on and

restored from a cloud storage. All mysqlbackup options compatible with single-file operations (including,

for example, the incremental, compression, partial, and encryption options) can be used with cloud

backups or restores.

Note

See Appendix B, Limitations of MySQL Enterprise Backup for some limitations

regarding the support for cloud storage by mysqlbackup.

Making a Single-File Backup

Currently, MySQL Enterprise Backup supports two types of cloud storage services: OpenStack Swift or

compatible object storage services (for example, Oracle Cloud Infrastructure Object Storage and Oracle

Cloud Infrastructure Object Storage Classic) and Amazon S3.

MySQL Enterprise Backup 4.1 supports the Swift v1.0 API, and also the OpenStack Identity (Keystone)

API v2.0 for authentication. It also supports authentication using Swift's TempAuth system or HTTP Basic

Authentication.

A cloud backup is created using the cloud options for mysqlbackup, which are described in details in

Section 16.15, “Cloud Storage Options”. Here are some sample commands for creating a cloud backup:

Example 4.16 Creating a Cloud Backup on Oracle Cloud Infrastructure Object Storage Classic

This example creates a cloud backup in an Oracle Cloud Infrastructure (OCI) Object Storage Classic

container, using the TempAuth system for authenticating the user's credentials.

mysqlbackup \

--cloud-service=openstack --cloud-container=<OCI Object Storage Classic container> \

--cloud-user-id=<serviceInstanceName>-<identityDomainName>:<userName> --cloud-password='<password>' \

--cloud-tempauth-url=https://<dataCenterCode>.storage.oraclecloud.com \

--cloud-trace=1 --cloud-object=image_900.mbi \

--backup-dir=/home/user/dba/orbackuptmpdir \

--backup-image=- \

backup-to-image

Example 4.17 Creating a Cloud Backup on Oracle Cloud Infrastructure Object Storage

This example creates a cloud backup in an Oracle Cloud Infrastructure (OCI) Object Storage bucket, using

HTTP basic authentication.

mysqlbackup \

--cloud-service=openstack --cloud-container=<OCI Object Storage bucket> \

--cloud-user-id=<OCI userName> --cloud-password='<OCI auth token>' \

--cloud-ca-info=/etc/ssl/certs/ca-certificates.crt

--cloud-basicauth-url=https://swiftobjectstorage.<region>.oraclecloud.com/v1/<OCI Object Storage namespace> \

--cloud-trace=1 --cloud-object=backup_image_900.mbi \

--backup-dir=/home/user/dba/orbackuptmpdir \

--backup-image=- \

backup-to-image

Example 4.18 Creating a Cloud Incremental Backup on Oracle Cloud Infrastructure Object Storage

This example creates an incremental cloud backup in an Oracle Cloud Infrastructure (OCI) Object Storage

bucket, using HTTP basic authentication.

mysqlbackup \

--cloud-service=openstack --cloud-container=<OCI Object Storage bucket> \

--cloud-user-id=<OCI userName> --cloud-password='<OCI auth token>' \

--cloud-ca-info=/etc/ssl/certs/ca-certificates.crt

--cloud-basicauth-url=https://swiftobjectstorage.<region>.oraclecloud.com/v1/<OCI Object Storage namespace> \

--cloud-object=backup_incr_image_900.mbi \

--backup-dir=/home/user/dba/orincrbackuptmpdir \

--incremental --incremental-base=history:last_backup \

--backup-image=- \

backup-to-image

Example 4.19 Creating a Cloud Backup on an OpenStack Object Storage

This example creates a cloud backup on an OpenStack object storage, using the Keystone identity service

to authenticate the user's credentials.

mysqlbackup \

--include-tables=testdb.t1 --use-tts=with-full-locking \

Making a Full Backup

--cloud-service=openstack --cloud-container=<swift container> \

--cloud-user-id=<keystone user> --cloud-password=<keystone password> \

--cloud-region=<keystone region> --cloud-tenant=<keystone tenant> \

--cloud-identity-url=<keystone url> \

--cloud-trace=1 --cloud-object=image_800.mbi \

--backup-dir=/home/user/dba/opbackuptmpdir \

--backup-image=- \

backup-to-image

Example 4.20 Creating a Cloud Backup on Amazon S3

mysqlbackup\

--cloud-service=s3 --cloud-aws-region=<aws region> \

--cloud-access-key-id=<aws access key id> --cloud-secret-access-key=< aws secret access key> \

--cloud-bucket=<s3 bucket name> --cloud-object-key=<aws object key> \

--backup-dir=/home/user/dba/s3backuptmpdir \

--backup-image=- \

backup-to-image

A cloud backup always uses one write thread.

Besides backup-to-image, all other mysqlbackup operations for single-file backups (backup-dir-

to-image, list-image, validate, image-to-backup-dir, extract, copy-back, and copy-

back-and-apply-log) can also be performed with cloud storage. For example:

Example 4.21 Extract an Existing Image from an Oracle Cloud Infrastructure Object Storage Classic

Container to a Backup Directory

Extract a backup image from an Oracle Cloud Infrastructure Object Storage Classic container, using the --

backup-dir option to specify the directory into which the image will be extracted:

mysqlbackup \

--cloud-service=openstack --cloud-container=<OCI Object Storage Classic container> \

--cloud-user-id=<serviceInstanceName>-<identityDomainName>:<userName> --cloud-password=<password> \

--cloud-tempauth-url=https://<dataCenterCode>.storage.oraclecloud.com \

--cloud-object=image_930.mbi \

--backup-dir=/home/user/dba/orbackupdir \

--backup-image=- \

image-to-backup-dir

Example 4.22 Extract an Existing Image from Amazon S3 Cloud Storage to a Backup Directory

Extract a backup image from Amazon S3, using the --backup-dir option to specify the directory into

which the image will be extracted:

mysqlbackup\

--cloud-service=s3 --cloud-aws-region=<aws region> \

--cloud-access-key-id=<aws access key id> --cloud-secret-access-key=< aws secret access key> \

--cloud-bucket=<s3 bucket name> --cloud-object-key=<aws object key> \

--backup-dir=/home/user/dba/s3backupdir \

--backup-image=- \

image-to-backup-dir

See Section 5.1.5, “Restoring a Backup from Cloud Storage to a MySQL Server” on how to restore a

backup image from a cloud storage.

4.3.2 Making a Full Backup

Most backup strategies start with a complete backup of the MySQL server, from which you can restore all

databases and tables. After you have created a full backup, you might perform incremental backups (which

are smaller and faster) for the next several backup tasks. You then make a full backup periodically to begin

the cycle again.

Making a Differential or Incremental Backup

For sample commands for making a full backup, see Section 4.2.2, “Backing Up an Entire MySQL

Instance”.

This section outlines some of the things to consider when deciding on a strategy for creating full backups.

As we shall see, factors like speed, capacity, and convenience are all relevant for your decisions.

Options on Command Line or in Configuration File?

For clarity, the examples in this manual often show some of the command-line options that are used with

the mysqlbackup commands. For convenience and consistency, you can include those options that

remain unchanged for most backup jobs into the [mysqlbackup] section of the MySQL configuration file

that you supply to mysqlbackup. mysqlbackup also picks up the options from the [mysqld] section

if they are present there. Putting the options into a configuration file can simplify backup administration

for you: for example, putting port information into a configuration file, you can avoid the need to edit your

backup scripts each time the database instance switches to a different port. See Chapter 17, Configuration

Files and Parameters for details about the use of configuration files.

Using a Single Backup Directory or Timestamped Subdirectories?

For convenience, the --with-timestamp option creates uniquely named subdirectories under the

backup directory to hold the backup data (permanent or temporary) and metadata. The timestamped

subdirectories make it simpler to establish retention periods, allowing easy removal and archiving of

backup data that has passed a certain age.

If you do use a single backup directory (that is, if you omit the --with-timestamp option), specify a new,

unique directory name for each backup job.

For incremental backups that uses the --incremental-base option to specify the directory containing

the previous backup, in order to make the directory names predictable, you might prefer to not use the --

with-timestamp option and generate a sequence of directory names with your backup script instead .

Always Full Backup, or Full Backup plus Incremental Backups?

If your InnoDB data volume is small, or if your database is so busy that a high percentage of data changes

between backups, you might want to run a full backup each time. However, you can usually save time and

storage space by running periodic full backups and then several incremental backups in between them, as

described in Section 4.3.3, “Making a Differential or Incremental Backup”.

Use Compression or Not?

Creating a compressed backup can save you considerable storage space and reduce I/O usage

significantly. And with the LZ4 compression method, the overhead for processing compression is quite low.

In cases where database backups are moving from a faster disk system where the active database files sit

to a possibly slower storage, compression will often significantly lower the overall backup time. It can result

in reduced restoration time as well. In general, we recommend LZ4 compression over no compression

for most users, as LZ4-based backups often finish in a shorter time period. However, test out MySQL

Enterprise Backup within your environment to determine what is the most efficient approach. For more

discussions on compressed backups, see Section 4.3.4, “Making a Compressed Backup”.

4.3.3 Making a Differential or Incremental Backup

Assuming a good portion of the data on your MySQL server remains unchanged over time, you can

increase the speed and reduce the required storage space for your regular backups by backing up not all

the data on the server each time, but only the changes to the data which have taken place over time. In

order to that, after making first a full backup that contains all data, you can do one of the following:

Making a Differential or Incremental Backup

• Performing a series of differential backups. Each differential backups includes all the changes

made to the data since the last full backup was performed. To restore data up to, for example, time t,

you simply restore first the full backup, and then, on top of it, the differential backup taken for time t.

• Perform a series of incremental backup. Each incremental backup only includes the changes since

the previous backup, which can itself be a full or incremental backup. The first backup in an incremental

series is always then a differential backup; but after that, each incremental backup only contains the

changes made since that last incremental backup. Each subsequent incremental backup is thus usually

smaller in size than a differential backup, and is faster to make; that allows you to make very frequent

incremental backups, and then enables you to restore the database to a more precise point in time

when necessary. However, restoring data with incremental backups might take longer and more work:

in general, to restore data up to, for example, time t, you start with restoring the full backup, and then

restore the incremental backups one by one, until you are finished with the incremental backup taken for

time t.

MySQL Enterprise Backup supports both incremental and differential backups. You should decide on

which backup strategy to adopt by looking at such factors like how much storage space you have, how

quickly you have to be able to restore data, and so on.

MySQL Enterprise Backup treats differential backup as a special case of incremental backup that has a

full backup as its base. To create a differential backup, simply follow the instructions below for performing

incremental backups, and make sure you specify a full backup as the base of your incremental backup

using the methods we describe below; you should also ignore any instructions that only apply to the

handling of multiple incremental backups.

See Section 16.7, “Incremental Backup Options”, for descriptions of the mysqlbackup options used for

incremental backups. An Incremental backup is enabled with one of the two options: --incremental

and --incremental-with-redo-log-only option. See Creating Incremental Backups Using Only the

Redo Log for their differences.

When creating an incremental backup, you have to indicate to mysqlbackup the point in time of the

previous full or incremental backup. For convenience, you can use the --incremental-base option to

automatically derive the necessary log sequence number (LSN) from the metadata stored in a previous

backup directory or on the server. Or, you can specify an explicit LSN value using the --start-lsn

option, providing to mysqlbackup the ending LSN from a previous full or incremental backup (see Other

Considerations for Incremental Backups on some limitation that applies when using the --start-lsn

option).

To prepare the backup data to be restored, you combine all incremental backups with an original full

backup. Typically, you perform a new full backup after a designated period of time, after which you can

discard the older incremental backup data.

Creating Incremental Backups Using Only the Redo Log

The --incremental-with-redo-log-only might offer some benefits over the --incremental

option for creating an incremental backup:

• The changes to InnoDB tables are determined based on the contents of the InnoDB redo log. Since the

redo log files have a fixed size that you know in advance, it can require less I/O to read the changes from

them than to scan the InnoDB tablespace files to locate the changed pages, depending on the size of

your database, amount of DML activity, and size of the redo log files.

• Since the redo log files act as a circular buffer, with records of older changes being overwritten as new

DML operations take place, you must take new incremental backups on a predictable schedule dictated

by the size of the log files and the amount of redo data generated for your workload. Otherwise, the redo

log might not reach back far enough to record all the changes since the previous incremental backup,

Making a Differential or Incremental Backup

in which case mysqlbackup will quickly determine that it cannot proceed and will return an error. Your

backup script should be able to catch that error and then perform an incremental backup with the --

incremental option instead.

For example:

• To calculate the size of the redo log, issue the command SHOW VARIABLES LIKE

'innodb_log_file%' and, based on the output, multiply the innodb_log_file_size setting

by the value of innodb_log_files_in_group. To compute the redo log size at the physical level,

look into the datadir directory of the MySQL instance and sum up the sizes of the files matching the

pattern ib_logfile*.

• The InnoDB LSN value corresponds to the number of bytes written to the redo log. To check the LSN

at some point in time, issue the command SHOW ENGINE INNODB STATUS and look under the LOG

heading. While planning your backup strategy, record the LSN values periodically and subtract the

earlier value from the current one to calculate how much redo data is generated each hour, day, and

so on.

• This type of incremental backup is not so forgiving of too-low --start-lsn values as the standard --

incremental option is. For example, you cannot make a full backup and then make a series of --

incremental-with-redo-log-only backups all using the same --start-lsn value. Make sure to

specify the precise end LSN of the previous backup as the start LSN of the next incremental backup; do

not use arbitrary values.

Note

To ensure the LSN values match up exactly between successive incremental

backups, it is recommended that you always use the --incremental-base

option when you use the --incremental-with-redo-log-only option.

• To judge whether this type of incremental backup is practical and efficient for a particular MySQL

instance:

• Measure how fast the data changes within the InnoDB redo log files. Check the LSN periodically to

decide how much redo data accumulates over the course of some number of hours or days.

• Compare the rate of redo log accumulation with the size of the redo log files. Use this ratio to see how

often to take an incremental backup, in order to avoid the likelihood of the backup failing because the

historical data are not available in the redo log. For example, if you are producing 1GB of redo log data

per day, and the combined size of your redo log files is 7GB, you would schedule incremental backups

more frequently than once a week. You might perform incremental backups every day or two, to avoid

a potential issue when a sudden flurry of updates produced more redo log data than usual.

• Benchmark incremental backup times using both the --incremental and --incremental-with-

redo-log-only options, to confirm if the redo log backup technique performs faster and with less

overhead than the traditional incremental backup method. The result could depend on the size of your

data, the amount of DML activity, and the size of your redo log files. Do your testing on a server with

a realistic data volume and a realistic workload. For example, if you have huge redo log files, reading

them in the course of an incremental backup could take as long as reading the InnoDB data files using

the traditional incremental technique. Conversely, if your data volume is large, reading all the data files

to find the few changed pages could be less efficient than processing the much smaller redo log files.

Other Considerations for Incremental Backups

The incremental backup feature is primarily intended for InnoDB tables, or non-InnoDB tables that are

read-only or rarely updated. Incremental backups detect changes at the level of pages in the InnoDB

Making a Differential or Incremental Backup

data files, as opposed to table rows; each page that has changed is backed up. Thus, the space and time

savings are not exactly proportional to the percentage of changed InnoDB rows or columns.

For non-InnoDB files, the entire file is always included in an incremental backup, which means the savings

for backup resources are less significant when comparing with the case with InnoDB tables.

You cannot perform incremental backups with the --compress option.

When making an incremental backup that is based on a backup (full or incremental) created using the --

no-locking option, use the --skip-binlog option to skip the backing up of the binary log, as binary

log information will be unavailable to mysqlbackup in that situation.

No binary log files are copied into the incremental backup if the --start-lsn option is used. To include

binary log files for the period covered by the incremental backup, use the --incremental-base option

instead, which provides the necessary information for mysqlbackup to ensure that no gap exists between

binary log data included in the previous backup and the current incremental backup.

Full-scan versus Optimistic Incremental Backup

By default, when the --incremental option is used for a backup without any value specified,

mysqlbackup scans all InnoDB data files in the server's data directory to find pages that have been

changed since the last backup was made and then saves those pages. This is called a full-scan

incremental backup, which might not be very efficient when not many tables have been modified since

the last back up. An optimistic incremental backup, on the other hand, only scans for changed pages in

InnoDB data files that have been modified since the last backup, thus saving some unnecessary scan time.

An optimistic incremental backup can be performed by specifying --incremental=optimistic. While

an optimistic increment backup might shorten the backup time, it has the following limitations:

• Since this feature makes use of the modification times of the files in the server's data directory, two

things must have remained unchanged since the previous backup: (1) the system time on the server,

and (2) the location of the data directory. Otherwise, the backup might fail, or an inconsistent incremental

backup might be produced.

• Optimistic incremental backups cannot be performed with the --incremental-with-redo-log-

only, for which mysqlbackup reads the redo log files instead of scanning the files in the data directory.

• If the --start-lsn option is used, a full scan is performed even if --incremental=optimistic is

specified since, in that case, mysqlbackup cannot determine the point in time for which the previous

backup is consistent, and thus has no time frame to determine which files have been modified recently.

For these and other cases in which an optimistic backup is not desirable, perform a full-scan incremental

backup by using the --incremental option without specifying a value, or use --incremental=full-

scan, which has the same effect.

See Section 4.1.2, “Grant MySQL Privileges to Backup Administrator” on the privileges required for

mysqlbackup to perform an optimistic incremental backup. Also see Using Optimistic Backups and

Optimistic Incremental Backups Together on how to utilize the two features together in a backup schedule.

Examples of Incremental Backups

These examples use mysqlbackup to make an incremental backup of a MySQL server, including all

databases and tables. We show two alternatives, one using the --incremental-base option and the

other using the --start-lsn option.

With the --incremental-base option, you do not have to keep track of LSN values between one

backup and the next. Instead, you can do one of the following:

Making a Differential or Incremental Backup

• Tell mysqlbackup to query the end_lsn value from the last successful non-TTS backup as recorded in

the backup_history table on the server using --incremental-base=history:last_backup.

• Advanced: For directory backups, specify the directory of the previous backup (either full or incremental)

with --incremental-base=dir:directory_path, and mysqlbackup will figure out the starting

point for this backup based on the metadata of the earlier one. Because you need a known set of

directory names, you might want to use hardcoded names or generate a sequence of names in your

own backup script, rather than using the --with-timestamp option. If your last backup was a single-

file, you can still use --incremental-base=dir:directory_path to provide the location of the

temporary directory you supplied with the --backup-dir option during the last backup

In the following example, the --incremental-base=history:last_backup option is used, given

which mysqlbackup fetches the LSN of the last successful (non-TTS) full or partial backup from the

mysql.backup_history table and performs an incremental backup basing on that.

mysqlbackup --defaults-file=/home/dbadmin/my.cnf \

--incremental --incremental-base=history:last_backup \

--backup-dir=/home/dbadmin/temp_dir \

--backup-image=incremental_image1.bi \

backup-to-image

In the following example, an incremental backup similar to the one in the last example but optimistic in

nature is performed.

mysqlbackup --defaults-file=/home/dbadmin/my.cnf \

--incremental=optimistic --incremental-base=history:last_backup \

--backup-dir=/home/dbadmin/temp_dir \

--backup-image=incremental_image1.bi \

backup-to-image

Advanced: Use the following command to create an incremental directory backup using the --

incremental-base=dir:directory_path option; the backup is saved at the location specified by --

incremental-backup-dir:

mysqlbackup --defaults-file=/home/dbadmin/my.cnf --incremental \

--incremental-base=dir:/incr-backup/wednesday \

--incremental-backup-dir=/incr-backup/thursday \

backup

You can also use the --start-lsn option to specify where the incremental backup should start. You

have to record the LSN of the previous backup reported by mysqlbackup at the end of the backup:

mysqlbackup: Was able to parse the log up to lsn 2654255716

The number is also recorded in the meta/backup_variables.txt file in the folder specified by --

backup-dir during the backup. Supply then that number to mysqlbackup using the --start-lsn

option. The incremental backup then includes all changes that came after the specified LSN.

To create an incremental backup image with the --start-lsn option, use the following command,

specifying with --backup-dir the backup directory, which, in this case, is a directory for storing the

metadata for the backup and some temporary files:

mysqlbackup --defaults-file=/home/dbadmin/my.cnf --incremental \

--start-lsn=2654255716 \

--with-timestamp \

--backup-dir=/incr-tmp \

--backup-image=/incr-backup/incremental_image.bi \

backup-to-image

In the following example though, because --backup-image does not provide a full path to the image file

to be created, the incremental backup image is created under the folder specified by --backup-dir:

Making a Compressed Backup

mysqlbackup --defaults-file=/home/dbadmin/my.cnf --incremental \

--start-lsn=2654255716 \

--with-timestamp \

--backup-dir=/incr-images \

--backup-image=incremental_image1.bi \

backup-to-image

Maintaining a backup schedule:

• On a regular schedule determined by date or amount of database activity, take more incremental or

differential backups.

• Optionally, periodically start the cycle over again by taking a full, uncompressed or compressed backup.

Typically, this milestone happens when you can archive and clear out your oldest backup data.

On how to restore your database using the incremental backups, see Section 5.1.3, “Restoring an

Incremental Backup”

4.3.4 Making a Compressed Backup

To save disk space, you can compress InnoDB backup data files by using the --compress option of

mysqlbackup. Compression lets you keep more sets of backup data on hand or save transmission

time when sending the backup data to another server. Also, compression often results in faster backups

because of reduced IO.

The backup compression feature works only for InnoDB tables. After the InnoDB tablespace files are

compressed during backup, they receive the .ibz extension. To avoid wasting CPU cycles without saving

additional disk space, --compress does not attempt to compress tables that were already-compressed on

the server (see Creating Compressed Tables); nevertheless, such tablespace files are also saved with the

.ibz extension inside the backup.

Note

When there is unused space within an InnoDB tablespace file, the entire file is

copied during an uncompressed backup. Perform a compressed backup to avoid

the storage overhead for the unused space.

The binary log and relay log files are compressed and saved with the .bz extension when being included

in a compressed backup.

You can only use the --compress option for full backups, not for incremental backups.

You can also select the compression algorithm to use by the --compress-method option and, when

using the ZLIB or LZMA compression algorithm, the level of compression by the --compress-level

option. See Section 16.6, “Compression Options” for details.

This is a sample command for making a compressed single-file backup:

mysqlbackup --defaults-file=/etc/my.cnf --compress --backup-image=backup.img backup-to-image

Advanced: This is a sample command for making a compressed directory backup:

mysqlbackup --defaults-file=/etc/my.cnf --compress-method=zlib --compress-level=5 backup

This is a sample command for making a compressed and prepared directory backup (only supported for

MySQL Enterprise Backup 4.0.1 and later):

Making a Partial Backup

mysqlbackup --defaults-file=/etc/my.cnf --compress-method=zlib --compress-level=5 backup-and-apply-log

Note

See the limitation that applies to compressed backups in Appendix B, Limitations of

MySQL Enterprise Backup.

4.3.5 Making a Partial Backup

Note

To facilitate the creation of partial backups, MySQL Enterprise Backup has

introduced two new options for partial backup since version 3.10: --include-

tables and --exclude-tables. The new options are intended for replacing

the older options of --include, --databases, --databases-list-file,

and --only-innodb-with-frm, which are incompatible with the new options

and will be deprecated in the upcoming releases. In the discussions below we

assume the new options are used for partial backups. For reference purpose, we

have included information on the older options at the end of this section in Making a

Partial Backup with Legacy Options.

By default, all the files under the database subdirectories in the data directory are included in the backup,

so that the backup includes data from all MySQL storage engines, any third-party storage engines, and

even any non-database files in that directory. This section explains options you can use to selectively back

up or exclude data.

There are various ways to create different kinds of partial backup with MySQL Enterprise Backup:

• Including or excluding specific tables by their names. This uses the --include-tables or --

exclude-tables option.

Each table is checked against the regular expression specified with the --include-tables or --

exclude-tables option. If the regular expression matches the fully qualified name of the table (in

the form of db_name.table_name), the table is included or excluded for the backup. The regular

expression syntax used is the extended form specified in the POSIX 1003.2 standard. The options have

been implemented with the RE2 regular expression library.

• Including some or all InnoDB tables, but not other table types. This uses the --only-innodb option.

• Leaving out files that are present in the MySQL data directory but not actually part of the MySQL

instance. This uses the --only-known-file-types option.

• Achieving a multiple of selection effects by using a combination of the above mentioned options.

• Backing up a selection of InnoDB tables using transportable tablespaces (TTS). This uses the --use-

tts and the --include-tables or --exclude-tables (or both) options.

For syntax details on all the options involved, see Section 16.8, “Partial Backup and Restore Options”.

Important

Typically, a partial backup is more difficult to restore than a full backup, because

the backup data might not include the necessary interrelated pieces to constitute

a complete MySQL instance. In particular, InnoDB tables have internal IDs and

other data values that can only be restored to the same instance, not a different

MySQL server. Always fully test the recovery procedure for any partial backups to

understand the relevant procedures and restrictions.

Making a Partial Backup

Important

Because the InnoDB system tablespace holds metadata about InnoDB tables from

all databases in an instance, restoring a partial backup on a server that includes

other databases could cause the system to lose track of those InnoDB tables in the

other databases. Always restore partial backups on a fresh MySQL server instance

without any other InnoDB tables that you want to preserve.

The following are some command samples for partial backups.

Including all tables with names starting with “emp” into the backup:

mysqlbackup \

--host=localhost --user=mysqluser --protocol=TCP --port=3306 \

--backup-dir=$MEB_TEMP_BACKUP_DIR --backup-image=$MEB_BACKUPS_DIR/my.mbi \

--include-tables="\.emp" \

backup-to-image

Taking a backup of all tables except tables from the “mysql” and “performance_schema” databases:

mysqlbackup \

--host=localhost --user=mysqluser --protocol=TCP --port=3306 \

--backup-dir=$MEB_TEMP_BACKUP_DIR --backup-image=$MEB_BACKUPS_DIR/my.mbi \

--exclude-tables="^(mysql|performance_schema)\." \

backup-to-image

Taking a backup of all tables in the “sales” database, but excludes the table with the name “hardware”

mysqlbackup \

--host=localhost --user=mysqluser --protocol=TCP --port=3306 \

---backup-dir=$MEB_TEMP_BACKUP_DIR --backup-image=$MEB_BACKUPS_DIR/my.mbi \

--include-tables="^sales\." --exclude-tables="^sales\.hardware$" \

backup-to-image

Taking a backup of all tables in the “sales reps” database, but excludes the table with the name “euro-

asia” (special characters like spaces or dashes are supported by the partial backup options since release

3.12.1):

mysqlbackup \

--host=localhost --user=mysqluser --protocol=TCP --port=3306 \

--backup-dir=$MEB_TEMP_BACKUP_DIR --backup-image=$MEB_BACKUPS_DIR/my.mbi \

--include-tables="^sales reps\." --exclude-tables="^sales reps\.euro-asia" \

backup-to-image

Backing up all InnoDB tables, but not .frm files:

mysqlbackup --defaults-file=/home/dbadmin/my.cnf --only-innodb backup-to-image

You can also make compressed and other kinds of selective backups by using the appropriate command

options.

Making a Partial Backup with the Legacy Options

Important

Information in this subsection is only for using the legacy options of --include,

--databases, --databases-list-file, and --only-innodb-with-frm,

which will be deprecated in the upcoming issues. For creating partial backups, it

is strongly recommended that the new options of --include-tables and --

Making a Partial Backup

exclude-tables be used instead. Note that you cannot combine the legacy and

the new partial-backup options in a single command.

MySQL Enterprise Backup can make different kinds of partial backup using the legacy partial-backup

options:

• Including certain InnoDB tables but not others. This operation involves the --include, --only-

innodb, and --only-innodb-with-frm options.

• Including certain non-InnoDB tables from selected databases but not others. This operation involves the

--databases and --databases-list-file options.

For syntax details on all these options, see Legacy Partial Backup Options.

Note

Typically, a partial backup is more difficult to restore than a full backup, because

the backup data might not include the necessary interrelated pieces to constitute

a complete MySQL instance. In particular, InnoDB tables have internal IDs and

other data values that can only be restored to the same instance, not a different

MySQL server. Always fully test the recovery procedure for any partial backups to

understand the relevant procedures and restrictions.

With its --include option, mysqlbackup can make a backup that includes some InnoDB tables but not

others:

• A partial backup with the --include option always contains the InnoDB system tablespace and all the

tables inside it.

• For the InnoDB tables stored outside the system tablespace, the partial backup includes only those

tables whose names match the regular expression specified with the --include option.

This operation requires the tables being left out to be stored in separate table_name.ibd files. To put

an InnoDB table outside the system tablespace, create it while the innodb_file_per_table MySQL

configuration option is enabled. Each .ibd file holds the data and indexes of one table only.

Those InnoDB tables created with innodb_file_per_table turned off are stored as usual in the

InnoDB system tablespace, and cannot be left out of the backup.

For each table with a per-table data file a string of the form db_name.table_name is checked against

the regular expression specified with the --include option. If the regular expression matches the

complete string db_name.table_name, the table is included in the backup. The regular expression

syntax used is the extended form specified in the POSIX 1003.2 standard. On Unix-like systems, quote the

regular expression appropriately to prevent interpretation of shell meta-characters. This feature has been

implemented with the RE2 regular expression library.

The backup directory produced contains a backup log file and copies of InnoDB data files.

IMPORTANT: Although mysqlbackup supports taking partial backups, be careful when restoring a

database from a partial backup. mysqlbackup copies also the .frm files of those tables that are not

included in the backup, except when you do partial backups using, for example, the --databases option.

If you use mysqlbackup with the --include option, before restoring the database, delete from the

backup data the .frm files for any tables that are not included in the backup.

IMPORTANT: Because the InnoDB system tablespace holds metadata about InnoDB tables from all

databases in an instance, restoring a partial backup on a server that includes other databases could cause

Making a Partial Backup

the system to lose track of those InnoDB tables in other databases. Always restore partial backups on a

fresh MySQL server instance without any other InnoDB tables that you want to preserve.

The --only-innodb and --only-innodb-with-frm options back up InnoDB tables only, skipping

those of other storage engines. You might also use them together with the --include option to make

selective backup of InnoDB tables while excluding all other files created by other storage engines.

Example 4.23 Making an Uncompressed Partial Backup of InnoDB Tables

In this example, we have configured MySQL so that some InnoDB tables have their own tablespaces.

We make a partial backup including only those InnoDB tables in test database whose name starts with

ib. The contents of the database directory for test database are shown below. The directory contains

a MySQL description file (.frm file) for each of the tables (alex1, alex2, alex3, blobt3, ibstest0,

ibstest09, ibtest11a, ibtest11b, ibtest11c, and ibtest11d) in the database. Of these 10

tables six (alex1, alex2, alex3, blobt3, ibstest0, ibstest09) are stored in per-table data files

(.ibd files).

$ ls /sqldata/mts/test

alex1.frm alex2.ibd blobt3.frm ibstest0.ibd ibtest11a.frm ibtest11d.frm

alex1.ibd alex3.frm blobt3.ibd ibtest09.frm ibtest11b.frm

alex2.frm alex3.ibd ibstest0.frm ibtest09.ibd ibtest11c.frm

We run the mysqlbackup with the --include option:

# Back up some InnoDB tables but not any .frm files.

$ mysqlbackup --defaults-file=/home/dbadmin/my.cnf --include="^test\.ib.*" --only-innodb backup

# Contents in the backup directory's subdirectory for the test database:

$ ls /sqldata-backup/test

ibstest0.ibd ibtest09.ibd

# Back up some InnoDB tables and the .frm files for the backed-up tables only.

$ mysqlbackup --defaults-file=/home/dbadmin/my.cnf --include="^test\.ib.*" \

--only-innodb-with-frm=related backup

# Contents in the backup directory's subdirectory for the test database:

$ ls /sqldata-backup/test

ibstest0.frm ibtest09.frm

ibstest0.ibd ibtest09.ibd

The backup directory's subdirectory for the test database contains only backups of ibstest0 and

ibtest09 tables, because other InnoDB tables do not match the include pattern ^test\.ib.*.

The related .frm files are included in the second case. Notice that, however, the tables ibtest11a,

ibtest11b, ibtest11c, ibtest11d are in the backup even though they are not visible in the directory

shown below, because they are stored in the system tablespace (ibdata1 file) which is always included in

the backup.

Example 4.24 Making a Compressed Partial Backup

We have configured MySQL so that every InnoDB table has its own tablespace. We make a partial backup

including only those InnoDB tables whose name starts with alex or blob. The contents of the database

directory for test database is shown below.

$ ls /sqldata/mts/test

alex1.frm alex2.ibd blobt3.frm ibstest0.ibd ibtest11a.frm ibtest11d.frm

alex1.ibd alex3.frm blobt3.ibd ibtest09.frm ibtest11b.frm

alex2.frm alex3.ibd ibstest0.frm ibtest09.ibd ibtest11c.frm

We run mysqlbackup with the --compress and --include options:

Making an Optimistic Backup

$ mysqlbackup --defaults-file=/home/dbadmin/my.cnf --compress \

--include=".*\.(alex|blob).*" --only-innodb backup

The backup directory for the database test is shown below. The .ibz files are compressed per-table

data files.

$ ls /sqldata-backup/test

alex1.ibz alex2.ibz alex3.ibz blobt3.ibz

The --databases and --databases-list-file options of mysqlbackup let you back up non-

InnoDB tables only from selected databases, rather than across the entire MySQL instance. (To filter

InnoDB tables, use the --include option instead.) With --databases, you specify a space-separated

list of database names, with the entire list enclosed in double quotation marks. With --databases-list-

file, you specify the path of a file containing the list of database names, one per line.

Some or all of the database names can be qualified with table names, to only back up selected non-

InnoDB tables from those databases.

If you specify this option, make sure you include the same set of databases for every backup (especially

incremental backups), so that you do not restore out-of-date versions of any databases.

4.3.6 Making an Optimistic Backup

Optimistic backup is a feature for improving performance for backing up and restoring huge databases in

which only a small number of tables are modified frequently.

During a hot backup of a huge database (say, in the order of terabytes), huge redo log files could be

generated on the server when the backup is in progress. As the redo log files grow faster than they can be

processed by mysqlbackup, the backup operation can actually fail when mysqlbackup cannot catch up

with the redo log cycles and LSNs get overwritten by the server before they are read by mysqlbackup.

Moreover, the apply-log step for preparing a backup for restoration can take a very long time as

mysqlbackup has huge ibbackup_logfile files (created from the big redo log files) to apply to the

backup. The problems are intensified when the I/O resources available for reading and writing the redo

logs are scarce during the backup and restoration processes.

Optimistic backup relieves the problems by dividing the backup process into two internal phases, which are

transparent to the users:

1. Optimistic phase: In this first phase, tables that are unlikely to be modified during the backup process

(referred to as the “inactive tables” below, identified by the user with the optimistic-time option or,

by exclusion, with the optimistic-busy-tables option) are backed up without locking the MySQL

instance. And because those tables are not expected to be changed before the backup is finished, redo

logs, undo logs, and system table spaces are not backed up by mysqlbackup in this phase.

2. Normal phase: In this second phase, tables that are not backed up in the first phase (referred to as the

“busy tables” below) are being backed up in a manner similar to how they are processed in an ordinary

backup: the InnoDB files are copied first, and then other relevant files and copied or processed with the

MySQL instance locked. The redo logs, undo logs, and the system tablespace are also backed up in

this phase.

An optimistic backup occurs whenever the optimistic-time or optimistic-busy-tables option

is used. For how to use the options, see detailed descriptions for them in Section 16.10, “Performance /

Scalability / Capacity Options”. If, as expected, the list of inactive tables identified by the optimistic options

do not change during the backup (or, even if it changes by a small percentage), most users will find that

the overall backup time is reduced significantly compared to an ordinary backup, as the size of the redo

log data to be backed up will be far smaller. Additionally, restore time for the backup will also be reduced,

Making an Optimistic Backup

as the apply-log operation will be much faster because of the smaller redo log. However, if it turns out

that the list of inactive tables identified changed by a significant portion during the backup process, benefits

of performing an optimistic back up will become limited and, in the worst case, an optimistic backup might

actually take longer to perform and, for a single-file backup, the size of the backup will be larger when

comparing with an ordinary backup. Therefore, users should be careful in identifying which tables are

“inactive” and which are “busy” when trying to perform an optimistic backup.

Note

An optimistic backup cannot be performed for an incremental backup or a backup

using transportable tablespaces (TTS).

The following examples illustrate how to make an optimistic backup.

Example 4.25 Optimistic Backup Using the Option optimistic-time=YYMMDDHHMMSS

In this example, tables that have been modified since the noon of May 16, 2011 are treated as busy tables

and backed up in the normal phase of an optimistic backup, and all other tables are backed up in the

optimistic phase:

mysqlbackup --defaults-file=/etc/my.cnf --optimistic-time=110516120000 backup-to-image

Example 4.26 Optimistic Backup Using the Option optimistic-time=now

In this example, all tables are treated as inactive tables and backed up in the optimistic phase of an

optimistic backup:

mysqlbackup --defaults-file=/etc/my.cnf --optimistic-time=now backup-to-image

Example 4.27 Optimistic Backup Using the optimistic-busy-tables Option

In this example, tables in mydatabase that are prefixed by mytables- in their names are treated as busy

tables and backed up in the normal phase of an optimistic backup, and all other tables are backed up in the

optimistic phase:

mysqlbackup --defaults-file=/etc/my.cnf --optimistic-busy-tables="^mydatabase\.mytables-.*" backup

When you use both the optimistic-time and optimistic-busy-tables options and they come into

conflict on determining which tables are to be busy tables, optimistic-busy-tables takes precedence

over optimistic-time. For example:

Example 4.28 Optimistic and Partial Backup Using both the optimistic-busy-tables and

optimistic-time Options

In this example, tables in mydatabase that are prefixed by mytables- in their names are treated as busy

tables and backed up in the normal phase, even if they have not been modified since May 16, 2010, the

time specified by optimistic-time:

mysqlbackup --defaults-file=/etc/my.cnf --optimistic-busy-tables="^mydatabase\.mytables-.*" \

--optimistic-time=100516 backup

Using Optimistic Backups and Optimistic Incremental Backups Together

By utilizing optimistic backup and optimistic incremental backup together in your backup schedule, you

can speed up backups for huge databases, especially when only a relatively small number of tables have

been modified since a certain time and not many tables are being modified on a frequent basis. Below is a

Making an Optimistic Backup

sample sequence of commands illustrating a weekly backup schedule that makes use of the two features;

it also includes the steps for restoring the data to a certain day.

# A full optimistic backup performed on 2017/02/04, Sat, at 1130 PM.

# The --optimistic-time option is used to specify an optimistic time of 2016/08/16, 0800 PM

mysqlbackup --defaults-file=/home/admin/my.cnf --optimistic-time=160816200000 \

--backup-dir=/home/admin/temp_dir --backup-image=/home/admin/backups/mydb_full_201702042330.bi \

--with-timestamp \

backup-to-image

# A sequence of optimistic incremental backups are then performed on each the following six days at 1130 PM

# On Sunday, 2017/02/05

mysqlbackup --defaults-file=/home/admin/my.cnf \

--incremental=optimistic --incremental-base=history:last_backup \

--backup-dir=/home/admin/temp_dir \

--backup-image=/home/admin/backups/mydb_incremental__201702052330.bi \

--with-timestamp \

backup-to-image

# On Monday, 2017/02/06

mysqlbackup --defaults-file=/home/admin/my.cnf \

--incremental=optimistic --incremental-base=history:last_backup \

--backup-dir=/home/admin/temp_dir \

--backup-image=/home/admin/backups/mydb_incremental__201702062330.bi \

--with-timestamp \

backup-to-image

# On Tuesday, 2017/02/07

mysqlbackup --defaults-file=/home/admin/my.cnf \

--incremental=optimistic --incremental-base=history:last_backup \

--backup-dir=/home/admin/temp_dir \

--backup-image=/home/admin/backups/mydb_incremental__201702072330.bi \

--with-timestamp \

backup-to-image

# On Wednesday, 2017/02/08

mysqlbackup --defaults-file=/home/admin/my.cnf \

--incremental=optimistic --incremental-base=history:last_backup \

--backup-dir=/home/admin/temp_dir \

--backup-image=/home/admin/backups/mydb_incremental__201702082330.bi \

--with-timestamp backup-to-image

# On Thursday, 2017/02/09

mysqlbackup --defaults-file=/home/admin/my.cnf \

--incremental=optimistic --incremental-base=history:last_backup \

--backup-dir=/home/admin/temp_dir \

--backup-image=/home/admin/backups/mydb_incremental__201702092330.bi \

--with-timestamp \

backup-to-image

# On Friday, 2017/02/10

mysqlbackup --defaults-file=/home/admin/my.cnf \

--incremental=optimistic --incremental-base=history:last_backup \

--backup-dir=/home/admin/temp_dir \

--backup-image=/home/admin/backups/mydb_incremental__201702102330.bi \

--with-timestamp \

backup-to-image

# Another full optimistic backup is performed on Saturday, 2017/02/11

mysqlbackup --defaults-file=/etc/my.cnf --optimistic-time=110516200000 \

--backup-dir=/home/admin/temp_dir --backup-image=/home/admin/backups/mydb_full_201702112330.bi \

--with-timestamp \

backup-to-image

# Restore the database to its state at Tuesday, 2017/02/07, at 11:30 PM

# First, restore the full optimistic backup taken on the Saturday before, which was 2017/02/04:

mysqlbackup --defaults-file=/etc/my.cnf --backup-image=/home/admin/backups/mydb_full_201702042330.bi \

--backup-dir=/home/admin/temp_dir --datadir=/var/lib/mysql \

--with-timestamp \

copy-back-and-apply-log

# Next, restore the optimistic incremental taken on the Sunday, Monday, and Tuesday that follow:

Making a Back Up of In-Memory Database Data

mysqlbackup --defaults-file=/etc/my.cnf --backup-image=/home/admin/backups/mydb_incremental__201702052330.bi \

--backup-dir=/home/admin/temp_dir --datadir=/var/lib/mysql --incremental \

--with-timestamp \

copy-back-and-apply-log

mysqlbackup --defaults-file=/etc/my.cnf --backup-image=/home/admin/backups/mydb_incremental__201702062330.bi \

--backup-dir=/home/admin/temp_dir --datadir=/var/lib/mysql --incremental \

--with-timestamp \

copy-back-and-apply-log

mysqlbackup --defaults-file=/etc/my.cnf --backup-image=/home/admin/backups/mydb_incremental__201702072330.bi \

--backup-dir=/home/admin/temp_dir --datadir=/var/lib/mysql --incremental \

--with-timestamp \

copy-back-and-apply-log

4.3.7 Making a Back Up of In-Memory Database Data

The --exec-when-locked option of mysqlbackup lets you specify a command (together with the

desired command arguments) to run near the end of the backup while the database is still locked. This

command can copy or create additional files in the backup directory. For example, you can use this option

to back up MEMORY tables with the mysqldump command, storing the output in the backup directory. To

delay any redirection or variable substitution until the command is executed, enclose the entire option

value within single quotes.

4.3.8 Making Scheduled Backups

Maintaining a regular backup schedule is an important measure for preventing data loss for you MySQL

server. This section discusses some simple means for setting up a schedule for running MySQL Enterprise

Backup.

For Linux and other Unix-like platforms: you can set up a cron job on your system for scheduled backups.

There are two types of cron jobs. To set up a user cron job, which is owned and run by a particular user, do

the following:

• Log on as the user who runs MySQL Enterprise Backup and use the following command to invoke an

editor for creating (or modifying) a crontab:

shell> crontab -e

• In the editor, add an entry similar to the following one to the crontab, and then save your changes (make

sure contents in both lines below appear in a single line in the crontab):

@daily /path-to-mysqlbackup/mysqlbackup -uroot --backup-dir=/path-to-backup-folder/cronbackups

--with-timestamp --backup-image=my.mib backup-to-image &>/dev/null

This crontab entry invokes mysqlbackup to create a backup under the cronbackups directory at

00:00:00 everyday. Outputs from the stderr and stdout streams are redirected to /dev/null/, so they

will not invoke other actions on the part of the Cron server (for example, email notifications to the user).

To set up a system cron job, which is owned and run by root, create a file under the /etc/cron.d folder

and put into it a similar crontab entry as the one above, adding the user (root in the following example)

before the mysqlbackup command:

@daily root /path-to-mysqlbackup/mysqlbackup -uroot --backup-=/path-to-backup-folder/cronbackups \

--with-timestamp --backup-image=my.mib backup-to-image &>/dev/null

Check your platform's documentation for further details on the different ways to set up cron jobs for various

types of schedules.

For Windows platforms: Use the Task Scheduler for the purpose. Check the documentation for your

Windows platform for instructions.

Making Backups with a Distributed File System (DFS) or Storage Access Network (SAN)

4.4 Making Backups with a Distributed File System (DFS) or Storage

Access Network (SAN)

When system administrators attempt to set up MySQL and MySQL Enterprise Backup in an environment

that uses a distributed file system (DFS) or a storage access network (SAN), the MySQL server, the

server's data directory, MySQL Enterprise Backup, and the backup directory may end up existing on

different physical servers. When that happens, the operations of mysqlbackup might be impacted. The

operation most likely to be adversely affected is hot backup, the success of which depends on:

1. Each page of a data file is copied consistently, that is, all the bytes in the page correspond to the same

LSN.

2. No copied page is older than the time that marks the beginning of the temporal duration the backup is

supposed to cover.

3. The redo log is copied consistently, meaning a continuous segment of redo log is copied, and it

includes all the changes from the beginning of the temporal period that the backup is to cover until the

end of the backup operation. Each block of the copied redo log has to be consistent.

Condition 1 is easily achievable with most DFSs or SANs of reasonable performance. Condition 2 though

can remain unfulfilled even when condition 1 has been satisfied: for example, mysqlbackup could copy

all the pages of a tablespace correctly except for one page for which mysqlbackup has included an

old version into the copy. If the LSN of that old version of the page is smaller than the LSN first seen by

mysqlbackup at the beginning of the backup process, the resulting backup will be defective. This example

shows that mysqlbackup may have problem performing a hot backup unless it can see the writes to the

file system being executed in the correct order, that is, the order in which the server executed them.

Regarding condition 3, unlike data file pages, redo log blocks are written sequentially, which means

condition 3 is easier to fulfill than conditions 1 and 2. However, if mysqlbackup reaches the highest LSN

in the copied data file pages before encountering the end of the redo log, the backup fails. A failure occurs

also if mysqlbackup reads a corrupted log block at any time during the copying of the redo log. Both

these failures can occur if mysqlbackup does not see the same history of the file system states as the

MySQL server does.

Therefore, to use mysqlbackup with a DFS or SAN, it is important to make sure that mysqlbackup sees

all the writes to the file system in the same order as the MySQL server does. The condition is most likely

to be satisfied when mysqlbackup and the MySQL server are running on the same server node, and it is

unlikely to be always fulfilled when it is otherwise.

Chapter 5 Recovering or Restoring a Database

Table of Contents

5.1 Performing a Restore Operation .................................................................................................. 61

5.1.1 Restoring a Compressed Backup ..................................................................................... 62

5.1.2 Restoring an Encrypted Backup Image ............................................................................. 63

5.1.3 Restoring an Incremental Backup ..................................................................................... 63

5.1.4 Restoring Backups Created with the --use-tts Option ................................................... 64

5.1.5 Restoring a Backup from Cloud Storage to a MySQL Server .............................................. 65

5.1.6 Restoring External InnoDB Tablespaces to Different Locations ........................................... 66

5.1.7 Advanced: Preparing and Restoring a Directory Backup .................................................... 66

5.2 Point-in-Time Recovery ............................................................................................................... 67

5.3 Restoring a Backup with a Database Upgrade or Downgrade ....................................................... 69

The ultimate purpose of backup data is to help recover from a database issue or to create a clone of

the original database in another location (typically, to run report queries or to create a new replica). This

section describes the procedures to handle those scenarios.

After a serious database issue, you might need to perform a recovery under severe time pressure. It is

critical to confirm in advance:

• How long the recovery will take, including any steps to transfer, unpack, and otherwise process the data.

• That you have practiced and documented all steps of the recovery process, so that you can do it

correctly in one try. If a hardware issue requires restoring the data to a different server, verify all

privileges, storage capacity, and so on, on that server ahead of time.

• That you have periodically verified the accuracy and completeness of the backup data, so that the

system will be up and running soon after being recovered.

5.1 Performing a Restore Operation

The mysqlbackup commands to perform a restore operation are copy-back-and-apply-log and

copy-back (for directory backup only; see Section 5.1.7, “Advanced: Preparing and Restoring a Directory

Backup”). Normally, the restoration process requires the database server to be already shut down (or,

at least not operating on the directory you are restoring the data to), except for restorations of backups

created with the --use-tts option; see explanations below. The process copies the data files, logs, and

other backed-up files from the backup directory back to their original locations, and performs any required

post-processing on them.

Example 5.1 Restoring a Database

mysqlbackup --defaults-file=<my.cnf> -uroot --backup-image=<image_name> \

--backup-dir=<backupTmpDir> --datadir=<restoreDir> copy-back-and-apply-log

The copy-back-and-apply-log command achieves two things:

• Extracts the backup from the image file and copies it to the data directory on the server to be restored.

• Performs an apply log operation to the restored data to bring them up-to-date.

Restoring a Compressed Backup

See Section 4.2.4, “Restoring a Database” for an explanation of the important options used in a restore

operation like --defaults-file, --datadir, --backup-image, and --backup-dir.

The restored data includes the backup_history table, where MySQL Enterprise Backup records details

of each backup. The table allows you to perform future incremental backups using the --incremental-

base=history:last_backup option.

Important

• When performing a restore, make sure the target directories for restore data

are all clean, containing no old or unwanted data files (this might require

manual removal of files at the locations specified by the --datadir, --

innodb_data_home_dir, --innodb_log_group_home_dir, and --

innodb_undo_directory options). The same cleanup is not required for

restoring backups created with the--use-tts option (in which case other

requirements described in Section 5.1.4, “Restoring Backups Created with the --

use-tts Option” apply though).

• After a full restore, depending on how you are going to start the restored server,

you might need to adjust the ownership of the restored data directory. For

example, if the server is going to be started by the user mysql, use the following

command to change the owner attribute of the data directory and the files under it

to the mysql user, and the group attribute to the mysql group.

$ chown -R mysql:mysql /path/to/datadir

• Because mysqlbackup does not support MySQL server's online DDL feature

feature, a backup might become corrupted if, during the backup process, a

DDL operation that took advantage of the feature occurred. To avoid the issue,

mysqlbackup turns the server's system variable old_alter_table to “1” at

the beginning of a backup if it is “0,” so that any DDL operations that take place

during the backup are handled with the old table copy method. mysqlbackup

then turns the variable back to “0” near the end of the backup operation.

Notice that in cases where mysqlbackup quits unexpectedly and does not turn

old_alter_table back to its original value, the user will have to turn the value

back to “0” manually on the server, in order to return the server to its original

configuration. This should be performed if the statement “Server system variable

'old_alter_table' was set to '0'. Setting it to '1'” appears in the early output of

mysqlbackup, but the statement “Setting server system variable 'old_alter_table'

back to '0'” does not appear before mysqlbackup quits.

The following subsections describe a number of different scenarios for restoring a backup.

5.1.1 Restoring a Compressed Backup

Restore a compressed backup image named <image_name>, using the --backup-dir option to specify

the temporary directory into which temporary output, status files, and backup metadata will be saved:

Example 5.2 Restoring a Compressed Backup

mysqlbackup --defaults-file=<my.cnf> -uroot --backup-image=<image_name> \

--backup-dir=<backupTmpDir> --datadir=<restoreDir> --uncompress copy-back-and-apply-log

Advanced: Do the same for a compressed directory backup at <backupDir> to <restoreDir> on the

server using copy-back-and-apply-log:

Restoring an Encrypted Backup Image

Example 5.3 Restoring a Compressed Directory Backup

mysqlbackup --defaults-file=<my.cnf> -uroot --backup-dir=<backupDir> --datadir=<restoreDir> \

--uncompress copy-back-and-apply-log

To restore a compressed and prepared directory backup created with the backup-and-apply-log

command, use the copy-back command and the --uncompress option:

Example 5.4 Restoring a Compressed and Prepared Directory Backup

mysqlbackup --defaults-file=<my.cnf> -uroot --backup-dir=<backupDir> --datadir=<restoreDir> \

--uncompress copy-back

See Section 4.3.4, “Making a Compressed Backup” and Section 16.6, “Compression Options” for more

details on compressed backups.

5.1.2 Restoring an Encrypted Backup Image

Restore an encrypted backup image named <image_name> to <restoreDir> on the server with copy-

back-and-apply-log, using the encryption key contained in a file named <keyFile> :

Example 5.5 Restoring an Encrypted Backup Image

mysqlbackup --defaults-file=<my.cnf> --backup-image=<image_name> \

--backup-dir=<backupTmpDir> --datadir=<restoreDir> --decrypt --key-file=<keyFile> copy-back-and-apply-log

See Section 16.13, “Encryption Options” for more details on backup encryption and decryption.

5.1.3 Restoring an Incremental Backup

There are different ways to use incremental backups to restore a database under different scenarios.

The preferred method is to first restore the full backup and make it up-to-date to the time at which the full

backup was performed using the copy-back-and-apply-log command (see Example 5.1, “Restoring

a Database” on how to do it), then use copy-back-and-apply-log again to restore the incremental

backup image on top of the full backup that was just restored:

Example 5.6 Restoring an Incremental Backup Image

mysqlbackup --defaults-file=<my.cnf> -uroot --backup-image=<inc_image_name> \

--backup-dir=<incBackupTmpDir> --datadir=<restoreDir> --incremental \

copy-back-and-apply-log

In this example, the incremental backup image named <inc_image_name> is restored to <restoreDir>

on the server (where the full backup that the incremental backup image was based on has already been

restored). The --backup-dir option is used to specify the temporary directory into which temporary

output, status files, and backup metadata are saved. Repeat the step with other incremental backup

images that you have, until the data has been restored to a desired point in time.

Advanced: Restoring an Incremental Backup Directory

Incremental directory backups can be restored in a series of copy-back-and-apply-log command, as

illustrated above for single-file backups. Alternatively, at anytime after an incremental backup is taken and

before the data is restored, you can bring your full backup up-to-date with your incremental backup. First,

apply to the full backup any changes that occurred while the backup was running:

$ mysqlbackup --backup-dir=/full-backup/2010-12-08_17-14-11 apply-log

Restoring Backups Created with the --use-tts Option

..many lines of output...

101208 17:15:10 mysqlbackup: Full backup prepared for recovery successfully!

101208 17:15:10 mysqlbackup: mysqlbackup completed OK!

Then, we apply the changes from the incremental backup using the apply-incremental-backup

command:

$ mysqlbackup --incremental-backup-dir=/incr-backup/2010-12-08_17-14-48

--backup-dir=/full-backup/2010-12-08_17-14-11 apply-incremental-backup

...many lines of output...

101208 17:15:12 mysqlbackup: mysqlbackup completed OK!

Now, the data files in the full backup directory are fully up-to-date, as of the time of the last incremental

backup. You can keep updating it with more incremental backups, so it is ready to be restored anytime.

Binary Log and Relay Log Restore

When an incremental backup is being restored using either the copy-back-and-apply-log or apply-

incremental-backup command, the binary log (and also the relay log, in the case of a replica server), if

included in the incremental backup, is also restored to the target server by default. This default behavior is

overridden when either (1) the --skip-binlog option (or the --skip-relaylog option for the relay log)

is used with the restore command, or (2) if the full backup the incremental backup was based on or any

prior incremental backup that came in between the full backup and this incremental backup has the binary

log (or relay log) missing.

For release 4.1.2 and later: Location of the binary log (or relay log) after an incremental backup is restored

is, by default, the same as the log's location on the backed-up server when the incremental backup was

taken, or as specified by the --log-bin (or --relay-log) option during the restore of the incremental

backup.

For release 4.1.1 and earlier: Location of the binary log (or relay log) after an incremental backup is

restored is the data directory of the restored server.

See Section 4.3.3, “Making a Differential or Incremental Backup”, and Section 16.7, “Incremental Backup

Options”, for more details on incremental backups.

5.1.4 Restoring Backups Created with the --use-tts Option

There are some special requirements for restoring backups created with transportable tablespaces (TTS)

(that is, created with the --use-tts option):

• The destination server must be running.

• Make sure that the required parameters for connecting to the server (port number, socket name, etc.)

are provided as command-line options for mysqlbackup, or are specified in the [client] section of a

defaults file.

• The destination server must be using the same page size that was used on the server on which the

backup was made.

• The innodb_file_per_table option must be enabled on the destination server.

• The tables being restored must not exist on the destination server.

• A restore fails if the InnoDB file format of a per-table data file (.ibd file) to be restored does not match

the value of the innodb_file_format system variable on the destination server. In that case, use the

Restoring a Backup from Cloud Storage to a MySQL Server

--force option with the restore commands to change temporarily the value of innodb_file_format

on the server, in order to allow restores of per-table data files regardless of their format.

When restoring a single-file backup created with the option setting --use-tts=with-minimum-

locking, the folder specified with --backup-dir, besides holding temporary output, status files, and

metadata, is also used for extracting temporarily all the tables in the backup and for performing an apply-

log operation to make the data up-to-date before restoring them to the server's data directory.

Selected tables can be restored from a backup created with transportable tablespaces (TTS) using the --

include-tables and --exclude-tables options. The following command restores all tables in the

“sales” database from the backup, but excludes the table with the name “hardware” :

Example 5.7 Restoring Selected Tables from a TTS Image Backup

mysqlbackup --socket=/tmp/restoreserver.sock --datadir=/logs/restoreserverdata --backup-dir=/logs/backup \

--include-tables="^sales\." --exclude-tables="^sales\.hardware$" copy-back-and-apply-log

This following commands rename a table when restoring it from a TTS Backup by using the --rename

option:

Example 5.8 Restoring and Renaming a Table from a TTS Backup

# Using fully qualified table names:

mysqlbackup --socket=/tmp/restoreserver.sock --datadir=/logs/restoreserverdata \

--backup-dir=/BackupDirTemp --backup-image=/home/dbadmin/backups/tts-backup.mbi \

--include-tables="^sales\.cars" --rename="sales.cars to sales.autos" copy-back-and-apply-log

# It works the same if database names are omitted in the argument for --rename:

mysqlbackup --socket=/tmp/restoreserver.sock --datadir=/logs/restoreserverdata \

--backup-dir=/BackupDirTemp --backup-image=/home/dbadmin/backups/tts-backup.mbi \

--include-tables="^sales\.cars" --rename="cars to autos" copy-back-and-apply-log

5.1.5 Restoring a Backup from Cloud Storage to a MySQL Server

To restore a backup image from cloud storage to datadir on the server, use the cloud storage options,

and also the --backup-dir option to specify the temporary directory into which temporary output, status

files, and backup metadata will be saved:

Example 5.9 Restoring a Single-file Backup from an Oracle Cloud Infrastructure (OCI) Object

Storage Classic Container to a MySQL Server

mysqlbackup \

--defaults-file=<my.cnf> \

--cloud-service=openstack --cloud-container=<OCI Object Storage Classic container> \

--cloud-user-id=<serviceInstanceName>-<identityDomainName>:<userName> --cloud-password=<password> \

--cloud-tempauth-url=https://<dataCenterCode>.storage.oraclecloud.com \

--cloud-object=<backup_image_name> \

--datadir=/home/user/dba/datadir \

--backup-dir=/home/user/dba/orbackuptmpdir \

--backup-image=- \

copy-back-and-apply-log

Example 5.10 Restoring a Single-file Backup from an Oracle Cloud Infrastructure (OCI) Object

Storage to a MySQL Server

mysqlbackup \

--defaults-file=<my.cnf> \

--cloud-service=openstack --cloud-container=<OCI Object Storage bucket> \

Restoring External InnoDB Tablespaces to Different Locations

-cloud-user-id=<OCI userName> --cloud-password='<OCI auth token>' \

--cloud-ca-info=/etc/ssl/certs/ca-certificates.crt

--cloud-basicauth-url=https://swiftobjectstorage.<region>.oraclecloud.com/v1/<OCI Object Storage namespace> \

--cloud-object=<backup_image_name> \

--datadir=<server_datadir> \

--backup-dir=/home/user/dba/orbackuptmpdir \

--backup-image=- \

copy-back-and-apply-log

Example 5.11 Restoring a Cloud Incremental Backup from an Oracle Cloud Infrastructure (OCI)

Object Storage Service to a MySQL Server

mysqlbackup \--defaults-file=<my.cnf> \

--cloud-service=openstack --cloud-container=<OCI Object Storage bucket> \

-cloud-user-id=<OCI userName> --cloud-password='<OCI auth token>' \

--cloud-ca-info=/etc/ssl/certs/ca-certificates.crt

--cloud-basicauth-url=https://swiftobjectstorage.<region>.oraclecloud.com/v1/<OCI Object Storage namespace> \

--cloud-object=<backup_image_name> \

--backup-image=- --datadir=<server_datadir> \

--backup-dir=/home/user/dba/orincrbackuptmpdir \

--incremental

copy-back-and-apply-log

Example 5.12 Restoring a Single-file Backup from an OpenStack Object Storage to a MySQL Server

mysqlbackup \

--defaults-file=<my.cnf> \

--cloud-service=openstack --cloud-container=<swift container> \

--cloud-user-id=<keystone user> --cloud-password=<keystone password> \

--cloud-region=<keystone region> --cloud-tenant=<keystone tenant> \

--cloud-identity-url=<keystone url> --cloud-object=image_800.mbi \

--backup-dir=/home/user/dba/swiftbackuptmpdir \

--datadir=/home/user/dba/datadir \

--backup-image=- \

copy-back-and-apply-log

Example 5.13 Restoring a Single-file Backup from Amazon S3 to a MySQL Server

mysqlbackup\

--defaults-file=<my.cnf> \

--cloud-service=s3 --cloud-aws-region=<aws region> \

--cloud-access-key-id=<aws access key id> --cloud-secret-access-key=< aws secret access key> \

--cloud-bucket=<s3 bucket name> --cloud-object-key=<aws object key> \

--backup-dir=/home/user/dba/s3backuptmpdir \

--datadir=/home/user/dba/datadir \

--backup-image=- \

copy-back-and-apply-log

5.1.6 Restoring External InnoDB Tablespaces to Different Locations

When a backup contains external InnoDB tablespaces that resided outside of the backed-up server's data

directory, you can restore them to locations different from their original ones by updating their path names

in the .bl file inside the backup; see description of the file in Table 1.1, “Types of Files in a Backup” for

details.

5.1.7 Advanced: Preparing and Restoring a Directory Backup

A directory backup, just like a single-file backup, can be prepared and restored using the copy-back-

and-apply-log command as explained at the beginning of Section 5.1, “Performing a Restore

Operation”.

Point-in-Time Recovery

Example 5.14 Restoring a Backup Directory using copy-back-and-apply-log

mysqlbackup --defaults-file=/usr/local/mysql/my.cnf \

--backup-dir=/export/backups/full \

copy-back-and-apply-log

However, two alternatives exist for directory backups:

• Perform the apply log operation on the raw backup right after the backup, or anytime before restore,

using the apply-log command. You can run this step on the same database server where you did the

backup, or transfer the raw backup files to a different system first, to limit the CPU and storage overhead

on the database server. Here are some examples of doing that, on different kinds of directory backups:

Example 5.15 Applying the Log to a Backup

This example runs mysqlbackup to roll forward the data files so that the data is ready to be restored:

mysqlbackup --backup-dir=/export/backups/2011-06-21__8-36-58 apply-log

That command creates InnoDB log files (ib_logfile*) within the backup directory and applies log

records to the InnoDB data files (ibdata* and *.ibd).

Example 5.16 Applying the Log to a Compressed Backup

If the backup is compressed, as in Section 4.3.4, “Making a Compressed Backup”, specify the --

uncompress option to mysqlbackup when applying the log to the backup:

mysqlbackup --backup-dir=/export/backups/compressed --uncompress apply-log

Note

Since the apply log operation does not modify any of the original files in the

backup, nothing is lost if the operation fails for some reason (for example,

insufficient disk space). After fixing the problem, you can safely retry apply-log

and by specifying the --force option, which allows the data and log files created

by the failed apply log operation to be overwritten.

• For backups that are non-incremental, you can combine the initial backup and the apply-log steps

using the backup-and-apply-log command.

After the backup has been prepared, you can now restore it using the copy-back command:

mysqlbackup --defaults-file=/usr/local/mysql/my.cnf \

--backup-dir=/export/backups/full \

copy-back

5.2 Point-in-Time Recovery

You can restore your database to its state at an arbitrary time using the binary log files included in the

backups. The process assumes that following conditions are met:

• The backed-up MySQL Server has had its binary logging enabled. To check if this condition has been

satisfied, perform this query on the server:

mysql> SHOW VARIABLES LIKE 'log_bin';

+---------------+-------+

| Variable_name | Value |

+---------------+-------+

Point-in-Time Recovery

| log_bin | ON |

+---------------+-------+

1 row in set (0.00 sec)

If the value of log_bin is OFF, binary logging has not been enabled. See The Binary Log on how to

enable binary logging for the server.

• A series of backups, consisting typically of a full backup followed by a series of incremental backups, has

been created for the server. The last backup in the series covers the targeted point in time for recovery.

The example below illustrates such a typical case.

• The last backup in the backup series you have taken include in itself the relevant binary log files. (To

ensure this requirement is satisfied, do not use any of the following MySQL Enterprise Backup options

when creating the backup: --skip-binlog, --use-tts, --no-locking, or --start-lsn.)

These are the steps for a point-in-time recovery:

1. Restore the series of backups to the server, except for the last incremental backup in the series (which

covers the targeted point in time for recovery). When finished, note the binary log position to which you

have restored the server. The information is available from the backup_variables.txt file in the

restored data directory of the server: look for the value of the entry binlog_position in the file. For

example:

binlog_position=binlog.000012:426

This means after the restore of the backup series, the server is now at log position 426 found in the

binary log file binlog.000012. You will need the information later.

Note

While the last binary log position recovered is also displayed by InnoDB after the

restore, that is not a reliable means for obtaining the ending log position of your

restore, as there could be DDL events and non-InnoDB changes that have taken

place after the time reflected by the displayed position.

2. Extract the binary log from the last incremental backup in the backup series (that is, the backup that

covers the targeted point in time for recovery). You do this by unpacking the incremental backup image

into a backup directory using the image-to-backup-dir command; for example:

mysqlbackup --backup-dir=incr-backup-dir2 --backup-image=incremental_image2.bi image-to-backup-dir

Next, go into the resulting backup directory (incr-backup-dir2 in this example) and, under the data

directory inside, find the binary log file[s] (binlog.000012 in this example):

incr-backup-dir2$ ls datadir

binlog.000012 ibbackup_logfile mysql pets undo_002

...

3. Roll forward the database to its state at the targeted point in time for recovery, identified as t

in this

example, using the binary log file extracted in the last step. Then, using the mysqlbinlog utility,

replay to the server the SQL activities recorded in the binary log file[s], from the log position the server

has been restored to in Step 1 above (which is 426 in our example) all the way to time t

. Specify the

range of binary log events to replay using the --start-position option and the --stop-position

option (which indicates the corresponding binary log position for t

), and pipe the output to the mysql

client:

mysqlbinlog --start-position="binary-log-position-at-the-end-of-backup-restores" \

--stop-position="binary-log-position-corresponding-to-t

" \

Restoring a Backup with a Database Upgrade or Downgrade

binary-log-filename | mysql -uadmin -p

Notes

• Using the --start-datetime or --stop-datetime option to specify the

range of binary log segment to replay is not recommended: there is a higher

risk of missing binary log events when using the option. Use --start-

position and --stop-position instead.

• If you have more than one binary log files in your incremental backup and

they are all needed for bringing the server up to its state at t

, you need to

pipe all of them to the server in a single connection; for example:

mysqlbinlog --start-position="426" --stop-position="binary-log-position-corresponding-to-t

" \

binlog.000012 binlog.000013 binlog.000014 | mysql -u admin -p

You can also dump all the mysqlbinlog output to a single file first, and then pipe or play the file to the

mysql client.

For more explanations on using the binary log for point-in-time recovery, see Point-in-Time

(Incremental) Recovery.

4. Check that the server has been restored to the desired point in time.

5.3 Restoring a Backup with a Database Upgrade or Downgrade

Important

You may encounter technical challenges during a server upgrade or downgrade,

and it is beyond the function of MySQL Enterprise Backup, as a backup tool, to

ensure a successful server upgrade or downgrade. Users interested in the topic are

advised to consult the MySQL server manual, especially the Upgrading MySQL and

Downgrading MySQL sections, and pay careful attention to the requirements and

restrictions discussed there.

You can facilitate a server upgrade or downgrade by using MySQL Enterprise Backup to make a backup

of your data from a source server, restore it to a new target server, and, after some preparations, start a

different version of MySQL Server on the restored data. Here are a number of things that users should pay

attention to when restoring a backup with a database upgrade or downgrade:

• Restoring a database with a server downgrade should only be performed when the MySQL servers on

the source and the target servers are in the same release series. Downgrading to a lower series (for

example, from 5.7.21 to 5.6.39) might cause server crashes or data corruption.

• Restoring a database with a server upgrade requires the following steps, the skipping of any of which

might crash the restored server:

1. Back up the data on the source server.

2. Using the same version of MySQL Enterprise Backup with which the backup was taken, restore the

data to the target server by running a copy-back-and-apply-log operation on the backup.

3. Install on the target server the same version of MySQL Server that was running on the source server

when your backup was created.

4. Start the MySQL Server you just installed. Your restored data go through an abbreviated crash

recovery process in preparation for a server upgrade.

Restoring a Backup with a Database Upgrade or Downgrade

5. Perform a slow shutdown of the MySQL Server you just started in the last step by issuing the SET

GLOBAL innodb_fast_shutdown=0 statement and then shutting the server down. This ensures

that all dirty pages are flushed, and hence there will be no redo log processing later for the upgraded

server.

6. Install the newer MySQL Server version on the target server.

7. Start the newer MySQL Server version you just installed on the data directory you have restored and

prepared in the earlier steps.

8. Perform any other additional upgrade steps that might be required for your platform or distribution as

documented in the MySQL reference manual. Make sure the mysql_upgrade that comes with your

newer server version is applied.

After performing these steps, check your data to make sure that your restore has been successful.

Chapter 6 Working with Encrypted InnoDB Tables

MySQL Enterprise Backup supports encrypted InnoDB tablespaces. For details on how the MySQL server

encrypts and decrypts InnoDB tables, see InnoDB Data-at-Rest Encryption—it explains concepts like

master key and tablespace keys, which are important for understanding how MySQL Enterprise Backup

works with encrypted InnoDB tablespaces.

When InnoDB tablespace encryption uses Oracle Key Vault (OKV) for encryption key management, the

feature is referred to as “MySQL Enterprise Transparent Data Encryption (TDE).”

The following is a brief description on how encrypted InnoDB tables are handled by MySQL Enterprise

Backup in backup, restore, and apply-log operations.

Backing up a database with encrypted InnoDB tables. The following is a typical command for

backing up a database containing encrypted InnoDB tables:

$ mysqlbackup --user=root --password --backup-image=/home/admin/backups/my.mbi --backup-dir=/home/admin/backup-tmp \

--encrypt-password="password" backup-to-image

During the backup operation, mysqlbackup copies the encrypted InnoDB tablespace files into the backup,

and also performs the following actions:

• For MySQL Enterprise Backup 4.1.0, MySQL Enterprise Backup 4.1.1 and later working with MySQL

Enterprise Server 5.7.20 and earlier, or MySQL Enterprise Backup 4.1.1 and later working with MySQL

Community Server 5.7:

• For online backups, mysqlbackup contacts the MySQL server to determine the keyring plugin

the server is using, which, currently, is either one of keyring_file or keyring_okv (for offline

backups, the --keyring option must be used to convey the same information to mysqlbackup).

mysqlbackup also finds out from the server where to access the keyring (for offline backup, the --

keyring_file_data or --keyring_okv_conf_dir option must be used to supply the same

information). Once mysqlbackup has access to the keyring, it obtains the master key and uses it to

decrypt the encrypted tablespace keys, which were used to encrypt the InnoDB tables on the server.

• Using the user password supplied with the option --encrypt-password, mysqlbackup reencrypts

the tablespace keys. For each encrypted table, the reencrypted tablespace key, together with other

information, is stored into a transfer file (with the .bkt extension), which is saved into the backup.

• For MySQL Enterprise Backup 4.1.1 and later working with MySQL Enterprise Server 5.7.21 and later:

• MySQL Enterprise Backup always stores the master key for encryption in an encrypted file inside the

backup, irrespective of the kind of keyring plugin the server uses.

• mysqlbackup contacts the MySQL server to determine the keyring plugin the server is using, which,

currently, is one of keyring_encrypted_file, keyring_file, keyring_okv, or keyring_aws.

• If the server is using the keyring_encrypted_file plugin, the user must use the option --

encrypt-password to supply to mysqlbackup the keyring file encryption password that has been

set on the server with the keyring_encrypted_file_password option. mysqlbackup then

copies over from the server the encrypted keyring data file, which contains the master key used to

encrypt all the tablespace keys, into the meta folder in the backup. The encrypted tablespace files are

also copied into the backup.

• If the server uses a keyring plugin other than keyring_encrypted_file, mysqlbackup accesses

the keyring to obtain the master key and uses it to decrypt the encrypted tablespace keys, which were

used to encrypt the InnoDB tables on the server. The master key is then put into a keyring data file

encrypted with the user password supplied with the option --encrypt-password and saves the file,

named keyring_kef, into the meta folder in the backup.

Note

Users who do not want to supply the password on the command line or in a defaults

file may use the --encrypt-password option without specifying any value;

mysqlbackup then asks the user to type in the password before the operation

starts. This applies to all commands that use the --encrypt-password option.

An extract or image-to-backup-dir command for an image backup containing encrypted InnoDB

tables does not require the --encrypt-password option.

Restoring a backup with encrypted InnoDB tables. The following is a typical command for restoring

a single-file back up containing encrypted InnoDB tables:

$ mysqlbackup --defaults-file=/usr/local/mysql/my.cnf --backup-image=/home/admin/backups/my.mbi \

--backup-dir=/home/admin/restore-tmp --encrypt-password="password" copy-back-and-apply-log

For MySQL Enterprise Backup 4.1.0, or MySQL Enterprise Backup 4.1.1 and later working with MySQL

5.7.20 and earlier: During the restore operation, mysqlbackup copies the encrypted InnoDB tablespace

files onto the server. mysqlbackup also performs the following actions:

• Using the user password supplied with the option --encrypt-password, which should be the same

password used for backing up the database, mysqlbackup decrypts the tablespace keys, which were

encrypted using the password when the backup was performed earlier.

• If the --generate-new-master-key option is used, mysqlbackup generates a new master key and

uses it to reencrypt the tablespace keys. To use the --generate-new-master-key option, the --

keyring option, as well as the --keyring_file_data option (when --keyring=keyring_file)

or --keyring_okv_conf_dir option (when --keyring=keyring_okv) must be specified, so

mysqlbackup can access the keyring and add the new master key to it.

$ mysqlbackup --defaults-file=/usr/local/mysql/my.cnf --backup-image=/home/admin/backups/my.mbi \

--backup-dir=/home/admin/restore-tmp --encrypt-password="password" \

--generate-new-master-key --keyring=keyring_file --keyring-file-data=path-to-keyring-file \

copy-back-and-apply-log

The keyring parameters should then be supplied to the restored server.

If the --generate-new-master-key is not used, mysqlbackup assumes that the same keyring used

on the server when it was backed up continues to be valid and is available to the restored server.

For MySQL Enterprise Backup 4.1.1 and later working with MySQL 5.7.21 and later: The same password

used for backing up the database must be supplied with the --encrypt-password option for a restore

operation. During a restore, mysqlbackup copies the encrypted InnoDB tablespace files and the

encrypted file containing the master keys (keyring_kef) onto the server. It also performs the following

actions:

• For a MySQL Enterprise Server: mysqlbackup restores the encrypted keyring data file to its proper

location on the server. The restored server has to be started with keyring_encrypted_file plugin

and with the options keyring_encrypted_file_data and keyring_encrypted_file_password

(which should supply the server with the same password used with the --encrypt-password option

during the restore).

• For a MySQL Community Server: The keyring_file plugin is the only keyring plugin supported by the

MySQL Community Server; therefore mysqlbackup uses the password supplied with the --encrypt-

password option to decrypt keyring data file and then restores it to the proper location on the server for

the keyring_file plugin to use.

For Incremental Backups. For a series of incremental backups, if a keyring plugin other than

keyring_encrypted_file is being used on the server, users can provide a different value for --

encrypt-password for any of the full or incremental backup in the backup sequence. However,

the password used to make the specific full or incremental backup must be provided to restore that

backup. When starting the server after restoring a series of incremental backups, the password

used for the restore of the last incremental backup should be supplied to the server (except for a

MySQL Community Server, which will start with the keyring_file plugin and does not require the

keyring_encrypted_file_password option to start).

Advanced: Creating and Restoring a directory backup with encrypted InnoDB tables. The

following is a typical command for creating a directory backup containing encrypted InnoDB tables:

$ mysqlbackup --user=root --password --backup-dir=/home/admin/backup \

--encrypt-password="password" backup

The following is a typical command for preparing the backup with the apply-log command:

$ mysqlbackup --backup-dir=/home/admin/backup --encrypt-password="password" apply-log

Notice that the user password must be supplied with the --encrypt-password option, as the tablespace

keys must be decrypted before the log can be applied. The same requirement applies when you try to

update a backup with an incremental backup using the apply-incremental-backup command:

$ mysqlbackup --backup-dir=/home/admin/backup --incremental-backup-dir=/home/admin/backup-in \

--encrypt-password="password" apply-incremental-backup

If you used different values for --encrypt-password for the full or incremental backups in the backup

sequence, make sure you supply the very password you used to create an individual backup when you

perform an apply-log or apply-incremental-backup operation with it.

Next, a copy-back command restores the prepared backup onto the server:

$ mysqlbackup --defaults-file=/usr/local/mysql/my.cnf --backup-dir=/home/admin/backup copy-back

Notice that the --encrypt-password option is not required for this step.

You can combine the two steps of apply-log and copy-back into one by running the copy-back-

and-apply-log command, for which the --encrypt-password option is required:

$ mysqlbackup --defaults-file=/usr/local/mysql/my.cnf --backup-dir=/home/admin/backup \

--encrypt-password="password" copy-back-and-apply-log

For MySQL Enterprise Backup 4.1.0, or MySQL Enterprise Backup 4.1.1 and later working with MySQL

5.7.20 and earlier: You can also use the --generate-new-master-key option, just like when you are

restoring a single-file backup:

$ mysqlbackup --defaults-file=/usr/local/mysql/my.cnf --backup-dir=/home/admin/backup \

--generate-new-master-key --keyring=keyring_file --keyring-file-data=path-to-keyring-file \

--encrypt-password="password" copy-back-and-apply-log

Limitations. Certain limitations apply when MySQL Enterprise Backup works with encrypted InnoDB

tables:

• For MySQL 5.7.11 and earlier, backup for InnoDB tablespaces encrypted with “MySQL Enterprise

Transparent Data Encryption (TDE)” is not supported by mysqlbackup. To perform a backup for

those tables, upgrade the server to the latest MySQL 5.7 release, paying attention to any upgrade

requirements explained in Changes in MySQL 5.7, especially the one regarding the --early-plugin-

load option, and rotate the master key on the upgraded server using the ALTER INSTANCE ROTATE

INNODB MASTER KEY statement. Proceed then with the backup process.

• During a validate operation, if mysqlbackup encounters any encrypted InnoDB tables, it issues a

warning and then skips over them.

• For partial backups using transportable table spaces (that is, when the --use-tts option is used),

encrypted InnoDB tables are never included in a backup. A warning is issued in the log file whenever an

encrypted InnoDB table that matches the table selection criteria has been skipped over.

• The --skip-unused-pages option has no effect on encrypted InnoDB tables during a backup (that is,

empty pages for those tables are not skipped).

• Offline backups of encrypted InnoDB tables are not supported by MySQL Enterprise Backup 4.1.1 and

later when working with MySQL Enterprise Server 5.7.21 and later.

Chapter 7 Using MySQL Enterprise Backup with Replication

Table of Contents

7.1 Setting Up a New Replica ........................................................................................................... 75

7.2 Backing up and Restoring a Replica Database ............................................................................ 77

7.3 Restoring a Source Database ..................................................................................................... 78

Backup and restore operations are especially important in systems that use MySQL replication to

synchronize data across a source server and a set of replica servers. In a replication configuration, MySQL

Enterprise Backup helps you manage images for the entire system, set up new replica servers, or restore a

source server in an efficient way that avoids unnecessary work for the replica servers. On the other hand,

having multiple replica servers to choose from gives you more flexibility about where to perform backups.

When the binary log is enabled, you have more flexibility about restoring the database to a specific point in

time, even a time that is later than that of the last backup.

7.1 Setting Up a New Replica

MySQL Enterprise Backup allows you to set up a replica by backing up the source and restoring the

backup on a new replica server, without having to stop the source.

For servers NOT using GTID:

1. Take a full backup of the source and then use, for example, the copy-back-and-apply-log

command, to restore the backup and the log files to the right directories on the new replica and prepare

the data.

Note

Do not use the --no-locking option when backing up the server, or you will

be unable to get a proper binary log position in Step 4 below for initializing the

replica.

2. Edit the my.cnf file of the new replica and put skip-slave-start and event_scheduler=off (if

the source uses the Event Scheduler) under the [mysqld] section.

3. Start the new replica mysqld. You see the following in the server's output:

…

InnoDB: Last MySQL binlog file position 0 128760007, file name ./hundin-bin.000006

…

While a Last MySQL binlog file position has been displayed, it is NOT necessarily the latest

binary log position on the backed up server, as InnoDB does not store binary log position information

for any DDL operations or any changes to non-InnoDB tables. Do not use this binary log position to

initialize the replica. The next step explains how to find the correct binary log position to use.

4. Look for the file datadir/meta/backup_variables.txt where datadir is the data directory of

the new replica. Look into the file to retrieve the latest binary log position and the corresponding log file

number stored inside:

binlog_position=hundin-bin.000006:128760128

5. Use the CHANGE MASTER TO SQL statement and the information you have retrieved in the last step to

initialize the replica properly:

Setting Up a New Replica

CHANGE MASTER TO

MASTER_LOG_FILE='hundin-bin.000006',

MASTER_LOG_POS=128760128;

6. Set the statuses of any events that were copied from the source to SLAVESIDE_DISABLED. For

example:

mysql> UPDATE mysql.event SET status = 'SLAVESIDE_DISABLED';

7. Remove the line skip-slave-start and event_scheduler=off entries you added to the my.cnf

file of the replica in step 2. (You can also leave the skip-slave-start entry in, but then you will

always need to use the START SLAVE statement to start replication whenever you restart the replica

server.)

8. Restart the replica server. Replication starts.

For servers using GTIDs (see Setting Up Replication Using GTIDs on how to enable servers to use

GTIDs):

1. Take a full backup of the source and then use, for example, the copy-back-and-apply-log

command, to restore the backup and the log files to the right directories on a new GTID-enabled replica

and prepare the data.

2. Edit the my.cnf file of the new replica and put skip-slave-start and event_scheduler=off (if

the source uses the Event Scheduler) under the [mysqld] section.

3. Start the new replica server.

4. Connect to the replica server with the mysql client. Then, execute the following statement to reset the

binary log:

mysql> RESET MASTER;

And execute the following statement to stop the binary logging:

mysql> SET sql_log_bin=0;

5. When a server using the GTID feature is backed up, mysqlbackup produces a file named

backup_gtid_executed.sql, which can be found in the restored data directory of the new replica

server. The file contains a SQL statement that sets the GTID_PURGED configuration option on the

replica:

# On a new replica, issue the following command if GTIDs are enabled:

SET @@GLOBAL.GTID_PURGED='f65db8e2-0e1a-11e5-a980-080027755380:1-3';

It also contains a commented-out CHANGE MASTER TO statement for initializing the replica:

# Use the following command if you want to use the GTID handshake protocol:

# CHANGE MASTER TO MASTER_AUTO_POSITION = 1;

Uncomment the command and add any needed connection and authentication parameters to it (for

example, MASTER_HOST, MASTER_USER, MASTER_PASSWORD, and MASTER_PORT):

# Use the following command if you want to use the GTID handshake protocol:

CHANGE MASTER TO MASTER_HOST='127.0.0.1', MASTER_USER='muser', MASTER_PASSWORD='mpass', MASTER_PORT=18675, MASTER_AUTO_POSITION = 1;

Execute the file with the mysql client

mysql> source /path-to-backup_gtid_executed.sql/backup_gtid_executed.sql

Backing up and Restoring a Replica Database

6. Set the statuses of any events that were copied from the source to SLAVESIDE_DISABLED. For

example:

mysql> UPDATE mysql.event SET status = 'SLAVESIDE_DISABLED';

7. Remove the skip-slave-start and event_scheduler=off entries you added to the my.cnf file

of the replica in step 2. (You can also leave the skip-slave-start entry in, but then you will always

need to use the START SLAVE statement to start replication whenever you restart the replica server.)

8. Restart the replica server. Replication starts.

For more information on the GTIDs, see GTID feature.

7.2 Backing up and Restoring a Replica Database

To backup a replica database, add the --slave-info option to your backup command.

To restore the backup on a replica server, follow the same steps outlined in Section 7.1, “Setting Up a New

Replica”.

Temporary tables on statement-based replication (SBR) replica. MySQL Enterprise Backup

does not include temporary tables inside a backup. As a result, for a replica server in a statement-based

replication (SBR) or a mixed-based replication setup (see Replication Formats for details), any temporary

tables still open at the end of the backup process will be missing in the restored replica server, making

the replication state of the replica inconsistent, and any subsequent replicated statements that refer to the

temporary tables will fail. To avoid the issue, after the hot backup phase in which mysqlbackup copies all

the InnoDB tables, it enters into a loop, in which the following happens:

1. mysqlbackup waits until all temporary tables have been closed by the replication SQL thread.

mysqlbackup tells if that is the case by checking if the variable Slave_open_temp_tables has a

zero value.

2. After Slave_open_temp_tables=0 is detected, mysqlbackup stops the replication SQL thread to

prevent more changes to the tables on the replica.

3. To avoid the unexpected consequence by a race condition, after the replication SQL thread has been

stopped, mysqlbackup checks once more if Slave_open_temp_tables=0 is still true:

• If it is true, mysqlbackup exits the loop and finishes the backup by asserting a global read lock and

copies all the non-InnoDB tables.

• If it is not true, new temporary tables have just been created and opened on the replica.

mysqlbackup then restarts the replication SQL thread, so more updates can be made on the replica

servers. mysqlbackup then goes back to step 1 of this loop

Besides the exit condition described in step (3) above (which is, there really are no more open temporary

tables and mysqlbackup is ready to complete the backup), mysqlbackup will time out after staying in the

above loop for too long to wait for all temporary tables to be closed. The duration mysqlbackup waits until

it times out is specified by the --safe-slave-backup-timeout option.

In addition, mysqlbackup also runs an initial check at the beginning of a replica backup to see if

Slave_open_temp_tables=0 becomes true within the duration set by --safe-slave-backup-

timeout. See description for --safe-slave-backup-timeout on details about the check.

Notice that the above-described issue with temporary tables does not exist for a row-based replication

(RBR) setup, for which temporary tables are not replicated onto the replica. User who are certain that

SBR is not occurring for the replica can set --safe-slave-backup-timeout=0, which will prevent

mysqlbackup from entering the above-mentioned loop.

Restoring a Source Database

Note

See the limitation that applies when backing up a replica in Appendix B, Limitations

of MySQL Enterprise Backup.

7.3 Restoring a Source Database

To fix a corruption problem in a replication source database, you can restore the backup, taking care not to

propagate unnecessary SQL operations to the replica servers:

1. Shut down the source database and then use, for example, the copy-back-and-apply-log

command, to restore a backup of it and prepare the data.

2. Edit the source my.cnf file and comment out log-bin, so that the replicas do not receive twice the

binary log needed to recover the source.

3. Replication in the replicas must be stopped temporarily while you pipe the binary log to the source. In

the replicas, do:

mysql> STOP SLAVE;

4. Start the source mysqld on the restored backup:

$ mysqld

…

InnoDB: Doing recovery: scanned up to log sequence number 0 64300044

InnoDB: Last MySQL binlog file position 0 5585832, file name

./omnibook-bin.000002

…

InnoDB prints the binary log file (./omnibook-bin.000002 in this case) and the position (5585832

in this case) it was able to recover to.

5. Pipe the remaining of the binary log files to the restored server. The number of remaining binary log

files varies depending on the length of the timespan between the last backup and the time to which you

want to bring the database up to date. The longer the timespan, the more remaining binary log files

there may be. All the binary log files, containing all the continuous binary log positions in that timespan,

are required for a successful restore.

You also need to supply the starting position in the binary log by which the piping of the events should

start. Deduce that information from the meta/backup_variables.txt file in the backup you just

restored in step 1 above (access backup_variables.txt by, for example, going to the temporary

backup directory you specified with --backup-dir during the restore, and find the file under the meta

folder): look for the entry binlog_position=value in meta/backup_variables.txt, and supply

value to mysqlbinlog with the --start-position option.

Note

While the last binary log position recovered is also displayed by InnoDB after

the restore (see step 4 above), that is not a reliable number for deducing the

start position for mysqlbinlog to use, as there could be DDL events and non-

InnoDB changes that have taken place after the time reflected by the displayed

position.

For example, if there are two more binary log files, omnibook-bin.000003 and omnibook-

bin.000004 that come after omnibook-bin.000002 and the recovery in step 4 above has ended by

5585834 according to the backup_variables.txt file, pipe the binary log with a single connection

to the server with this command:

Restoring a Source Database

$ mysqlbinlog --start-position=5585834 /mysqldatadir/omnibook-bin.000002 \

/mysqldatadir/omnibook-bin.000003 /mysqldatadir/omnibook-bin.000004 | mysql

See Point-in-Time (Incremental) Recovery for more instructions on using mysqlbinlog.

6. The source database is now recovered. Shut down the source and edit my.cnf to uncomment log-

bin.

7. Start the source again.

8. Start replication in the replicas again:

mysql> START SLAVE;

Chapter 8 Using MySQL Enterprise Backup with Group

Replication

For how to use MySQL Enterprise Backup to back up and subsequently restore a Group Replication

member, see Using MySQL Enterprise Backup with Group Replication.

For MySQL Enterprise Backup 4.1.2 and later: When working with a Group Replication setup,

mysqlbackup makes the backup history available to all members of the server group by making sure

that the backup_history table is updated on a primary node after each mysqlbackup operation. The

feature requires the following to be true:

• Host names or host addresses contained in the member_host column in the

performance_schema.replication_group_members table must be resolvable by mysqlbackup.

• In addition to the mysqlbackup@localhost account created on each host as described in

Section 4.1.2, “Grant MySQL Privileges to Backup Administrator”, accounts for the user mysqlbackup

to access all hosts in the group have been created on every member of the server group. These are

sample statements for creating the accounts:

CREATE USER 'mysqlbackup'@'host1' IDENTIFIED BY 'password';

CREATE USER 'mysqlbackup'@'host2' IDENTIFIED BY 'password';

CREATE USER 'mysqlbackup'@'host3' IDENTIFIED BY 'password';

...

Or, if the host names fall into a pattern, use a wildcard for the host names (if allowed by your

environment's security policy) to perform the same task with a single statement; for example:

CREATE USER 'mysqlbackup'@'host_' IDENTIFIED BY 'password';

• The account mysqlbackup@localhost AND all other accounts for accessing the hosts

(i.e., mysqlbackup@host1, mysqlbackup@host2,mysqlbackup@host3, etc, or simply

mysqlbackup@host_ in our example) use the same password for server access. This is needed

because, depending on how host names are resolved, mysqlbackup will likely be using two accounts

to finish a backup job on a Group Replication set up (one for the actual backup operation, the other for

updating the history and progress tables on a primary host), but only one password can be specified in

the backup command. Therefore, the accounts must use the same password.

• All the privileges granted to mysqlbackup@localhost as described in Section 4.1.2, “Grant MySQL

Privileges to Backup Administrator” are also granted to all other accounts with which mysqlbackup

connects to the hosts:

GRANT SELECT ON performance_schema.replication_group_members TO 'mysqlbackup'@'host1';

GRANT SELECT ON performance_schema.replication_group_members TO 'mysqlbackup'@'host2';

GRANT SELECT ON performance_schema.replication_group_members TO 'mysqlbackup'@'host3';

[More grant statements for other privileges for all hosts ...]

Or, if the host names fall into a pattern, use a wildcard for the host names (if allowed by your

environment's security policy) to perform the same task; for example:

GRANT SELECT ON performance_schema.replication_group_members TO 'mysqlbackup'@'host_';

[More grant statements for other privileges...]

Chapter 9 Encryption for Backups

In order to enhance security for backed up data, MySQL Enterprise Backup provides encryption for single-

file backups. The encryption can also be applied when creating a partial, compressed, or incremental

single-file backups, and for streaming backup data to another device or server.

The encryption is performed with Advanced Encryption Standard (AES) block cipher in CBC mode, with a

key string of 64 hexadecimal digits supplied by the user. Decryption is performed using the same key. The

key can be created manually just by putting together 64 random hexadecimal bytes, or it can be generated

by shasum (or similar programs for hash calculations that work on your platform) by supplying it with a

keyphrase:

$ echo -n "my secret passphrase" | shasum -a 256

a7e845b0854294da9aa743b807cb67b19647c1195ea8120369f3d12c70468f29 -

Note that the “-” at the end is not part of the key and should be ignored. Supply the key to mysqlbackup

with the --key option, or paste the key into a key file and supply the file's pathname to mysqlbackup with

the --key-file option.

To generate a key randomly, you can use tools like OpenSSL:

$ openssl rand -hex 32

8f3ca9b850ec6366f4a54feba99f2dc42fa79577158911fe8cd641ffff1e63d6

To put an OpenSSL-generated key into a key file, you can do the following:

$ openssl rand -hex 32 >keyfile

$ cat keyfile

6a1d325e6ef0577f3400b7cd624ae574f5186d0da2eeb946895de418297ed75b

The encryption function uses MySQL Enterprise Backup's own encryption format, which means decryption

is possible only by using MySQL Enterprise Backup. For Unix-like operating systems, different magic

numbers are used to identify encrypted and unencrypted backup files. For example, you can add these

lines to the /etc/magic file of your operating system:

0 string MBackuP\n MySQL Enterprise Backup backup image

0 string MebEncR\n MySQL Enterprise Backup encrypted backup

The file command can then be used to identify the file types:

$ file /backups/image1 /backups/image2

/backups/image1: MySQL Enterprise Backup backup image

/backups/image2: MySQL Enterprise Backup encrypted backup

The command options used for encryption and decryption are --encrypt, --decrypt, --key, and --

key-file. These options can be used with various operations on backup images. See Section 16.13,

“Encryption Options” for details.

The following is a sample command for creating an encrypted backup:

mysqlbackup --backup-image=/backups/image.enc --encrypt

--key=23D987F3A047B475C900127148F9E0394857983645192874A2B3049570C12A34

--backup-dir=/var/tmp/backup backup-to-image

To use a key file for the same task:

mysqlbackup --backup-image=/backups/image.enc --encrypt

--key-file=/meb/key --backup-dir=/var/tmp/backup backup-to-image

To decrypt a backup when extracting it:

mysqlbackup --backup-image=/backups/image.enc --decrypt

--key-file=/meb/key --backup-dir=/backups/extract-dir extract

To validate an encrypted backup image:

mysqlbackup --backup-image=/logs/encimage.bi --decrypt --key-file=/meb/enckey validate

Chapter 10 Using MySQL Enterprise Backup with Media

Management Software (MMS) Products

Table of Contents

10.1 Backing Up to Tape with Oracle Secure Backup ........................................................................ 85

This section describes how you can use MySQL Enterprise Backup in combination with media

management software (MMS) products for creating backups for your database. Such products are typically

used for managing large volumes of backup data, often with high-capacity backup devices such as tape

drives. Also see this whitepaper for more discussions on using an MMS product with mysqlbackup

through the System Backup to Tape (SBT) interface.

10.1 Backing Up to Tape with Oracle Secure Backup

Tape drives are affordable, high-capacity storage devices for backup data. MySQL Enterprise Backup can

interface with media management software (MMS) such as Oracle Secure Backup (OSB) to drive MySQL

backup and restore jobs. The media management software must support Version 2 or higher of the System

Backup to Tape (SBT) API.

On the MySQL Enterprise Backup side, you run the backup job as a single-file backup using the --

backup-image parameter, with the prefix sbt: in front of the filename, and optionally pass other --sbt-

* parameters to mysqlbackup to control various aspects of the SBT processing. The --sbt-* options

are listed in Section 16.9, “Single-File Backup Options”.

On the OSB side, you can schedule MySQL Enterprise Backup jobs by specifying a configurable command

that calls mysqlbackup. You control OSB features such as encryption by defining a “storage selector” that

applies those features to a particular backup, and passing the name of the storage selector to OSB using

the MySQL Enterprise Backup parameter --sbt-database-name=storage_selector.

To back up MySQL data to tape:

• Specify the --backup-image=sbt:name parameter of mysqlbackup to uniquely identify the backup

data. The sbt: prefix sends the backup data to the MMS rather than a local file, and the remainder of

the argument value is used as the unique backup name within the MMS.

• Specify the --sbt-database-name parameter of mysqlbackup to enable the OSB operator to

configure a storage selector for backups from this MySQL source. (This parameter refers to a “storage

selector” defined by the OSB operator, not to any MySQL database name.) By default, mysqlbackup

supplies a value of MySQL for this MMS parameter. The argument to this option is limited to 8 bytes.

• If you have multiple media management programs installed, to select the specific SBT library to use,

specify the --sbt-lib-path parameter of the mysqlbackup command. If you do not specify the --

sbt-lib-path parameter, mysqlbackup uses the normal operating system paths and environment

variables to locate the SBT library, which is named libobk.so on Linux and Unix systems and

ORASBT.DLL on Windows systems. When you specify --sbt-lib-path, you can use a different

filename for the library in addition to specifying the path.

• Specify any other product-specific settings that are normally controlled by environment variables using

the --sbt-environment option.

Backing Up to Tape with Oracle Secure Backup

Each time an online backup is made to a tape using the SBT API , besides recording the backup in

the mysql.backup_history and the mysql.backup_progress tables, an entry is also made

to the mysql.backup_sbt_history table on the backed up MySQL instance. That facilitates the

management of tape backups by allowing easy look-ups for information on them. The definition of the

backup_sbt_history table is shown below:

mysql> DESCRIBE `backup_sbt_history`;

+--------------------+---------------+------+-----+---------------------+-------+

+--------------------+---------------+------+-----+---------------------+-------+

| file_creation_time | timestamp | NO | | 0000-00-00 00:00:00 | |

| file_expiry_time | timestamp | NO | | 0000-00-00 00:00:00 | |

| sbt_error_code | int(11) | NO | | NULL | |

--------------------------------------------------------------------------------

Here are the descriptions for the fields in the table:

• backup_id: The backup's ID, which is also recorded in the backup's entries in the

mysql.backup_history and the mysql.backup_progress tables.

• backup_file_name: The file name provided by the user through the --backup-image=sbt:name

option.

• file_creation_time: Creation date and time for the tape backup.

• file_expiry_time: Expiration date and time for the tape backup.

• volume_label: Volume label for the physical medium which contains the tape backup.

• sbt_error_msg: Error message, when an error occurs while retrieving information for the tape backup.

• sbt_error_code: Error code, when an error occurs while retrieving information for the tape backup.

Multiple entries, one for each volume label, are created in the mysql.backup_sbt_history table, if the

backup is split across multiple volumes.

Here are some sample entries in the mysql.backup_sbt_history table:

mysql> select * from mysql.backup_sbt_history;

+-------------------+------------------+---------------------+---------------------+--------------+---------------+----------------+

+-------------------+------------------+---------------------+---------------------+--------------+---------------+----------------+

| 14377247250385175 | test1.mbi | 2015-07-24 09:59:00 | 0000-00-00 00:00:00 | MySQL-000018 | | 0 |

| 14377259679196369 | test2.mbi | 2015-07-24 10:19:43 | 0000-00-00 00:00:00 | MySQL-000018 | | 0 |

| 14377265350728997 | test3.mbi | 2015-07-24 10:29:10 | 0000-00-00 00:00:00 | MySQL-000018 | | 0 |

| 14377265350728997 | test3.mbi | 2015-07-24 10:29:10 | 0000-00-00 00:00:00 | MySQL-000019 | | 0 |

+-------------------+------------------+---------------------+---------------------+--------------+---------------+----------------+

4 rows in set (0.00 sec)

A backup to tape always uses one write thread.

To restore MySQL data from tape:

• Specify the --backup-image=sbt:name parameter of mysqlbackup as part of the restore operation.

Use the same name value which was used during the backup. This single parameter retrieves the

appropriate data from the appropriate tape device.

• Optionally use the --sbt-lib-path option, using the same value as for the backup operation.

Backing Up to Tape with Oracle Secure Backup

• Specify any other product-specific settings that are normally controlled by environment variables using

the --sbt-environment option.

For product-specific information about Oracle Secure Backup, see the Oracle Secure Backup

documentation.

Example 10.1 Sample mysqlbackup Commands Using MySQL Enterprise Backup with Oracle

Secure Backup

# Uses libobk.so or ORASBT.DLL, at standard locations:

mysqlbackup --port=3306 --protocol=tcp --user=root --password \

--backup-image=sbt:backup-shoeprod-2011-05-30 \

--backup-dir=/backup backup-to-image

# Associates this backup with storage selector 'shoeprod':

mysqlbackup --port=3306 --protocol=tcp --user=root --password \

--backup-image=sbt:backup-shoeprod-2011-05-30 \

--sbt-database-name=shoeprod \

--backup-dir=/backup backup-to-image

# Uses an alternative SBT library, /opt/Other-MMS.so:

mysqlbackup --port=3306 --protocol=tcp --user=root --password \

--backup-image=sbt:backup-shoeprod-2011-05-30 \

--sbt-lib-path=/opt/Other-MMS.so \

--backup-dir=/backup backup-to-image

Chapter 11 Performance Considerations for MySQL Enterprise

Backup

Table of Contents

11.1 Optimizing Backup Performance ................................................................................................ 89

11.2 Optimizing Restore Performance ............................................................................................... 92

This chapter describes the performance considerations for backing up and restoring databases using

MySQL Enterprise Backup.

11.1 Optimizing Backup Performance

This section describes the performance considerations for backing up a database with MySQL Enterprise

Backup. When optimizing and tuning the backup procedure, measure both the raw performance (how long

it takes the backup to complete) and the amount of overhead on the database server. When measuring

backup performance, consider:

• The limits imposed by your backup procedures. For example, if you take a backup every 8 hours, the

backup must take less than 8 hours to finish.

• The limits imposed by your network and storage infrastructure. For example, if you need to fit many

backups on a particular storage device, you might use compressed backups, even if that made the

backup process slower.

• The tradeoff between backup time and restore time. You might choose a set of options resulting in

a slightly slower backup, if those options enable the restore to be much faster. See Section 11.2,

“Optimizing Restore Performance” for performance information for the restore process.

Full or Incremental Backup

After taking a full backup, subsequent backups can be performed more quickly by doing incremental

backups, where only the changed data is backed up. For an incremental backup, specify the --

incremental or --incremental-with-redo-log-only option to mysqlbackup. See Section 16.7,

“Incremental Backup Options” for information about these options. For usage instructions for the backup

and apply stages of incremental backups, see Section 4.3.3, “Making a Differential or Incremental Backup”.

Compressed Backup

Compressing the backup data before transmitting it to another server involves additional CPU overhead

on the database server where the backup takes place, but less network traffic and less disk I/O on the

server that is the final destination for the backup data. Consider the load on your database server, the

bandwidth of your network, and the relative capacities of the database and destination servers when

deciding whether or not to use compression. See Section 4.3.4, “Making a Compressed Backup” and

Section 16.6, “Compression Options” for information about creating compressed backups.

Compression involves a tradeoff between backup performance and restore performance. In an emergency,

the time needed to uncompress the backup data before restoring it might be unacceptable. There

might also be storage issues if there is not enough free space on the database server to hold both the

compressed backup and the uncompressed data. Thus, the more critical the data is, the more likely that

Single-File Backups

you might choose not to use compression: accepting a slower, larger backup to ensure that the restore

process is as fast and reliable as possible.

Single-File Backups

The single-file backup by itself is not necessarily faster than the traditional type of backup that produces

a directory tree of output files. Its performance advantage comes from combining different steps that you

might otherwise have to perform in sequence, such as combining the backup data into a single output

file and transferring it to another server. See Section 15.5, “Other Single-File Backup Operations” for

the options related to single-file backups, and Section 4.3.1, “Making a Single-File Backup” for usage

instructions.

InnoDB Configuration Options Settings

As discussed later, there are a number of reasons why you might prefer to run with the setting

innodb_file_per_table=1.

Parallel Backup

mysqlbackup can take advantage of modern multicore CPUs and operating system threads to perform

backup operations in parallel. See Section 16.10, “Performance / Scalability / Capacity Options” for the

options to control how many threads are used for different aspects of the backup process. If you see that

there is unused system capacity during backups, consider increasing the values for these options and

testing whether doing so increases backup performance:

• When tuning and testing backup performance using a RAID storage configuration, consider the

combination of option settings --read-threads=3 --process-threads=6 --write-threads=3.

Compare against the combination --read-threads=1 --process-threads=6 --write-

threads=1.

• When tuning and testing backup performance using a non-RAID storage configuration, consider the

combination of option settings --read-threads=1 --process-threads=6 --write-threads=1.

• When you increase the values for any of the 3 “threads” options, also increase the value of the --

limit-memory option, to give the extra threads enough memory to do their work.

• If the CPU is not too busy (less than 80% CPU utilization), increase the value of the --process-

threads option.

• If the storage device that you are backing up from (the source drive) can handle more I/O requests,

increase the value of the --read-threads option.

• If the storage device that you are backing up to (the destination drive) can handle more I/O requests,

increase the value of the --write-threads option.

Depending on your operating system, you can measure resource utilization using commands such as top,

iostat, sar, dtrace, or a graphical performance monitor. Do not increase the number of read or write

threads iowait once the system iowait value reaches approximately 20%.

MyISAM Considerations

Important

• Although mysqlbackup backs up InnoDB tables without interrupting database

use, the final stage that copies non-InnoDB files (such as MyISAM tables and

Network Performance

.frm files) temporarily puts the database into a read-only state, using the

statement FLUSH TABLES WITH READ LOCK. For best backup performance

and minimal impact on database processing:

1. Do not run long SELECT queries or other SQL statements at the time of the

backup run.

2. Keep your MyISAM tables relatively small and primarily for read-only or read-

mostly work.

Then the locked phase at the end of a mysqlbackup run is short (maybe a

few seconds), and does not disturb the normal processing of mysqld much. If

the preceding conditions are not met in your database application, use the --

only-innodb or --only-innodb-with-frm option to back up only InnoDB

tables, or use the --no-locking option to back up non-InnoDB files. Note that

MyISAM, .frm, and other files copied under the --no-locking setting cannot

be guaranteed to be consistent, if they are updated during this final phase of the

backup.

• For a large database, a backup run might take a long time. Always check

that mysqlbackup has completed successfully, either by verifying that

mysqlbackup returned exit code 0, or by observing that mysqlbackup has

printed the text “mysqlbackup completed OK!”.

• Schedule backups during periods when no DDL operations involving tables are

running. See Appendix B, Limitations of MySQL Enterprise Backup for restrictions

on backups at the same time as DDL operations.

Network Performance

For data processing operations, you might know the conventional advice that Unix sockets are faster

than TCP/IP for communicating with the database. Although the mysqlbackup command supports the

options --protocol=tcp, --protocol=socket, and --protocol=pipe, these options do not have

a significant effect on backup or restore performance. These processes involve file-copy operations rather

than client/server network traffic. The database communication controlled by the --protocol option is

low-volume. For example, mysqlbackup retrieves information about database parameters through the

database connection, but not table or index data.

Data Size

If certain tables or databases contain non-critical information, or are rarely updated, you can leave them

out of your most frequent backups and back them up on a less frequent schedule. See Section 16.8,

“Partial Backup and Restore Options” for information about the relevant options, and Section 4.3.5,

“Making a Partial Backup” for instructions about leaving out data from specific tables, databases, or

storage engines. Partial backups are faster because they copy, compress, and transmit a smaller volume

of data.

To minimize the overall size of InnoDB data files, consider enabling the MySQL configuration option

innodb_file_per_table. This option can minimize data size for InnoDB tables in several ways:

• It prevents the InnoDB system tablespace from ballooning in size, allocating disk space that

can afterwards only be used by MySQL. For example, sometimes huge amounts of data are

only needed temporarily, or are loaded by mistake or during experimentation. Without the

innodb_file_per_table option, the system tablespace expands to hold all this data, and never

shrinks afterward.

Advanced: Apply-Log Phase (for Directory Backups only)

• It immediately frees the disk space taken up by an InnoDB table and its indexes when the table is

dropped or truncated. Each table and its associated indexes are represented by a .ibd file that is deleted

or emptied by these DDL operations.

• It allows unused space within a .ibd file to be reclaimed by the OPTIMIZE TABLE statement, when

substantial amounts of data are removed or indexes are dropped.

• It enables partial backups where you back up some InnoDB tables and not others, as discussed in

Section 4.3.5, “Making a Partial Backup”.

• It allows the use of table compression for InnoDB tables.

In general, using table compression by having ROW_FORMAT=COMPRESSED decreases table sizes and

increase backup and restore performance. However, as a trade-off, table compression can potentially

increase redo log sizes and thus slow down incremental backups and restores, as well as apply-log

operations. See How Compression Works for InnoDB Tables for details.

Avoid creating indexes that are not used by queries. Because indexes take up space in the backup data,

unnecessary indexes slow down the backup process. (The copying and scanning mechanisms used by

mysqlbackup do not rely on indexes to do their work.) For example, it is typically not helpful to create an

index on each column of a table, because only one index is used by any query. Because the primary key

columns are included in each InnoDB secondary index, it wastes space to define primary keys composed

of numerous or lengthy columns, or multiple secondary indexes with different permutations of the same

columns.

Advanced: Apply-Log Phase (for Directory Backups only)

If you store the backup data on a separate machine, and that machine is not as busy the machine hosting

the database server, you can offload some postprocessing work (the apply-log phase) to that separate

machine. Apply-log Operation

There is always a performance tradeoff between doing the apply-log phase immediately after the initial

backup (makes restore faster), or postponing it until right before the restore (makes backup faster). In

an emergency, restore performance is the most important consideration. Thus, the more crucial the data

is, the more important it is to run the apply-log phase immediately after the backup. Either combine the

backup and apply-log phases on the same server by specifying the backup-and-apply-log option, or

perform the fast initial backup, transfer the backup data to another server, and then perform the apply-log

phase using one of the options from Apply-log Operation.

11.2 Optimizing Restore Performance

This section describes the performance considerations for restoring a database with MySQL Enterprise

Backup. This subject is important because:

• The restore operation is the phase of the backup-restore cycle that tends to vary substantially between

different backup methods. For example, backup performance might be acceptable using mysqldump,

but mysqldump typically takes much longer than MySQL Enterprise Backup for a restore operation.

• The restore operation is often performed during an emergency, where it is critical to minimize the

downtime of the application or web site.

• The restore operation is always performed with the database server shut down.

• The restore operation is mainly dependent on low-level considerations, such as I/O and network speed

for transferring files, and CPU speed, processor cores, and so on for uncompressing data.

Restoring Different Classes of Backup Data

For the combination of options you can specify for a restore job, see Section 15.3, “Restore Operations”.

Restoring Different Classes of Backup Data

Restoring a partial backup takes less time than restoring a full backup, because there is less data to

physically copy. See Section 16.8, “Partial Backup and Restore Options” for information about making

partial backups.

Restoring a compressed backup takes more time than restoring an uncompressed backup, because the

time needed to uncompress the data is typically greater than any time saved by transferring less data

across the network. If you need to rearrange your storage to free up enough space to uncompress the

backup before restoring it, include that administration work in your estimate of the total time required. In

an emergency, the time needed to uncompress the backup data before restoring it might be unacceptable.

on the database server to hold both the compressed backup and the uncompressed data. Thus, the more

critical the data is, the more likely that you might choose not to use compression: accepting a slower,

larger backup to ensure that the restore process is as fast and reliable as possible. See Section 16.6,

“Compression Options” for information about making compressed backups.

The unpacking process to restore a single-file backup is typically not expensive either in terms of raw

speed or extra storage. Each file is unpacked directly to its final destination, the same as if it was copied

individually. Thus, if you can speed up the backup substantially or decrease its storage requirements by

using single-file backups, that typically does not involve a tradeoff with restore time. See Section 15.5,

“Other Single-File Backup Operations” for information about making single-file backups.

The Apply-Log Phase (for Directory Backups only)

See Advanced: Apply-Log Phase (for Directory Backups only) for performance considerations regarding

the apply-log phase.

Network Performance

For data processing operations, you might know the conventional advice that Unix sockets are faster

than TCP/IP for communicating with the database. Although the mysqlbackup command supports the

options --protocol=tcp, --protocol=socket, and --protocol=pipe, these options do not have

a significant effect on backup or restore performance. These processes involve file-copy operations rather

than client/server network traffic. The database communication controlled by the --protocol option is

low-volume. For example, mysqlbackup retrieves information about database parameters through the

database connection, but not table or index data.

Parallel Restore

mysqlbackup can take advantage of modern multicore CPUs and operating system threads to perform

backup operations in parallel. See Section 16.10, “Performance / Scalability / Capacity Options” for the

options to control how many threads are used for different aspects of the restore process. If you see that

there is unused system capacity during a restore, consider increasing the values for these options and

testing whether doing so increases restore performance:

• When tuning and testing backup performance using a RAID storage configuration, consider the

combination of option settings --read-threads=3 --process-threads=6 --write-threads=3.

Compare against the combination --read-threads=1 --process-threads=6 --write-

threads=1.

• When tuning and testing backup performance using a non-RAID storage configuration, consider the

combination of option settings --read-threads=1 --process-threads=6 --write-threads=1.

Parallel Restore

• When you increase the values for any of the 3 “threads” options, also increase the value of the --

limit-memory option, to give the extra threads enough memory to do their work.

• If the CPU is not too busy (less than 80% CPU utilization), increase the value of the --process-

threads option.

• If the storage device that you are restoring from (the source drive) can handle more I/O requests,

increase the value of the --read-threads option.

• If the storage device that you are restoring to (the destination drive) can handle more I/O requests,

increase the value of the --write-threads option.

For an apply-log operation, the --process-threads option controls the number of threads that read

and write modified datafile pages in parallel; those threads are usually I/O bound, even though they also

perform some in-memory processing.

Depending on your operating system, you can measure resource utilization using commands such as top,

iostat, sar, dtrace, or a graphical performance monitor. Do not increase the number of read or write

threads iowait once the system iowait value reaches approximately 20%.

Chapter 12 Monitoring Backups with MySQL Enterprise Monitor

The MySQL Enterprise Monitor is a companion product to the MySQL Server that enables monitoring

of MySQL instances and their hosts, notification of potential issues and problems, and advice on how to

correct issues. Among its other functions, it can be used to monitor the progress and history of backup

jobs. Check the MySQL Enterprise Monitor User's Guide for detail.

Chapter 13 Troubleshooting for MySQL Enterprise Backup

Table of Contents

13.1 Exit codes of MySQL Enterprise Backup .................................................................................... 97

13.2 Working Around Corruption Problems ........................................................................................ 98

13.3 Using the MySQL Enterprise Backup Logs ................................................................................ 99

13.4 Using the MySQL Enterprise Backup Manifest ......................................................................... 101

To troubleshoot issues regarding backup and restore with the MySQL Enterprise Backup product, consider

the following aspects:

• Before troubleshooting any problem, familiarize yourself with the known limits and restrictions on the

product, in Appendix B, Limitations of MySQL Enterprise Backup.

• If mysqlbackup encounters problems during operating system calls, it returns the corresponding OS

error codes. You might need to consult your operating system's documentation for the meaning of those

error codes and how to handle them.

• The output from mysqlbackup is sent to stderr rather than stdout. By default, the same output is

also saved to a log file in the backup_dir for use in error diagnosis. See Section 16.11, “Message

Logging Options” for details on how to configure this logging feature.

• Incremental backups, when performed using the --start-lsn option, require care to specify a

sequence of time periods. You must record the final LSN value at the end of each backup, and specify

that value in the next incremental backup. You must also make sure that the full backup you restore is

prepared correctly first, so that it contains all the changes from the sequence of incremental backups.

• As mysqlbackup proceeds, it writes progress information into the mysql.backup_progress

table. When the command finishes the backup operation, it records status information in the

mysql.backup_history table. You can query those tables to monitor ongoing backup jobs, see how

much time has been used for various stages, and check if any errors have occurred.

13.1 Exit codes of MySQL Enterprise Backup

MySQL Enterprise Backup returns one of the following exit codes as it exists an operation. The meaning

of each code is explained in Table 13.1, “MySQL Enterprise Backup Exit Codes and Messages” by its

associated exit message.

Table 13.1 MySQL Enterprise Backup Exit Codes and Messages

Exit Code Exit Message

0 No error

1 Unknown Error

2 Internal Error

3 One of the required files is corrupt

4 One of the required files not found

5 Corrupt page or header encountered

6 Mismatch in config and the value obtained

Working Around Corruption Problems

Exit Code Exit Message

7 Illegal Argument

8 One or more of the arguments are unknown

9 IO operation failed

10 Error allocating memory

11 Connection to server failed

12 Ongoing operation interrupted by user

13 User doesn't have sufficient privileges

14 No space left on device

15 Image version is not supported by this version of meb

16 The value is out of range

17 Innodb Error

18 Timedout while waiting for resource

19 Server returned error while executing sql

The mysqlbackup command print-message takes an exit code supplied with the --error-code

option and returns the corresponding exit message in the stdout stream. Users can, for example, use

a script to catch the exit code returned by mysqlbackup, and then pass it onto the print-message

command to obtain an exit message. See the description for print-message for details.

13.2 Working Around Corruption Problems

Sometimes the operating system or the hardware can corrupt a data file page at a location that does not

cause a database error, but prevents mysqlbackup from completing:

170225 10:46:18 PCR1 INFO: Re-reading page at offset 0 in D:/temp/5.7_master/test/emp2.ibd

...

170225 10:46:26 PCR1 ERROR: Page at offset 0 in D:/temp/5.7_master/test/emp2.ibd seems corrupt!

A corruption problem can have different causes. Here are some suggestions for dealing with it:

• The problem can occur if the MySQL server is too busy. Before trying other solutions, you might want to

perform the backup again using some non-default settings for the following mysqlbackup options:

• --page-reread-time=MS. Try set the value to, for example, “0.05”, for faster rereads during

checksum failures.

• --page-reread-count=retry_limit. Try set the value to, for example, “1000”, to allow more

rereads during checksum failures before MySQL Enterprise Backup gives up and throws an error.

• Scrambled data in memory can cause the problem even though the data on disk is actually uncorrupted.

Reboot the database server and the storage device to see if the problem persists.

• If the problem persists after the database server and the storage device have been restarted, you might

really have a corruption on your disk. You might consider restoring data from an earlier backup and "roll

forward" the recent changes to bring the database back to its current state.

• If you want to make MySQL Enterprise Backup finish a backup anyway before you go and investigate

the root cause of the issue, you can rewrite the checksum values on the disk by running the

innochecksum utility on the server:

Using the MySQL Enterprise Backup Logs

innochecksum --no-checksum --write=crc32

The option --no-checksum disable the verification function of the tool, and the option --

write=crc32 makes innochecksum rewrite the checksum values on the disk.

IMPORTANT: Do not treat corruption problems as a minor annoyance. Find out what is wrong with the

system that causes the corruption—however, such troubleshooting is beyond the scope of this manual.

13.3 Using the MySQL Enterprise Backup Logs

Besides the message output of MySQL Enterprise Backup to the stderr stream and the log file,

progress and history of each backup are also logged into the mysql.backup_progress and

mysql.backup_history tables on the backed-up servers (to skip updating the two tables, use the --

no-history-logging option with the backup command).

backup_progress Table

Each row in the backup_progress table records a state change or message from a running backup job.

The backup_progress table has the following columns:

mysql> DESCRIBE mysql.backup_progress;

+---------------+---------------+------+-----+-------------------+-----------------------------------------------+

+---------------+---------------+------+-----+-------------------+-----------------------------------------------+

| error_code | int(11) | NO | | NULL | |

+---------------+---------------+------+-----+-------------------+-----------------------------------------------+

6 rows in set (0.00 sec)

The backup_progress table is in CSV format. You can query the table with the mysql client, or parse

the corresponding .CSV file with an application or script.

Here are some ways to make use of the information in the backup_progress table:

• Use the backup_id value to query all the information for different stages of a single backup operation,

and to find the corresponding row in the backup_history table for the same backup (the row is written

to the backup_history table only after the backup is finished).

• Use the error_code and error_message values to track any errors that have occurred, and to see if

the backup operation should be terminated because of any serious errors.

• Use the current_time and current_state values to track the progress of the operation. They also

allow you to measure how long each stage of the backup takes, which helps you plan for your future

backups.

backup_history Table

Each row in the backup_history table records the details of one completed backup produced by a

mysqlbackup command. The backup_history table has the following columns:

mysql> DESC backup_history;

+---------------------------+---------------+------+-----+---------------------+-------+

+---------------------------+---------------+------+-----+---------------------+-------+

backup_history Table

| start_time | timestamp | NO | | 0000-00-00 00:00:00 | |

| end_time | timestamp | NO | | 0000-00-00 00:00:00 | |

| compression_level | int(11) | NO | | NULL | |

| last_error_code | int(11) | NO | | NULL | |

| meb_version | varchar(20) | NO | | 0.0.0 | |

+---------------------------+---------------+------+-----+---------------------+-------+

30 rows in set (0.00 sec)

Warning

Because a successful backup is always recorded as such in the backup_history

table, a failure in the apply-log phase of a backup-and-apply-log command

is not reflected in the backup_history table. It is always important to check the

output of mysqlbackup to see if an operation is completed fully without an error.

Here is information on some columns of the backup_history table, and some ways to make use of the

information:

• The tool_name column records the full mysqlbackup command that triggers the backup, including all

the options used.

• You can use the end_lsn value of your latest backup as the starting LSN value for you next incremental

backup by specifying it with the --start-lsn option. (An alternative to specifying the start LSN value

for an incremental backup is to use the --incremental-base option).

• The binlog_pos column gives the position of the binary log up to where log events have been covered

by the backup. Because the backup_history table used to be in the CSV format, which cannot

the same applies to other columns for the logging of NULL values.

• The value for backup_type is one of FULL, PARTIAL, INCREMENTAL, or TTS.

• The value for backup_format is one of IMAGE (for single-file backups) or DIRECTORY (for directory

backups).

• Use the values that show the backup's settings such as mysql_data_dir, innodb_data_home_dir,

and backup_destination to confirm that the backups are using the right source and destination

directories.

100

Using the MySQL Enterprise Backup Manifest

• The value for exit_state is either SUCCESS or FAILURE. If the exit_state is SUCCESS and

last_error is 'NO_ERROR', the backup operation has been successful; when it is not the case, see

last_error and last_error_code for the latest error of the operation. To retrieve the full list of

errors for that backup operation, go to the backup_progress table.

13.4 Using the MySQL Enterprise Backup Manifest

Each backup directory includes some files in the meta subdirectory that detail how the backup was

produced, and what files it contains. The files containing this information are known collectively as the

manifest.

mysqlbackup produces these files for use by database management tools; it does not consult or modify

the manifest files after creating them. Management tools can use the manifest during diagnosis and

troubleshooting procedures, for example where the original MySQL instance has been lost entirely and the

recovery process is more involved than copying files back to a working MySQL server.

The files in the manifest include:

• backup_create.xml: information about the backup operation.

• backup_content.xml: information about the files in the backup. This information is only complete

and consistent when the backup operation succeeds. A management tool might use this information to

confirm which tables are part of a backup. A management tool might compare the checksum recorded

in the manifest for a single-file backup against the checksum for the file after the single-file backup is

unpacked. The file also contains details of all the plugins defined on the backed-up server, by which

users should make sure the same plugins are defined in the same manner on the target server for

restoration.

• image_files.xml: information about the files in a single-file backup. (Only produced for backups

taken with the backup-to-image and backup-dir-to-image commands.) A management tool

might use the paths recorded in this file to plan or automate the unpacking of a single-file backup using

the image-to-backup-dir or extract commands, or to remap the paths of extracted files with the

--src-entry and --dst-entry options.

101

102

Part III mysqlbackup Command Reference

Table of Contents

14 mysqlbackup ........................................................................................................................... 107

15 mysqlbackup commands ......................................................................................................... 109

15.1 Backup Operations ......................................................................................................... 109

15.2 Update Operations .......................................................................................................... 110

15.3 Restore Operations ......................................................................................................... 111

15.4 Validation Operations ...................................................................................................... 114

15.5 Other Single-File Backup Operations ............................................................................... 115

15.6 Other Operations ............................................................................................................ 117

16 mysqlbackup Command-Line Options ...................................................................................... 119

16.1 Standard Options ............................................................................................................ 125

16.2 Connection Options ......................................................................................................... 127

16.3 Server Repository Options ............................................................................................... 129

16.4 Backup Repository Options ............................................................................................. 132

16.5 Metadata Options ............................................................................................................ 137

16.6 Compression Options ...................................................................................................... 137

16.7 Incremental Backup Options ............................................................................................ 139

16.8 Partial Backup and Restore Options ................................................................................ 141

16.9 Single-File Backup Options ............................................................................................. 148

16.10 Performance / Scalability / Capacity Options ................................................................... 150

16.11 Message Logging Options ............................................................................................. 158

16.12 Progress Report Options ............................................................................................... 159

16.13 Encryption Options ........................................................................................................ 163

16.14 Options for Working with Encrypted InnoDB Tablespaces ............................................... 163

16.15 Cloud Storage Options .................................................................................................. 165

16.16 Options for Special Backup Types ................................................................................. 168

16.17 Other Options ............................................................................................................... 170

17 Configuration Files and Parameters ............................................................................................ 173

105

106

Chapter 14 mysqlbackup

The mysqlbackup client is an easy-to-use tool for all backup and restore operations. During backup

operations, mysqlbackup backs up:

• All InnoDB tables and indexes, including:

• The InnoDB system tablespace, which, by default contains all the InnoDB tables.

• Any separate data files produced with the InnoDB file-per-table setting. Each one contains one table

and its associated indexes. Each data file can use either the original Antelope or the new Barracuda

file format.

• All MyISAM tables and indexes.

• Tables managed by other storage engines.

• Other files underneath the MySQL data directory, such as the .frm files that record the structure of each

table.

• Any other files in the database subdirectories under the server's data directory.

In addition to creating backups, mysqlbackup can pack and unpack backup data, apply to the backup

data any changes to InnoDB tables that occurred during the backup operation, and restore data, index, and

log files back to their original locations, or to other places.

Here are some sample commands to start a backup operation with mysqlbackup are:

# Information about data files can be retrieved through the database connection.

# Specify connection options on the command line.

mysqlbackup --user=dba --password --port=3306 \

--with-timestamp --backup-dir=/export/backups \

backup

# Or we can include the above options in the configuration file

# under the [mysqlbackup] section, and just specify the configuration file

# and the 'backup' operation.

mysqlbackup --defaults-file=/usr/local/mysql/my.cnf backup

# Or we can specify the configuration file as above, but

# override some of those options on the command line.

mysqlbackup --defaults-file=/usr/local/mysql/my.cnf \

--compress --user=backupadmin --password --port=18080 \

backup

The --user and the --password you specify are used to connect to the MySQL server. This MySQL

user must have certain privileges in the MySQL server, as described in Section 4.1.2, “Grant MySQL

Privileges to Backup Administrator”.

The --with-timestamp option places the backup in a subdirectory created under the directory you

specified above. The name of the backup subdirectory is formed from the date and the clock time of the

backup run.

For the meanings of other command-line options, see Chapter 16, mysqlbackup Command-Line Options.

For information about configuration files, see Chapter 17, Configuration Files and Parameters.

Make sure that the user or the cron job running mysqlbackup has the rights to copy files from the MySQL

database directories to the backup directory.

107

Make sure that your connection timeouts are long enough so that the mysqlbackup command can keep

the connection to the server open for the duration of the backup run. mysqlbackup pings the server after

copying each database to keep the connection alive.

Important

• Although mysqlbackup backs up InnoDB tables without interrupting database

use, the final stage that copies non-InnoDB files (such as MyISAM tables and

.frm files) temporarily puts the database into a read-only state, using the

statement FLUSH TABLES WITH READ LOCK. For best backup performance

and minimal impact on database processing:

1. Do not run long SELECT queries or other SQL statements at the time of the

backup run.

2. Keep your MyISAM tables relatively small and primarily for read-only or read-

mostly work.

Then the locked phase at the end of a mysqlbackup run is short (maybe a few

seconds), and does not disturb the normal processing of mysqld much. If the

preceding conditions are not met in your database application, use the --only-

innodb option to back up only InnoDB tables, or use the --no-locking option

to back up non-InnoDB files. Note that MyISAM, .frm, and other files copied

under the --no-locking setting cannot be guaranteed to be consistent, if they

are updated during this final phase of the backup.

• For a large database, a backup run might take a long time. Always check that

the mysqlbackup command has been completed successfully by verifying that

mysqlbackup has returned the exit code 0, or by observing that mysqlbackup

has printed the text “mysqlbackup completed OK!”.

• mysqlbackup is not the same as the former “MySQL Backup” open source

project from the MySQL 6.0 source tree. The MySQL Enterprise Backup product

supersedes the MySQL Backup initiative.

• Schedule backups during periods when no DDL operations involving tables are

running. See Appendix B, Limitations of MySQL Enterprise Backup for restrictions

on creating backups in parallel with the DDL operations.

108

Chapter 15 mysqlbackup commands

Table of Contents

15.1 Backup Operations ................................................................................................................. 109

15.2 Update Operations .................................................................................................................. 110

15.3 Restore Operations ................................................................................................................. 111

15.4 Validation Operations .............................................................................................................. 114

15.5 Other Single-File Backup Operations ....................................................................................... 115

15.6 Other Operations .................................................................................................................... 117

These are commands for the major operations for mysqlbackup. Only one of them can be specified

for each mysqlbackup invocation, and, unlike the command options, the name of a command is not

preceded by any dashes.

Each of these commands has its own set of required or allowed command options. For example, the

backup command typically requires connection information to the database server. The apply-log and

other commands that operate on the backup data after it is produced require the options that specify where

the backup data is located.

The major groups of commands are:

• Backup operations: backup, backup-and-apply-log, backup-to-image

• Update operations: apply-log, apply-incremental-backup

• Restore operations: copy-back, copy-back-and-apply-log

• Validation operation: validate

• Single-file backup operations: image-to-backup-dir, backup-dir-to-image, list-image,

extract

15.1 Backup Operations

The backup operations are the most frequently performed tasks by MySQL Enterprise Backup.

Various kinds of backups can be performed by adding different options, like using --compress or

--incremental for compressed or incremental backups. Here is the syntax for the mysqlbackup

commands for performing a backup operation:

mysqlbackup [STD-OPTIONS]

[CONNECTION-OPTIONS]

[SERVER-REPOSITORY-OPTIONS]

[BACKUP-REPOSITORY-OPTIONS]

[METADATA-OPTIONS]

[COMPRESSION-OPTIONS]

[SPECIAL-BACKUP-TYPES-OPTIONS]

[INCREMENTAL-BACKUP-OPTIONS]

[PARTIAL-BACKUP-RESTORE-OPTIONS]

[SINGLE-FILE-BACKUP-OPTIONS]

[PERFORMANCE-SCALABILITY-CAPACITY-OPTIONS]

[MESSAGE-LOGGING-OPTIONS]

[PROGRESS-REPORT-OPTIONS]

[ENCRYPTION-OPTIONS]

[CLOUD-STORAGE-OPTIONS]

[ENCRYPTED-INNODB-OPTIONS]

backup-to-image

109

Update Operations

mysqlbackup [STD-OPTIONS]

[CONNECTION-OPTIONS]

[SERVER-REPOSITORY-OPTIONS]

[BACKUP-REPOSITORY-OPTIONS]

[METADATA-OPTIONS]

[COMPRESSION-OPTIONS]

[SPECIAL-BACKUP-TYPES-OPTIONS]

[INCREMENTAL-BACKUP-OPTIONS]

[PARTIAL-BACKUP-RESTORE-OPTIONS]

[PERFORMANCE-SCALABILITY-CAPACITY-OPTIONS]

[MESSAGE-LOGGING-OPTIONS]

[PROGRESS-REPORT-OPTIONS]

[ENCRYPTED-INNODB-OPTIONS]

backup | backup-and-apply-log

• backup-to-image

Produces a single-file backup holding the backup data. In most cases, single-file backups are preferred

over directory backups, which are created using the backup command.

The command requires the --backup-image option to specify the destination file. Can be used to

stream the backup to a storage device or another system without ever storing the data on the database

server. You can specify --backup-image=-, representing standard output, allowing the output to be

piped to another command. To avoid mixing normal informational messages with backup output, the --

help message, errors, alerts, and normal informational messages are always printed to standard error

stream.

The command also requires the use of the --backup-dir option to supply a temporary folder to save

the backup metadata (including the mysqlbackup message log, the start and end LSN, and so on) and

some temporary output. Note that, however, except when streaming the backup image with --backup-

image=-, if --backup-image does not give a full path name, mysqlbackup will actually take the

value of --backup-image as a path relative to the directory specified by --backup-dir, and thus

store the single-file backup under --backup-dir (or, if the --with-timestamp option is used, under

a subdirectory created under --backup-dir that bears the timestamp in its name).

• backup

Backs up data to a directory. In most cases, single-file backups, which are created using the backup-

to-image command, are preferred over directory backups.

The command only performs the initial phase of a complete backup process. The second phase is

performed later by running mysqlbackup again with the apply-log command, which makes the

backup consistent.

• backup-and-apply-log

A combination of backup and apply-log. It cannot be used for an incremental backup.

15.2 Update Operations

There are two types of operations to bring your backup data up-to-date:

Apply-log Operation

After a backup job was first completed, the backup data might not be in a consistent state, because data

could have been inserted, updated, or deleted while the backup was running. This initial backup file is

known as the raw backup. During a backup, mysqlbackup also copies the accumulated InnoDB log to

a file called ibbackup_logfile. In an apply-log operation, the ibbackup_logfile file is used to “roll

110

Apply-incremental-backup Operation

forward” the raw data files, so that every page in the data files corresponds to the same log sequence

number of the InnoDB log. This is similar to the operation that takes place during a crash recovery.

For single-file backups, the apply-log operation is usually performed as part of the copy-back-and-

apply-log command. For directory backups, the copy-back-and-apply-log command can also be

used, but you also have the two alternatives of

• Performing the apply-log operation together with the back up using the backup-and-apply-log

command (not applicable for incremental or compressed directory backups)

• Performing the apply-log operation separately with the apply-log command on the raw backup, before

running the copy-back command.

mysqlbackup [STD-OPTIONS]

[--limit-memory=MB] [--uncompress] [--backup-dir=PATH]

[MESSAGE-LOGGING-OPTIONS]

[PROGRESS-REPORT-OPTIONS]

[ENCRYPTED-INNODB-OPTIONS]

apply-log

• apply-log

Advanced: Brings the InnoDB tables in the directory backup up-to-date, including any changes made to

the data while the backup was running.

Example 15.1 Apply Log to Full Backup

mysqlbackup --backup-dir=/path/to/backup apply-log

It reads the backup-my.cnf file inside backup-dir to understand the backup. The my.cnf defaults files

have no effect other than supplying the limit-memory=MB value, which limits usage of memory while

doing the apply-log operation.

Apply-incremental-backup Operation

Advanced: Use the apply-incremental-backup to update a backup directory with data in an

incremental backup directory:

mysqlbackup [STD-OPTIONS]

[--incremental-backup-dir=PATH] [--backup-dir=PATH]

[--limit-memory=MB] [--uncompress]

[MESSAGE-LOGGING-OPTIONS]

[PROGRESS-REPORT-OPTIONS]

[ENCRYPTED-INNODB-OPTIONS]

apply-incremental-backup

• apply-incremental-backup

Advanced: Brings up-to-date a directory backup, specified by the --backup-dir option, using the data

from an incremental backup directory, specified with the --incremental-backup-dir option. See

Section 5.1.3, “Restoring an Incremental Backup” for instructions on restoring incremental backups.

For a single-file incremental backup, you typically use the copy-back-and-apply-log command with

the --incremental option to apply the data in the incremental image backup to the full backup that

has already been restored to the data directory of the target server.

15.3 Restore Operations

The restore operations restores the data files from a backup to their original locations on the database

server, or to other desired locations. Normally, the restoration process requires the database server

111

Restore Operations

to be already shut down (or, at least not operating on the directory you are restoring the data to),

except for restorations of backups created with the --use-tts option. The options datadir,

innodb_log_files_in_group, and innodb_log_file_size must be specified either in the target

server's configuration file, in the file specified by the --defaults-file option, or as command-line

options. For usage examples, see Chapter 5, Recovering or Restoring a Database.

mysqlbackup [STD-OPTIONS]

[SERVER-REPOSITORY-OPTIONS]

[--backup-image=IMAGE]

[--backup-dir=PATH]

[--uncompress]

[MESSAGE-LOGGING-OPTIONS]

[PARTIAL-BACKUP-RESTORE-OPTIONS]

[PROGRESS-REPORT-OPTIONS]

[ENCRYPTION-OPTIONS]

[CLOUD-STORAGE-OPTIONS]

[ENCRYPTED-INNODB-OPTIONS]

copy-back-and-apply-log

mysqlbackup [STD-OPTIONS]

[SERVER-REPOSITORY-OPTIONS]

[--backup-dir=PATH]

[--uncompress]

[MESSAGE-LOGGING-OPTIONS]

[PARTIAL-BACKUP-RESTORE-OPTIONS]

[PROGRESS-REPORT-OPTIONS]

[CLOUD-STORAGE-OPTIONS]

[ENCRYPTED-INNODB-OPTIONS]

copy-back

• copy-back-and-apply-log

In a single step, restores a single-file backup specified by the --backup-image option or a backup

from the directory specified by the --backup-dir option to a server's data directory and performs

an apply-log operation to the restored data to bring them up-to-date. Comparing with a multi-step

approach for restoring a single-file backup (which typically consists of performing the successive steps

of extract, uncompress, apply-log, and copy-back for restoring compressed image, or extract ,apply-

log, and copy-back for uncompressed image), the command makes the restoration process simpler and

faster, and also saves the disk space required.

The following are some special requirements for different kinds of backup restoration using copy-back-

and-apply-log:

• To restore a compressed directory or image, include the --uncompress option in the command line.

• To restore a single-file backup, besides specifying the location of the backup image with the --

backup-image option, also supply with the --backup-dir option the location of a folder that will be

used for storing temporary files produced during the restoration process.

• To restore a single-file incremental backup, assuming the full backup (on which the incremental

backup was based) has already been restored:

• Include the --incremental option in the command line.

• Specifies the location of the incremental backup image with the --backup-image option

• Supplies with the --backup-dir option the location of a folder that will be used for storing

temporary files produced during the restoration process.

• Advanced: To restore an incremental backup directory, assuming the full backup (on which the

incremental backup was based) has already been restored:

112

Restore Operations

• Include the --incremental option in the command line.

• Use either the --backup-dir or --incremental-backup-dir option to specify the incremental

backup directory.

• To restore a backup created with the --use-tts option:

• See the general requirements described in Section 5.1.4, “Restoring Backups Created with the --

use-tts Option”.

• When restoring a single-file backup created with the option setting --use-tts=with-minimum-

locking, the folder specified with --backup-dir is also used for extracting temporarily all the

tables in the backup and for performing an apply-log operation to make the data up-to-date

before restoring them to the server's data directory.

• Advanced: When restoring a backup directory created with the option --use-tts=with-

minimum-locking, an apply-log operation will be performed on the backup directory. That

means the backup taken will be altered during the process, and users might want to make an extra

copy of the backup directory before proceeding with the restoration, in order to prevent the loss of

backup data in case something goes wrong.

Also note that:

• Backups created with the --skip-unused-pages option cannot be restored using copy-back-

and-apply-log.

• For image backups taken with MySQL Enterprise Backup 3.8.2 or earlier, per-table .ibd files pointed

to by .isl files in a backup are restored by copy-back-and-apply-log to the server's data

directory rather than the locations pointed to by the .isl files.

• Due to a known issue, when restoring a compressed backup created with MySQL Enterprise Backup

3.9 or earlier and containing any InnoDB tables that were created on the server as compressed tables

(by using the ROW_FORMAT=COMPRESSED option, the KEY_BLOCK_SIZE= option, or both), do not use

copy-back-and-apply-log; instead, perform an apply-log first, and then a copy-back. See

entry for Bug# 17992297 in the MySQL Enterprise Backup 3.10.0 changelog for details.

At the end of the copy-back-and-apply-log operation, the file backup_variables.txt is being

created or updated in the data directory. This file contains metadata about the restored contents and

is being used by successive single-step restores of incremental backups; it should not be deleted or

modified by users.

For some sample commands for restoring different kinds of backups with the copy-back-and-apply-

log command, see Section 5.1, “Performing a Restore Operation”.

• copy-back

Restores files from a directory backup to their original locations within the MySQL server.

Before restoring a hot backup using the copy-back command, the backup has to be prepared and

made consistent using the apply-log command. See Section 5.1.7, “Advanced: Preparing and

113

Validation Operations

Restoring a Directory Backup” for details. You can also perform apply-log and copy-back together

with a single copy-back-and-apply-log command.

Some clean-up efforts on the target directory for restoration might be needed before performing a full

restore (for example, when the backup data is used to set up a new MySQL server or to replace all data

of an existing MySQL server). See the discussions here for details.

There are some special requirements when restoring backups created with the --use-tts option; see

Section 5.1.4, “Restoring Backups Created with the --use-tts Option” for details.

Warning

When restoring a server for replication purpose, if the backed-up server has used

the innodb_undo_directory option to put the undo logs outside of the data

directory, when using the file server-my.cnf or server-all.cnf for the --

defaults-file option with copy-back or copy-back-and-apply-log, care

should be taken to configure correctly the innodb_undo_directory option in the

file. Otherwise, the data or log files on the original server might be overwritten by

accident.

15.4 Validation Operations

To ensure the integrity of the backup data, MySQL Enterprise Backup provides a validate command for

validating a backup by the checksum values of its data pages after the backup is created or transferred to

another system.

mysqlbackup [STD-OPTIONS]

[--backup-dir=PATH][--backup-image=IMAGE]

[MESSAGE-LOGGING-OPTIONS]

[PROGRESS-REPORT-OPTIONS]

[CLOUD-STORAGE-OPTIONS]

validate

• validate

Verifies that a backup is not corrupted, truncated, or damaged. This operation validates the checksum

value for each data page in a backup.

To avoid spending excessive time and resources on files that are too heavily corrupted, mysqlbackup

stops validating a .ibd file after more than twenty corrupted pages are found in it, and proceeds to the

next file instead. In that case, the operation's summary will not give a full count of corrupted pages, but

only says “at least 20 pages are corrupted.”

The operation also has the following limitations:

• If any .ibd or .frm files are missing from the data directory during a backup or have been deleted

from a backup after the backup was made, the validate operation will not be able to detect the

problem.

• If a backup has been corrupted by removing or truncating pages from any of the .ibd files inside , the

validate operation will not be able to detect the problem.

• For any backup directory, the operation can only validate the InnoDB data files (ibdata* and *.ibd

files) in it. Problems with other file types within a backup directory (for example, .frm file corruptions)

are not detected.

Here is a sample command for validating a backup image:

114

Other Single-File Backup Operations

mysqlbackup -uroot --backup-image=/logs/fullimage.mi validate

Advanced: Here is a sample command for validating a backup directory:

mysqlbackup -uroot --backup-dir=/logs/backupext validate

For more usage examples for the validate command, see Section 4.2.3, “Verifying a Backup”

15.5 Other Single-File Backup Operations

Besides the commands for creating and restoring single-file backups (namely, backup-to-image and

copy-back-and-apply-log), mysqlbackup provides a number of other commands for you to work

with single-file backups. They are explained below.

mysqlbackup [STD-OPTIONS]

[--backup-image=IMAGE] [--backup-dir=PATH]

[MESSAGE-LOGGING-OPTIONS]

[PROGRESS-REPORT-OPTIONS]

[ENCRYPTION-OPTIONS]

[CLOUD-STORAGE-OPTIONS]

image-to-backup-dir

mysqlbackup [STD-OPTIONS]

[--backup-dir=PATH] [--backup-image=IMAGE]

[MESSAGE-LOGGING-OPTIONS]

[PROGRESS-REPORT-OPTIONS]

[ENCRYPTION-OPTIONS]

[CLOUD-STORAGE-OPTIONS]

backup-dir-to-image

mysqlbackup [STD-OPTIONS]

[--backup-image=IMAGE]

[MESSAGE-LOGGING-OPTIONS]

[ENCRYPTION-OPTIONS]

[CLOUD-STORAGE-OPTIONS]

list-image

mysqlbackup [STD-OPTIONS]

[--backup-image=IMAGE]

[--backup-dir=PATH]

[--src-entry=PATH] [--dst-entry=PATH]

[--uncompress]

[MESSAGE-LOGGING-OPTIONS]

[PROGRESS-REPORT-OPTIONS]

[ENCRYPTION-OPTIONS]

[CLOUD-STORAGE-OPTIONS]

extract

• image-to-backup-dir

Unpacks a single-file backup to a full backup directory structure. You specify the paths to both the

image file and the destination directory for the unpacking. For usage examples, see Section 4.3.1,

“Making a Single-File Backup”.

Note

image-to-backup-dir only creates a raw backup directory, which is NOT

ready to be restored by the copy-back command. To become a prepared

backup, the backup directory has to go through an apply-log operation, executed

either by a stand-alone apply-log command or as a part of a copy-back-

and-apply-log command.

115

Other Single-File Backup Operations

• backup-dir-to-image

Packs an existing backup directory into a single file. The value for the --backup-image parameter

should either be “-”(stands for standard output) or an absolute path outside of the backup-dir

directory. Specify a --backup-image value of - (standard output) to stream an existing backup

directory structure to a tape device or a command that transfers the backup to another server. For usage

examples, see Section 4.3.1, “Making a Single-File Backup”.

• list-image

Display the contents of a single-file backup. Lists all files and directories in the image. For usage

examples, see Section 4.3.1, “Making a Single-File Backup”.

Note

The list-image operation can be performed on a cloud backup only if the

cloud proxy supports HTTP range headers.

• extract

Unpacks individual files or directories from a single-file backup. It is useful for troubleshooting, or for

restorations that do not require the full set of backup data. The resulting file or directory goes into the

current directory, or into the backup directory, if specified with --backup-dir; in either case, the

destination directory must be empty. For usage examples, see Section 4.3.1, “Making a Single-File

Backup”.

The --src-entry=string option can be used for selective extraction of files or directories whose path

names in the image contain the string specified with the option.

Notes

• Some items are always extracted from the backup; see the descriptions of --

src-entry for details.

• The option is currently not supported for the extraction of cloud backups, which

can only be extracted in full.

The --dst-entry=path option, along with --src-entry=path option, can be used to extract files or

directories into user-specified locations; see the description for the option for details.

For MySQL Enterprise Backup 4.1.4 and later: Use the --uncompress option to extract files from a

compressed single-file backup.

The default destination for the extract is the current working directory. All the files with relative

pathnames in the image are extracted to pathnames relative to the destination directory. If the image

contains some entries with absolute pathnames, those entries are extracted to the same absolute

pathnames on the local system even if the --backup-dir option is specified. The --dst-entry

option must be used to relocate an absolute pathname; see Example 4.12, “Dealing with Absolute Path

Names”.

Important

Even with all files extracted from the backup image, extract only creates a

raw backup directory, which is NOT ready to be restored by the copy-back

command. To become a prepared backup, the backup directory has to go

through an apply-log operation, executed either by a stand-alone apply-log

command or as a part of a copy-back-and-apply-log command.

116

Other Operations

15.6 Other Operations

This group of operations consists of any mysqlbackup commands not covered in other sections of this

chapter.

mysqlbackup --error-code=CODE print-message

• print-message

Prints the associated exit message for a mysqlbackup exit code to the stdout stream.

Use the --error-code option to supply the exit code for which you want to receive the associated exit

message:

$ mysqlbackup print-message --error-code=4 2> /dev/null

One of the required files not found

For a list of mysqlbackup exit codes and messages, see Section 13.1, “Exit codes of MySQL

Enterprise Backup”.

117

118

Chapter 16 mysqlbackup Command-Line Options

Table of Contents

16.1 Standard Options .................................................................................................................... 125

16.2 Connection Options ................................................................................................................. 127

16.3 Server Repository Options ...................................................................................................... 129

16.4 Backup Repository Options ..................................................................................................... 132

16.5 Metadata Options ................................................................................................................... 137

16.6 Compression Options .............................................................................................................. 137

16.7 Incremental Backup Options .................................................................................................... 139

16.8 Partial Backup and Restore Options ........................................................................................ 141

16.9 Single-File Backup Options ..................................................................................................... 148

16.10 Performance / Scalability / Capacity Options ........................................................................... 150

16.11 Message Logging Options ..................................................................................................... 158

16.12 Progress Report Options ....................................................................................................... 159

16.13 Encryption Options ................................................................................................................ 163

16.14 Options for Working with Encrypted InnoDB Tablespaces ....................................................... 163

16.15 Cloud Storage Options .......................................................................................................... 165

16.16 Options for Special Backup Types ......................................................................................... 168

16.17 Other Options ....................................................................................................................... 170

The following sections describe the command-line options for the different modes of operation of

mysqlbackup.

The table below list all the command options for mysqlbackup. Use the hyperlinks at the option names to

jump to the detailed descriptions for the options.

Note

The command options can also be specified in configuration files; see explanations

in Chapter 17, Configuration Files and Parameters. mysqlbackup follows the

MySQL standard practice for handling duplicate options, whether specified in a

configuration file, on the command line, or both. Options are processed first from

configuration files, then from the command line. If an option is specified more than

once, the last instance takes precedence.

Table 16.1 List of All Options

Option Name Description IntroducedDeprecated

--backup-dir The directory to store the backup data.

--backup-image Specifies the path name of the backup image.

backup_innodb_checksum_algorithm

The name of the checksum algorithm used for

validating InnoDB tablespaces.

backup_innodb_data_file_path

Specifies Innodb system tablespace files' path and

size in backup.

backup_innodb_data_home_dir

Backup base directory for all InnoDB data files in the

system tablespace.

backup_innodb_log_file_size

The size in bytes of each InnoDB backup log file.

119

Option Name Description IntroducedDeprecated

backup_innodb_log_files_in_group

Number of InnoDB log files in backup.

backup_innodb_log_group_home_dir

Backup directory for InnoDB log files.

backup_innodb_page_size

The page size for all InnoDB tablespaces in a

MySQL instance.

backup_innodb_undo_directory

The relative or absolute directory path where

InnoDB creates separate tablespaces for the undo

logs.

backup_innodb_undo_logs

Number of rollback segments in the system

tablespace that InnoDB uses within a transaction.

backup_innodb_undo_tablespaces

The number of tablespace files that the undo

logs are divided between when a non-zero

innodb_undo_logs setting is used.

--character-sets-dir Directory for character set files.

--cloud-access-key-id AWS access key ID for logging onto Amazon S3.

--cloud-aws-region Region for Amazon Web Services that mysqlbackup

access for S3.

--cloud-basicauth-url The URL for HTTP Basic Authentication for

accessing Swift.

4.1.1

--cloud-bucket The storage bucket for the backup image.

--cloud-buffer-size Size of buffer for cloud operations. 4.1.1

--cloud-ca-info Absolute path to the CA bundle file for host

authentication for SSL connections.

--cloud-ca-path CA certificate directory, in addition to the system's

default folder.

--cloud-chunked-transfer Use chunked transfer with cloud storage service. 4.1.1

--cloud-container The Swift container for the backup image.

--cloud-identity-url The URL of the Keystone identity service.

--cloud-object The storage object for the backup image.

--cloud-object-key The Amazon S3 object key for the backup image.

--cloud-password Password for user specified by --cloud-user-id.

--cloud-proxy Proxy address and port number for overriding the

environment's default proxy settings for accessing

cloud service.

--cloud-region The Keystone region for the user specified by --

cloud-user-id.

--cloud-secret-access-key AWS secret access key.

--cloud-service Cloud service for data backup or restoration.

--cloud-tempauth-url The URL of the identity service for authenticating

user credentials with Swift's TempAuth

authentication system.

120

Option Name Description IntroducedDeprecated

--cloud-tenant The Keystone tenant for the user specified by --

cloud-user-id.

--cloud-trace Print trace information for cloud operations.

--cloud-user-id User ID for accessing Swift.

--comments Specifies comments string.

--comments-file Specifies path to comments file.

--compress Create backup in compressed format.

--compress-level Specifies the level of compression.

--compress-method Specifies the compression algorithm.

--connect-if-online Use connection only if available.

--connect_timeout Connection timeout in seconds.

--databases [Legacy] Specifies the list of non-InnoDB tables to

back up.

--databases-list-file [Legacy] Specifies the pathname of a file that lists

the non-InnoDB tables to be backed up.

--datadir Path to mysql server data directory.

--debug Print debug information.

--decrypt Decrypt backup image written in an MEB Secure

File.

--default-character-set Set the default character set.

--defaults-extra-file Read this file after the global files are read.

--defaults-file Only read default options from the given file.

--defaults-group-suffix Also read option groups with the usual names and a

suffix of str.

--disable-manifest Disable generation of manifest files for a backup

operation.

--dst-entry Used with single-file backups to extract a single file

or directory to a user-specified path.

--encrypt Encrypt backup image and write it in an MEB

Secure File.

--encrypt-password The user-supplied password by which mysqlbackup

encrypts the encryption keys for encrypted InnoDB

tablespaces.

--error-code The exit code for which the print-message command

prints the corresponding exit message.

--exclude-tables Exclude in a backup or restore tables whose names

match the regular expression REGEXP.

--exec-when-locked Execute the specified utility in the lock phase near

the end of the backup operation.

--force Force overwriting of data, log, or image files,

depending on the operation.

--free-os-buffers Free filesystem cache by syncing the buffers

121

Option Name Description IntroducedDeprecated

--generate-new-master-key Generate new master key for encrypted InnoDB

tablespaces.

--help Display help.

--host Host name to connect.

--include [Legacy] Backup only those per-table innodb data

files which match the regular expression REGEXP.

--include-tables Include in a backup or a restore tables whose

names match the regular expression REGEXP.

--incremental Specifies that the associated backup or backup-to-

image operation is incremental.

--incremental-backup-dir Specifies the location for an incremental directory

backup.

--incremental-base The specification of base backup for --incremental

option.

--incremental-with-redo-log-

only

Specifies the incremental backup of InnoDB tables

to be based on copying redo log to the backup,

without including any InnoDB data files in the

backup.

innodb_checksum_algorithm

The name of the checksum algorithm used for

validating InnoDB tablespaces.

--innodb_data_file_path Specifies InnoDB system tablespace files' path and

size.

--innodb_data_home_dir Specifies base directory for all InnoDB data files in

the shared system tablespace.

--innodb_log_file_size The size in bytes of each InnoDB log file in the log

group.

--innodb_log_files_in_group The number of InnoDB log files.

innodb_log_group_home_dir

The directory path to InnoDB log files.

--innodb_page_size The page size for all InnoDB tablespaces in a

MySQL instance.

--innodb_undo_directory The directory path to InnoDB undo tablespaces.

--key The symmetric key used for encryption and

decryption.

--key-file The pathname of a file that contains the symmetric

key used for encryption and decryption.

--keyring The kind of keyring plugin used for master

encryption key management.

--keyring_file_data Path to the keyring file.

--keyring_okv_conf_dir Path to the Oracle Key Vault (OKV) endpoint

directory.

--limit-memory The memory in MB available for the MEB operation.

122

Option Name Description IntroducedDeprecated

--lock-wait-retry-count Specify the maximum number of retries to be

attempted by mysqlbackup after the FLUSH

TABLES WITH READ LOCK statement fails due to

a timeout.

4.1.4

--lock-wait-timeout Specify the timeout in seconds for the FLUSH

TABLES WITH READ LOCK statement that

mysqlbackup issues during the final stage of a

backup.

--log-bin Specify the location for the binary log to be restored. 4.1.2

--log-bin-index Specifies the absolute path of the index file that lists

all the binary log files.

--login-path Read options from the named login path in

the .mylogin.cnf login file.

--master-info-file Specifies the absolute path of the information file in

which a replica records information about its source

(for offline backups of replica servers only).

--messages-logdir Specifies the path name of an existing directory for

storing the message log.

--no-connection Do not connect to server.

--no-defaults Do not read default options from any given file.

--no-history-logging Disable history logging even if connection is

available.

--no-locking Disable all locking of tables during backups.

--number-of-buffers Specifies the exact number of memory buffers to be

used for the backup operation.

--on-disk-full Specifies the behavior when a backup process

encounters a disk-full condition.

--only-innodb Back up only InnoDB data and log files.

--only-innodb-with-frm [Legacy] Back up only InnoDB data, log files, and

the .frm files associated with the InnoDB tables.

--only-known-file-types Includes only files of a list of known types in the

backup.

--optimistic-busy-tables Perform an optimistic backup, using the regular

expression specified with the option to select tables

that will be skipped in the first phase of an optimistic

backup.

--optimistic-time Perform an optimistic backup with the value

specified with the option as the optimistic time—a

time after which tables that have not been modified

are believed to be inactive tables.

--page-reread-count Maximum number of page re-reads.

--page-reread-time Wait time before a page re-read.

--password Connection password.

--pipe alias for –protocol=pipe.

123

Option Name Description IntroducedDeprecated

--port TCP portnumber to connect to.

--print-defaults Print a list of option values supplied by defaults files

and exit.

--process-threads Specifies the number of process-threads for the

backup operation.

--progress-interval Interval between progress reports in seconds.

--protocol Connection protocol.

--read-threads Specifies the number of read-threads for the backup

operation.

--relay-log Specify the location for the relay log to be restored

on a replica server.

4.1.2

--relay-log-index Specifies the absolute path of the index file that lists

all the relay log files.

--relaylog-info-file Specifies the absolute path of the information file in

which a replica records information about the relay

logs (for offline backups of replica servers only).

--rename Rename a single table when it is selected by the --

include-tables option to be restored

--safe-slave-backup-

timeout

When backing up a replica server, the timeout value

for waiting for the replication SQL thread to drop its

temporary tables.

--sbt-database-name Used as a hint to the Media Management Software

(MMS) for the selection of media and policies for

tape backup.

--sbt-environment Comma separated list of environment variable

assignments to be given to the SBT library.

--sbt-lib-path Path name of the SBT library used by software that

manages tape backups.

--secure-auth Refuse client connecting to server if it uses old

(pre-4.1.1) protocol.

--shared-memory-base-

name

It designates the shared-memory name used by a

Windows server to permit clients to connect using

shared memory (Windows only).

--show-progress Instructs mysqlbackup to periodically output short

progress reports known as progress indicators on its

operation.

--skip-binlog Do not include binary log files during backup, or do

not restore binary log files during restore.

--skip-final-rescan Skip the final rescan for InnoDB tables that are

modified by DDL operations.

--skip-messages-logdir Disable logging to teelog file.

--skip-relaylog Do not include relay log files during backup, or do

not restore relay log files during a restore.

124

Standard Options

Option Name Description IntroducedDeprecated

--skip-unused-pages Skip unused pages in tablespaces when backing up

InnoDB tables.

--slave-info Capture information needed to set up an identical

replica server.

--sleep Time to sleep in milliseconds after copying each

1MB of data.

--socket Socket file to use to connect.

--src-entry Identifies a file or directory to extract from a single-

file backup.

--ssl Enable SSL for connection (automatically enabled

with other --ssl- flags).

Yes

--ssl-ca CA file in PEM format (implies –ssl).

--ssl-capath CA directory (check OpenSSL docs,implies --ssl).

--ssl-cert X509 cert in PEM format (implies --ssl).

--ssl-cipher SSL cipher to use (implies --ssl).

--ssl-key X509 key in PEM format (implies --ssl).

--ssl-mode Security state of connection to server.

--ssl-verify-server-cert Verify server's "Common Name" in its cert against

hostname used when connecting.

Yes

--start-lsn Specifies the highest LSN value included in a

previous backup.

--suspend-at-end Pauses the mysqlbackup command when the

backup procedure is close to ending.

--trace Trace level of messages by mysqlbackup.

--uncompress Uncompress a backup during an operation.

--use-tts Enable selective backup of InnoDB tables using

transportable tablespaces (TTS).

--user Database user name to connect.

--verbose Print more verbose information.

--version Display version information.

--with-timestamp Create a subdirectory underneath the backup

directory with a name formed from the timestamp of

the backup operation.

--write-threads Specifies the number of write-threads for the backup

operation.

16.1 Standard Options

The standard options are options of a general nature, or options that are not classified under any other

specific option group:

• The following standard options also exist for the mysql command. Full descriptions for these options

can be found in the MySQL reference manual, accessible through, e.g., Server Option, System Variable,

125

Standard Options

and Status Variable Reference. These options must be specified ahead of any other mysqlbackup

options, including the rest of the standard options:

--print-defaults Print the program argument list and exit.

--no-defaults Don't read default options from any option file.

--defaults-file=PATH Only read default options from the given file. It has to be the first option to be specified, if used.

--defaults-extra-file=PATH Read this file after the global files are read.

--defaults-group-suffix=str Also read option groups with the usual names and a suffix of str.

• The following options are also common between mysqlbackup and mysql, and full descriptions for

them can be found in the MySQL reference manual, accessible through, e.g., Server Option, System

Variable, and Status Variable Reference. However, mysqlbackup does not accept any short forms for

these options as mysql does (for example, you must use --help instead of -h for mysqlbackup):

--help Display help.

--version Display version information.

• More standard options are available for mysqlbackup:

--verbose: Print more verbose information.

--debug=STRING: Print additional debug information. The option accepts the following arguments:

• all: Print additional debug information for all operations

• SBT: Print additional debug information for operations using the System Backup to Tape (SBT)

interface

• null: When a null string or no argument at all is specified for the option, mysqlbackup behaves as if

the --verbose option is used.

--force: By default, some of the operations halt rather than overwrite any user data or log files when

told to write to existing files. --force allows the following:

Warning

For any restore operations, do NOT attempt to restore data to a non-empty data

directory using the --force option; doing so may cause data corruption and

other unexpected behaviors. Do not use the --force option with a copy-back

126

Connection Options

or a copy-back-and-apply-log operation, except for the special cases

described below.

• Overwriting of InnoDB data and log files during the apply-log and apply-incremental-backup

operations.

• When restoring a TTS backup, changing temporarily the value of innodb_file_format on the

server, in order to allow restores of per-table InnoDB data files regardless of their format.

• Replacing of an image file during an backup-to-image or backup-dir-to-image operation.

• Restoring a partial image backup created with MySQL Enterprise Backup 3.11 or earlier; the --force

option is required, due to a known issue (Bug# 20485910).

• Restoring a backup onto a server where the directory pointed to by the .bl file in the backup (a copy

of the .isl file from the backed-up server) already contains .ibd data files.

--trace=level

Command-Line Format --trace=LEVEL

Type Enumeration

Default Value 0

Valid Values 0

Trace level of mysqlbackup messages. The permissible levels, in the order of increasing fineness, are:

• 0 - INFO (information, warnings, errors)

• 1 - FINE (more information given than at trace level 0)

• 2 - FINER (finer level of information given than at trace level 1)

• 3 - FINEST (finest level of information that can be given)

16.2 Connection Options

When mysqlbackup creates a backup, it sends SQL commands to MySQL server using a database

connection. The general connection details are the same as described in Connecting to the MySQL Server

Using Command Options in the MySQL Reference Manual.

As part of the mysqlbackup invocation, specify the appropriate --user, --password, --port, and/or

--socket options that are necessary to connect to the MySQL server.

You can specify the following connection-specific options in the [mysqlbackup] or [client] sections

of a MySQL configuration file, or through mysqlbackup command-line options. mysqlbackup reads your

default configuration files and then the my.cnf file specified on the command line.

127

Connection Options

Note

• mysqlbackup reads only --user, --password, --port, and --socket

options from the [client] group, and ignores any other connection options.

• If you do not provide a value for the --password, the command prompts for one

from the keyboard.

• The --host option is allowed in the configuration file for compatibility, but

currently it has no effect. mysqlbackup always connects to the local server's IP

address.

Options Common to mysqld

========================

--login-path=name

--port=port-num

--protocol=tcp|socket|pipe|memory

--pipe [ alias for --protocol=pipe ]

--user=name [ short option: -u ]

--host=hostname

--socket=name

--shared-memory-base-name=value [Windows only]

--character-sets-dir=PATH

--default-character-set=VALUE

--secure-auth [ Don't connect to pre-4.1.1 server ]

--password[=value] [ short option: -p ]

--connect-timeout

--ssl [ Enable SSL for connection. Deprecated. ]

--ssl-mode=mode

--ssl-key=file_name

--ssl-cert=file_name

--ssl-ca=file_name

--ssl-capath=directory_name

--ssl-cipher=cipher_list

--ssl-verify-server-cert [ Deprecated. ]

Connection Options Specific to mysqlbackup

==========================================

--no-connection

--connect-if-online

Most other connection parameters used by the mysql command are recognized, but silently ignored.

Unknown connection parameters cause mysqlbackup to stop.

The following connections options are specific to mysqlbackup:

• --no-connection

The --no-connection option supersedes the other connection options and uses file-level operations

to perform the backup. When you use this option, you must specify in the configuration file or on the

command line many options whose values are normally retrieved automatically through the database

connection.

Warning

This option also turns on the --no-history-logging and --no-locking

options, which might result in inconsistencies in non-InnoDB data if the tables are

modified during the backup operation. It might also affect subsequent incremental

backups; see the description for the --incremental-base option for details.

128

Server Repository Options

• --connect-if-online

By default, a database connection is used for backup operations both during the initial stage to retrieve

source repository configuration, and to lock tables while copying non-InnoDB data. This option allows

mysqlbackup to make connection attempts in both phases, but continues even if the connection cannot

be established. If a connection cannot be established, the processing is the same as with the --no-

connection option. This option can be useful in emergency situations: for example, when the database

server goes down during the backup operation.

16.3 Server Repository Options

These repository options specify various parameters related to the database server, from which the data is

backed up or to which a backup is restored.

These options are used only with the following operations:

• Backup creation operations: backup, backup-and-apply-log, backup-to-image.

• Restore operations: copy-back, copy-back-and-apply-log.

When a database connection is available during a backup, the parameters describing the source repository

are ignored, overridden by the corresponding values retrieved from the database connection.

For information about how these options are specified for the MySQL server, click the option names to see

the descriptions in the MySQL Reference Manual.

• datadir=PATH

This is the datadir value used by the MySQL instance. The .frm files reside here inside

subdirectories named after the databases inside the instance.

When a database connection exists, the value is retrieved automatically and overrides any value you

specify. This is a crucial parameter for both the MySQL server and MySQL Enterprise Backup.

• log-bin[=basename]

For MySQL Enterprise Backup 4.1.2 and later: Specify the location for the binary log to be restored. By

default, during a restore, the binary log is restored to the same location it was found on the backed-up

server. Use this option to specify a different target location for the binary log. The option works similarly

as the --log-bin option of the MySQL server in determining the location and the name of the binary

log files—see description of the server --log-bin option for details. As a summary:

• Using this option without supplying a basename puts the binary log in the target server's data directory

with the default basename host_name-bin.

• Using this option while supplying a basename puts the binary log in the target server's data directory

with the specified basename.

• Using this option while supplying a basename containing a full file path (for example, /home/admin/

db/binlogdir/binlog) puts the binary log on the target server in the specified folder (/home/

admin/db/binlogdir/) using the supplied basename (binlog).

The option is only for the copy-back-and-apply-log and copy-back operations. Using it with any

other operations makes the command fail.

• relay-log[=basename]

129

Server Repository Options

For MySQL Enterprise Backup 4.1.2 and later: Specify the location for the relay log to be restored on

a replica server. By default, during a restore for a replica server, the relay log is restored to the same

location it was found on the backed-up replica server. Use this option to specify a different target location

for the relay log. The option works similarly as the --log-bin option for mysqlbackup in determining

the location and the name of the relay log—see description of the --log-bin option for details.

The option is only for the copy-back-and-apply-log and copy-back operations. Using it with any

other operations makes the command fail.

• innodb_data_home_dir=PATH

Specifies the directory where InnoDB data files reside. Usually the same as datadir, but can be

different.

This parameter, together with innodb_data_file_path=SIZE, determines where the InnoDB data

files such as ibdata1, ibdata2, and so on, are situated within the MySQL server.

For backups: Typically, you do not need to specify this option, because its value is retrieved

automatically using the database connection. Specifies the option for an offline backup if the InnoDB

data files do not reside under the server's data directory.

For restores: The directory where InnoDB data files are to be restored. Specify the option only if the

InnoDB data files are to be restored outside of the server's data directory. For release 4.1.3 and later: the

specified directory must be non-existent or empty, or the restore operation will fail, even if the --force

option is used.

Its value is derived as follows:

• If innodb_data_home_dir is not specified, it inherits the value of datadir.

• If innodb_data_home_dir is a relative path, the path is located relative to (that is, underneath) the

datadir value.

• An innodb_data_home_dir of "" refers to the / root directory.

• If innodb_data_home_dir is an absolute path, its value is used as-is.

• innodb_data_file_path=VALUE

Specifies InnoDB data file names and sizes. Examples:

ibdata1:32M;ibdata2:32M:autoextend

/abs/path/ibdata1:32M:autoextend

innodb-dir/ibdata1:32M:autoextend

When a database connection exists, the value is retrieved automatically and overrides any value you

specify.

This parameter together with innodb_data_home_dir determines where the InnoDB data files (such as

ibdata1, ibdata2, and so on) reside in server repository.

Typically, you do not need to specify this option, because its value is retrieved automatically using the

database connection. If no database connection is available, you must specify it.

Whether the initial file name begins with a / character or not, the files are located relative to the

innodb_data_home_dir value.

130

Server Repository Options

• innodb_log_group_home_dir=PATH

Specifies where the InnoDB redo log reside within the server repository. Usually the same as datadir,

but can be different.

For backups: Typically, you do not need to specify this option, because its value is retrieved

automatically using the database connection. Specifies the option for an offline backup if the InnoDB

redo log files do not reside under the server's data directory.

For restores: The directory where InnoDB redo log files are to be restored. Specify the option only if the

InnoDB redo log files are to be restored outside of the server's data directory. For release 4.1.3 and later:

the specified directory must be non-existent or empty, or the restore operation will fail, even if the --

force option is used.

Its value is derived as follows:

• If innodb_log_group_home_dir is not specified, it inherits the value of datadir.

• If innodb_log_group_home_dir is a relative path, the path is taken to be relative to (that is,

underneath) the datadir value.

• If innodb_log_group_home_dir is an absolute path, its value is used as-is.

• innodb_log_files_in_group=N

Specifies the number of InnoDB log files before being rotated.

Typically, you do not need to specify this option, because its value is retrieved automatically using the

database connection. If no database connection is available, you must specify it.

When a database connection exists, the value is retrieved automatically and overrides any value you

specify.

• innodb_log_file_size=SIZE

Specifies maximum single InnoDB log file size before switching to next log file. Example: 20M.

Typically, you do not need to specify this option, because its value is retrieved automatically using the

database connection. If no database connection is available, you must specify it.

When a database connection exists, the value is retrieved automatically and overrides any value you

specify.

• innodb_page_size=SIZE

Specifies the page size for all InnoDB tablespaces.

Typically, you do not need to specify this option, because its value is retrieved automatically using the

database connection. If no database connection is available, you must specify it.

When a database connection exists, the value is retrieved automatically and overrides any value you

specify.

131

Backup Repository Options

• innodb_undo_directory=PATH

Specifies where the InnoDB undo log reside within the server repository. Usually the same as datadir,

but can be different.

For backups: Typically, you do not need to specify this option, because its value is retrieved

automatically using the database connection. Specifies the option for an offline backup if the InnoDB

undo log files do not reside under the server's data directory.

For restores: The directory where InnoDB undo log files are to be restored. Specify the option only if the

undo log files are to be restored outside of the server's data directory. For release 4.1.3 and later: the

specified directory must be non-existent or empty, or the restore operation will fail, even if the --force

option is used.

Its value is derived as follows:

• If innodb_undo_directory is not specified, it inherits the value of datadir.

• If innodb_undo_directory is a relative path, the path is taken to be relative to (that is, underneath)

the datadir value.

• If innodb_undo_directory is an absolute path, its value is used as-is.

Warning

When using this option, make sure the undo log location does not change

between successive restores of a full and an incremental backups, or of two

incremental backups. Otherwise, the restore is going to fail.

• innodb_checksum_algorithm=NAME

Specifies the name of the checksum algorithm used for validating InnoDB tablespaces. Default is

innodb.

Typically, you do not need to specify this option, because its value is retrieved automatically using the

database connection. If no database connection is available, you must specify it.

When a database connection exists, the value is retrieved automatically and overrides any value you

specify.

16.4 Backup Repository Options

These options specify various parameters related to the backup image or directory, or to how the backup

will be restored. Typically, --backup-image and --backup-dir are the only options from the group that

you need to specify when using mysqlbackup.

The backup repository options are used with the following operations:

• Backup operations: backup, backup-and-apply-log, backup-to-image.

• Restore operations: copy-back, copy-back-and-apply-log.

The backup repository options are divided into two groups: the first one determines the structure of the

backup, and the second one provides information on the original structure of the data on the backed-up

server for future operations on the backup.

The following options determine the structure of the backup:

132

Backup Repository Options

• --backup-image=IMAGE

Command-Line Format --backup-image=IMAGE

Type File name

Specify the path name of the file used for a single-file backup, restore, or another single-file operation.

Note that if --backup-image does not give a full path name, mysqlbackup will actually take the value

of --backup-image as a path relative to the directory specified by --backup-dir, and thus store

the single-file backup under --backup-dir (or, if the --with-timestamp option is used, under a

subdirectory created under --backup-dir that bears the timestamp in its name).

By default, the single-file backup is streamed to standard output, so that you can pipe it directly to other

commands such as a tape backup or an ssh-related network command.

You can optionally prefix the image name with file: to signify a file I/O (the default). For tape backups,

prefix the image name with sbt:. See Section 4.3.1.2, “Backing Up to Tape” for details about tape

backups.

• --backup_dir=PATH

Same as --backup-dir. The backup directory under which the backup data and metadata are stored,

permanently or temporarily. It is a crucial parameter required for most kinds of backup and restore

operations.

The option is used differently for different operations and under different situations:

• For backup to a single file (including incremental, compressed, encrypted, and cloud backups): Use --

backup-dir to supply a temporary folder to save the backup metadata (including the mysqlbackup

message log, the start and end LSN, and so on) and some temporary output. The backup data,

together with a copy of the metadata, will be stored in a singe file whose name is specified with the

--backup-image option. Note that, however, if --backup-image does not give a full path name,

mysqlbackup will actually take the value of --backup-image as a path relative to the directory

specified by --backup-dir, and thus store the single-file backup under --backup-dir (or, if the --

with-timestamp option is used, under a subdirectory created under --backup-dir, which bears

the timestamp in its name).

• For backup to a directory: Use --backup-dir to specify the directory to store the backup data and

metadata (including the mysqlbackup message log, the start and end LSN, and so on). The directory

specified by --backup-dir cannot be a subdirectory of the directory specified by --datadir.

When the --with-timestamp option is also specified, an additional level of subdirectory, with the

timestamp in its name, is created under --backup-dir (see description for the --with-timestamp

option for details). Unless the --with-timestamp option is used, the directory specified by --

backup-dir must be empty, or the backup operation will fail with an error.

• For restoring a single-file backup (including incremental, compressed, encrypted, and cloud backups):

When using copy-back-and-apply-log to restore a single-file backup, use --backup-dir to

supply a temporary folder to store the temporary data of the restore operation. The directory specified

by --backup-dir should be empty—if a non-empty directory is used, the restore operation will still

be carried out, but the restore data might be corrupted.

When restoring a single-file backup created with the option setting use-tts=with-minimum-

locking, the folder specified with --backup-dir is also used for extracting temporarily all the

tables in the backup and for performing an apply-log operation to make the data up-to-date before

restoring them to the server's data directory.

133

Backup Repository Options

• For restoring a backup directory: Use --backup-dir to specify the location of the backup directory,

from which data will be restored to the server.

• backup_innodb_data_home_dir=PATH

The directory under which the backup's InnoDB data files are to be stored. Specify the option if you

want to put the data files at somewhere other than the default location (which is backup-dir/

datadir). If the value of the parameter is different from backup-dir/datadir, it is stored into

the backup-my.cnf file as innodb_data_home_dir for information, so that mysqlbackup can

understand the structure of the backup when it performs various operations on the backup. Together

with backup_innodb_data_file_path option, it determines the actual file paths of the InnoDB data

files inside the backup.

The value for the parameter is derived as follows:

• If backup_innodb_data_home_dir is not specified, its value will be backup-dir/datadir.

• If backup_innodb_data_home_dir is an absolute path, its value is used as-is.

• If backup_innodb_data_home_dir is a relative path, the path is taken to be relative to (that is,

underneath) backup-dir.

• An empty string (“”) for backup_innodb_data_home_dir means the value of

innodb_data_file_path is to be taken as an absolute path..

This parameter is applicable only for backup operations; during a restore, the InnoDB data files are

restored under the data directory specified by --datadir, unless another location is specified using the

--innodb_data_home_dir option during restore.

• backup_innodb_data_file_path=VALUE

The InnoDB data file names and sizes. Examples:

ibdata1:32M;ibdata2:32M:autoextend

/abs/path/ibdata1:32M:autoextend

innodb-dir/ibdata1:32M:autoextend

This parameter, together with backup_innodb_data_home_dir, determines where the InnoDB

data files are stored within the backup repository. Any file path specified with this option is taken to

be relative to the value of the backup_innodb_data_home_dir option (that is true even if the file

path is specified in the form of an absolute path, like /abs/path/ibdata1:32M:autoextend).

To specify truly absolute paths for InnoDB data files in the backup with this option, you must set the

backup_innodb_data_home option to "" [empty string], in addition to using an absolute path for this

option.

When the parameter is not specified, it inherits the value from the value of the

innodb_data_file_path option on the backed-up server. If both the source and destination of the

backup attempt to use the same absolute paths that resolves to the same files, the backup is cancelled.

The value of the parameter is stored into the backup-my.cnf file as innodb_data_file_path for

information, so that mysqlbackup can understand the structure of the backup when it performs various

operations on the backup.

134

Backup Repository Options

• backup_innodb_log_group_home_dir=PATH

The directory under which the backup's InnoDB logs will be stored. Specify this option only if you want to

put the logs at somewhere other than the default location (which is backup-dir/datadir). If the value

of the parameter is different from backup-dir/datadir, it is stored in the backup-my.cnf file as

innodb_log_group_home_dir for information, so that mysqlbackup can understand the structure

of the backup when it performs various operations on the backup. Note that while you can specify a

directory for saving the logs, the names of the log files are fixed and not reconfigurable.

This parameter is applicable only for backup operations; during a restore, the InnoDB log files are

restored under the data directory specified by --datadir, unless another location is specified using

the --innodb_log_group_home_dir option during restore. The value of the parameter is derived as

follows:

• If backup_innodb_log_group_home_dir is not specified, its value will be backup-dir/

datadir.

• If backup_innodb_log_group_home_dir is an absolute path, its value is used as-is.

• If backup_innodb_log_group_home_dir is a relative path, the path is taken to be relative to (that

is, underneath) backup-dir.

• An empty string (“”) for the option produces an error.

• backup_innodb_undo_directory=PATH

The relative or absolute directory path where separate tablespaces are created for the InnoDB

undo logs during the backup. When unspecified, the option takes up the same value as

backup_innodb_log_group_home_dir; specify this option only if you want to put the undo logs at

some other location. If the value of the parameter is different from backup-dir/datadir, it is stored

in the backup-my.cnf file as innodb_undo_directory for information, so that mysqlbackup can

understand the structure of the backup when it performs various operations on the backup.

This parameter is applicable only for backup operations; during a restore, the InnoDB undo log

tablespaces are restored under the data directory specified by --datadir, unless another location is

specified by the --innodb_undo_directory option during restore.

• --with-timestamp

Creates a subdirectory underneath the backup directory, with a name formed with the timestamp of

the backup operation. It is useful for maintaining a single backup directory containing many backup

snapshots put under different subdirectories.

Default: no timestamped subdirectory is created. To reuse the same backup directory for a new backup

without using this option, either remove the previous backup files manually or, for a single-file backup,

specify the --force option to overwrite the old backup file.

The following parameters provide information on the original structure of the data on the backed-up server

for future operations on the backup, but do not affect the structure of the backup itself:

• backup_innodb_log_files_in_group=N

The number of InnoDB log files in a log group on the restored server. See the description for

innodb_log_files_in_group in the MySQL server manual. The value for this parameter, saved as

innodb_log_files_in_group in the backup-my.cnf file, is derived as follows:

135

Backup Repository Options

• Use the backup_innodb_log_files_in_group value from command line or configuration file, if

specified.

• Else, use the innodb_log_files_in_group value from the backed-up server, if it is an online

backup.

• Else, use the innodb_log_files_in_group value from the mysqlbackup command line or

configuration file.

• backup_innodb_log_file_size=SIZE

The maximum single InnoDB log file size in backup before switching to next log file, on the restored

server. See the description for innodb_log_file_size in the MySQL server manual. The value for

this parameter, saved as innodb_log_file_size in the backup-my.cnf file, is derived as follows:

• Use the backup_innodb_log_file_size value from command line or configuration file, if

specified.

• Else, use the innodb_log_file_size value from the backed-up server, if it is an online backup.

• Else, use the specified innodb_log_file_size value from the mysqlbackup command line or

configuration file.

• backup_innodb_page_size=SIZE

Specifies, for an offline backup, the page size for all InnoDB tablespaces on the restored server. This

option should be specified carefully, because the page size must be the same as that of the backed up

MySQL instance, or the backup could become useless. For an online backup, the value is taken from the

value of the innodb_page_size option on the backed-up server.

Value of this option is saved in the backup-my.cnf file, to be used for restoring the database.

• backup_innodb_undo_logs=NUMBER

Specifies, for an offline backup, the number of rollback segments in the InnoDB system tablespace on

the restored server. This option should be specified carefully, because the value must be the same as

that of innodb_undo_logs on the backed-up MySQL instance, or the backup could become useless.

For an online backup, the value for the parameter is taken from the value of the innodb_undo_logs

option on the backed-up server.

• backup_innodb_undo_tablespaces=NUMBER

Specifies the number of tablespace files that the undo logs are divided between, when you use a non-

zero backup_innodb_undo_logs setting. This option should be specified carefully, because the value

must be the same as that of innodb_undo_tablespaces on the backed-up MySQL instance, or the

backup could become useless. For an online backup, the value for the parameter is taken from the value

of the innodb_undo_tablespaces option on the backed-up server. By default, all the undo logs are

part of the system tablespace, and the system tablespace will always contain one undo tablespace in

addition to those configured by innodb_undo_tablespaces.

• backup_innodb_checksum_algorithm=NAME

Specifies, for an offline backup, the name of the checksum algorithm used for validating the InnoDB

tablespaces on the restored server. This option should be specified carefully, because the checksum

algorithm must be the same use on the backed-up MySQL instance, or the backup could become

136

Metadata Options

useless. For an online backup, the value is taken from the value of the innodb_checksum_algorithm

option on the backed-up server.

Default value of the option is “innodb”.

Value of this option is saved in the backup-my.cnf file, to be used for restoring the database.

16.5 Metadata Options

These options control the generation of metadata about backups. Some metadata is stored in the backup

directory, other metadata is stored in tables within the mysql database of the backed-up instance.

• --no-history-logging

Turns off the recording of backup progress and history in logging tables inside the backed-up database.

See Section 13.3, “Using the MySQL Enterprise Backup Logs” for details about these tables.

Default: history logging is enabled. When --no-connection is specified, history logging is

automatically disabled. When --connect-if-online is specified, history logging only works if a

database connection is successfully established during the backup.

• --comments=STRING

Command-Line Format --comments=STRING

Type String

Specifies a comment string that describes or identifies the backup. Surround multi-word comments

with appropriate quotation marks. The string is saved in a file meta/comments.txt in the backup. For

example: --comments="Backup of HR data on 2010/12/10".

• --comments-file=PATH

Command-Line Format --comments-file=PATH

Type File name

Specifies path to a file containing comments describing the backup. This file is saved as meta/

comments.txt in the backup. For example: --comments-file=/path/to/comments.txt.

This option overrides the --comments option if both are specified.

16.6 Compression Options

For an overview on backup compression, see Section 4.3.4, “Making a Compressed Backup”.

• --compress

Create backup in compressed format. For a regular backup, among all the storage engines supported

by MySQL, only data files of the InnoDB format are compressed, and they bear the .ibz extension after

the compression. Similarly, for a single-image backup, only data files of the InnoDB format inside the

backup image are compressed. The binary log and relay log files are compressed and saved with the

.bz extension when being included in a compressed backup.

Default: compression is disabled.

• --compress-method=ALGORITHM

137

Compression Options

Command-Line Format --compress-method=ALGORITHM

Type Enumeration

Default Value lz4

Valid Values zlib

lz4

lzma

punch-hole

none

Specifies the compression algorithm. The supported arguments for the option and the algorithms they

represent are:

• lz4: LZ4 r109. Out of the three algorithms that are supported, this is the most efficient one, typically

taking the shortest backup and restore times with the lowest CPU cost. See lz4—Extremely Fast

Compression algorithm for more details, including a comparison with other compression algorithms.

• lzma: LZMA 9.20. Out of the three supported algorithms, this typically provides the highest

compression ratio; but it is also far more expensive in terms of CPU cost than the other two options.

Thus we do not recommend this for active systems, but only for off-hour or inactive databases, or

where I/O rates are extremely low.

• zlib: ZLIB v1.2.3. This is in between the other two supported algorithms in terms of both speed and

compression ratio. ZLIB was the only compression algorithm available for MySQL Enterprise Backup

versions prior to 3.10.

Default: lz4. Explicitly specifying a value for the option through a configuration file or command line

automatically enables the --compress option.

• --compress-level=LEVEL

Command-Line Format --compress-level=LEVEL

Type Numeric

Default Value 1

Minimum Value 0

Maximum Value 9

Specifies the level of compression, ranging from “0” to “9”: “0 ”disables compression; “1” is fastest

compression, and “9” is highest (and slowest) compression. The option is only meaningful for

compression using the ZLIB or LZMA algorithm; it is ignored when any other algorithms are selected by

the --compress-method option.

Default: 1 (lowest and fastest compression). Explicitly specifying a non-zero value through a

configuration file or command line automatically enables the --compress option.

• --uncompress

When used with the apply-log or copy-back-and-apply-log operation, uncompresses a

compressed backup before applying the InnoDB log. When used with the copy-back operation,

138

Incremental Backup Options

uncompresses a compressed prepared backup (created by the backup-and-apply-log command

with the --compress option) before restoring it to a server.

For MySQL Enterprise Backup 4.1.4 and later: When used with the extract operation, uncompresses

files that are extracted from a compressed single-file backup.

16.7 Incremental Backup Options

For an overview of incremental backups and usage examples for these options, see Section 4.3.3, “Making

a Differential or Incremental Backup” and Section 5.1.3, “Restoring an Incremental Backup”.

To take an incremental backup, specify the --incremental or --incremental-with-redo-

log-only, along with the --backup-dir option. Depending on whether --incremental or --

incremental-with-redo-log-only is used, other options are required or recommended. All InnoDB

data modified after the a certain LSN (specified directly or indirectly by the options you use) is copied

into the incremental backup. To restore an incremental backup, specify the --incremental and the --

backup-dir option.

• --incremental[={full-scan|optimistic}]

Command-Line Format --incremental

Type Enumeration

Default Value full-scan

Valid Values page-track

full-scan

optimistic

Specifies that the associated backup, backup-to-image, copy-back-and-apply-log, or copy-

back operation is incremental. When performing an incremental backup, there are two possible values

for this option:

• full-scan: mysqlbackup scans all InnoDB data files in the server's data directory to find pages that

have been changed since the last backup and saves them.

• optimistic: mysqlbackup only scans for changed pages in the InnoDB data files that have been

modified since the last backup and then saves them. In general, optimistic incremental backups are

faster than full-scan ones when not many tables in the database have been modified; however a few

restrictions apply to this feature. See Full-scan versus Optimistic Incremental Backup for details.

Default: full-scan. The value for the option have no meaning when the option is used for restoring an

incremental backup.

During a backup, the option also requires the use of either the --incremental-base option or the --

start-lsn option. Only InnoDB tables are backed up incrementally. By default, all non-InnoDB and

.frm files are included into the incremental backup and in their fullness. To exclude non-InnoDB data in

an incremental backup, use the --only-innodb option.

• --incremental-with-redo-log-only

Specifies that an incremental backup is to be created using only the redo log. This alternate type of

incremental backup has different performance characteristics and operational limitations comparing to

backups created with the --incremental option; see Creating Incremental Backups Using Only the

Redo Log for a discussion on their differences.

139

Incremental Backup Options

To use this option, you also need to specify the --incremental-base option or the --start-lsn

option. Just like with the --incremental option, only InnoDB tables are backed up incrementally. By

default, all non-InnoDB and .frm files are included into the incremental backup and in their fullness. To

exclude non-InnoDB data in an incremental backup, use the --only-innodb option.

• --incremental-base=mode:argument

Command-Line Format --incremental-base=mode:argument

Type String

With this option, the mysqlbackup retrieves the information needed to perform incremental backups

from the metadata inside the backup directory rather than from the --start-lsn option. It saves

you from having to specify an ever-changing, unpredictable LSN value when doing a succession of

incremental backups. Instead, you specify a way to locate the previous backup directory through the

combination of mode and argument in the option syntax. The alternatives are:

• dir:directory_path

You specify the prefix dir: followed by a directory path. The path argument points to the directory

where the data from the previous backup is stored. With the first incremental backup, you specify the

directory holding the full backup; with the second incremental backup, you specify the directory holding

the first incremental backup, and so on.

• history:last_backup

You specify the prefix history: followed by last_backup, the only valid argument for this mode.

This makes mysqlbackup query the end_lsn value from the last successful non-TTS backup as

recorded in the backup_history table of the server instance that is being backed up.

Note

If the last full or partial backup made was a TTS backup, mysqlbackup skips

it, and keeps searching the backup history until it finds the last non-TTS backup

and then returns its end_lsn value.

Warning

Do not use the history: mode if the previous backup was a full backup taken

with the --no-connection option, which always turns off the recording of

backup history and might cause errors for a subsequent incremental backup

using this mode of the --incremental-base option.

• --start-lsn=LSN

Command-Line Format --start-lsn=LSN

Type Numeric

In an incremental backup, specifies the highest LSN value included in a previous backup. You can

get this value from the output of the previous backup operation, or from the backup_history

table's end_lsn column for the previous backup operation. Always used in combination with the --

140

Partial Backup and Restore Options

incremental option; not needed when you use the --incremental-base option; not recommended

when you use the --incremental-with-redo-log-only mechanism for incremental backups.

Note

No binary log files are copied into the incremental backup if the --start-

lsn option is used. To include binary log files for the period covered by the

incremental backup, instead of --start-lsn, use the --incremental-base

option, which provides the necessary information for mysqlbackup to ensure

that no gap exists between binary log data included in the previous backup and

the current incremental backup.

• --incremental-backup-dir=PATH

Advanced: Specifies the location for data of an incremental directory backup. When creating or

restoring an incremental directory backup, the option serves the same function as --backup-dir does

for backups and restores in general, and the option can in fact be used interchangeably with --backup-

dir for directory backups. See the description for --backup-dir for details.

For an apply-incremental-backup operation, the option specifies the incremental backup directory

whose data is used to update a directory backup specified by the --backup-dir option.

Note

Do not use this option with any operations for image backups, for which the

option has no meaning.

16.8 Partial Backup and Restore Options

Note

Since MySQL Enterprise Backup 3.10, the two options --include-tables and

--exclude-tables have been introduced. These were intended for replacing

the older options of --include, --databases, --databases-list-file, and

--only-innodb-with-frm, which are incompatible with the new options and

will be deprecated in future releases. For references purpose, we have included

information on the older options at the end of this section in Legacy Partial Backup

Options.

To select specific data to be backed up or restored, use the partial backup and restore options described in

this section.

For an overview of partial backup and usage examples on the following options, see Section 4.3.5, “Making

a Partial Backup”. See also Section 5.1.4, “Restoring Backups Created with the --use-tts Option”, on

selective restore of tables from a backup.

• --include-tables=REGEXP

Command-Line Format --include-tables=REGEXP

Type String

Include for backup or restoration only those tables (both Innodb and non-Innodb) whose fully qualified

names (in the form of db_name.table_name) match the regular expression REGEXP. The regular

expression syntax used is the extended form specified in the POSIX 1003.2 standard. For example, --

include-tables=^mydb\.t[12]$ matches the tables t1 and t2 in the database mydb. On Unix-like

systems, quote the regular expression appropriately to prevent interpretation of shell meta-characters.

141

Partial Backup and Restore Options

mysqlbackup throws an error when the option is used without a regular expression being supplied with

it.

While mysqlbackup understands the MySQL convention of quoting the database or the table name (or

both) by backticks (see Schema Object Names), there is no need to include the backticks in the regular

expression for --include-tables.

For a backup operation, mysqlbackup throws an error when no table matches the regular expression

specified with --include-tables.

While the option can be used for different kinds of backups, selective restore is only supported for

backups created using transportable tablespace (TTS) (that is, backups created with the --use-tts

option). The option can also be used with the backup-dir-to-image and image-to-backup-dir

commands to select tables when creating or unpacking a backup image.

The option cannot be used together with the legacy --include, --databases, --databases-list-

file, or --only-innodb-with-frm option.

When used together with the --exclude-tables option, --include-tables is applied first,

meaning mysqlbackup first selects all tables specified by --include-tables and then excludes from

the set those tables specified by --exclude-tables.

• --exclude-tables=REGEXP

Command-Line Format --exclude-tables=REGEXP

Type String

Exclude for backup or restoration all tables (both Innodb and non-Innodb) whose fully qualified

names (in the form of db_name.table_name) match the regular expression REGEXP. The regular

expression syntax is the extended form specified in the POSIX 1003.2 standard. For example, --

exclude-tables=^mydb\.t[12]$ matches the tables t1 and t2 in the database mydb. On Unix-like

systems, quote the regular expression appropriately to prevent interpretation of shell meta-characters.

mysqlbackup throws an error when the option is used without a regular expression being supplied with

it.

While mysqlbackup understands the MySQL convention of quoting the database or the table name (or

both) by backticks (see Schema Object Names), there is no need to include the backticks in the regular

expression for --exclude-tables.

While the option can be used for different kinds of backups, selective restore is only supported for

backups created using transportable tablespaces (TTS) (that is, backups created with the --use-tts

option). The option can also be used with the backup-dir-to-image and image-to-backup-dir

commands to select tables when creating or unpacking a backup image.

The option cannot be used together with the --include, --databases, --databases-list-file,

or --only-innodb-with-frm option.

When used together with the --include-tables option, --include-tables is applied first,

meaning mysqlbackup first select all tables specified by --include-tables, and then exclude from

the set those tables specified by --exclude-tables.

• --only-known-file-types

For back up only. By default, all files in the database subdirectories under the data directory of the

server are included in the backup (see Table 1.1, “Types of Files in a Backup” for details). If the --

142

Partial Backup and Restore Options

only-known-file-types option is specified, mysqlbackup only backs up those types of files that

are data files for MySQL or its built-in storage engines, which, besides the ibdata* files, have the

following extensions:

• .ARM: ARCHIVE table metadata

• .ARZ: ARCHIVE table data

• .CSM: CSV table metadata

• .CSV: CSV table data

• .frm: table definitions

• .ibd: InnoDB tablespace created using the file-per-table mode

• .MRG: Merge storage engine references to other tables

• .MYD: MyISAM data

• .MYI: MyISAM indexes

• .opt: database configuration information

• .par: partition definitions

• .TRG: trigger parameters

• .TRN: trigger namespace information

• --only-innodb

For back up only. When this option is used, only InnoDB data and log files are included in the backup,

and all files created by other storage engines are excluded. Typically used when no connection to

mysqld is allowed or when there is no need to copy MyISAM files.

The option is not compatible with the --slave-info option.

• --use-tts[={with-minimum-locking|with-full-locking}]

Command-Line Format --use-tts[={with-minimum-locking|with-full-locking}]

Type Enumeration

Default Value with-minimum-locking

Valid Values with-minimum-locking

with-full-locking

Enable selective backup of InnoDB tables using transportable tablespaces (TTS). This is to be used in

conjunction with the --include-tables and --exclude-tables options for selecting the InnoDB

tables to be backed up by regular expressions. Using TTS for backups offers the following advantages:

• Backups can be restored to a different server

• The system tablespace is not backed up, saving disk space and I/O resources

143

Partial Backup and Restore Options

• Data consistency of the tables is managed by MySQL Enterprise Backup

However, the option has the following limitations:

• Backing up and restoring of partitioned tables are only supported for MySQL 5.7.4 and later. Also,

individual partitions cannot be selectively backed up or restored. Tables selected by the --include-

tables and --exclude-tables options are always backed up or restored in full.

• Can only backup tables that are stored in their own individual tablespaces (i.e., tables created with the

innodb_file_per_table option enabled)

• Non-InnoDB tables are not backed up

• For partial backups, encrypted InnoDB tables are never included (a warning is issued in the log file

whenever an encrypted InnoDB table that matches the table selection criteria has been skipped over).

• Cannot be used for incremental backups

• Does not include the binary log or the relay log in the backup

See also Appendix B, Limitations of MySQL Enterprise Backup for some more minor limitations.

There are two possible values for the option:

• with-minimum-locking: Hot copies of the selected tables are backed up, and the tables are then

locked in read-only mode while the redo log (with only the portion containing the relevant changes

made after the hot backup) is being included in the backup. Any tables created during the locking

phase are ignored.

• with-full-locking: The selected tables are locked in read-only mode while they are being backed

up. The redo log is not included in the backup. Any tables created during the locking phase are

ignored.

Note

Due to a known issue, when creating a backup using TTS for a server

containing tables with a mix of the Antelope and Barracuda file formats, do

NOT apply full locking on the tables.

Default: with-minimum-locking

To use the --use-tts option, extra privileges are required of the user through which mysqlbackup

connects to the server; see Section 4.1.2, “Grant MySQL Privileges to Backup Administrator” for details.

There are some special requirements for restoring backups created with the --use-tts option; see

Section 5.1.4, “Restoring Backups Created with the --use-tts Option” for details.

• --rename=“old_table_name to new_table_name”

Rename a single table when it is selected by the --include-tables or --exclude-tables option

(or both together) to be restored to a database from a backup created using the --use-tts option. The

table named old_table_name is renamed to new_table_name. Note that when using the option:

• The --include-tables or --exclude-tables option (or both together) must be used in the

restore command for the --rename option to work, unless there is only one table in the backup. Also,

144

Legacy Partial Backup Options

the --include-tables or --exclude-tables option (or both together) should specify one and

only one table for restore when --rename is used, or the restore will fail.

• old_table_name and new_table_name can be fully qualified (containing the database names, in

the format of db_name.tb_name) or not. Regular expressions are not accepted for old_table_name

and new_table_name.

• The restore fails if old_table_name does not match with the table specified using the --include-

tables or --exclude-tables option (or both together), or if new_table_name already exists in

the target database.

• The requirements listed in Section 5.1.4, “Restoring Backups Created with the --use-tts Option”

apply.

See Section 5.1.4, “Restoring Backups Created with the --use-tts Option”, for more information on

selective restores, and for an example of table renaming.

Legacy Partial Backup Options

Important

Information in this subsection is only for using the legacy options of --include,

--databases, --databases-list-file, and --only-innodb-with-frm,

which will be deprecated in the upcoming issues. For creating partial backups, it

is strongly recommended that the new options of --include-tables and --

exclude-tables be used instead. Note that you cannot combine the legacy and

the new partial-backup options in a single command.

Besides the legacy options, some other options are also discussed below, but the

information is only for using the options together with the legacy partial-backup

options.

For an overview of partial backups and usage examples for these legacy options, see Making a Partial

Backup with the Legacy Options.

• --include=REGEXP

This option is for filtering InnoDB tables for backup. The InnoDB tables' fully qualified names

are checked against the regular expression specified by the option. If the REGEXP matches

db_name.table_name, the table is included. The regular expression syntax used is the extended form

specified in the POSIX 1003.2 standard. For example, --include=mydb\.t[12] matches the tables

t1 and t2 in the database mydb. mysqlbackup throws an error when the option is used without a

regular expression being supplied with it.

This option only applies to InnoDB tables created with the MySQL option innodb_file_per_table

enabled (which is the default setting for MySQL 5.6 and after), in which case the tables are in separate

files that can be included or excluded from the backup. All tables in the InnoDB system tablespace are

always backed up.

When no InnoDB table names match the specified regular expression, an error is thrown with a message

indicating there are no matches.

Default: Backs up all InnoDB tables.

145

Legacy Partial Backup Options

Note

This option does not filter non-InnoDB tables, for which options like --

databases and --databases-list-file can be used.

Important

This option does not filter the .frm files associated with InnoDB tables, meaning

that regardless of the option’s value, all the .frm files for all InnoDB tables are

always backed up unless they are excluded by other options. Those .frm files

for InnoDB tables that are not backed up should be deleted before the database

backup is restored. See Making a Partial Backup with the Legacy Options for

details.

• --databases=LIST

Specifies the list of non-InnoDB tables to back up. The argument specifies a space-separated list of

database or table names of the following form:

"db_name[.table_name] db_name1[.table_name1] ...".

If the specified values do not match any database or table, then no non-InnoDB data files are backed up.

See Making a Partial Backup with the Legacy Options for details.

By default, all non-InnoDB tables from all databases are backed up.

Note

The option has no filtering effects on the InnoDB data files (.ibd files) for the

databases or tables it specifies. To filter InnoDB data files, use the --include

option instead.

• --databases-list-file=PATH

Specifies the pathname of a file that lists the non-InnoDB tables to be backed up. The file contains

entries for databases or fully qualified table names separated by newline or space. The format of the

entries is the same as for the --databases option:

db_name[.table_name]

db_name1[.table_name1]

...

Remove any white spaces surrounding the database or table names, as the white spaces are not

removed automatically. Begin a line with the # character to include a comment. No regular expressions

are allowed.

If the specified entries do not match any database or table, then no non-InnoDB data files are backed up.

Note

The option has no filtering effects on the InnoDB data files (.ibd files) for the

databases or tables it specifies. To filter InnoDB data files, use the --include

option instead.

146

Legacy Partial Backup Options

• --only-innodb-with-frm[={all|related}]

Back up only InnoDB data, log files, and the .frm files associated with the InnoDB tables.

• --only-innodb-with-frm=all includes the .frm files for all InnoDB tables in the backup.

• --only-innodb-with-frm=related, in combination with the --include option, copies only the

.frm files for the tables that are included in the partial backup.

• --only-innodb-with-frm with no argument is the same as --only-innodb-with-

frm=related.

Note

For incremental backups, even only changed .ibd files are backed up, .frm

files associated with all specified InnoDB tables are included.

This option saves you having to script the backup step for InnoDB .frm files, which you would normally

do while the server is put into a read-only state by a FLUSH TABLES WITH READ LOCK statement.

The .frm files are copied without putting the server into a read-only state, so that the backup operation

is a true hot backup and does not interrupt database processing. You must ensure that no ALTER

TABLE or other DDL statements change .frm files for InnoDB tables while the backup is in progress.

If mysqlbackup detects changes to any relevant .frm files during the backup operation, it halts with

an error. If it is not practical to forbid DDL on InnoDB tables during the backup operation, use the --

only-innodb option instead and use the traditional method of copying the .frm files while the server is

locked.

All files created by other storage engines are excluded. Typically used when no connection to mysqld

is allowed or when there is no need to copy MyISAM files, for example, when you are sure there are no

DDL changes during the backup. See Making a Partial Backup with the Legacy Options for instructions

and examples.

The option is not compatible with the --slave-info option.

Default: backups include files from all storage engines.

• --use-tts[={with-minimum-locking|with-full-locking}]

Enable selective backup of InnoDB tables using transportable tablespaces (TTS). This is to be used in

conjunction with the --include option, which selects the InnoDB tables to be backed up by a regular

expression. Using TTS for backups offers the following advantages:

• Backups can be restored to a different server

• The system tablespace is not backed up, saving disk space and I/O resources

• Data consistency of the tables is managed by MySQL Enterprise Backup

See important discussions here on the limitations with using the --use-tts option.

There are two possible values for the option:

• with-minimum-locking: Hot copies of the selected tables are backed up, and the tables are then

locked in read-only mode while the redo log (with only the portion containing the relevant changes

147

Single-File Backup Options

made after the hot backup) is being included in the backup. Any tables created during the locking

phase are ignored.

• with-full-locking: The selected tables are locked in read-only mode while they are being backed

up. The redo log is not included in the backup. Any tables created during the locking phase are

ignored.

Default: back up with minimum locking

There are some special requirements for restoring backups created with the --use-tts option; see the

explanations in Section 5.1, “Performing a Restore Operation” for details.

16.9 Single-File Backup Options

These options are associated with single-file backups. You use them in combination with the

mysqlbackup commands backup-to-image, image-to-backup-dir, backup-dir-to-image,

copy-back-and-apply-log, list-image, and extract. For usage examples, see Section 4.3.1,

“Making a Single-File Backup”.

• --backup-image=IMAGE

See description of the option in Section 16.4, “Backup Repository Options”

• --src-entry=STRING

Command-Line Format --src-entry=STRING

Type Path name

Identifies files or directories whose pathnames contain the STRING to be extracted from a single-file

backup. This option is used with the extract command. Optionally, you can also specify the --dst-

entry option to extract a file or directory to a location different from its original path name.

For example: src-entry=d1/f2 extracts only one file, f1, while src-entry=d1/ extracts the entire

directory tree for the d1 folder (notice the slash (/) at the end of the argument, without which all files or

folders containing the string d1 in their pathnames will be extracted).

Default: All entries are extracted.

Notes

• The following items are always extracted from the backup, irrespective of the

value of --src-entry (and the locations of their extraction are unaffected by

the --dst-entry option):

• The file backup-my.cnf.

• A datadir folder (which only contains items matched by the --src-entry

option).

• A meta folder, which contains the file backup_variables.txt, a log

file for the extract operation, and also items matched by the --src-entry

option.

• The option is currently not supported for the extract command for cloud

backups, which can only be extracted in full.

148

Single-File Backup Options

• --dst-entry=PATH

Command-Line Format --dst-entry=PATH

Type Path name

Used with single-file backups to extract a single file or directory to a user-specified path. Use of this

option requires specifying the --src-entry option. This option specifies the destination path for the

entry selected from the backup image by --src-entry. The entry could point to a single file or single

directory. For example, to retrieve the comments file from a backup image and store it as /tmp/my-

comments.txt, use a command like the following:

mysqlbackup --src-entry=meta/comments.txt \

--dst-entry=/tmp/my-comments.txt \

--backup-image=/var/myimage.bki extract

Similarly, to extract all the contents of the datadir/pets/ directory in a single-file backup as /pets-

extracted/, use a command like the following:

mysqlbackup --src-entry=datadir/pets/ \

--dst-entry=/pets-extracted/ \

--backup-image=/var/myimage.bki extract

The specified path is a simple path name without any wildcard expansion or regular expressions.

In case the argument for --src-entry matches multiple files or folders, they are all extracted into a

folder whose pathname, relative to the destination folder, is given by the argument of --dst-entry

(unless the argument specifies an absolute path).

Default: Original pathnames are used to create files under the destination folder.

• --sbt-database-name=NAME

Command-Line Format --sbt-database-name=NAME

Type String

Default Value MySQL

For tape backups, this option can be used as a hint to the Media Management Software (MMS) for the

selection of media and policies. This name has nothing to do with MySQL database names. It is a term

used by the MMS. See Section 4.3.1.2, “Backing Up to Tape” for usage details.

• --sbt-lib-path=PATH

Command-Line Format --sbt-lib-path=PATH

Type File name

Path name of the SBT library used by the software that manages tape backups. If this is not specified,

operating system-specific search methods are used to locate libobk.so (UNIX) or orasbt.dll

(Windows). See Section 4.3.1.2, “Backing Up to Tape” for usage details.

• --sbt-environment=VAR=value,...

Command-Line Format --sbt-environment=VAR1=value1[,VAR2=value2[,...]] SBT

API provider)

149

Performance / Scalability / Capacity Options

Type String

Passes product-specific environment variables to Oracle Secure Backup or another SBT-compliant

backup management product, as an alternative to setting and unsetting environment variables before

and after each mysqlbackup invocation.

The parameter to this option is a comma-separated list of key-value pairs, using

syntax similar to that of the RMAN tool for the Oracle Database. For example, --sbt-

environment=VAR1=val1,VAR2=val2,VAR3=val3.

Consult the documentation for your backup management product to see which of its features can be

controlled through environment variables. For example, the Oracle Secure Backup product defines

environment variables such as OB_MEDIA_FAMILY, OB_DEVICE, and OB_RESOURCE_WAIT_TIME.

You might set such variables with the mysqlbackup by specifying an option such as --sbt-

environment="OB_MEDIA_FAMILY=my_mf,OB_DEVICE=my_tape".

If the argument string contains any whitespace or special characters recognized by the

command shell, enclose the entire argument string in quotation marks. To escape an equal

sign or comma, use the \ character. For example, --sbt-environment="VAR1=multiple

words,VAR2=<angle_brackets>,VAR3=2+2\=4".

• --disable-manifest

Disable generation of manifest files for a backup operation, which are backup_create.xml and

backup_content.xml present in the meta subdirectory.

16.10 Performance / Scalability / Capacity Options

These options limit the resources used by the backup process, in order to minimize backup overhead for

busy or huge databases, or specify behaviors of the process when encountering resource issues.

• --number-of-buffers=num_buffers

Command-Line Format --number-of-buffers=NUMBER

Type Numeric

Default Value 14

Minimum Value 1

Specifies the number of buffers, each 16MB in size, to use during multithreaded options.

Use a high number for CPU-intensive processing such as backup, particularly when using compression.

Use a low number for disk-intensive processing such as restoring a backup. This value should be at

least as high as the number of read threads or write threads, depending on the type of operation.

Default: currently 14.

For compression or incremental backup operations, the buffer size is slightly more than 16MB to

accommodate the headers.

One additional buffer is used for single-file incremental backup and single-file compressed backup.

Compressed backup, compressed single-file backup, and uncompress apply-log operations require one

additional buffer for each process thread.

150

Performance / Scalability / Capacity Options

If you change the number of read, write, and processing threads, you can experiment with changing this

value so that it is slightly larger than the total number of threads specified by those other options. See

Section 11.1, “Optimizing Backup Performance” and Section 11.2, “Optimizing Restore Performance”

for additional advice about recommended combinations of values for this and other performance-related

options for various hardware configurations, such as RAID or non-RAID storage devices.

• --read-threads=num_threads

Command-Line Format --read-threads=NUMBER

Type Numeric

Default Value 1

Minimum Value 1

Maximum Value 15

Specifies the number of threads to use for reading data from disk. This option applies to these kinds

of operations: copy-back, copy-back-and-apply-log, extract, backup, and backup-and-

apply-log. If you specify a value of 0, it is silently adjusted to 1. The maximum is 15. If you supply

a negative value, it is silently adjusted to 15. For apply-log operations on backup directories and

the apply-log phase of copy-back-and-apply-log, backup-and-apply-log, or apply-

incremental-backup, the number of read threads is always 1 regardless of this option's setting. See

Section 11.1, “Optimizing Backup Performance” and Section 11.2, “Optimizing Restore Performance” for

advice about recommended combinations of values for --read-threads, --process-threads, and

--write-threads for various hardware configurations, such as RAID or non-RAID storage devices.

Default: 1.

• --process-threads=num_threads

Command-Line Format --process-threads=NUMBER

Type Numeric

Default Value 6

Minimum Value 1

Maximum Value 15

Specifies the number of threads to use for processing data, including compression and uncompression,

encryption and decryption, apply-log operations, and packing and extracting of backup images. For

backup-and-apply-log, copy-back-and-apply-log, and apply-incremental-backup, --

process-threads sets the worker threads number for the apply-log phase of the operation. The option

is ignored for those operations that do not involve data processing like copy-back (unless decryption

or uncompression is involved ), backup-dir-to-image , or a backup operation that uses the --

incremental-with-redo-log-only option.

Default: 6 for all operations to which the option is applicable.

If you specify a value of 0, it is silently adjusted to 1. The maximum is 15. If you supply a negative value,

it is silently adjusted to 15. See Section 11.1, “Optimizing Backup Performance” and Section 11.2,

“Optimizing Restore Performance” for advice about recommended combinations of values for --read-

151

Performance / Scalability / Capacity Options

threads, --process-threads, and --write-threads for various hardware configurations, such as

RAID or non-RAID storage devices.

• --write-threads=num_threads

Command-Line Format --write-threads=NUMBER

Type Numeric

Default Value 1

Minimum Value 1

Maximum Value 15

Specifies the number of threads to use for writing data to disk. This option applies to these kinds of

operations: copy-back, copy-back-and-apply-log, extract, backup-to-image, backup, and

backup-and-apply-log. Multiple write threads are supported for any write target that is seekable; --

write-threads is forced to be 1 only when the write target is non-seekable (e.g., when the backup is

written to stdout, to tape, or to cloud storage). The option is ignored when used with other single-file

backup operations like list-image or validate

If you specify a value of 0, it is silently adjusted to 1. The maximum is 15. If you supply a negative

value, it is silently adjusted to 15. For apply-log operations on backup directories and the apply-

log phase of copy-back-and-apply-log, backup-and-apply-log, or apply-incremental-

backup, the number of write threads is always 0 regardless of this option's setting. See Section 11.1,

“Optimizing Backup Performance” and Section 11.2, “Optimizing Restore Performance” for advice about

recommended combinations of values for --read-threads, --process-threads, and --write-

threads for various hardware configurations, such as RAID or non-RAID storage devices.

Default: 1.

• --limit-memory=MB

Command-Line Format --limit-memory=MB

Type Numeric

Default Value 100 for apply-log (without uncompression), 300 for

other operations

Minimum Value 0

Maximum Value 999999

Unit megabyte

Specify maximum memory in megabytes that can be used by mysqlbackup. Formerly applied only to

apply-log operation, but in MySQL Enterprise Backup 3.8 and higher it applies to all operations. Do

not include any suffixes such as mb or kb in the option value.

Default: 100 for apply-log not used with --uncompress, 400 for all operations (in megabytes).

The memory limit specified by this option also caps the number of 16MB buffers available for

multithreaded processing. For example, with a 400 MB limit, the maximum number of buffers is 25

(except for a cloud backup, for which extra memory is needed, and the maximum number of buffers

is 18). If additional buffers are required because you increased the values for --read-threads, --

process-threads, --write-threads, and/or --number-of-buffers, increase the --limit-

memory value proportionally.

152

Performance / Scalability / Capacity Options

• --sleep=MS

Command-Line Format --sleep=MS

Type Numeric

Default Value 0

Unit millisecond

Specify the number in milliseconds to sleep after copying a certain amount of data from InnoDB tables.

Each block of data is 1024 InnoDB data pages, typically totalling 16MB. This is to limit the CPU and I/O

overhead on the database server.

Default: 0 (no voluntary sleeps).

• --no-locking

Disables locking during backup of non-InnoDB files, even if a connection is available. Can be

used to copy non-InnoDB data with less disruption to normal database processing. There could be

inconsistencies in non-InnoDB data if any changes are made while those files are being backed up.

• --lock-wait-timeout

Command-Line Format --lock-wait-timeout

Type Numeric

Default Value 60

Minimum Value 1

Unit second

Specify the timeout in seconds for the FLUSH TABLES WITH READ LOCK statement, which

mysqlbackup issues during the final stage of a backup to temporarily put the database into a read-

only state. If the timeout is exceeded, the statement is failed and the lock on the tables is released, so

that queries held up by the lock can then be executed. mysqlbackup then retries the statement and

continues with the backup. The timeout prevents the case in which a long query running on the server

prevents the FLUSH TABLES WITH READ LOCK statement from finishing, holding up further queries

and eventually bringing down the server. Default is 60. Minimum value is 1.

• --lock-wait-retry-count

Command-Line Format --lock-wait-retry-count

Introduced 4.1.4

Type Integer

Default Value 2147483647

Minimum Value 0

Maximum Value 2147483647

Specify the maximum number of retries to be attempted by mysqlbackup after the FLUSH TABLES

WITH READ LOCK statement, issued during the final stage of a backup to temporarily put the database

into a read-only state, fails due to a timeout. The timeout is specified by the mysqlbackup option --

lock-wait-timeout, or by the system variable --innodb_lock_wait_timeout on the server. This

option prevents an excessive number of retries of the lock statement, during which all transactions are

blocked on the server. Default and maximum value is 2147483647. Minimum value is 0 (no retry).

153

Performance / Scalability / Capacity Options

• --page-reread-time=MS

Command-Line Format --page-reread-time=MS

Type Numeric

Default Value 100

Unit millisecond

Interval in milliseconds that mysqlbackup waits before re-reading a page that fails a checksum test.

A busy server could be writing a page at the same moment that mysqlbackup is reading it. Can

be a floating-point number, such as 0.05 meaning 50 microseconds. Best possible resolution is 1

microsecond, but it could be worse on some platforms. Default is 100 milliseconds (0.1 second).

• --page-reread-count=retry_limit

Command-Line Format --page-reread-count=number

Type Numeric

Default Value 500

Maximum number of re-read attempts, when a page fails a checksum test. A busy server could be

writing a page at the same moment that mysqlbackup is reading it. If the same page fails this many

checksum tests consecutively, with a pause based on the --page-reread-time option between each

attempt, the backup fails. Default is 500.

• --on-disk-full={abort|abort_and_remove|warn}

Command-Line Format --on-disk-full=option

Type Enumeration

Default Value abort

Valid Values abort

warn

abort_and_remove

Specifies the behavior when a backup process encounters a disk-full condition. This option is only for

backup operations (backup, backup-and-apply-log, and backup-to-image).

• abort: Abort backup, without removing the backup directory. The disk remains full.

• abort_and_remove: Abort backup and remove the backup directory.

• warn: Write a warning message every 30 seconds and retry backup until disk space becomes

available.

Default: abort.

• --skip-unused-pages

Skip unused pages in tablespaces when backing up InnoDB tables. This option is applicable to the

backup and backup-to-image operations, but not to incremental backups. The option is ignored by

the backup-and-apply-log operation.

Note that backups created with the --skip-unused-pages option cannot be restored using copy-

back-and-apply-log.

154

Performance / Scalability / Capacity Options

Unused pages are free pages often caused by bulk delete of data. By skipping the unused pages during

backups, this option can reduce the backup sizes and thus the required disk space and I/O resources

for the operations. However, subsequent apply-log operations on the backups will take more time to

complete, as the unused pages are inserted back into the tables during the operations.

• --skip-binlog

Skip including the binary log files in the backup during a backup operation, or skip copying the binary log

files onto a server during a restore operation.

Binary log files, together with the binary log index file, are included by default for all kinds of online

backups (full, incremental, compressed, partial, single-file, etc.). See Table 1.1, “Types of Files in a

Backup”, for details. Use this option to skip backing up binary logs for the following situations:

• If resource or performance issues arise.

• If any binary log files are missing on the server you are backing up, in order to avoid mysqlbackup

throwing an error for the missing files.

• If you are making an incremental backup that is based on a backup (full or incremental) created using

the --no-locking option, as binary log information will then be unavailable to mysqlbackup in that

situation.

• --skip-relaylog

When working with a replica server, skip including the relay log files in the backup during a backup

operation, or skip copying the relay log files onto a server during a restore operation.

Relay log files, together with the relay log index file and the master.info and the slave.info files,

are included by default for all kinds of online backups (full, incremental, compressed, partial, single-file,

etc.) of a replica server. See Table 1.1, “Types of Files in a Backup”, for details. Use this option to skip

backing up relay logs if resource, performance, or other issues arise.

Note

If a user runs a FLUSH LOGS statement while backup is in progress for a replica,

the backup process will fail. Use the--skip-relaylog option if you expect a

FLUSH LOGS statement will be run during the backup and it is not necessary to

include the relay logs in the backup.

The relay log files and the files backed up together with them, when included in a backup, are always

copied into the restored server's data directory during a restore operation; if that is not the behavior you

desire, use this option to skip the restoring of the relay log.

• --skip-final-rescan

Skip the final rescan for InnoDB tables that are modified by DDL operations after the database has been

read-locked near the end of a backup operation. This potentially shortens the duration for the lock and

reduces the backup's impact on the server's normal operation, especially when many tables are being

backed up.

Warning

This option can cause an incomplete or inconsistent backup if, during the backup

operation, DDL operations are executed on any InnoDB tables whose file-per-

155

Performance / Scalability / Capacity Options

table tablespaces are outside the MySQL data directory (i.e., any InnoDB tables

created using the DATA DIRECTORY table option).

The option is ignored for backups using the --incremental-with-redo-log-only option and for

non-backup operations.

• --log-bin-index[=PATH]

Command-Line Format --log-bin-index=FILENAME

Type File name

Default Value data_dir/host_name-bin.index

For all offline backups: specify the absolute path (including file name and extension) of the index file on

the MySQL server that lists all the used binary log files, if it is different from the default path given below,

in order to include the binary log files in the backup.

Default: data_dir/host_name-bin.index.

• --relay-log-index[=PATH]

Command-Line Format --relay-log-index=FILENAME

Type File name

Default Value data_dir/host_name-relay-bin.index

For offline backups of replica servers only: specify the absolute path (including file name and extension)

of the index file on the MySQL server that lists all the used relay log files, if it is different from the default

path given below, in order to include the relay log files in the backup.

Default: data_dir/host_name-relay-bin.index.

• --master-info-file[=PATH]

Command-Line Format --master-info-file=FILENAME

Type File name

Default Value data_dir/master.info

For offline backups of replica servers only: specify the absolute path (including file name and extension)

of the information file in which a replica records information about its source, if it is different from the

default path given below, in order to include the information file in the backup.

Default: data_dir/master.info.

• --relaylog-info-file[=PATH]

Command-Line Format --relaylog-info-file=FILENAME

Type File name

Default Value data_dir/relay-log.info

For offline backups of replica servers only: specify the absolute path (including file name and extension)

of the information file in which a replica records information about the relay logs, if it is different from the

default path given below, in order to include the information file in the backup.

Default: data_dir/relay-log.info.

156

Performance / Scalability / Capacity Options

• --optimistic-time[=DATE-TIME]

Command-Line Format --optimistic-time=DATE-TIME

Type String

Default Value now

Perform an optimistic backup with the value specified with the option as the “optimistic time”—a

time after which the tables that have not been modified are taken as “inactive tables.” The “inactive

tables”are believed to be unlikely to change during the backup process. The inactive tables are backed

up in the optimistic phase of the backup, and all other tables are backed up in the normal phase. See

Section 4.3.6, “Making an Optimistic Backup” for details on the concept, use cases, and command

samples for an optimistic backup.

Accepted formats for specifying the option include:

• now: This includes all tables into the optimistic phase of the backup process. It is the default value for

the option when no value is specified.

• {Number}{Unit}: Indicates the optimistic time as a time at a certain duration into the past. {Unit}

can be any one of years, months, hours, and minutes. Some examples for option strings in this

format include: 5years, 2days,13months, 23hours, and 35minutes.

• A date-time format in any of the following forms: YYMMDD, YYYYMMDD, YYMMDDHHMMSS,

YYYYMMDDHHMMSS, YY-MM-DD, YYYY-MM-DD, YY-MM-DD HH.MM.SS, or YYYYMMDDTHHMMSS (where

T is the character T).

When both the optimistic-time and the optimistic-busy-tables options are used and

they come into conflict on determining which tables are to be backed up in the optimistic phase,

optimistic-busy-tables takes precedence over optimistic-time.

• --optimistic-busy-tables=REGEXP

Command-Line Format --optimistic-busy-tables=REGEXP

Type String

Perform an optimistic backup, using the regular expression specified with the option to select

tables that will be skipped in the first phase of an optimistic backup, because they are likely

to be modified during the backup process. Tables whose fully qualified names (in the form of

database_name.table_name) are matched by the regular expression are taken as “busy tables”,

which will be backed up in the second or the “normal” phase of the backup. Tables whose fully qualified

names are NOT matched by the regular expression are taken as “inactive tables”, which will be backed

up in the first or the “optimistic” phase of the backup. See Section 4.3.6, “Making an Optimistic Backup”

for details on the concept, use cases, and command samples for an optimistic backup.

MySQL Enterprise Backup will throw an error if the option is used but no regular expression is supplied

with it.

When both the optimistic-time and the optimistic-busy-tables options are used and they

come into conflict on determining which tables are to be “optimistic”, optimistic-busy-tables takes

precedence over optimistic-time.

157

Message Logging Options

• --free-os-buffers

Free the system buffer cache at the end of a backup operation by syncing all data from the buffer cache

to the hard disk. Using the option might increase the backup time significantly for systems with slow

storage devices and databases with many tables.

Default: Automatic syncing disabled.

16.11 Message Logging Options

mysqlbackup writes important progress and error information to the stderr stream. The information

is often very valuable for tracking down problems that occur during an operation. Starting from MySQL

Enterprise Backup 3.9, the output to the stderr stream is also saved to a log file by default (for most

mysqlbackup operations), so that the error information can be easily accessed in any debug process.

The message logging works like a tee process on a Unix-like system, in which the output of a program is

split to be both displayed and saved to a file. The log file thus produced is named in the following format:

MEB_timestamp_operation.log, where operation is the mysqlbackup operation that was run

(e.g., backup, apply-log, etc.), and timestamp is the date and time at which the operation was run.

Here are some examples of names for the log files:

MEB_2013-06-24.16-32-43_backup.log

MEB_2013-06-28.11-07-18_apply_log.log

MEB_2013-06-29.10-08-06_list_image.log

The following options control the message logging function:

• --skip-messages-logdir

Skip message logging. Logging is turned on by default (except for the list-image and validate

operations; see the description for the --messages-logdir option for details), and it is turned off by

this option.

• --messages-logdir=path

Command-Line Format --messages-logdir=PATH

Type Directory name

Default Value backup_dir/meta

Specifies the path name of an existing directory for storing the message log. If the specified directory

does not exist, message logging fails and returns an error message. When this option is omitted, the

default directory of backup_dir/meta is used, where backup_dir is the directory specified with the

--backup-dir option.

Note

Use this option to turn on message logging for the list-image and validate

operations. Message logging is turned off by default for the two operations,

because they do not modify any files and a message log is usually not required

for debugging them. And because the default path name of backup_dir/meta

is not meaningful for the two operations, this option is required for both turning on

message logging and for supplying the path name of a directory in which to save

the log file. However, if the --skip-messages-logdir option is also specified,

it takes precedence and message logging is skipped.

The following are some examples showing how the message logging is controlled.

158

Progress Report Options

This creates a log file for the backup operation in the directory /home/backup_dir/meta due to the

default settings:

mysqlbackup -uroot --port=3306 --backup-dir=/home/backup_dir backup

This skips message logging for the backup operation:

mysqlbackup -uroot --port=3306 --backup-dir=/home/backup_dir \

--skip-messages-logdir backup

This creates a log file for the apply-log operation in an existing directory named /home/teelog_dir,

rather than the default location:

mysqlbackup -uroot --port=3306 --backup-dir=/home/backup_dir \

--messages-logdir=/home/teelog_dir apply-log

This creates a log file for the list-image operation in an existing directory named /home/teelog_dir:

mysqlbackup -uroot --port=3306 --backup-image=/backup/my.mbi \

--messages-logdir=/home/teelog_dir list-image

16.12 Progress Report Options

There are two options for controlling the progress reporting function of mysqlbackup: --show-progress

and --progress-interval:

• --show-progress[={stderr|stdout|file:FILENAME|fifo:FIFONAME|table|variable}]

Command-Line Format --show-progress[=destinations]

Type Enumeration

Valid Values stderr

stdout

file:FILENAME

fifo:FIFONAME

table

variable

The option instructs mysqlbackup to periodically output short progress reports known as progress

indicators on its operation.

The argument of the option controls the destination to which the progress indicators are sent:

• stderr: Progress indicators are sent to the standard error stream. The report is embedded in a time-

stamped mysqlbackup INFO message. For example:

130607 12:22:38 mysqlbackup: INFO: Progress: 191 of 191 MB; state: Completed

• stdout: Progress indicators are sent to the standard output stream. A single newline character is

printed after each progress indicator.

• file:FILENAME: Progress indicators are sent to a file. Each new progress report overwrites the file,

and the file contains the most recent progress indicator followed by a single newline character.

159

Progress Report Options

• fifo:FIFONAME: Progress indicators are sent to a file system FIFO. A single newline character is

printed after each progress indicator.

Warning

If there is no process reading the FIFO, the mysqlbackup process hangs at

the end of the execution.

• table: Progress indicators are sent to the mysql.backup_progress table. This requires a

connection to the MySQL server, and therefore, only works when backing up a running MySQL

instance. mysqlbackup first adds one row of the progress report to the mysql.backup_progress

table, and then updates the row afterwards with the latest progress indicator. The progress indicator is

stored in the current_status column of the table.

If the backup locks the MySQL instance (for example, by issuing a FLUSH TABLES WITH READ

LOCK statement), the progress reports are not delivered to the mysql.backup_progress table until

the MySQL instance is unlocked.

• variable: Progress indicators are sent to the system variable backup_progress.

Warning

The system variable backup_progress is not yet defined for the MySQL

Server. Users need to create their own plugin to define the variable. See The

MySQL Plugin API for more information on user plugins.

When there is no argument specified for --show-progress, progress indicators are sent to stderr.

Progress can be reported to multiple destinations by specifying the --show-progress option several

times on the command line. For example the following command line reports progress of the backup

command to stderr and to a file called meb_output:

mysqlbackup --show-progress --show-progress=file:meb_output --backup-dir=/full-backup

160

Progress Report Options

backup

The progress indicators are short strings that indicate how far the execution of a mysqlbackup

operation has progressed. A progress indicator consists of one or more meters that measure the

progress of the operation. For example:

Progress: 100 of 1450 MB; state: Copying .ibd files

This shows that 100 megabytes of a total of 1450 megabytes have been copied or processed so far, and

mysqlbackup is currently copying InnoDB data files (.ibd files).

The progress indicator string begins with Progress:, followed by one or more meters measuring the

progress. If multiple meters are present, they are separated by semicolons. The different types of meters

include:

• Total data meter: It is always the first meter in the progress indicator. It is in the format of:

DATA of TOTAL UNIT

DATA and TOTAL are unsigned decimal integers, and UNIT is either MB (megabytes), KB (kilobytes),

or bytes (1MB=1024KB and 1KB=1024 bytes).

The total data meter has two slightly different meanings depending on the mysqlbackup operation:

• The amount of data copied or processed and the total amount of data to be copied or processed by

the mysqlbackup operation. For example:

Progress: 200 of 1450 MB

When the operation is for, e.g., backup, the indicator means 200MB is copied of 1450MB. But

when the operation is for, e.g., validate or incremental, it means 200MB is processed out of

1450MB.

• Total amount of data copied or processed and an estimate for the total that will be copied by the end

of the operation. The estimated total is updated as per the data on the server, as the execution of

the command progresses.

For some operations such as backup, it is not possible to know exactly at the start of the execution

how much data will be copied or processed. Therefore, the total data meter shows the estimated

161

Progress Report Options

amount of the total data for a backup. The estimate is updated during the execution of the

command. For example:

Progress: 200 of 1450 MB

is followed by:

Progress: 200 of 1550 MB

when 100MB of data is added on the server.

If the operation is successful, the final progress indicator shows the actual amount of data copied at

the end of the operation.

• Compression meter: It indicates the sliding average of the compression ratio, which is defined for

each block of data that is compressed as (orig_size - compressed_size) / orig_size. For

example:

compression: 40%

This means that after compression, the data takes 40% less space (calculated as an average over the

last 10 data blocks).

The compression meter is included in the progress indicator if the --compress option is enabled for

the mysqlbackup operation. The value of the compression meter is undefined until at least 10 data

blocks have been compressed. The undefined meter value is denoted by the '-' in the meter:

compression: -

• State meter: It is a short description of the major step the command is currently executing. For

example:

state: Copying InnoDB data

state: Waiting for locks

state: Copying system tablespace

state: Copying .ibd files

state: Copying non-InnoDB data

state: Completed

Here are some examples of progress indicators with different meters:

Progress: 300 of 1540 MB; state: Waiting for locks

Progress: 400 of 1450 MB; state: Copying InnoDB data: compression: 30%

The exact set of meters included in the progress indicator depends on the command and the options

used for it.

• --progress-interval=SECONDS

Command-Line Format --progress-interval=SECONDS

Type Numeric

Default Value 2

Minimum Value 1

162

Encryption Options

Maximum Value 100000

Unit second

Interval between progress reports in seconds. Default value is two seconds. The shortest interval is 1

second and the longest allowed interval is 100000 seconds.

16.13 Encryption Options

These options are for creating encrypted single-file backups and for decrypting them. See Chapter 9,

Encryption for Backups for more details and usage examples for the encryption and decryption functions of

MySQL Enterprise Backup.

• --encrypt

Encrypt the data when creating a backup image by a backup-to-image operation, or when packing

a backup directory into a single file with the backup-dir-to-image command. It cannot be used with

the backup or backup-and-apply-log command.

• --decrypt

Decrypt an encrypted backup image when performing an extract, image-to-backup-dir, or

copy-back-and-apply-log operation. It is also used for performing a validate or list-image

operation on an encrypted backup image.

The option cannot be used in a apply-log, backup-and-apply-log, or copy-back operation.

For restoration using the copy-back command, the encrypted backup image has to be unpacked and

decrypted first using the image-to-backup-dir or extract command, together with the --decrypt

option.

• --key=STRING

Command-Line Format --key=KEY

Type String

The symmetric key for encryption and decryption of a backup image. It should be a 256-bit key,

encoded as a string of 64 hexadecimal digits. See Chapter 9, Encryption for Backups on how to create a

key. The option is incompatible with the --key-file option.

• --key-file=PATH

Command-Line Format --key-file=FILE

Type File name

The pathname to file that contains a 256-bit key, encoded as a string of 64 hexadecimal digits, for

encryption and decryption of a backup image. The option is incompatible with the --key option.

16.14 Options for Working with Encrypted InnoDB Tablespaces

Starting with release 4.0.2, MySQL Enterprise Backup supports encrypted InnoDB tablespaces. For details

on how the MySQL server encrypts and decrypts InnoDB tables, see InnoDB Data-at-Rest Encryption. See

Chapter 6, Working with Encrypted InnoDB Tables on how mysqlbackup commands handle encrypted

InnoDB tables.

163

Options for Working with Encrypted InnoDB Tablespaces

When InnoDB tablespace encryption uses Oracle Key Vault (OKV) for encryption key management, the

feature is referred to as “MySQL Enterprise Transparent Data Encryption (TDE).”

The following are the command-line options for working with encrypted InnoDB tables:

• --keyring=VALUE

Command-Line Format --keyring=VALUE

Type String

(Only for MySQL Enterprise Backup 4.1.0, or MySQL Enterprise Backup 4.1.1 working with MySQL

5.7.20 and earlier) The kind of keyring plugin used for master encryption key management. Currently,

there are two possible values:

• keyring_file: The keyring_file plugin is used, which means the master encryption key is

stored in a keyring file, the location of which is specified by the --keyring_file_data option.

• keyring_okv: The keyring_okv plugin is used, which means the master keys are managed by

an Oracle Key Vault (OKV); the location of the key vault's endpoint directory is specified by the --

keyring_okv_conf_dir option.

The option must be specified for copy-back-and-apply-log operations, copy-back operations that use the

--generate-new-master-key option, and offline backups. For online backups, the keyring plugin

setting on the server overrides the value set by this option.

• --keyring_file_data=PATH

Command-Line Format --keyring_file_data=PATH

Type File name

(Only for MySQL Enterprise Backup 4.1.0, or MySQL Enterprise Backup 4.1.1 working with MySQL

5.7.20 and earlier) Path to the keyring file. The option is required when the option --keyring has the

value keyring_file. For online backups, the --keyring_file_data setting on the server overrides

the value set by this option.

• --keyring_okv_conf_dir=PATH

Command-Line Format --keyring_okv_conf_dir=PATH

Type Directory name

(Only for MySQL Enterprise Backup 4.1.0, or MySQL Enterprise Backup 4.1.1 working with

MySQL 5.7.20 and earlier) Path to the Oracle Key Vault (OKV) endpoint directory. The option is

required when the option --keyring has the value keyring_okv. For online backups, the --

keyring_okv_conf_dir setting on the server overrides the value set by this option.

• --encrypt-password[=STRING]

Command-Line Format --encrypt-password=STRING

Type String

For MySQL Enterprise Backup 4.1.0, or MySQL Enterprise Backup 4.1.1 working with MySQL 5.7.20

and earlier: The user-supplied password by which mysqlbackup encrypts the encryption keys for the

tablespaces.

164

Cloud Storage Options

For MySQL Enterprise Backup 4.1.1 working with MySQL 5.7.21 and later: The user-supplied

password by which mysqlbackup encrypts the master encryption key, which is used to encrypt the

encryption keys for the InnoDB tablespaces. If the server is using the keyring_encrypted_file

plugin, the password supplied with the option must match the value of the system variable

keyring_encrypted_file_password on the server. If different passwords were used for different

backups in a sequence of full and incremental backups, make sure the very password used to create

an individual backup is supplied when performing an apply-log, apply-incremental-backup, or

copy-back-and-apply-log operation with it.

The option must be used when backing up a server that has a keyring plugin enabled for InnoDB table

encryption and for restoring a backup containing encrypted InnoDB tables. The same password supplied

during backup must be supplied again during a copy-back-and-apply-log, apply-log, or an

apply-incremental-backup operation for the backup, or mysqlbackup will error out when it

encounters encrypted InnoDB tables during the operation.

Users who do not want to supply the password on the command line or in a defaults file may use the

option without specifying any value; mysqlbackup then asks the user to type in the password before the

operation starts.

• --generate-new-master-key

Command-Line Format --generate-new-master-key

(Only for MySQL Enterprise Backup 4.1.0, or MySQL Enterprise Backup 4.1.1 working with MySQL

5.7.20 and earlier) Generate new master key. During a restore, when the option is used, mysqlbackup

generates a new master key and uses it to reencrypt all the tablespace keys. To use the --

generate-new-master-key option, the --keyring option, as well as the --keyring_file_data

option (when --keyring=keyring_file) or --keyring_okv_conf_dir option (when --

keyring=keyring_okv) must be specified, so that mysqlbackup can access the keyring.

16.15 Cloud Storage Options

These options are for using cloud storage for single-file operations. See Section 4.3.1.3, “Backing Up to

Cloud Storage”, and Section 5.1.5, “Restoring a Backup from Cloud Storage to a MySQL Server”, for more

information and instructions on using cloud storage with MySQL Enterprise Backup.

• --cloud-service=SERVICE

Cloud service for data backup or restoration. Currently, there are two types of cloud storage services

supported by mysqlbackup, represented by the following values for the options:

• openstack: OpenStack Swift or compatible object storage services (for example, Oracle Cloud

Infrastructure Object Storage and Oracle Cloud Infrastructure Object Storage Classic).

• s3: Amazon Simple Storage Service (S3).

• --cloud-trace

Print trace information for cloud operations. It works independently of --trace, which specifies the

trace level for the non-cloud operations of mysqlbackup. Any non-zero value for the option enables the

trace function.

Default value is “0.”

• --cloud-proxy=proxy-url:port

165

Cloud Storage Options

Proxy address and port number for overriding the environment's default proxy settings for accessing a

cloud storage service.

Note

The list-image operation can be performed on a cloud backup only if the

cloud proxy supports HTTP range headers.

• --cloud-ca-info=PATH

Absolute path to the CA bundle file for host authentication for SSL connections. When the option is

specified, the usage of the CA bundle file is preferred over the usage of individual .pem files for host

authentication.

• --cloud-ca-path=PATH

CA certificate directory, in addition to the system's default folder.

• --cloud-buffer-size=MB

(For release 4.1.1 and later) Size of the buffer for cloud operations in megabytes. mysqlbackup

accumulates data up to the size specified by this option before initiating a cloud transfer. The value has

to be between 16 to 4096.

Default: 64

• Options used only for OpenStack Swift or compatible object storage services (using them when the

argument for --cloud-service is anything other than openstack will cause mysqlbackup to throw

an error):

• --cloud-container=SWIFT_CONTAINER

The Swift container for the backup image. For Oracle Cloud Infrastructure (OCI) Object Storage, this

is the object storage bucket.

• --cloud-object=SWIFT_OBJECT

The Swift object for the backup image. Note that names of objects within the same container (or

bucket, for OCI Object Storage) have to be unique.

• --cloud-user-id=SWIFT_USER_ID

User ID for accessing Swift. The user credentials are authenticated by the Swift TempAuth identity

system when the --cloud-tempauth-url option is used, by the OpenStack Keystone identity

service when the --cloud-identity-url option is used, and by HTTP Basic Authentication when

the --cloud-basicauth-url option is used.

• --cloud-password=SWIFT_PASSWORD

Password for accessing Swift for the user specified by the --cloud-user-id option. The user

credentials are authenticated by the Swift TempAuth identity system when the --cloud-tempauth-

url option is used, by the OpenStack Keystone identity service when the --cloud-identity-url

166

Cloud Storage Options

option is used, and by HTTP Basic Authentication when the --cloud-basicauth-url option is

used.

• --cloud-tempauth-url=SWIFT_TEMPAUTH-URL

The TempAuth URL for authenticating user credentials.

• --cloud-basicauth-url=SWIFT_BASICAUTH-URL

(For release 4.1.1 and later) The URL for HTTP Basic Authentication.

• --cloud-storage-url=SWIFT_STORAGE-URL

(For release 4.1.2 and later) The URL for Oracle Cloud Infrastructure Object Storage when using

OAuth for client authentication.

• --cloud-oauth-token=SWIFT_OAUTH-TOKEN

(For release 4.1.2 and later) The OAuth token for Oracle Cloud Infrastructure Object Storage access.

The option value must be in one of the following formats:

• --cloud-oauth-token=oauth_token, where oauth_token is the OAuth token itself

• --cloud-oauth-token=file:oauth_filepath, where oauth_filepath points to a file

containing only the OAuth token.

• --cloud-identity-url=SWIFT_KEYSTONE-URL

The URL of the Keystone identity service, when it is used for authenticating user credentials.

• --cloud-tenant=SWIFT_KEYSTONE-TENANT

The Keystone tenant for the user specified by --cloud-user-id, when the Keystone identity

service is used for authenticating user credentials.

• --cloud-region=SWIFT_KEYSTONE-REGION

The Keystone region for the user specified by --cloud-user-id, when the Keystone identity

service is used for authenticating user credentials.

• --cloud-chunked-transfer=VALUE

(For release 4.1.1 and later) Use chunked transfer with Swift service.

If the option is set to true, backups are transferred and stored as Dynamic Large Objects (DLOs),

for which multiple file segments are considered as a single file. Each backup larger than 5GB is split

167

Options for Special Backup Types

into multiple parts with names in the form of <object_name>_part_<number>. See the OpenStack

documentation for details.

If the option is set to false, mysqlbackup uploads the backup in segments in the size of the buffer,

and the maximum number of segments a backup can have is determined by the Swift service.

Default: true

Warning

Set the option to false if chunked transfer is not supported by your cloud

storage; otherwise, the mysqlbackup operation may fail.

One and only one of --cloud-tempauth-url, --cloud-identity-url, --cloud-basicauth-

url, or --cloud-storage-url should be used when accessing a Swift service, or mysqlbackup will

throw an error.

• Options used only for Amazon S3 (using them when the argument for --cloud-service is anything

other than s3 will cause mysqlbackup to throw an error):

• --cloud-bucket=S3_BUCKET

The storage bucket on Amazon S3 for the backup image.

In order to perform cloud backups and restores with the bucket, the user identified by the --cloud-

access-key-id option must have at least the following permissions on the bucket:

• s3:ListBucket: For listing information on items in the bucket.

• s3:ListBucketMultipartUploads: For listing multipart uploads in progress to the bucket.

• s3:GetObject: For retrieving objects from the bucket.

• s3:PutObject: For adding objects to the bucket.

• --cloud-object-key=S3_OBJECT_KEY

The Amazon S3 object key for the backup image.

• --cloud-access-key-id=S3_KEY-ID

AWS access key ID for logging onto Amazon S3.

• --cloud-secret-access-key=S3_ACCESS-KEY

AWS secret access key for the AWS access key id specified with --cloud-access-key-id.

• --cloud-aws-region=S3_REGION

Region for Amazon Web Services that mysqlbackup accesses for S3.

16.16 Options for Special Backup Types

These options are for backing up database servers that play specific roles in replication, or contain certain

kinds of data that require special care in backing up.

• --slave-info

168

Options for Special Backup Types

When backing up a replica server, this option captures information needed to set up an identical

replica server. It creates a file meta/ibbackup_slave_info inside the backup directory, containing

a CHANGE MASTER statement with the binary log position and name of the binary log file of the source

server. This information is also printed in the mysqlbackup output. To set up a new replica for this

source, restore the backup data on another server, start a replica server on the backup data, and issue a

CHANGE MASTER command with the binary log position saved in the ibbackup_slave_info file. See

Section 7.1, “Setting Up a New Replica” for instructions.

Note

Only use this option when backing up a replica server. Its behavior is undefined

when used on a source or non-replication server.

This option is not compatible with the --no-locking option; using both options

together will make mysqlbackup throw an error.

This option is not compatible with the --only-innodb or --only-innodb-

with-frm options.

• --safe-slave-backup-timeout=SECONDS

For a statement-based replication (SBR) or a mixed-based replication setup, the option specifies the

time (in seconds) mysqlbackup will wait for Slave_open_temp_tables to become “0” (which is true

when no temporary tables are open) to complete the backup for a replica server by asserting a global

read lock and copies all the non-InnoDB tables. If the duration of the wait exceeds that specified with

the option, mysqlbackup times out and throws an error. The wait is for preventing mysqlbackup from

finishing a replica backup when there are temporary tables still open. See descriptions in Temporary

tables on statement-based replication (SBR) replica for details on how mysqlbackup deals with

temporary tables on a replica server.

In addition, mysqlbackup also runs an initial check at the beginning of a replica backup to see if

Slave_open_temp_tables=0 becomes true within the duration set by --safe-slave-backup-

timeout. If it does not, mysqlbackup takes it as an early sign that before the backup is completed,

some temporary tables are likely to remain open after the timeout limit is exceeded; mysqlbackup

then throws an error, instead of continuing with the backup. When that happens, you can either restart

the backup with a higher value for --safe-slave-backup-timeout, or retry at a time when fewer

temporary tables are being used.

Default: 300

Warning

Proper setting of this value depends on the use case, and it can vary a lot

according to the situation. Setting the value for this option either too high or too

low will affect adversely the performance of the backup operation:

• Too high: If you need to wait for a long time for there to be no more temporary

tables, the chance is that the change rate for your database is quite high, which

means the amount of redo log data to be included in the backup will be large

and the restore time for the backup will be long. In such a case, it would have

been better to have let mysqlbackup timeout and then restart the backup

operation, so the tables are copied in their final states. It is therefore not helpful

to set a high timeout value for the option. As a very general rule of thumb: even

for busy databases that use many contemporary tables, do not set the value to

more than an a few hours.

169

Other Options

• Too low: Setting the wait time value too low would make the backup process

time out very easily and when that happens, the process has to be restarted.

With a repeating cycle of restarts, the backup might then take a long time to

complete, and resources used on the failed backups will be wasted. As a very

general rule of thumb, do not set the timeout to below the default value of 300s.

For a row-based replication (RBR) setup, temporary tables are not replicated onto the replica. Users who

are certain that SBR is not occurring for the replica can set --safe-slave-backup-timeout=0, with

which mysqlbackup will not check for any open temporary tables before finishing the backup.

• --suspend-at-end

This option pauses the mysqlbackup command when the backup procedure is close to ending. It

creates a file called ibbackup_suspended in the backup log group home directory and waits until you

delete that file before proceeding. This option is useful to customize locking behavior and backup of non-

InnoDB files through custom scripting.

All tables are locked before suspending, putting the database into a read-only state, unless you turn

off locking with the --no-locking or --no-connection option. The --only-innodb and --

only-innodb-with-frm options also prevent the locking step. Because locking all tables could be

problematic on a busy server, you might use a combination of --only-innodb and --suspend-at-

end to back up only certain InnoDB tables.

• --exec-when-locked="utility arg1 arg2 ..."

Command-Line Format --exec-when-locked="utility arg1 arg2 ..."

Type String

You can use this option to run a script that backs up any information that is not included as part of the

usual backup. For example, with --exec-when-locked, you can use mysqldump to back up tables

from the MEMORY storage engine, which are not on disk.

Set any variable you want to use within your script before you run mysqlbackup. In the following

example, the BACKUP_DIR environment variable is set to point to the current backup directory (quotes

are used for the argument of --exec-when-locked, to prevent premature expansion of the variable

BACKUP_DIR):

On Unix or Linux systems:

export BACKUP_DIR=path_to_backupdir

mysqlbackup --exec-when-locked="mysqldump mydb t1 > $BACKUP_DIR/t1.sql" other_options mysqlbackup_command

Or on Windows systems:

set BACKUP_DIR=path_to_backupdir

mysqlbackup --exec-when-locked="mysqldump mydb t1 > %BACKUP_DIR%/t1.sql" other_options mysqlbackup_command

If the utility cannot be executed or returns a non-zero exit status, the whole backup process is cancelled.

If you also use the --suspend-at-end option, the utility specified by --exec-when-locked is

executed after the suspension is lifted.

16.17 Other Options

This section explains any options that are not covered by other sections in this chapter.

• --error-code=CODE

170

Other Options

Command-Line Format --error-code

Type Numeric

Minimum Value 0

Maximum Value 19

Specifies the exit code for which the print-message command prints the corresponding exit

message. See Section 13.1, “Exit codes of MySQL Enterprise Backup” for details.

171

172

Chapter 17 Configuration Files and Parameters

You can specify mysqlbackup options either on the command line or as configuration parameters inside a

configuration file. This section describes the use of configuration files.

In general, mysqlbackup follows the mysql style of processing configuration options: [mysqlbackup]

and [client] group options are passed as command-line options. Any command-line options that you

specify when you run mysqlbackup override the values from the configuration file, and in the case of

duplicate options, the last instance takes precedence. mysqlbackup also reads options in the [mysqld]

group to detect parameters related to the source repository when no connection to mysqld is available.

Within mysqlbackup option names, dashes (-) and underscores (_) may be used interchangeably, similar

to mysqld parameters that use this same convention (see Using Options on the Command Line in the

MySQL Reference Manual for details). The MySQL server's reference manual typically lists the parameter

names with underscores, to match the output of the SHOW VARIABLES statement.

Options Files

mysqlbackup reads the location of the MySQL data to be backed up from the following sources (listed in

order of priority):

• The connection information from the running database, whenever possible. Thus, in most cases, you can

avoid specifying most options on the command line or in a configuration file.

• Parameters you specify on the mysqlbackup command line. You can specify certain options for

individual backup jobs this way.

• The MySQL configuration file (by default, my.cnf on Unix and my.ini on Windows). The parameters

are searched for first under the [mysqlbackup] group, then under the [client] group. You can put

common parameters that apply to most of your backup jobs into the configuration file.

Configuration Files Stored with the Backup Data

Each set of backup data includes a configuration file, backup-my.cnf, containing a minimal set of

configuration parameters. The mysqlbackup command generates this file to record the settings that apply

to this backup data. Subsequent operations, such as the apply-log process, read options from this file to

determine how the backup data is structured.

Example 17.1 Example backup-my.cnf file

Here is an example backup-my.cnf file generated by mysqlbackup:

# Generated backup-my.cnf file.

# Auto generated by mysqlbackup program.

[mysqld]

innodb_data_file_path=ibdata1:12M:autoextend

innodb_log_file_size=50331648

innodb_log_files_in_group=2

innodb_page_size=16384

innodb_checksum_algorithm=crc32

innodb_buffer_pool_filename=ib_buffer_pool

All paths in the generated backup-my.cnf file point to a single backup directory. For ease of verification

and maintenance, you typically store all data for a backup inside a single directory rather than scattered

among different directories.

173

Configuration Files Stored with the Backup Data

During a backup, the configuration parameters that are required for later stages (such as the restore

operation) are recorded in the backup-my.cnf file that is generated in the backup directory. Only

the minimally required parameters are stored in backup-my.cnf, to allow you to restore the backup

to a different environment without having to make extensive changes to that file. For example, the

innodb_data_home_dir and innodb_log_group_home_dir options are omitted from the backup-

my.cnf file when their values are the same as the value for backup-dir .

174

Part IV Appendixes

Table of Contents

A Frequently Asked Questions for MySQL Enterprise Backup ........................................................... 179

B Limitations of MySQL Enterprise Backup ...................................................................................... 181

C Compatibility Information for MySQL Enterprise Backup ................................................................ 183

C.1 Supported Platforms ......................................................................................................... 183

C.2 Cross-Platform Compatibility ............................................................................................. 183

C.3 Compatibility with MySQL Versions ................................................................................... 183

C.4 Compatibility with Older MySQL Enterprise Backup ........................................................... 183

D Backup History Table Update ...................................................................................................... 185

MySQL Enterprise Backup Glossary ................................................................................................ 187

177

178

Appendix A Frequently Asked Questions for MySQL Enterprise

Backup

This section lists some common questions about MySQL Enterprise Backup, with answers and pointers to

further information.

Questions

• A.1: What versions of the MySQL server does MySQL Enterprise Backup 4.1 support?

• A.2: What is the big ibdata file that is in all the backups?

• A.3: Can I back up non-InnoDB data with MySQL Enterprise Backup?

• A.4: What happens if the apply-log or apply-incremental-backup step is interrupted?

• A.5: Why is the option --defaults-file not recognized?

• A.6: Can I back up a database on one OS platform and restore it on another one using MySQL

Enterprise Backup?

• A.7: What if I have included the binary log or relay log in my backup but do not want to restore it?

• A.8: What would happen if I start a server directly using a raw directory backup, without running either

the copy-back or the apply-log operation?

Questions and Answers

A.1: What versions of the MySQL server does MySQL Enterprise Backup 4.1 support?

See Section C.3, “Compatibility with MySQL Versions” for details of compatibility between different

releases of MySQL Enterprise Backup and MySQL Server.

A.2: What is the big ibdata file that is in all the backups?

You might find your backup data taking more space than expected because of a large file with a name

such as ibdata1. This file represents the InnoDB system tablespace, which grows but never shrinks as a

database operates, and is included in every full and incremental backup. To reduce the space taken up by

this file in your backup data:

• After doing a full backup, do a succession of incremental backups, which take up less space. The

ibdata1 file in the incremental backups is typically much smaller, containing only the portions of the

system tablespace that changed since the full backup.

• Set the configuration option innodb_file_per_table=1 before creating your biggest or most active

InnoDB tables. Those tables are split off from the system tablespaces into separate .ibd files; the

tables can then be individually included or excluded from backups, and disk space is freed when the

tables are dropped or truncated.

• If your system tablespace is very large because you created a high volume of InnoDB data before

turning on the innodb_file_per_table setting, you might use mysqldump to create a dump of your

entire server instance, then turn on innodb_file_per_table before re-creating the database, so that

all the table data is kept outside the system tablespace.

A.3: Can I back up non-InnoDB data with MySQL Enterprise Backup?

179

While MySQL Enterprise Backup can back up non-InnoDB data (like MYISAM tables), the MySQL server

to be backed up must support InnoDB (i.e., the backup process will fail if the server was started up with the

--innodb=OFF or --skip-innodb option), and the server must contain at least one InnoDB table.

A.4: What happens if the apply-log or apply-incremental-backup step is interrupted?

If mysqlbackup is interrupted during the apply-log or apply-incremental-backup stage, the

backup data is OK. The file operations performed by those options can be performed multiple times without

harming the consistency of the backup data. Just run the same mysqlbackup command again, and when

it completes successfully, all the necessary changes are present in the backup data.

A.5: Why is the option --defaults-file not recognized?

When you specify the --defaults-file option, it must be the first option going after mysqlbackup.

Otherwise, the error message makes it look as if the option name is not recognized.

A.6: Can I back up a database on one OS platform and restore it on another one using MySQL

Enterprise Backup?

See Section C.2, “Cross-Platform Compatibility” for details.

A.7: What if I have included the binary log or relay log in my backup but do not want to restore it?

If you want to skip the restore of the binary log, relay log, or both during a restore, use the --skip-

binlog option, the --skip-relaylog option, or both with your copy-back or copy-back-and-

apply-log command.

A.8: What would happen if I start a server directly using a raw directory backup, without running

either the copy-back or the apply-log operation?

This should never be attempted. Not only would the server crash, but the backup would likely get corrupted

and become unusable. This is because the directory backup contains metadata created by mysqlbackup

that the MySQL server would not understand; also, the raw backup might be inconsistent and need to be

brought up-to-date by an apply-log operation, so that changes made to the database during the backup

process can be applied.

180

Appendix B Limitations of MySQL Enterprise Backup

Please refer to the MySQL Enterprise Backup 4.1 Release Notes for a list of fixed bugs for mysqlbackup.

Here is a list of limitations of MySQL Enterprise Backup:

• In some cases, backups of non-transactional tables such as MyISAM tables could contain additional

uncommitted data. If autocommit is turned off, and both InnoDB tables and non-transactional tables are

modified within the same transaction, data can be written to the non-transactional table before the binary

log position is updated. The binary log position is updated when the transaction is committed, but the

non-transactional data is written immediately. If the backup occurs while such a transaction is open, the

backup data contains the updates made to the non-transactional table.

• If the mysqlbackup process is interrupted by, for example, a Unix kill -9 command, a FLUSH

TABLES WITH READ LOCK operation might remain running. In this case, use the KILL QUERY

statement from the mysql command line to kill the FLUSH TABLES WITH READ LOCK statement.

This issue is more likely to occur if the FLUSH TABLES operation is stalled by a long-running query or

transaction. Refer to Section 11.1, “Optimizing Backup Performance” for guidelines about backup timing

and performance.

• Do not run the DDL operations (for example, ALTER TABLE, TRUNCATE TABLE, OPTIMIZE TABLE,

REPAIR TABLE, RESTORE TABLE or CREATE INDEX) while a backup operation is going on. The

resulting backup might become corrupted.

• The engines column in the mysql.backup_history table does not correctly reflect the storage

engines of the backed-up databases.

• Hot backups for large databases with heavy writing workloads (say, in the order of gigabytes per minute)

can take a very long time to complete due to the huge redo log files that are generated on the server

while the backup is running. However, when it is a relatively small subset of tables in the database that

are being modified frequently, the Optimistic Backup feature can be used to improve performance and

reduce backup size, as well as backup and recovery times. See Section 4.3.6, “Making an Optimistic

Backup” for details.

• While it is possible to backup to or restore from a Network Attached Storage (NAS) device using MySQL

Enterprise Backup, due to networking issues that might arise, the consistency of the backups and the

performance of the backup or restore operations might be compromised.

• When creating a backup using transportable tablespace (TTS) for a server containing tables with a

mix of the Antelope and Barracuda file formats, do not apply full locking on the tables (that is, do not

specify --use-tts=with-full-locking). Instead, just specify --use-tts or --use-tts=with-

minimum-locking, both of which will apply minimum locking to the tables.

• Backup of a partitioned table using transportable tablespace (TTS) would fail when any (or all) of its

partitions were created in a shared tablespace.

• Restoring a partitioned table backed up using transportable tablespace (TTS), even when the --force

option is used, would fail if any of the partitions was created outside of the backed-up server's data

directory.

• If Data Definition Language (DDL) statements are executed on the server when a backup is being

created using transportable tablespace (TTS), the backup might fail. It is because tables not being

backed up are not locked during the backup process, but mysqlbackup still checks the status of

those tables at the end of the process, and an error might occur if definitions of those tables have been

changed. To avoid the problem, do not perform any DDL operation, especially DROP TABLE, when a

TTS backup is going on.

181

• If a table containing full-text search (FTS) index is backed up using transportable tablespace (TTS), after

it is restored, the FTS index will be corrupted. Users will need to recreate the index with the following

command:

mysql> ALTER TABLE mytable ENGINE = INNODB;

Then, check that there are no more errors with the table:

mysql> CHECK TABLE mytable;

• Tables created on the MySQL server with the ANSI_QUOTES SQL mode cannot be backed up using

transportable tablespace (TTS).

• MySQL Enterprise Backup does not include the .pem files from the server into the backup. The files are

part of the server instance when SSL connections are enabled.

• When backing up an instance of MySQL 5.7, if a CREATE INDEX statement with ALGORITHM =

INPLACE is issued when the backup process is going on, because the statement will not go into the redo

log of the MySQL 5.7 server (see Sorted Index Builds for details), it cannot be recorded in the backup,

and the index will not be recreated by mysqlbackup when the backup is restored.

• When a file of an unrecognized file type exists under a subdirectory in the server's data directory, it will

be backed up by mysqlbackup unless the --only-known-file-types option is used. However, if

the name of the file does not have an extension, it will cause mysqlbackup to throw an error when it

tries to restore the backup to a server.

• Cloud operations by MySQL Enterprise Backup are not supported on macOS and Windows platforms,

and also on Linux platforms when generic Linux builds are used for both the server and MySQL

Enterprise Backup (i.e., when both the server and MySQL Enterprise Backup have been installed using

generic Linux tarballs).

• Using the --src-entry option with the extract command on cloud backups will cause the command

to fail. Cloud backups can only be extracted in full.

• Some limitations apply when mysqlbackup works with encrypted InnoDB tables. See the discussion

here for details.

• Backup operations may fail if checksums for redo log pages are disabled (i.e., if --

innodb_log_checksums is OFF or FALSE or 0) on the server.

• A compressed directory backup fails when a general tablespace bears the same basename as the

database's system tablespace (usually ibdata1) and exists in the same directory with it (usually the

server's data directory). A compressed single-file backup created under the same situation will be

corrupted, and cannot be restored. To avoid the problem, the server administrator should not put into

the same directory the system tablespace and a general tablespace of the same basename; if that is

unavoidable, do not perform a compressed backup for the database.

• When working with a replication set up whose source server also belongs to a separate Group

Replication setup, over time, create backups consistently either from the source or the replica, but not

from both. Otherwise, there will be conflicts between the id values generated by the source and the

replica, causing backups to fail.

182

Appendix C Compatibility Information for MySQL Enterprise

Backup

Table of Contents

C.1 Supported Platforms ................................................................................................................. 183

C.2 Cross-Platform Compatibility ..................................................................................................... 183

C.3 Compatibility with MySQL Versions ........................................................................................... 183

C.4 Compatibility with Older MySQL Enterprise Backup ................................................................... 183

This section describes information related to compatibility issues for MySQL Enterprise Backup releases.

C.1 Supported Platforms

See Supported Platforms: MySQL Database (MySQL platform support evolves over time; please refer to

the page for the latest updates).

C.2 Cross-Platform Compatibility

MySQL Enterprise Backup is cross-platform compatible when running on the Linux and Windows

operating systems: backups on a Linux machine can be restored on a Windows machine, and vice versa.

However, to avoid data transfer problems arising from letter cases of database or table names, the

variable lower_case_table_names must be properly configured on the MySQL servers. For details, see

Identifier Case Sensitivity.

C.3 Compatibility with MySQL Versions

MySQL Enterprise Backup 4.1 only supports MySQL 5.7. For MySQL 5.6 and earlier, use MySQL

Enterprise Backup 3.12.

For MySQL 5.7.11 and earlier, backup for InnoDB tablespaces encrypted with “MySQL Enterprise

Transparent Data Encryption (TDE)” is not supported by MySQL Enterprise Backup 4.1. See the

discussion here for details.

C.4 Compatibility with Older MySQL Enterprise Backup

Currently, MySQL Enterprise Backup 4.1 is incompatible with earlier MySQL Enterprise Backup versions.

183

184

Appendix D Backup History Table Update

The mysql.backup_history table has been updated with the release of MySQL Enterprise Backup

4.1.2 in the following ways:

• Changed the storage engine from CSV to InnoDB

• Added a new column for server UUIDs

When MySQL Enterprise Backup 4.1.2 tries to perform its first full backup on a database, it attempts to

perform a format update on the table automatically with the following steps:

• Create in the new format a table named mysql.backup_history_new and copy into it data from the

original mysql.backup_history table.

• Rename the original mysql.backup_history table to mysql.backup_history_old, and the

mysql.backup_history_new table to mysql.backup_history.

For the migration to the new table format to succeed, before performing the first backup task with MySQL

Enterprise Backup 4.1.2 , grant the required privileges to the mysqlbackup user on the server by issuing

these statements at the mysql client: :

GRANT CREATE, INSERT, DROP ON mysql.backup_history_old TO 'mysqlbackup'@'localhost';

GRANT CREATE, INSERT, DROP, ALTER ON mysql.backup_history_new TO 'mysqlbackup'@'localhost';

If these privileges are not granted, the first full backup on the upgraded server will fail with an error

message similar to the following:

180612 08:40:45 MAIN ERROR: MySQL query 'DROP TABLE IF EXISTS mysql.backup_history_old': 1142, DROP command denied to user 'mysqlbackup'@'localhost' for table 'backup_history_old'

mysqlbackup failed with errors!

These privileges are no longer needed after the first full backup has been performed by MySQL Enterprise

Backup 4.1.2 , by which point they can be revoked.

Note

If you are working with a multiprimary Group Replication setting, make sure these

privileges are granted on all primary nodes; see also Chapter 8, Using MySQL

Enterprise Backup with Group Replication.

185

186

MySQL Enterprise Backup Glossary

These terms are commonly used in information about the MySQL Enterprise Backup product.

.ARM file

Metadata for ARCHIVE tables. Contrast with .ARZ file. Files with this extension are always included in backups

produced by the mysqlbackup command of the MySQL Enterprise Backup product.