Master – 192.168.56.3

Slave – 192.168.56.4

Percona Toolkit Version – 3.0.12

MySQL Version -MySQL GA 5.7.23

I have created a user in the master to check and repair:

GRANT ALL ON *.* to chksum@'%' identified by 'Password@2018';

In case you have non default ports (3306) for MySQL, Please set the following variables on the slaves:

# my.cnf
[mysqld]
report_host = slave
report_port = 3307

Confirm data inconsistencies in Master (192.168.56.3) and Slave (192.168.56.4):

Master

mysql> select count(1) from titles; 
+----------+
| count(1) |
+----------+
|   443308 |
+----------+
1 row in set (0.09 sec)

Slave

mysql> select count(1) from titles; 
+----------+
| count(1) |
+----------+
|   443311 |
+----------+
1 row in set (0.09 sec)

Check data inconsistencies using pt-table-checksum:

Check for data inconsistencies by executing the following command** on the Master:

** command I have executed below :- pt-table-checksum h=192.168.56.3,u=chksum,p=Password@2018,P=3306 –set-vars innodb_lock_wait_timeout=30 –no-check-binlog-format –databases=employees –tables=titles

[root@localhost ~]# pt-table-checksum h=192.168.56.3,u=chksum,p=Password@2018,P=3306 --set-vars innodb_lock_wait_timeout=30 --no-check-binlog-format --databases=employees --tables=titles
Checking if all tables can be checksummed ...
Starting checksum ...
            TS ERRORS  DIFFS     ROWS  DIFF_ROWS  CHUNKS SKIPPED    TIME TABLE
09-20T22:53:02      0      2   443308          5       6       0   1.319 employees.titles

Fixing data inconsistencies using pt-table-checksum

We are checking data inconsistency from Master (192.168.56.3) to Slave (192.168.56.3) by executing the following command** on the Master:

** command I have executed below :- pt-table-sync –sync-to-master h=192.168.56.4,u=chksum,p=Password@2018,P=3306 –databases=employees –tables=titles –print

[root@localhost ~]# pt-table-sync --sync-to-master h=192.168.56.4,u=chksum,p=Password@2018,P=3306 --databases=employees --tables=titles --print
REPLACE INTO `employees`.`titles`(`emp_no`, `title`, `from_date`, `to_date`) VALUES ('10144', 'Senior Staff', '1992-10-14', '1993-08-10') /*percona-toolkit src_db:employees src_tbl:titles src_dsn:P=3306,h=192.168.56.3,p=...,u=chksum dst_db:employees dst_tbl:titles dst_dsn:P=3306,h=192.168.56.4,p=...,u=chksum lock:1 transaction:1 changing_src:1 replicate:0 bidirectional:0 pid:3789 user:root host:localhost.localdomain*/;
REPLACE INTO `employees`.`titles`(`emp_no`, `title`, `from_date`, `to_date`) VALUES ('10144', 'Staff', '1985-10-14', '1992-10-14') /*percona-toolkit src_db:employees src_tbl:titles src_dsn:P=3306,h=192.168.56.3,p=...,u=chksum dst_db:employees dst_tbl:titles dst_dsn:P=3306,h=192.168.56.4,p=...,u=chksum lock:1 transaction:1 changing_src:1 replicate:0 bidirectional:0 pid:3789 user:root host:localhost.localdomain*/;
DELETE FROM `employees`.`titles` WHERE `emp_no`='87000' AND `title`='Staff Engineer' AND `from_date`='1990-01-01' LIMIT 1 /*percona-toolkit src_db:employees src_tbl:titles src_dsn:P=3306,h=192.168.56.3,p=...,u=chksum dst_db:employees dst_tbl:titles dst_dsn:P=3306,h=192.168.56.4,p=...,u=chksum lock:1 transaction:1 changing_src:1 replicate:0 bidirectional:0 pid:3789 user:root host:localhost.localdomain*/;
DELETE FROM `employees`.`titles` WHERE `emp_no`='97000' AND `title`='Database Engineer' AND `from_date`='1991-01-01' LIMIT 1 /*percona-toolkit src_db:employees src_tbl:titles src_dsn:P=3306,h=192.168.56.3,p=...,u=chksum dst_db:employees dst_tbl:titles dst_dsn:P=3306,h=192.168.56.4,p=...,u=chksum lock:1 transaction:1 changing_src:1 replicate:0 bidirectional:0 pid:3789 user:root host:localhost.localdomain*/;
DELETE FROM `employees`.`titles` WHERE `emp_no`='97500' AND `title`='Project Manager' AND `from_date`='1983-04-11' LIMIT 1 /*percona-toolkit src_db:employees src_tbl:titles src_dsn:P=3306,h=192.168.56.3,p=...,u=chksum dst_db:employees dst_tbl:titles dst_dsn:P=3306,h=192.168.56.4,p=...,u=chksum lock:1 transaction:1 changing_src:1 replicate:0 bidirectional:0 pid:3789 user:root host:localhost.localdomain*/;
DELETE FROM `employees`.`titles` WHERE `emp_no`='97501' AND `title`='Project Manager' AND `from_date`='1983-04-11' LIMIT 1 /*percona-toolkit src_db:employees src_tbl:titles src_dsn:P=3306,h=192.168.56.3,p=...,u=chksum dst_db:employees dst_tbl:titles dst_dsn:P=3306,h=192.168.56.4,p=...,u=chksum lock:1 transaction:1 changing_src:1 replicate:0 bidirectional:0 pid:3789 user:root host:localhost.localdomain*/;
DELETE FROM `employees`.`titles` WHERE `emp_no`='97502' AND `title`='Project Engineer' AND `from_date`='1993-04-11' LIMIT 1 /*percona-toolkit src_db:employees src_tbl:titles src_dsn:P=3306,h=192.168.56.3,p=...,u=chksum dst_db:employees dst_tbl:titles dst_dsn:P=3306,h=192.168.56.4,p=...,u=chksum lock:1 transaction:1 changing_src:1 replicate:0 bidirectional:0 pid:3789 user:root host:localhost.localdomain*/;
[root@localhost ~]# 

To fix inconsistencies on MySQL Master against the Slave execute the following command on the Master:

[root@localhost ~]# pt-table-sync --sync-to-master h=192.168.56.4,u=chksum,p=Password@2018,P=3306 --databases=employees --tables=titles --execute

Confirm the data inconsistencies in Master (192.168.56.3) and Slave (192.168.56.4) are fixed:

Master

mysql> select count(1) from titles;
+----------+
| count(1) |
+----------+
|   443308 |
+----------+
1 row in set (0.09 sec)

Slave

mysql> select count(1) from titles;
+----------+
| count(1) |
+----------+
|   443308 |
+----------+
1 row in set (0.09 sec)

Conclusion 

I recommend / encourage my customers to perform pt-table-checksum exercise regularly in their MySQL replication infrastructure to avoid unpleasant experiences due to data consistency issues.

The post How to check and fix MySQL replication inconsistencies ? appeared first on The WebScale Database Infrastructure Operations Experts.

How PT-ONLINE-SCHEMA-CHANGE works ?

PT-ONLINE-SCHEMA-CHANGE tool is developed by Percona to alter tables without locking them during ALTER TABLE operation. This  tool creates a new empty table like the original table with the necessary structure changes, copy the data from the original table in multiple smaller chunks to the new table, drop the original table and finally rename the new table to the original name. I have explained below PT-ONLINE-SCHEMA-CHANGE with simple example

Consider table city (I have copied below table creation script) , You want to index column “city” for some performance reason. Now what will be impact if you create index on this table while your transaction is still on ? MySQL blocks your transaction causing performance outage (may not be too long) so how can we create index with no impact on application performance and reliability ? The example below explains the step-by-step process of what I am referring to:

mysql> show create table city\G;

*************************** 1. row ***************************

       Table: city

Create Table: CREATE TABLE `city` (

  `city_id` smallint(5) unsigned NOT NULL AUTO_INCREMENT,

  `city` varchar(50) NOT NULL,

  `country_id` smallint(5) unsigned NOT NULL,

  `last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

  PRIMARY KEY (`city_id`),

  KEY `idx_fk_country_id` (`country_id`),

  CONSTRAINT `fk_city_country` FOREIGN KEY (`country_id`) REFERENCES `country` (`country_id`) ON UPDATE CASCADE

) ENGINE=InnoDB AUTO_INCREMENT=901 DEFAULT CHARSET=utf8

1 row in set (0.00 sec)

Now I am using PT-ONLINE-SCHEMA-CHANGE to create index on column “city” of table “city”:

[root@localhost ~]# pt-online-schema-change D=sakila,t=city,h=localhost,u=root --execute --alter="ADD INDEX city_idx (city)" --alter-foreign-keys-method="auto" --ask-pass

Enter MySQL password: 

No slaves found.  See --recursion-method if host localhost.localdomain has slaves.

Not checking slave lag because no slaves were found and --check-slave-lag was not specified.

Operation, tries, wait:

  analyze_table, 10, 1

  copy_rows, 10, 0.25

  create_triggers, 10, 1

  drop_triggers, 10, 1

  swap_tables, 10, 1

  update_foreign_keys, 10, 1

Child tables:

  `sakila`.`address` (approx. 603 rows)

Will automatically choose the method to update foreign keys.

Altering `sakila`.`city`...

Creating new table...

Created new table sakila._city_new OK.

Altering new table...

Altered `sakila`.`_city_new` OK.

2017-11-29T23:38:36 Creating triggers...

2017-11-29T23:38:36 Created triggers OK.

2017-11-29T23:38:36 Copying approximately 601 rows...

2017-11-29T23:38:36 Copied rows OK.

2017-11-29T23:38:36 Max rows for the rebuild_constraints method: 79360

Determining the method to update foreign keys...

2017-11-29T23:38:36   `sakila`.`address`: 603 rows; can use rebuild_constraints

2017-11-29T23:38:36 Analyzing new table...

2017-11-29T23:38:36 Swapping tables...

2017-11-29T23:38:36 Swapped original and new tables OK.

2017-11-29T23:38:36 Rebuilding foreign key constraints...

2017-11-29T23:38:36 Rebuilt foreign key constraints OK.

2017-11-29T23:38:36 Dropping old table...

2017-11-29T23:38:36 Dropped old table `sakila`.`_city_old` OK.

2017-11-29T23:38:36 Dropping triggers...

2017-11-29T23:38:36 Dropped triggers OK.

Successfully altered `sakila`.`city`.

PT-ONLINE-SCHEMA-CHANGE Limitations

The following are limitations while using PT-ONLINE-SCHEMA-CHANGE :

• The PRIMARY KEY or UNIQUE INDEX needs to be present in the table, This is because the tool creates a DELETE trigger to keep new table updated while the process is running. The exception is when a PRIMARY KEY or UNIQUE INDEX is being created from existing columns as part of the ALTER clause; In such cases it will be using these column(s) for the DELETE trigger
• The RENAME clause cannot be used to rename the table
• The columns cannot be rename by dropping and re-adding with new name. This tool will not copy the original column’s data to the new column
• The tool will fail if you add a column without a default value and make it NOT NULL
• We need to specify _constraint_name to DROP FOREIGN KEY constraint_name, i.e  –alter “DROP FOREIGN KEY _city_id_fk”
• The tool does not use LOCK IN SHARE MODE with MySQL 5.0 because it can cause a slave error which breaks replication

ONLINE DDL

MySQL 5.6 introduced online DDL for InnoDB tables, which means ALTER TABLE statements will no longer block access to database modification while running.  Typically if you want to make changes to your production servers please prefer during off hours, Especially when the server utilization is at its lowest for both CPU and disk. The Online DDL syntax is almost similar to the normal DDL statement with two additional parameters : ALGORITHM and LOCK

ALGORITHM:

• INPLACE: The table change will happen “in-place” (without rebuilding the entire table), Often copying data to a temporary table is not required
• COPY: This is actually traditional method of copying data to a temporary table, rebuilding the table and reconstructing the secondary indexes

LOCK: 

NONE: Read and write operations are allowed during the database altering process

SHARED: Only read operations are allowed during the database altering process

EXCLUSIVE: Neither SELECT nor DML are allowed, The entire  table will be locked for both reading and writing

ONLINE DDL Example

mysql> alter table city 

    -> add index city_idx(city), algorithm=inplace, lock=none;

Query OK, 0 rows affected (0.03 sec)

Records: 0  Duplicates: 0  Warnings: 0

The post Difference between ONLINE DDL and PT-ONLINE-SCHEMA-CHANGE appeared first on The WebScale Database Infrastructure Operations Experts.