Introduction – Redundant and fault-tolerant database systems

Most of our customers are 24*7 internet properties and their applications ( i.e. entire business operations) rely heavily on availability of respective database systems nodes, These systems going unavailable cause serious impact on reputation, customer experience and revenue. To ensure business continuity under foreseeable and unforeseeable man-made or natural disasters, the database system has to be designed and built with fault tolerance and disaster recovery capabilities, in other words the component can be removed and the system should continue to operate as expected. High availability refers to the characteristics of a system that allow the system to sustain continuous operation in the event of hardware and software failures due to natural or man made causes (Webb, 2008).

How MySQL 8.0 Group Replication works ?

• MySQL Group Replication is a plugin (first released with MySQL 5.7 Server) which can provide virtually synchronous replication, with built-in conflict detection / handling and consistency guarantee, So the individual machines and/or MySQL instances can fail or be taken offline for maintenance while the distributed MySQL service continues to operate and handle application traffic.
• MySQL Group Replication is based on a Paxos implementation, which ensures that all servers receive the same set of data in the same order. There is a built-in group membership service that keeps the view of the group consistent and available for all servers at any given point in time. Servers can leave and join the group and the view is updated accordingly. Sometimes servers can leave the group unexpectedly, in which case the failure detection mechanism detects this and notifies the group that the view has changed. This is all automatic.
• In MySQL Group Replication, For a transaction to commit, the majority of the group have to agree on the order of a given transaction in the global sequence of transactions. Deciding to commit or abort a transaction is done by each server individually, but all servers make the same decision. If there is a network partition, resulting in a split where members are unable to reach agreement, then the system does not progress until this issue is resolved. Hence there is also a built-in, automatic, split-brain protection mechanism.

MySQL 8.0 Group Replication Limitations

• Maximum number of MySQL servers that can be the members of a single replication group is 9, Any further attempt made by adding more members will be refused. Functionally this appears very rational to us, Why add several MySQL instances to make the Group Replication infrastructure operationally complex ?
• If you are operating Group Replication in multi-primary mode, The SELECT … FOR UPDATE statements can cause a deadlock, This is because the lock is not shared across the members of the group, therefore the expectation for such a statement might not be reached.
• We cannot use  replication filters on Group Replication. Because, Filtering the transactions  on selected servers would make the group enabled  to reach agreement on consistent state. But, You can implement  replication filters on the replication channels that are not directly involved with  Group Replication

Handling Transaction Limit in MySQL Group Replication

• Configure group_replication_transaction_size_limit system variable to control maximum transaction size the group will tolerate, The default value is approximately 143 MB (150000000 bytes). The time taken to process a transaction is proportional to its size.
• Group Replication use compression algorithm LZ4 and it is enabled by default with threshold of 1000000 bytes (1 MB), You can tune this value by configuring the system variable group_replication_compression_threshold . We have copied below the steps to set a compression threshold:

STOP GROUP_REPLICATION;
SET GLOBAL group_replication_compression_threshold= 5242880;
START GROUP_REPLICATION; 

In the above script, We have set the compression threshold to 5MB. Now if a transaction generates a replication  message  more  than  5MB, i.e., the binary log  transaction entry is larger than 5MB, then it is compressed.

Note: You can disable the compression by setting threshold to “0“

• In Group Replication large messages are automatically fragmented. Because, When an abnormally large message is sent between Group Replication group members, it can result in some group members being reported as failed and expelled from the group. So large messages are automatically split into fragments that are sent separately and reassembled by the recipients. You can configure system variable  group_replication_communication_max_message_size ( default value is 10485760 bytes / 10 MiB ) to specify a message size above which messages are fragmented. This value cannot be larger than the system variable slave_max_allowed_packet , which is 1073741824 bytes (1 GB). You can switch off fragmentation by setting value “0” for group_replication_communication_max_message_size .

Note: In order for a replication group to use fragmentation, all group members must be at MySQL 8.0.16 or above

Conclusion

Group Replication definitely  is an off-the-shelf solution for building highly available, fault tolerant  and self-healing MySQL infrastructure operations. Even if some of the servers subsequently fail, as long it is not all or a majority, the system is still available. Depending on the number of servers which fail the group might have degraded performance or scalability, but it is still available. To conclude, MySQL Group Replication guarantees that the database service is continuously available.  But, When you are using such compelling solution for MySQL Reliability, It’s alway better to also discount limitations and tunable parameter for better performance. Thanks  for taking  time in reading this  blog and we welcome your comments.

References 

https://www.percona.com/blog/2017/08/01/group-replication-sweet-sour/ 

https://mysqlhighavailability.com/mysql-group-replication-a-quick-start-guide/ 

https://dev.mysql.com/doc/refman/8.0/en/group-replication-limitations.html

The post MySQL 8 Group Replication Limitations appeared first on The WebScale Database Infrastructure Operations Experts.

I use MySQL replication extensively, It’s simple and robust, It’s also one main reason I love MySQL lot compared to other database systems. I basically use MySQL replication for performance, scalability, high availability, reliability, failover, fault tolerance etc. I have written blogs in the past on MySQL Master-Slave ( https://minervadb.com/index.php/2018/01/24/mysql-master-slave-replication-on-ubuntu-xenial-16-04/  & https://minervadb.com/index.php/2018/01/24/mysql-master-slave-replication-on-ubuntu-xenial-16-04-2/ ), Master-Master replication (https://minervadb.com/index.php/2018/01/24/mysql-master-slave-replication-on-ubuntu-xenial-16-04-2/) , delayed replication (https://minervadb.com/index.php/2018/01/29/how-to-lag-a-mysql-slave-behind-to-avoid-a-data-corruption/) etc.   MySQL introduced global transaction identifiers (GTIDs) in MySQL 5.6.5 to make failover better and easy, GTID is a global transaction identifier which consists of two entities separated by a colon (:) , GTID looks like this – {source_id:transaction_id} / {c03c4200-0659-11e8-94ab-0800279747b5:1-2},  The source_id is server’s UUID and transaction_id is a sequence number . I have explained below step-by-step GTID based chained replication, One master and two slaves

• Master – 192.168.56.15
• Salve 1 – 192.68.56.16
• Slave 2 – 192.168.56.17

MySQL My.CNF in the master – 192.168.56.15 

[mysqld]

!includedir /etc/mysql/conf.d/

!includedir /etc/mysql/mysql.conf.d/

server-id=900

log-bin=mysql-bin

bind-address=192.168.56.15

binlog-do-db=employees

gtid-mode = on

enforce-gtid-consistency

log-slave-updates

In this blog I have used mysqldump for backup so enabled system variable by setting “read_only” to ON

mysql> set global read_only=on;

Query OK, 0 rows affected (0.00 sec)

root@VBOX110:/home/shiv# mysqldump --all-databases --single-transaction --triggers --routines --events --user=root --password=HardPassword/1947 > backup.sql

Once successfully completed back-up please disable system variable  by setting “read_only” to OFF

mysql> set global read_only=off;

Query OK, 0 rows affected (0.00 sec)

MySQL My.CNF in the slave – 192.168.56.16

[mysqld]

!includedir /etc/mysql/conf.d/

!includedir /etc/mysql/mysql.conf.d/

server-id=910

log-bin=mysql-bin

skip-slave-start

gtid-mode=on

enforce-gtid-consistency

log-slave-updates

MySQL My.CNF in the slave – 192.168.56.17

!includedir /etc/mysql/conf.d/

!includedir /etc/mysql/mysql.conf.d/

server-id=930

log-bin=mysql-bin

skip-slave-start

gtid-mode=on

enforce-gtid-consistency

log-slave-updates

We need to create two pseudo-users in the master ( 192.168.56.15), This user will be used for replicating data between master and two slaves (192.168.56.16 and 192.168.56.17) .

mysql> create user 'repl_usr_16'@'192.168.56.16' identified by 'repl_usr_16' ;

Query OK, 0 rows affected (0.02 sec)

mysql> grant replication slave on *.* to 'repl_usr_16'@'192.168.56.16';

Query OK, 0 rows affected (0.00 sec)

mysql> create user 'repl_usr_17'@'192.168.56.17' identified by 'repl_usr_17' ;

Query OK, 0 rows affected (0.02 sec)

mysql> grant replication slave on *.* to 'repl_usr_17'@'192.168.56.17';

Query OK, 0 rows affected (0.00 sec)

 Restore mysql backup from master (192.168.56.15) on slaves (192.168.56.16 and 192.168.56.17)

root@VBOX125:/home/shiv# mysql -u root -p < backup.sql&nbsp;

Enter password:&nbsp;

Connect slaves (192.168.56.16 and 192.168.56.17) to the master (192.168.56.15)

Run “change master to” command in 192.168.56.16

mysql> change master to 

    -> master_host='192.168.56.15',

    -> master_user='repl_usr',

    -> master_password='repl_usr',

    -> master_auto_position=1; 

Query OK, 0 rows affected, 2 warnings (0.01 sec)

start slave in 192.168.56.16

mysql> start slave;

Query OK, 0 rows affected (0.01 sec)

Run “change master to” command in 192.168.56.17

mysql> change master to 

    -> master_host='192.168.56.15',

    -> master_user='repl_usr',

    -> master_password='repl_usr',

    -> master_auto_position=1; 

Query OK, 0 rows affected, 2 warnings (0.01 sec)

start slave in 192.168.56.17

mysql> start slave;

Query OK, 0 rows affected (0.01 sec)

Confirm replication is successful from both slaves (192.168.56.16 and 192.168.56.17)

mysql> show slave status\G; 

*************************** 1. row ***************************

               Slave_IO_State: Waiting for master to send event

                  Master_Host: 192.168.56.15

                  Master_User: repl_usr

                  Master_Port: 3306

                Connect_Retry: 60

              Master_Log_File: mysql-bin.000002

          Read_Master_Log_Pos: 529

               Relay_Log_File: VBOX-112-relay-bin.000002

                Relay_Log_Pos: 742

        Relay_Master_Log_File: mysql-bin.000002

             Slave_IO_Running: Yes

            Slave_SQL_Running: Yes

              Replicate_Do_DB: 

          Replicate_Ignore_DB: 

           Replicate_Do_Table: 

       Replicate_Ignore_Table: 

      Replicate_Wild_Do_Table: 

  Replicate_Wild_Ignore_Table: 

                   Last_Errno: 0

                   Last_Error: 

                 Skip_Counter: 0

          Exec_Master_Log_Pos: 529

              Relay_Log_Space: 952

              Until_Condition: None

               Until_Log_File: 

                Until_Log_Pos: 0

           Master_SSL_Allowed: No

           Master_SSL_CA_File: 

           Master_SSL_CA_Path: 

              Master_SSL_Cert: 

            Master_SSL_Cipher: 

               Master_SSL_Key: 

        Seconds_Behind_Master: 0

Master_SSL_Verify_Server_Cert: No

                Last_IO_Errno: 0

                Last_IO_Error: 

               Last_SQL_Errno: 0

               Last_SQL_Error: 

  Replicate_Ignore_Server_Ids: 

             Master_Server_Id: 900

                  Master_UUID: c03c4200-0659-11e8-94ab-0800279747b5

             Master_Info_File: /var/lib/mysql/master.info

                    SQL_Delay: 0

          SQL_Remaining_Delay: NULL

      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates

           Master_Retry_Count: 86400

                  Master_Bind: 

      Last_IO_Error_Timestamp: 

     Last_SQL_Error_Timestamp: 

               Master_SSL_Crl: 

           Master_SSL_Crlpath: 

           Retrieved_Gtid_Set: c03c4200-0659-11e8-94ab-0800279747b5:1-2

            Executed_Gtid_Set: c03c4200-0659-11e8-94ab-0800279747b5:1-2

                Auto_Position: 1

         Replicate_Rewrite_DB: 

                 Channel_Name: 

           Master_TLS_Version: 

1 row in set (0.00 sec)

ERROR: 

No query specified

mysql> show slave status \G

*************************** 1. row ***************************

               Slave_IO_State: Waiting for master to send event

                  Master_Host: 192.168.56.15

                  Master_User: repl_usr_17

                  Master_Port: 3306

                Connect_Retry: 60

              Master_Log_File: mysql-bin.000002

          Read_Master_Log_Pos: 1008

               Relay_Log_File: VBOX125-relay-bin.000002

                Relay_Log_Pos: 893

        Relay_Master_Log_File: mysql-bin.000002

             Slave_IO_Running: Yes

            Slave_SQL_Running: Yes

              Replicate_Do_DB: 

          Replicate_Ignore_DB: 

           Replicate_Do_Table: 

       Replicate_Ignore_Table: 

      Replicate_Wild_Do_Table: 

  Replicate_Wild_Ignore_Table: 

                   Last_Errno: 0

                   Last_Error: 

                 Skip_Counter: 0

          Exec_Master_Log_Pos: 1008

              Relay_Log_Space: 1102

              Until_Condition: None

               Until_Log_File: 

                Until_Log_Pos: 0

           Master_SSL_Allowed: No

           Master_SSL_CA_File: 

           Master_SSL_CA_Path: 

              Master_SSL_Cert: 

            Master_SSL_Cipher: 

               Master_SSL_Key: 

        Seconds_Behind_Master: 0

Master_SSL_Verify_Server_Cert: No

                Last_IO_Errno: 0

                Last_IO_Error: 

               Last_SQL_Errno: 0

               Last_SQL_Error: 

  Replicate_Ignore_Server_Ids: 

             Master_Server_Id: 900

                  Master_UUID: c03c4200-0659-11e8-94ab-0800279747b5

             Master_Info_File: /var/lib/mysql/master.info

                    SQL_Delay: 0

          SQL_Remaining_Delay: NULL

      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates

           Master_Retry_Count: 86400

                  Master_Bind: 

      Last_IO_Error_Timestamp: 

     Last_SQL_Error_Timestamp: 

               Master_SSL_Crl: 

           Master_SSL_Crlpath: 

           Retrieved_Gtid_Set: c03c4200-0659-11e8-94ab-0800279747b5:3-4

            Executed_Gtid_Set: c03c4200-0659-11e8-94ab-0800279747b5:1-4

                Auto_Position: 1

         Replicate_Rewrite_DB: 

                 Channel_Name: 

           Master_TLS_Version: 

1 row in set (0.00 sec)

The post MySQL Replication simplified with GTID – Step-by-step GTID replication setup appeared first on The WebScale Database Infrastructure Operations Experts.

Introduction

One major reason I really appreciate MySQL is it’s replication simplicity, I agree it’s not perfect but quite straightforward. There is only one way to do it so very minimal complexities .  All our customers are either an internet / mobile property or Internet of Things services provider so scaling their MySQL infrastructure for performance, availability and reliability is very important. MySQL replication is an quick and easy solution for performance, scalability and availability. I have written blogs on MySQL master – slave replication : https://minervadb.com/index.php/2018/01/24/step-by-step-mysql-master-slave-replication-on-centos/  (CentOS) and https://minervadb.com/index.php/2018/01/24/mysql-master-slave-replication-on-ubuntu-xenial-16-04/  (Ubuntu) . MySQL Master-Slave replication really works great if you are just looking for scaling reads or splitting read-write but what about availability in a simple Master-Slave replication ? You have to manually graduate / promote slave to master and there is downtime expected here so if you are looking for maximum availability of MySQL infrastructure operations MySQL Master-Master replication is recommended. I am not going to explain here about MySQL installation,  I will be starting with you my.cnf configuration in both master instances, Including the next steps for an successful Master-Master MySQL replication

Master 1 – 192.168.56.13

Master 2 – 192.168.56.23 

Configure my.cnf of Master 1 – 192.168.56.13 and Master 2 – 192.168.56.23 

my.cnf of MySQL instance 192.168.56.13

root@DA:/home/shiv/sakila# vi /etc/mysql/my.cnf

[mysqld]

server-id = 900

log-bin = mysql-bin

bind-address = 192.168.56.13

binlog-do-db = sakila

my.cnf of MySQL instance 192.168.56.23

root@DBA:/home/shiv/sakila# vi /etc/mysql/my.cnf

[mysqld]

server-id=901

log-bin = mysql-bin

bind-address = 192.168.56.23

binlog-do-db = sakila

Restart MySQL on both instances (192.168.56.13 and 192.168.56.23)

We need to create a pseudo-user that will be used for replicating data between our two VPS (192.168.56.13 and 192.168.56.23). The examples in this article will assume that you name this user “repl” and password used here is “repl” .

User created in 192.168.56.13 

mysql> create user 'repl'@'192.168.56.23' identified by 'repl' ;

Query OK, 0 rows affected (0.01 sec)

mysql> grant replication slave on *.* to 'repl'@'192.168.56.23';

Query OK, 0 rows affected (0.01 sec)

mysql> flush privileges; 

Query OK, 0 rows affected (0.00 sec)

User created in 192.168.56.23

mysql> create user 'repl'@'192.168.56.13' identified by 'repl';

Query OK, 0 rows affected (0.01 sec)

mysql> grant replication slave on *.* to 'repl'@'192.168.56.13';

Query OK, 0 rows affected (0.01 sec)

mysql> flush privileges; 

Query OK, 0 rows affected (0.01 sec)

The last thing we have to do before we complete the mysql master-master replication is to make note of the master log file and position. We need this data to hook both MySQL instances on each other, We are doing an Master-Master replication so we need details of both the MySQL instances, 192.168.56.13 and 192.168.56.23

Master log file and position of 192.168.56.13 

mysql> show master status;

+------------------+----------+--------------+------------------+-------------------+

| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |

+------------------+----------+--------------+------------------+-------------------+

| mysql-bin.000002 |      154 | sakila       |                  |                   |

+------------------+----------+--------------+------------------+-------------------+

1 row in set (0.00 sec)

Master log file and position of 192.168.56.23

mysql> show master status;

+------------------+----------+--------------+------------------+-------------------+

| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |

+------------------+----------+--------------+------------------+-------------------+

| mysql-bin.000001 |      771 | sakila       |                  |                   |

+------------------+----------+--------------+------------------+-------------------+

1 row in set (0.00 sec)

Completing MySQL Master-Master replication 

The following steps will start replicating data between both MySQL instances (192.168.56.13 and 192.168.56.23)

Run this command on MySQL instance in 192.168.56.13 (confirm replication is successful and no error is reported with command “show slave status\G” command)

mysql> stop slave;

mysql> change master to 

    -> MASTER_HOST='192.168.56.23',

    -> MASTER_USER='repl', 

    -> MASTER_PASSWORD='repl',

    -> MASTER_LOG_FILE='mysql-bin.000001',

    -> MASTER_LOG_POS=  771;

Query OK, 0 rows affected, 2 warnings (0.04 sec)

mysql> start slave;

Query OK, 0 rows affected (0.02 sec)
 
mysql> show slave status\G;

*************************** 1. row ***************************

               Slave_IO_State: Waiting for master to send event

                  Master_Host: 192.168.56.23

                  Master_User: repl

                  Master_Port: 3306

                Connect_Retry: 60

              Master_Log_File: mysql-bin.000001

          Read_Master_Log_Pos: 771

               Relay_Log_File: DA-relay-bin.000002

                Relay_Log_Pos: 320

        Relay_Master_Log_File: mysql-bin.000001

             Slave_IO_Running: Yes

            Slave_SQL_Running: Yes

              Replicate_Do_DB: 

          Replicate_Ignore_DB: 

           Replicate_Do_Table: 

       Replicate_Ignore_Table: 

      Replicate_Wild_Do_Table: 

  Replicate_Wild_Ignore_Table: 

                   Last_Errno: 0

                   Last_Error: 

                 Skip_Counter: 0

          Exec_Master_Log_Pos: 771

              Relay_Log_Space: 524

              Until_Condition: None

               Until_Log_File: 

                Until_Log_Pos: 0

           Master_SSL_Allowed: No

           Master_SSL_CA_File: 

           Master_SSL_CA_Path: 

              Master_SSL_Cert: 

            Master_SSL_Cipher: 

               Master_SSL_Key: 

        Seconds_Behind_Master: 0

Master_SSL_Verify_Server_Cert: No

                Last_IO_Errno: 0

                Last_IO_Error: 

               Last_SQL_Errno: 0

               Last_SQL_Error: 

  Replicate_Ignore_Server_Ids: 

             Master_Server_Id: 901

                  Master_UUID: cd1eedbe-fed3-11e7-a414-08002720a95a

             Master_Info_File: /var/lib/mysql/master.info

                    SQL_Delay: 0

          SQL_Remaining_Delay: NULL

      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates

           Master_Retry_Count: 86400

                  Master_Bind: 

      Last_IO_Error_Timestamp: 

     Last_SQL_Error_Timestamp: 

               Master_SSL_Crl: 

           Master_SSL_Crlpath: 

           Retrieved_Gtid_Set: 

            Executed_Gtid_Set: 

                Auto_Position: 0

         Replicate_Rewrite_DB: 

                 Channel_Name: 

           Master_TLS_Version: 

1 row in set (0.00 sec)

ERROR: 

No query specified

Run this command on MySQL instance in 192.168.56.23 (confirm replication is successful and no error is reported with command “show slave status\G” command)

mysql> stop slave;

Query OK, 0 rows affected (0.01 sec)

mysql> change master to 

    ->  MASTER_HOST='192.168.56.13', 

    -> MASTER_USER='repl',

    -> MASTER_PASSWORD='repl',

    ->  MASTER_LOG_FILE='mysql-bin.000002',

    -> MASTER_LOG_POS=  154;

Query OK, 0 rows affected, 2 warnings (0.01 sec)

mysql> start slave;

Query OK, 0 rows affected (0.01 sec)

mysql> show slave status\G;

*************************** 1. row ***************************

               Slave_IO_State: Waiting for master to send event

                  Master_Host: 192.168.56.13

                  Master_User: repl

                  Master_Port: 3306

                Connect_Retry: 60

              Master_Log_File: mysql-bin.000002

          Read_Master_Log_Pos: 154

               Relay_Log_File: DBA-relay-bin.000002

                Relay_Log_Pos: 320

        Relay_Master_Log_File: mysql-bin.000002

             Slave_IO_Running: Yes

            Slave_SQL_Running: Yes

              Replicate_Do_DB: 

          Replicate_Ignore_DB: 

           Replicate_Do_Table: 

       Replicate_Ignore_Table: 

      Replicate_Wild_Do_Table: 

  Replicate_Wild_Ignore_Table: 

                   Last_Errno: 0

                   Last_Error: 

                 Skip_Counter: 0

          Exec_Master_Log_Pos: 154

              Relay_Log_Space: 525

              Until_Condition: None

               Until_Log_File: 

                Until_Log_Pos: 0

           Master_SSL_Allowed: No

           Master_SSL_CA_File: 

           Master_SSL_CA_Path: 

              Master_SSL_Cert: 

            Master_SSL_Cipher: 

               Master_SSL_Key: 

        Seconds_Behind_Master: 0

Master_SSL_Verify_Server_Cert: No

                Last_IO_Errno: 0

                Last_IO_Error: 

               Last_SQL_Errno: 0

               Last_SQL_Error: 

  Replicate_Ignore_Server_Ids: 

             Master_Server_Id: 900

                  Master_UUID: a1546678-febf-11e7-9618-080027275ccc

             Master_Info_File: /var/lib/mysql/master.info

                    SQL_Delay: 0

          SQL_Remaining_Delay: NULL

      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates

           Master_Retry_Count: 86400

                  Master_Bind: 

      Last_IO_Error_Timestamp: 

     Last_SQL_Error_Timestamp: 

               Master_SSL_Crl: 

           Master_SSL_Crlpath: 

           Retrieved_Gtid_Set: 

            Executed_Gtid_Set: 

                Auto_Position: 0

         Replicate_Rewrite_DB: 

                 Channel_Name: 

           Master_TLS_Version: 

1 row in set (0.00 sec)

ERROR: 

No query specified

Test MySQL Master-Master replication

create this table in 192.168.56.13  MySQL instance

mysql> create table tab1 (

    -> col1   integer);

Query OK, 0 rows affected (0.02 sec)

confirm this table replicated to the MySQL instance in 192.168.56.23 successfully

mysql> show tables like 'tab1%';

+--------------------------+

| Tables_in_sakila (tab1%) |

+--------------------------+

| tab1                     |

+--------------------------+

1 row in set (0.00 sec)

create this table in 192.168.56.23 MySQL instance

mysql> create table tab11

    -> ( col11 integer);

Query OK, 0 rows affected (0.02 sec)

confirm this table replicated to the MySQL instance in 192.168.56.13 successfully

mysql> show tables like 'tab11%';

+---------------------------+

| Tables_in_sakila (tab11%) |

+---------------------------+

| tab11                     |

+---------------------------+

1 row in set (0.00 sec)

Conclusion

The successful completion of above tests confirms you have MySQL Master-Master replication in place !

The post MySQL Master – Master Replication on Ubuntu (xenial – 16.04 ) appeared first on The WebScale Database Infrastructure Operations Experts.

Introduction

MySQL master-slave replication is quite simple and direct way to scale-out reads optimally. Retaining multiple copies of master across slaves also guarantee reliability and maximum availability, Replication is never an substitute or alternative for database backup and disaster recovery. In this post I am explaining simple step-by-step MySQL master-salve replication.

This blog post will be using following IP addresses:

MySQL Master – 192.168.56.11

MySQL Slave – 192.168.56.12

Configure MySQL master (192.168.56.11)

To configure a master to use binary log file based replication, Configure binary logging first and include a unique server ID. Once done these steps restart MySQL server. Binary logging is enabled by default (log_bin variable is set to ON). It’s always a best practice to create binary log files with a non-default base name to avoid confusion as your database infrastructure grows. To configure binary logging, first shutdown the MySQL and edit my.cnf or my.ini

[mysqld]
server-id=1
bind-address=192.168.56.11
log-bin=mysql-bin
binlog-do-db=sakila

To guarantee the ultimate transaction durability and consistency in the replication setup please confirm innodb_flush_log_at_trx_commit=1 and sync_binlog=1 in the master my.cnf file. You must also confirm skip-networking option is not enabled on your replication master.

Configure MySQL slave (192.168.56.12)

MySQL slave must have a unique server ID, You must restart MySQL server after setting (this must be done in [mysqld] section of the configuration file) this in my.cnf . A slave is not required to have binary logging enabled to setup an simple MySQL master-slave  replication but if you are building an relay-slave MySQL replication where server A serves as the master for slave B, and B serves as the master for slave C then slave B binary logging must be enabled. In addition to binary logging, this kind of replication topology requires the –log-slave-updates option to be enabled. You can disable binary logging on slave by configuring –skip-log-bin and –skip-log-slave-updates options in my.cnf

[mysqld]
server-id=2

Creating user for MySQL replication

The slave connect to master using MySQL username and password, There must a user account on the master that slave can use to connect. This user account created in the master must be granted with REPLICATION SLAVE privilege. You may create different account for each slave or connect to master using the same account for each slave. Though it’s not compulsory to create user account specifically for replication, You must be aware that the replication user name and password is stored in plain text in the master info data dictionary table mysql.slave_master_info so from an MySQL security perspective it is recommended to have a dedicated user account for successful replication.

Please create the user below in MySQL master – 192.168.56.11

mysql> CREATE USER 'repl'@'192.168.56.12' IDENTIFIED BY 'MySQLDBA19/47';
mysql> GRANT REPLICATION SLAVE ON *.* TO 'repl'@'192.168.56.12';

Collecting MySQL replication Master Binary Log Coordinates

MySQL slave must be configured to start replication from the correct point, To do this task successfully you must note down master’s current coordinates with its binary log info. To collect MYSQL master binary log coordinates please FLUSH TABLES WITH READ LOCK, which blocks COMMIT operations for InnoDB tables (data consistency in replication is very important )Please run the below SQL in MySQL master – 192.168.56.11

mysql> FLUSH TABLES WITH READ LOCK;

Now from different MySQL session on the master run SHOW MASTER STATUS statement to collect current binary log file name and it’s position

mysql > SHOW MASTER STATUS;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000147 | 450      | sakila       | manual,mysql     |
+------------------+----------+--------------+------------------+

This is the position (Position 450) from which the slave database will start replication, Take a note the number as you need them while setting up slave for MySQL replication. Now you are ready to proceed with exporting your database using mysqldump from a new window  ( please confirm you are not typing this command from MySQL, It’s done from bash shell )

mysqldump -u root -p --opt sakila > sakila-backup.sql

Now unlock tables in the master

UNLOCK TABLES;

Restore the mysqldump backup to slave (192.168.56.12)

mysql -u root -p sakila < sakila-backup.sql

Connecting slave to master

Run the script below from MySQL terminal in the slave

CHANGE MASTER TO 
MASTER_HOST='192.168.56.11', 
MASTER_USER='repl', 
MASTER_PASSWORD='MySQLDBA19/47', 
MASTER_LOG_FILE='mysql-bin.000147', 
MASTER_LOG_POS=  450;

Start MySQL slave

START SLAVE;

To confirm the successful slave replication please run below command on slave terminal

SHOW SLAVE STATUS\G;

Congrats, All done for an successful MySQL master-slave replication on CentOS !

The post Step-by-step MySQL Master – Slave Replication on CentOS appeared first on The WebScale Database Infrastructure Operations Experts.

I spend most of my professional life consulting for Internet/Mobility/SaaS companies across Southeast Asia, MiddleEast and US as MySQL Architect, I am accountable for MySQL Infrastructure Operations, Site Reliability, Performance, Scalability and High Availability of my customer MySQL Infrastructure. My customers are from diversified industries like Mobile Advertisement Networks, Online Commerce, Mobile Social Media Gaming and SaaS, The DATA is business for them and Database Infrastructure Outage is the worst thing which could ever happen so “Maximum MySQL Availability” is something I am very serious about !!! Being an MySQL consultant I get to meet new customers almost every week and most of them ask me a very common question “How we can be Optimal, Scalable an Highly Available ? ” , I will ask them back “what can be maximum duration (in hours) of MySQL outage you could ever afford? ” Many of them will reply “We have  to be available 24*7*365”, WOW, Great to hear but in real life it’s IMPOSSIBLE !!! Being an independent MySQL consultant I am very serious about my professional reputation and I always feel comfortable in sharing good, bad and ugly side of MySQL infrastructure operations! There are some customers who appreciate being upfront and some who consider me pessimistic but eventually I will be signing common MySQL Infrastructure Operations Management SLA for all of them mentioning Possible MySQL Outage Scenarios , Time-To-Recover , Recovery & Healing, HealthCheck/Diagnostics/Forensics, Performance, Scalability and Emergency Response / On-Call . In this post I am addressing Cost-of-Data-Outage.    

Cost-of-Data-Outage is expensive for every corporation from all directions like revenue, brand reputation and eventually customer experience. Think about travel booking site going offline during holiday season or Ad-Network going down during shopping festival, The damage is not just limited to web/mobile property but even to strategic partners involved in it like hotels signed-up with travel site or E-commerce platform partnered with Ad-Network so everyone consider “Data Outage” serious but can you ever promise anyone 100% availability of your data infrastructure, The honest answer will be NO!!! This doesn’t mean we are ignorant about “Data Outage” but to architect & implement “Maximum Availability Architecture” you need to know Cost-of-Data-Outage. Building Self Healing / Auto Failover, Fault Tolerant, Highly Available and Responsive MySQL Infrastructure Operations is expensive ($$$), Resource Intensive and Operationally Complex so plan carefully for Budget, People, Roadmap and Housekeeping/Operations while concluding MySQL Maximum Availability Strategy.

What are possible Data Outage scenarios in MySQL Infrastructure?

The following are few possible (most common) Data Outage Scenarios in a MySQL Infrastructure:

1. Human error: The most common and even more dangerous because there is no limit (intentional or not) for an individual/group to damage your data.
2. Business: Often the potential internet marketing and consumer psychology are mistaken/underestimated. The poor capacity planning & sizing can bring down your MySQL infrastructure in no time !!!
3. Natural Catastrophe: This could be just anything Flood, Earthquake, Tsunami, Fire etc. Man still get defeated often by mother nature unfortunately !!!
4. Planned Outage: The advancement in Hardware is so compelling (PCIe SSDs are much faster than HDDs) so upgrades are unavoidable

What I consider while planning for MySQL Maximum Availability ?

I never standardise MySQL Infrastructure Operations (Performance, Scalability High Availability and 24*7 MySQL DBA Services ) for any of my customers!! They are completely different from each other for me (though all of them use either MySQL GA / Percona Server / MariaDB / WebScaleSQL). The following are few questions which I will have the answer before engaging personally to MySQL Infrastructure Operations of my customers:

1. MySQL flavour (MySQL GA / Percona Server / MariaDB / WebScaleSQL)
2. The current MySQL DBA Operations, SLA, Emergency Support and On-Call DBA
3. The current MySQL data growth rate / day
4. MySQL Infrastructure Operations Trending Report (generally two weeks of history data will be sufficient in most of the cases )
5. MySQL Health Check, Diagnostics & Forensics Report
6. Immediate goals & accomplishments for MySQL Infrastructure Operations
7. MySQL Infrastructure Operations Roadmap

How typical MySQL Maximum Availability Project Plan will look like ?

*** This is only the to show my readers how typical Maximum MySQL Availability Project Plan will look like !!!

1. 24*7 Service Operations (Monitoring & Trending MySQL ), MySQL Infrastructure Operations Workflow (Emergency Support & On-Call) and MySQL Operations SLA (escalation and maximum accepted duration in closing tickets successfully)
2. Planned MySQL HealthCheck, Diagnostics and Forensics Audit to be proactive !!
3. Planned MySQL Tuning Schedule
4. MySQL Backup Strategy, Execution and Sanity Testing of MySQL Backups
5. MySQL High Availability Architecture, Implementation and Operations
6. Planned MySQL Outage (hardware upgrades)
7. MySQL Upgrade Plan

Now what is really the Cost-of-Data-Outage ? 

The answer for this question is not just limited to a number but much beyond  that, It is about Customer Experience, Employee Satisfaction (people hate to work for frequently going down websites), Investor Faith, Partner Loyalty, Brand Value and last but not the least REVENUE!!!

What should I do to achieve Maximum MySQL Availability ?

1. Hire / Contract professional MySQL DBA or MySQL Consulting company ?
2. Invest for your in-house MySQL DBA training / research , MySQL is growing faster so staying updated is key to gain maximum benefits
3. Sponser your MySQL DBAs for Conferences, They enjoy learning and networking with peers and industry experts

The post Cost of Data Outage appeared first on The WebScale Database Infrastructure Operations Experts.