I use MySQL replication extensively, It’s simple and robust, It’s also one main reason I love MySQL lot compared to other database systems. I basically use MySQL replication for performance, scalability, high availability, reliability, failover, fault tolerance etc. I have written blogs in the past on MySQL Master-Slave ( https://minervadb.com/index.php/2018/01/24/mysql-master-slave-replication-on-ubuntu-xenial-16-04/  & https://minervadb.com/index.php/2018/01/24/mysql-master-slave-replication-on-ubuntu-xenial-16-04-2/ ), Master-Master replication (https://minervadb.com/index.php/2018/01/24/mysql-master-slave-replication-on-ubuntu-xenial-16-04-2/) , delayed replication (https://minervadb.com/index.php/2018/01/29/how-to-lag-a-mysql-slave-behind-to-avoid-a-data-corruption/) etc.   MySQL introduced global transaction identifiers (GTIDs) in MySQL 5.6.5 to make failover better and easy, GTID is a global transaction identifier which consists of two entities separated by a colon (:) , GTID looks like this – {source_id:transaction_id} / {c03c4200-0659-11e8-94ab-0800279747b5:1-2},  The source_id is server’s UUID and transaction_id is a sequence number . I have explained below step-by-step GTID based chained replication, One master and two slaves

• Master – 192.168.56.15
• Salve 1 – 192.68.56.16
• Slave 2 – 192.168.56.17

MySQL My.CNF in the master – 192.168.56.15 

[mysqld]

!includedir /etc/mysql/conf.d/

!includedir /etc/mysql/mysql.conf.d/

server-id=900

log-bin=mysql-bin

bind-address=192.168.56.15

binlog-do-db=employees

gtid-mode = on

enforce-gtid-consistency

log-slave-updates

In this blog I have used mysqldump for backup so enabled system variable by setting “read_only” to ON

mysql> set global read_only=on;

Query OK, 0 rows affected (0.00 sec)

root@VBOX110:/home/shiv# mysqldump --all-databases --single-transaction --triggers --routines --events --user=root --password=HardPassword/1947 > backup.sql

Once successfully completed back-up please disable system variable  by setting “read_only” to OFF

mysql> set global read_only=off;

Query OK, 0 rows affected (0.00 sec)

MySQL My.CNF in the slave – 192.168.56.16

[mysqld]

!includedir /etc/mysql/conf.d/

!includedir /etc/mysql/mysql.conf.d/

server-id=910

log-bin=mysql-bin

skip-slave-start

gtid-mode=on

enforce-gtid-consistency

log-slave-updates

MySQL My.CNF in the slave – 192.168.56.17

!includedir /etc/mysql/conf.d/

!includedir /etc/mysql/mysql.conf.d/

server-id=930

log-bin=mysql-bin

skip-slave-start

gtid-mode=on

enforce-gtid-consistency

log-slave-updates

We need to create two pseudo-users in the master ( 192.168.56.15), This user will be used for replicating data between master and two slaves (192.168.56.16 and 192.168.56.17) .

mysql> create user 'repl_usr_16'@'192.168.56.16' identified by 'repl_usr_16' ;

Query OK, 0 rows affected (0.02 sec)

mysql> grant replication slave on *.* to 'repl_usr_16'@'192.168.56.16';

Query OK, 0 rows affected (0.00 sec)

mysql> create user 'repl_usr_17'@'192.168.56.17' identified by 'repl_usr_17' ;

Query OK, 0 rows affected (0.02 sec)

mysql> grant replication slave on *.* to 'repl_usr_17'@'192.168.56.17';

Query OK, 0 rows affected (0.00 sec)

 Restore mysql backup from master (192.168.56.15) on slaves (192.168.56.16 and 192.168.56.17)

root@VBOX125:/home/shiv# mysql -u root -p < backup.sql&nbsp;

Enter password:&nbsp;

Connect slaves (192.168.56.16 and 192.168.56.17) to the master (192.168.56.15)

Run “change master to” command in 192.168.56.16

mysql> change master to 

    -> master_host='192.168.56.15',

    -> master_user='repl_usr',

    -> master_password='repl_usr',

    -> master_auto_position=1; 

Query OK, 0 rows affected, 2 warnings (0.01 sec)

start slave in 192.168.56.16

mysql> start slave;

Query OK, 0 rows affected (0.01 sec)

Run “change master to” command in 192.168.56.17

mysql> change master to 

    -> master_host='192.168.56.15',

    -> master_user='repl_usr',

    -> master_password='repl_usr',

    -> master_auto_position=1; 

Query OK, 0 rows affected, 2 warnings (0.01 sec)

start slave in 192.168.56.17

mysql> start slave;

Query OK, 0 rows affected (0.01 sec)

Confirm replication is successful from both slaves (192.168.56.16 and 192.168.56.17)

mysql> show slave status\G; 

*************************** 1. row ***************************

               Slave_IO_State: Waiting for master to send event

                  Master_Host: 192.168.56.15

                  Master_User: repl_usr

                  Master_Port: 3306

                Connect_Retry: 60

              Master_Log_File: mysql-bin.000002

          Read_Master_Log_Pos: 529

               Relay_Log_File: VBOX-112-relay-bin.000002

                Relay_Log_Pos: 742

        Relay_Master_Log_File: mysql-bin.000002

             Slave_IO_Running: Yes

            Slave_SQL_Running: Yes

              Replicate_Do_DB: 

          Replicate_Ignore_DB: 

           Replicate_Do_Table: 

       Replicate_Ignore_Table: 

      Replicate_Wild_Do_Table: 

  Replicate_Wild_Ignore_Table: 

                   Last_Errno: 0

                   Last_Error: 

                 Skip_Counter: 0

          Exec_Master_Log_Pos: 529

              Relay_Log_Space: 952

              Until_Condition: None

               Until_Log_File: 

                Until_Log_Pos: 0

           Master_SSL_Allowed: No

           Master_SSL_CA_File: 

           Master_SSL_CA_Path: 

              Master_SSL_Cert: 

            Master_SSL_Cipher: 

               Master_SSL_Key: 

        Seconds_Behind_Master: 0

Master_SSL_Verify_Server_Cert: No

                Last_IO_Errno: 0

                Last_IO_Error: 

               Last_SQL_Errno: 0

               Last_SQL_Error: 

  Replicate_Ignore_Server_Ids: 

             Master_Server_Id: 900

                  Master_UUID: c03c4200-0659-11e8-94ab-0800279747b5

             Master_Info_File: /var/lib/mysql/master.info

                    SQL_Delay: 0

          SQL_Remaining_Delay: NULL

      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates

           Master_Retry_Count: 86400

                  Master_Bind: 

      Last_IO_Error_Timestamp: 

     Last_SQL_Error_Timestamp: 

               Master_SSL_Crl: 

           Master_SSL_Crlpath: 

           Retrieved_Gtid_Set: c03c4200-0659-11e8-94ab-0800279747b5:1-2

            Executed_Gtid_Set: c03c4200-0659-11e8-94ab-0800279747b5:1-2

                Auto_Position: 1

         Replicate_Rewrite_DB: 

                 Channel_Name: 

           Master_TLS_Version: 

1 row in set (0.00 sec)

ERROR: 

No query specified

mysql> show slave status \G

*************************** 1. row ***************************

               Slave_IO_State: Waiting for master to send event

                  Master_Host: 192.168.56.15

                  Master_User: repl_usr_17

                  Master_Port: 3306

                Connect_Retry: 60

              Master_Log_File: mysql-bin.000002

          Read_Master_Log_Pos: 1008

               Relay_Log_File: VBOX125-relay-bin.000002

                Relay_Log_Pos: 893

        Relay_Master_Log_File: mysql-bin.000002

             Slave_IO_Running: Yes

            Slave_SQL_Running: Yes

              Replicate_Do_DB: 

          Replicate_Ignore_DB: 

           Replicate_Do_Table: 

       Replicate_Ignore_Table: 

      Replicate_Wild_Do_Table: 

  Replicate_Wild_Ignore_Table: 

                   Last_Errno: 0

                   Last_Error: 

                 Skip_Counter: 0

          Exec_Master_Log_Pos: 1008

              Relay_Log_Space: 1102

              Until_Condition: None

               Until_Log_File: 

                Until_Log_Pos: 0

           Master_SSL_Allowed: No

           Master_SSL_CA_File: 

           Master_SSL_CA_Path: 

              Master_SSL_Cert: 

            Master_SSL_Cipher: 

               Master_SSL_Key: 

        Seconds_Behind_Master: 0

Master_SSL_Verify_Server_Cert: No

                Last_IO_Errno: 0

                Last_IO_Error: 

               Last_SQL_Errno: 0

               Last_SQL_Error: 

  Replicate_Ignore_Server_Ids: 

             Master_Server_Id: 900

                  Master_UUID: c03c4200-0659-11e8-94ab-0800279747b5

             Master_Info_File: /var/lib/mysql/master.info

                    SQL_Delay: 0

          SQL_Remaining_Delay: NULL

      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates

           Master_Retry_Count: 86400

                  Master_Bind: 

      Last_IO_Error_Timestamp: 

     Last_SQL_Error_Timestamp: 

               Master_SSL_Crl: 

           Master_SSL_Crlpath: 

           Retrieved_Gtid_Set: c03c4200-0659-11e8-94ab-0800279747b5:3-4

            Executed_Gtid_Set: c03c4200-0659-11e8-94ab-0800279747b5:1-4

                Auto_Position: 1

         Replicate_Rewrite_DB: 

                 Channel_Name: 

           Master_TLS_Version: 

1 row in set (0.00 sec)

The post MySQL Replication simplified with GTID – Step-by-step GTID replication setup appeared first on The WebScale Database Infrastructure Operations Experts.

Introduction

MySQL master-slave replication is quite simple and direct way to scale-out reads optimally. Retaining multiple copies of master across slaves also guarantee reliability and maximum availability, Replication is never an substitute or alternative for database backup and disaster recovery. In this post I am explaining simple step-by-step MySQL master-salve replication.

This blog post will be using following IP addresses:

MySQL Master – 192.168.56.11

MySQL Slave – 192.168.56.12

Configure MySQL master (192.168.56.11)

To configure a master to use binary log file based replication, Configure binary logging first and include a unique server ID. Once done these steps restart MySQL server. Binary logging is enabled by default (log_bin variable is set to ON). It’s always a best practice to create binary log files with a non-default base name to avoid confusion as your database infrastructure grows. To configure binary logging, first shutdown the MySQL and edit my.cnf or my.ini

[mysqld]
server-id=1
bind-address=192.168.56.11
log-bin=mysql-bin
binlog-do-db=sakila

To guarantee the ultimate transaction durability and consistency in the replication setup please confirm innodb_flush_log_at_trx_commit=1 and sync_binlog=1 in the master my.cnf file. You must also confirm skip-networking option is not enabled on your replication master.

Configure MySQL slave (192.168.56.12)

MySQL slave must have a unique server ID, You must restart MySQL server after setting (this must be done in [mysqld] section of the configuration file) this in my.cnf . A slave is not required to have binary logging enabled to setup an simple MySQL master-slave  replication but if you are building an relay-slave MySQL replication where server A serves as the master for slave B, and B serves as the master for slave C then slave B binary logging must be enabled. In addition to binary logging, this kind of replication topology requires the –log-slave-updates option to be enabled. You can disable binary logging on slave by configuring –skip-log-bin and –skip-log-slave-updates options in my.cnf

[mysqld]
server-id=2

Creating user for MySQL replication

The slave connect to master using MySQL username and password, There must a user account on the master that slave can use to connect. This user account created in the master must be granted with REPLICATION SLAVE privilege. You may create different account for each slave or connect to master using the same account for each slave. Though it’s not compulsory to create user account specifically for replication, You must be aware that the replication user name and password is stored in plain text in the master info data dictionary table mysql.slave_master_info so from an MySQL security perspective it is recommended to have a dedicated user account for successful replication.

Please create the user below in MySQL master – 192.168.56.11

mysql> CREATE USER 'repl'@'192.168.56.12' IDENTIFIED BY 'MySQLDBA19/47';
mysql> GRANT REPLICATION SLAVE ON *.* TO 'repl'@'192.168.56.12';

Collecting MySQL replication Master Binary Log Coordinates

MySQL slave must be configured to start replication from the correct point, To do this task successfully you must note down master’s current coordinates with its binary log info. To collect MYSQL master binary log coordinates please FLUSH TABLES WITH READ LOCK, which blocks COMMIT operations for InnoDB tables (data consistency in replication is very important )Please run the below SQL in MySQL master – 192.168.56.11

mysql> FLUSH TABLES WITH READ LOCK;

Now from different MySQL session on the master run SHOW MASTER STATUS statement to collect current binary log file name and it’s position

mysql > SHOW MASTER STATUS;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000147 | 450      | sakila       | manual,mysql     |
+------------------+----------+--------------+------------------+

This is the position (Position 450) from which the slave database will start replication, Take a note the number as you need them while setting up slave for MySQL replication. Now you are ready to proceed with exporting your database using mysqldump from a new window  ( please confirm you are not typing this command from MySQL, It’s done from bash shell )

mysqldump -u root -p --opt sakila > sakila-backup.sql

Now unlock tables in the master

UNLOCK TABLES;

Restore the mysqldump backup to slave (192.168.56.12)

mysql -u root -p sakila < sakila-backup.sql

Connecting slave to master

Run the script below from MySQL terminal in the slave

CHANGE MASTER TO 
MASTER_HOST='192.168.56.11', 
MASTER_USER='repl', 
MASTER_PASSWORD='MySQLDBA19/47', 
MASTER_LOG_FILE='mysql-bin.000147', 
MASTER_LOG_POS=  450;

Start MySQL slave

START SLAVE;

To confirm the successful slave replication please run below command on slave terminal

SHOW SLAVE STATUS\G;

Congrats, All done for an successful MySQL master-slave replication on CentOS !

The post Step-by-step MySQL Master – Slave Replication on CentOS appeared first on The WebScale Database Infrastructure Operations Experts.

I spend most of my professional life consulting for Internet/Mobility/SaaS companies across Southeast Asia, MiddleEast and US as MySQL Architect, I am accountable for MySQL Infrastructure Operations, Site Reliability, Performance, Scalability and High Availability of my customer MySQL Infrastructure. My customers are from diversified industries like Mobile Advertisement Networks, Online Commerce, Mobile Social Media Gaming and SaaS, The DATA is business for them and Database Infrastructure Outage is the worst thing which could ever happen so “Maximum MySQL Availability” is something I am very serious about !!! Being an MySQL consultant I get to meet new customers almost every week and most of them ask me a very common question “How we can be Optimal, Scalable an Highly Available ? ” , I will ask them back “what can be maximum duration (in hours) of MySQL outage you could ever afford? ” Many of them will reply “We have  to be available 24*7*365”, WOW, Great to hear but in real life it’s IMPOSSIBLE !!! Being an independent MySQL consultant I am very serious about my professional reputation and I always feel comfortable in sharing good, bad and ugly side of MySQL infrastructure operations! There are some customers who appreciate being upfront and some who consider me pessimistic but eventually I will be signing common MySQL Infrastructure Operations Management SLA for all of them mentioning Possible MySQL Outage Scenarios , Time-To-Recover , Recovery & Healing, HealthCheck/Diagnostics/Forensics, Performance, Scalability and Emergency Response / On-Call . In this post I am addressing Cost-of-Data-Outage.    

Cost-of-Data-Outage is expensive for every corporation from all directions like revenue, brand reputation and eventually customer experience. Think about travel booking site going offline during holiday season or Ad-Network going down during shopping festival, The damage is not just limited to web/mobile property but even to strategic partners involved in it like hotels signed-up with travel site or E-commerce platform partnered with Ad-Network so everyone consider “Data Outage” serious but can you ever promise anyone 100% availability of your data infrastructure, The honest answer will be NO!!! This doesn’t mean we are ignorant about “Data Outage” but to architect & implement “Maximum Availability Architecture” you need to know Cost-of-Data-Outage. Building Self Healing / Auto Failover, Fault Tolerant, Highly Available and Responsive MySQL Infrastructure Operations is expensive ($$$), Resource Intensive and Operationally Complex so plan carefully for Budget, People, Roadmap and Housekeeping/Operations while concluding MySQL Maximum Availability Strategy.

What are possible Data Outage scenarios in MySQL Infrastructure?

The following are few possible (most common) Data Outage Scenarios in a MySQL Infrastructure:

1. Human error: The most common and even more dangerous because there is no limit (intentional or not) for an individual/group to damage your data.
2. Business: Often the potential internet marketing and consumer psychology are mistaken/underestimated. The poor capacity planning & sizing can bring down your MySQL infrastructure in no time !!!
3. Natural Catastrophe: This could be just anything Flood, Earthquake, Tsunami, Fire etc. Man still get defeated often by mother nature unfortunately !!!
4. Planned Outage: The advancement in Hardware is so compelling (PCIe SSDs are much faster than HDDs) so upgrades are unavoidable

What I consider while planning for MySQL Maximum Availability ?

I never standardise MySQL Infrastructure Operations (Performance, Scalability High Availability and 24*7 MySQL DBA Services ) for any of my customers!! They are completely different from each other for me (though all of them use either MySQL GA / Percona Server / MariaDB / WebScaleSQL). The following are few questions which I will have the answer before engaging personally to MySQL Infrastructure Operations of my customers:

1. MySQL flavour (MySQL GA / Percona Server / MariaDB / WebScaleSQL)
2. The current MySQL DBA Operations, SLA, Emergency Support and On-Call DBA
3. The current MySQL data growth rate / day
4. MySQL Infrastructure Operations Trending Report (generally two weeks of history data will be sufficient in most of the cases )
5. MySQL Health Check, Diagnostics & Forensics Report
6. Immediate goals & accomplishments for MySQL Infrastructure Operations
7. MySQL Infrastructure Operations Roadmap

How typical MySQL Maximum Availability Project Plan will look like ?

*** This is only the to show my readers how typical Maximum MySQL Availability Project Plan will look like !!!

1. 24*7 Service Operations (Monitoring & Trending MySQL ), MySQL Infrastructure Operations Workflow (Emergency Support & On-Call) and MySQL Operations SLA (escalation and maximum accepted duration in closing tickets successfully)
2. Planned MySQL HealthCheck, Diagnostics and Forensics Audit to be proactive !!
3. Planned MySQL Tuning Schedule
4. MySQL Backup Strategy, Execution and Sanity Testing of MySQL Backups
5. MySQL High Availability Architecture, Implementation and Operations
6. Planned MySQL Outage (hardware upgrades)
7. MySQL Upgrade Plan

Now what is really the Cost-of-Data-Outage ? 

The answer for this question is not just limited to a number but much beyond  that, It is about Customer Experience, Employee Satisfaction (people hate to work for frequently going down websites), Investor Faith, Partner Loyalty, Brand Value and last but not the least REVENUE!!!

What should I do to achieve Maximum MySQL Availability ?

1. Hire / Contract professional MySQL DBA or MySQL Consulting company ?
2. Invest for your in-house MySQL DBA training / research , MySQL is growing faster so staying updated is key to gain maximum benefits
3. Sponser your MySQL DBAs for Conferences, They enjoy learning and networking with peers and industry experts

The post Cost of Data Outage appeared first on The WebScale Database Infrastructure Operations Experts.