The scope of performance benchmarking

The core objective of this benchmarking exercise is to measure MySQL 8.0 performance, This include INSERTs , SELECTs and complex transaction processing (both INSERTs and SELECTs) without any tuning of MySQL 8 instance’s my.cnf. We agree tuning my.cnf will greatly improve performance but in this activity we wanted to benchmark MySQL 8 transaction processing capabilities and technically in MinervaDB we measure performance by Response Time and believe you can build high performance MySQL applications by writing optimal SQL. We have used Sysbench (https://github.com/MinervaDB/MinervaDB-Sysbench release 1.0.20) for this benchmarking activity. This is not a paid / sponsored benchmarking effort by any of the software or hardware vendors, We will remain forever an vendor neutral and independent web-scale database infrastructure operations company with core expertise in performance, scalability, high availability and database reliability engineering. You can download detailed copy of this benchmarking here

Note: This MySQL 8.0 performance benchmarking paper is published by MinervaDB Performance Engineering Team, You are free to copy the entire content for research and publishing without copyrighting the content. This document  is distributed in the hope that it will be useful but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

☛ A low cost and instant gratification health check-up for your MySQL infrastructure operations from MinervaDB

• Highly responsive and proactive MySQL performance health check-up, diagnostics and forensics.
• Detailed report on your MySQL configuration, expensive SQL, index operations, performance, scalability and reliability.
• Recommendations for building an optimal, scalable, highly available and reliable MySQL infrastructure operations.
• Per MySQL instance performance audit, detailed report and recommendations.
• Security Audit – Detailed Database Security Audit Report  which includes the results of the audit and an actionable Compliance and Security Plan for fixing vulnerabilities and ensuring the ongoing security of your data.

** You are paying us only for the MySQL instance we have worked for :

☛ MinervaDB contacts – Sales & General Inquiries

Business Function	Contact
CONTACT GLOBAL SALES (24*7)	(844) 588-7287 (USA) (415) 212-6625 (USA) (778) 770-5251 (Canada)
TOLL FREE PHONE (24*7)	(844) 588-7287
MINERVADB FAX	+1 (209) 314-2364
MinervaDB Email - General / Sales / Consulting	contact@minervadb.com
MinervaDB Email - Support	support@minervadb.com
MinervaDB Email -Remote DBA	remotedba@minervadb.com
Shiv Iyer Email - Founder and Principal	shiv@minervadb.com
CORPORATE ADDRESS: CALIFORNIA	MinervaDB Inc., 340 S LEMON AVE #9718 WALNUT 91789 CA, US
CORPORATE ADDRESS: DELAWARE	MinervaDB Inc., PO Box 2093 PHILADELPHIA PIKE #3339 CLAYMONT, DE 19703
CORPORATE ADDRESS: HOUSTON	MinervaDB Inc., 1321 Upland Dr. PMB 19322, Houston, TX 77043, US

The post Benchmarking MySQL 8.0 Performance on Amazon EC2 appeared first on The WebScale Database Infrastructure Operations Experts.

(Thursday, June 18, 2020 – 06:00 PM to 07:00 PM PDT)

There can be several reasons for a MySQL database outage: hardware failure, power outage, human error, natural disaster etc. We may not be able prevent all the disaster from happening but investing on a robust disaster recovery plan is very important for building fault-tolerant database infrastructure operations on MySQL.  Every MySQL DBA is accountable for developing a disaster recovery plan addressing data sensitivity, data loss tolerance and data security. Join Shiv Iyer, Founder and Principal of MinervaDB to lean about the best practices for building highly reliable MySQL DR strategy and operations on Thursday, June 18, 2020 – 06:00 PM to 07:00 PM PDT. Building DR for a high traffic MySQL database infrastructure means deep understanding of multiple backup strategies and choosing optimal ones which are best suited for performance and reliability. Most of the data intensive MySQL infrastructure will have a combination of multiple backup methods and tools, In this webinar Shiv talks about his experiences in the past and present on building MySQL DR Ops, tools and zero tolerance data loss methods.

Join this webinar to learn more about:

• Proactive MySQL DR – From strategy to execution
• Building capacity for reliable MySQL DR
• MySQL DR strategies
• MySQL Backup tools
• Managing MySQL DR Ops. for very large databases
• Testing MySQL Backups
• Biggest MySQL DR mistakes
• MySQL DR Best Practices and Checklist

The post MySQL Backup and Disaster Recovery Webinar appeared first on The WebScale Database Infrastructure Operations Experts.

This post is about changes in the implementation of NDB Cluster from MySQL NDB Cluster 8.0 through 8.0.20, as compared to earlier release series. We have included only those of MySQL NDB Cluster 8.0.20 new features which are really interesting and can directly influence / make an impact to performance, scalability and reliability:

• MySQL NDB Cluster  development is in parallel with the development of  MySQL Server going forward. What does that mean for MySQL customers globally NDB 8.0 is developed in, built from, and released with the MySQL 8.0 source code tree and numbering scheme for NDB Cluster 8.0 releases follows the scheme for MySQL 8.0 ( starting with version 8.0.13).
• As of NDB 8.0.18, The identifiers can use up to 64 bytes for databases and tables (the 63-byte limit on identifiers is removed ).
• Generated names for foreign keys similar to the patterns used by InnoDB – NDB version 8.0.18 and later uses the pattern tbl_name_fk_N
• Responsive schema and metadata distribution and synchronization – MySQL NDB 8.0 (version 8.0.18 and later) will now use MySQL data dictionary to distribute schema information to SQL nodes joining a cluster and to synchronize new schema changes between existing SQL nodes.
• More metadata information for intuitive troubleshooting and monitoring – Rather than storing the binary representation of the table as in previous NDB versions, The NDB 8.0.14 and later now use serialized metadata from the MySQL data dictionary (You can read more about MySQL Data Dictionary here ). This also means that NDB tables created in NDB Cluster 8.0.14 and later are not compatible with previous NDB Cluster releases.
• On-the-fly upgrades from NDB 7.6 to NDB 8.0 – The compressed .frm file created from NDB 7.6 is no longer supported in MySQL 8.0. The upgrade is made simpler by on-the-fly translation of this metadata and writes it into the MySQL Server’s data dictionary, which enables the mysqld in NDB Cluster 8.0 to work with the table without preventing subsequent use of the table by a previous version of the NDBsoftware.
• NDB_STORED_USER privilege for user privileges Synchronization – User roles and privileges can be synchronized across all SQL nodes in the cluster by issuing a GRANT NDB_STORED_USER statement.  A user or role having NDB_STORED_USER, along with its privileges, is shared with all SQL nodes as soon as they join a given NDB Cluster. The changes to the privileges of the user or role are synchronized immediately with all connected SQL nodes. It is possible to make such changes from any connected SQL node, but recommended practice is to do so from a designated SQL node only, since the order of execution of statements affecting privileges from different SQL nodes cannot be guaranteed to be the same on all SQL nodes.
• What’s new with MySQL NDB 8.0 INFORMATION_SCHEMA ?
  • INFORMATION_SCHEMA tables are now populated with tablespace statistics for MySQL Cluster tables.
  • Tablespaces and log file groups are no longer represented in the FILES table.
• Troubleshoot NDB Cluster error with ndb_perror – To know more details about NDB Cluster error codes and troubleshoot use ndb_perror  
• Maximum row size for NDB Cluster increased:
  • NDB 8.0.18 increases the maximum number of bytes that can be stored in an NDBCLUSTER table from 14000 to 30000 bytes.
  • The maximum offset for a fixed-width column of an NDB table is 8188 bytes (unchanged from releases previous to 8.0.18).
  • The maximum size of a BLOB or TEXT column is 264 bytes (unchanged from releases previous to 8.0.18).
• Rename NDB Cluster table columns online – Beginning with MySQL NDB Cluster 8.0.18, The columns of NDB tables can be renamed online, using ALGORITHM=INPLACE.
• The offline ordered index builds are now multithreaded by default from MySQL NDB Cluster 8.0.18,  This improve restart and restore times and performance, availability, and the user experience compared to index rebuild which used to consume all cores available. The default value for BuildIndexThreads is changed from 0 to 128.
• MySQL NDB Cluster 8.0.18 support more nodes – NDB 8.0.18 increases the maximum number of data nodes supported per cluster to 144 (previously, this was 48). Data nodes can now use node IDs in the range 1 to 144, inclusive.
• NDB 8.0.19 makes the following changes in configuration parameter maximum and default values:
  • Maximum value for DataMemory is increased to 16 terabytes.
  • Maximum value for DiskPageBufferMemory is also increased to 16 terabytes.
  • Default value for StringMemory is increased to 25%.
  • Default for LcpScanProgressTimeout is increased to 180 seconds.
• Tuning disk I/O for better NDB 8.0.19 performance:
  • Parameters introduced with NDB 8.0.19:
    • MaxDiskDataLatency – To manage degree of latency permitted for disk access and causes transactions taking longer than this length of time to be aborted.
    • DiskDataUsingSameDisk – Distribute disk I/O by benefitting from the advantage of housing Disk Data tablespaces on separate disks by increasing the rate at which checkpoints of such tablespaces can be performed.
  • Metadata available on ndbinfo database to monitor NDB 8.0.19 disk I/O performance:
    • Monitor write throughput performance from diskstat 
    • Monitor writes to Disk Data tablespaces for each of the last 20 seconds from diskstats_1sec
    • Monitor latency of disk operations relating to Disk Data tablespaces from pgman_time_track_stats
• Read from any replicas supported from NDB 8.0.19 – This means default value for ndb_read_backup system variable is now ON. The value of the NDB_TABLE comment option READ_BACKUP is 1 when creating a new NDB table. Enabling read from any replica significantly improves performance for reads from NDB tables, with minimal impact on writes.

☛ MinervaDB contacts for MySQL NDB Cluster Consulting, Support and Remote DBA Services

Business Function	Contact
CONTACT GLOBAL SALES (24*7)	(844) 588-7287 (USA) (415) 212-6625 (USA) (778) 770-5251 (Canada)
TOLL FREE PHONE (24*7)	(844) 588-7287
MINERVADB FAX	+1 (209) 314-2364
MinervaDB Email - General / Sales / Consulting	contact@minervadb.com
MinervaDB Email - Support	support@minervadb.com
MinervaDB Email -Remote DBA	remotedba@minervadb.com
Shiv Iyer Email - Founder and Principal	shiv@minervadb.com
CORPORATE ADDRESS: CALIFORNIA	MinervaDB Inc., 340 S LEMON AVE #9718 WALNUT 91789 CA, US
CORPORATE ADDRESS: DELAWARE	MinervaDB Inc., PO Box 2093 PHILADELPHIA PIKE #3339 CLAYMONT, DE 19703
CORPORATE ADDRESS: HOUSTON	MinervaDB Inc., 1321 Upland Dr. PMB 19322, Houston, TX 77043, US

The post What is New in MySQL NDB Cluster 8.0.20 appeared first on The WebScale Database Infrastructure Operations Experts.

Delayed Replication – You can deliberately execute transactions later than the master by a specific duration of time , Why you do that and for what ? Consider this, Accidentally someone did a wrong UPDATE / DELETE in the master and the transaction is committed, Now how can DBA rollback the database system to the last known good condition ? This is when we benefit from MySQL delayed slave replication investment. The default replication delay in MySQL is “0” seconds, To delay the slave by N seconds use the CHANGE MASTER TO MASTER_DELAY = N, The transactions received from the master is not executed until N seconds later than it’s commit on the immediate master. We have blogged here how to setup delayed slave replication in MySQL. In this blog post we have explained how MySQL 8.0 advanced Delayed Slave Replication features.

MySQL 8.0 and Delayed Slave Replication

In MySQL 8.0 the delayed replication is controlled by two system variables on timestamps – orginal_commit_timestamp and immediate_commit_timestamp ,  They depend on GTID of each transaction (instead of each event like in MySQL 5.7) written to the binary log.These two system variables are applicable only when your entire replication infrastructure is on MySQL 8.0.1 or above , If either Master or slave is not using these timestamps, then delayed replication from MySQL 5.7 is used. 

• original_commit_timestamp: The total number of microseconds since epoch when the transaction was written (committed) to the binary log of the original master.
• immediate_commit_timestamp: The total number of microseconds since epoch when the transaction was written (committed) to the binary log of the immediate master / slave.

The orginal_commit_timestamp will be always same on all replication when the transaction is applied. In a typical master-slave replication, the original_commit_timestamp of a transaction in the (original) master’s binary log is always the same as its immediate_commit_timestamp. In the slave’s relay log, the original_commit_timestamp and immediate_commit_timestamp of the transaction are the same as in the master’s binary log; whereas in its own binary log, the transaction’s immediate_commit_timestamp corresponds to when the slave committed the transaction.

Monitoring the Delayed Slave Replication

We strongly recommend following Performance Schema tables to monitor the replication delay (lag):

• replication_connection_status: The current status of connections to the master, This data dictionary table provides information on the last and current transaction the connection thread queued into the relay log.
• replication_applier_status_by_coordinator: The current status of the coordinator thread that only displays information when using a multithreaded slave, This data dictionary table also provides information on the last transaction buffered by the coordinator thread to a worker’s queue, as well as the transaction it is currently buffering.
• replication_applier_status_by_worker: The current status of the thread(s) applying transactions received from the master and it also provides information about the transactions applied by the applier thread, or by each worker when using a multithreaded slave.

The following two matrices from the output of SHOW SLAVE STATUS is also helpful to monitor Delayed Replication:

SQL_Delay – This is measured in seconds of replication delay which configured using CHANGE MASTER TO MASTER_DELAY=N

SQL_Remaining_Delay – This shows total number seconds left of the delay configured intentionally , i.e. Slave_SQL_Running_State is Waiting for MASTER_DELAY seconds

The post MySQL 8.0 Delayed Replication – New Features and Benefits appeared first on The WebScale Database Infrastructure Operations Experts.

Recently one of our customers in Fintech. business (among the largest one in the Asia) wanted to upgrade from MySQL 5.7 to MySQL. 8.0. and they approached us for a safest and durable MySQL upgrade strategy, roadmap and execution. In Fintech. business every transaction needs to durable from statutory regulatory compliance perspective and we at MinervaDB never wanted to go for unplanned / easy in-place MySQL 8.0 upgrade method here without proper pre-migration audit, We wanted to list down in detail what are the possible scenarios this MySQL 8.0 upgrade will fail and the compatibility issues between MySQL 5.7 and MySQL 8.0. Thankfully Upgrade Checker utility that comes with MySQL Shell 8.0 can be executed against MySQL 5.7 server to confirm upgrade readiness, We have written a blog on MySQL Shell 8.0 Upgrade Checker here   

Things to remember before MySQL. 8.0 upgrade:

• Metadata on Transactional Data Dictionary tables – MySQL 8.0 manages metadata in transactional data dictionary tables. In the previous MySQL releases, the metadata was stored in non-transactional system tables
  • With the introduction of the –innodb-directoriesfeature, the location of file-per-table and general tablespace files created with an absolute path or in a location outside of the data directory should be added to the innodb_directoriesargument value. Otherwise, InnoDB is not able to locate these files during recovery. To view tablespace file locations, query the INFORMATION_SCHEMA.FILEStable:

    SELECT TABLESPACE_NAME, FILE_NAME FROM INFORMATION_SCHEMA.FILES \G

• Caching_sha2_password– Starting from MySQL 8.0 the default value for the system variable –default-authentication-plugin is “caching_sha2_password” . But for existing accounts during upgrades to MySQL 8.0. the authentication plugin will remain unchanged, this is also application for the administrative account ‘root’@’localhost’ . To connect to the server following data directory initialization, you must therefore use a client or connector that supports caching_sha2_password. If you can do this but prefer that the root account use mysql_native_password after installation, install MySQL and initialize the data directory as you normally would. Then connect to the server as root and use ALTER USERas follows to change the account authentication plugin and password:

  ALTER USER 'root'@'localhost'
  IDENTIFIED WITH mysql_native_password
  BY 'NEW_PASSWORD';

• If ever the client or the connector the application use does not support caching_sha2_password , Please modify my.cnf system variable –default-authentication-plugin = mysql_native_password 
• Use mysqlcheck. to detect non-compatible datatypes, functions, orphaned .frm files – MySQL 8.0 in-place upgrade is not supported if the tables contain old temporal columns in pre-5.6.4 format (TIME, DATETIME and TIMESTAMP columns without support for fractional seconds precision )

  mysqlcheck -u root -p --all-databases --check-upgrade

• There must be no tables that have foreign key constraint names longer than 64 characters. To identify tables with too-long constraint names, execute this query:

  SELECT TABLE_SCHEMA, TABLE_NAME
  FROM INFORMATION_SCHEMA.TABLES
  WHERE TABLE_NAME IN
    (SELECT LEFT(SUBSTR(ID,INSTR(ID,'/')+1),
                 INSTR(SUBSTR(ID,INSTR(ID,'/')+1),'_ibfk_')-1)
     FROM INFORMATION_SCHEMA.INNODB_SYS_FOREIGN
     WHERE LENGTH(SUBSTR(ID,INSTR(ID,'/')+1))>64);

• To avoid a startup failure on MySQL 8.0, remove any instance of NO_AUTO_CREATE_USER from sql_modesystem variable settings in MySQL option files.
• Validate the execution plan of your optimizer hints after upgrade from MySQL 5.7 to 8.0 , Some optimizer hint may even be counterproductive
• There must be no queries and stored program definitions from MySQL 8.0.12 or lower that use ASCor DESCqualifiers for GROUP BYclauses. Otherwise, upgrading to MySQL 8.0.13 or higher may fail, as may replicating to MySQL 8.0.13 or higher slave servers.
• There must be no table partitions that reside in shared InnoDB tablespaces, which include the system tablespace and general tablespaces. Identify table partitions in shared tablespaces by querying INFORMATION_SCHEMA:

  SELECT DISTINCT NAME, SPACE, SPACE_TYPE FROM INFORMATION_SCHEMA.INNODB_SYS_TABLES 
    WHERE NAME LIKE '%#P#%' AND SPACE_TYPE NOT LIKE 'Single';

• To move table partitions from shared tablespaces to file-per-table tablespaces, You can run  ALTER TABLE … REORGANIZE PARTITION query:

  ALTER TABLE table_name REORGANIZE PARTITION partition_name 
    INTO (partition_definition TABLESPACE=innodb_file_per_table);

• Confirm your MySQL 5.7 is not configured (my.cnf) with deprecated or removed system variable. If any, Your MySQL 8.0 upgrade will fail. The list of removed, deprecated and new system / status variables are available here –https://dev.mysql.com/doc/refman/8.0/en/added-deprecated-removed.html  
• If you are doing in-place MySQL 8.0. upgrade, either commit or rollback the XA transactions by issuing XA COMMIT or XA ROLLBACK statement

How to upgrade to MySQL 8.0 ?

There are basically two ways to upgrade to MySQL 8.0:

1. In-Place Upgrade – In this upgrade methodology you will be shutting down the old MySQL Server and replacing the old MySQL binaries or other related packages with new ones. Once successfully completed
2. Logical Upgrade – In this upgrade methodology you will be exporting SQL from the older MySQL instance using logical backup utilities like mysqldump or mysqlpump  and importing to MySQL 8.0

We at MinervaDB are technically ok with both methods of MySQL upgrades so far you have done thorough due diligence of MySQL 5.7 to MySQL 8.0 compatibilities and conflicts.

mysql_upgrade is going way with MySQL 8.0.16

mysql_upgrade binary is deprecated with MySQL 8.0.16, Going forward it will be functionally known as “server upgrade”. This is added alongside the Data Dictionary upgrade (DD Upgrade), which is a process to update the data dictionary table definitions. From MySQL 8.0.16 mysqld binary takes care of entire upgrade procedure if needed.

Manual mysql_upgrade process is still possible !

You don’t want automatic upgrades from “server upgrade” ? It is possible by configuring the MySQL 8.0 system variable –upgrade 

Recommended values:

• AUTO –  MySQL 8.0 performs automatic upgrade from the older release, MySQL 8.0 default value for server option –upgrade is AUTO.
• MINIMAL – MySQL 8.0 performs the upgrade of the data dictionary, the Performance Schema and INFORMATION_SCHEMA. When the server option –upgrade is configured MINIMAL, Group Replication cannot be started, because system tables on which the replication internals depend are not updated. 
• FORCE – When server option –upgrade is set to FORCE , The server upgrades the he data dictionary, the Performance Schema, the INFORMATION_SCHEMA, the system tables in. the mysql schema, the sys schema and other user schemas. 
• NONE – The server performs no automatic upgrades when configured the server option –upgrade to NONE, This option prevents data dictionary upgrade and server exits with an error if the data dictionary is found out of date. 

Troubleshooting MySQL 8.0 upgrade – What usually can go wrong with MySQL 8.0. upgrade ?

• Conflicting with my.cnf of previous MySQL release / installation –If the new mysqld of MySQL 8.0 does not start, Please verify that you do not have an old my.cnf file from your previous installation.  You can check this with the –print-defaults option (for example, mysqld –print-defaults). If this command displays anything other than the program name, you have an active my.cnf file that affects server or client operation.
• Commands out of sync / unexpected core dumps – After MySQL 8.0 upgrade, you experience problems with compiled client programs, such as Commands out of sync or unexpected core dumps, you probably have used old header or library files when compiling your programs. In this case, check the date for your mysql.h file and libmysqlclient.a library to verify that they are from the new MySQL distribution. If not, recompile your programs with the new headers and libraries. Recompilation might also be necessary for programs compiled against the shared client library if the library major version number has changed
• Schema mismatch errors – A schema mismatch in a MySQL 5.7 instance between the .frm file of a table and the InnoDB data dictionary can cause an upgrade to MySQL 8.0 to fail. Such mismatches may be due to .frm file corruption. To address this issue, dump and restore affected tables before attempting the upgrade again.
• User-defined function (UDF) conflicts due to same name – If ever you have created a user-defined function (UDF) / stored functions in MySQL previous releases (eg. MySQL 5.7) with same name of MySQL 8.0 built-in function, the UDF becomes inaccessible.

Downgrading from MySQL 8.0

Downgrade from MySQL 8.0 to MySQL 5.7, or from a MySQL 8.0 release to a previous MySQL 8.0 release, is not supported. The only supported alternative is to restore a backup taken before upgrading. It is therefore imperative that you backup your data before starting the upgrade process.

Conclusion

MySQL 8.0 upgrades are not complex. But, If not carefully planned, there are high chances you will end-up with an unsuccessful upgrade. We strongly recommend to do detailed low-level MySQL 8.0 compatibility assessments before planning for an upgrade, Thanks for your comments.

The post How to plan for MySQL 8.0 upgrade ? appeared first on The WebScale Database Infrastructure Operations Experts.

SELECT * FROM mysql.component;

Currently the available log components are in lib/plugins:

• component_log_filter_dragnet.so
• component_log_sink_json.so
• component_log_sink_syseventlog.so
• component_log_sink_test.so

Error Log configuration / system variables

The log_error_services system variable controls which log components to enable for error logging

mysql> select @@log_error_services;
+----------------------------------------+
| @@log_error_services                   |
+----------------------------------------+
| log_filter_internal; log_sink_internal |
+----------------------------------------+
1 row in set (0.00 sec)

The default value indicates that log evens first pass through the built-in filter controller, log_filter_interval and later through the built-in log writer component, log_sink_interval. Typically, a sink processes log events into log messages that have a particular format and writes these messages to its associated output, such as a file or the system log. The combination of  log_filter_internal and log_sink_internal implements the default error log filtering and output behavior.

The output destination of error log can be collected from system variable log_error .  You can configure the destination of error log either to the system log or JSON file.

You can make mysqld to write the error log to system log (Event Log on Windows and syslog on Linux and Unix systems):

INSTALL COMPONENT 'file://component_log_sink_syseventlog';
SET GLOBAL log_error_services = 'log_filter_internal; log_sink_syseventlog';

You can enable JSON writer to record the error log by first loading the writer component and then modifying log_error_services system variable:

INSTALL COMPONENT 'file://component_log_sink_json';
SET GLOBAL log_error_services = 'log_filter_internal; log_sink_json';

traditional error log:

2019-03-10T08:36:59.950769Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.15) starting as process 13222
2019-03-10T08:37:00.253523Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2019-03-10T08:37:00.267812Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.15'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.
2019-03-10T08:37:00.429164Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/var/run/mysqld/mysqlx.sock' bind-address: '::' port: 33060
2019-03-10T08:37:37.635761Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.15)  MySQL Community Server - GPL.
2019-03-10T08:37:37.985380Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.15) starting as process 13410
2019-03-10T08:37:38.277912Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2019-03-10T08:37:38.291494Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.15'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.

JSON error log:

{ "prio" : 0, "err_code" : 10910, "source_line" : 2191, "source_file" : "mysqld.cc", "function" : "clean_up", "msg" : "/usr/sbin/mysqld: Shutdown complete (mysqld 8.0.15)  MySQL Community Server - GPL.", "time" : "2019-03-10T09:21:18.722864Z", "err_symbol" : "ER_SERVER_SHUTDOWN_COMPLETE", "SQL_state" : "HY000", "subsystem" : "Server", "label" : "System" }

• Filtering MySQL error with log_error_verbosity:
  • Errors only – 1
  • Errors and warnings  – 2
  • Errors, warnings and notes – 3
    • If log_error_verbosity is configured to 2 or higher, the MySQL server even logs the statement that are unsafe for statement-based logging / replication. if the value is 3, the server logs aborted connections and access-denied errors for fresh connection attempts. It is recommended to configure log_error_verbosity with 2 or higher to record detailed information about what is happening to MySQL infrastructure.
• How MySQL 8.0 component based error logging filters are different ? 

We are used to default built-in error log filters that are configured with MySQL system variable log_error_verbosity (default is 2). But, MySQL 8.0 has another component that allows you to filter on rules that you define: log_filter_dragnet. I have explained below step-by-step on how to setup Rule-Based Error Log Filtering using log_filter_dragnet :

INSTALL COMPONENT 'file://component_log_filter_dragnet';
SET GLOBAL log_error_services = 'log_filter_dragnet; log_sink_internal';

To limit information events to no more than one per 60 seconds:

mysql> SET GLOBAL dragnet.log_error_filter_rules =
    -> 'IF prio>=INFORMATION THEN throttle 1/60.';
Query OK, 0 rows affected (0.00 sec)

To throttle  plugin-shutdown messages to only 5 per 5 minutes (300 seconds):

IF err_code == ER_PLUGIN_SHUTTING_DOWN_PLUGIN THEN throttle 1.

To throttle errors and warnings to 1000 per hour and information messages to 100 per hour:

IF prio <= INFORMATION THEN throttle 1000/3600 ELSE throttle 100/3600.

and we can monitor the available dragnet rule:

mysql> select * from global_variables where VARIABLE_NAME like 'dragnet%'\G
*************************** 1. row ***************************
 VARIABLE_NAME: dragnet.log_error_filter_rules
VARIABLE_VALUE: IF prio>=INFORMATION THEN throttle 1/60.
1 row in set (0.00 sec)

Conclusion

MySQL 8.0 Error Logging Services are more powerful compared to the versions before and you can now filter error logging much better by creating your own components 

The post MySQL 8.0 all new Error Logging appeared first on The WebScale Database Infrastructure Operations Experts.

Using MySQL Shell 8.0.11 “upgrade checker”

Typical “upgrade checker” run will look similar to this:

 MySQL  JS >  util.checkForServerUpgrade("root@localhost:3306")
Please provide the password for 'root@localhost:3306': **********
The MySQL server at localhost:3306 will now be checked for compatibility issues for upgrade to MySQL 8.0...
MySQL version: 5.7.22-log - MySQL Community Server (GPL)

1) Usage of db objects with names conflicting with reserved keywords in 8.0
  No issues found

2) Usage of utf8mb3 charset
  Warning: The following objects use the utf8mb3 character set. It is recommended to convert them to use utf8mb4 instead, for improved Unicode support.

  sakila.actor.first_name - column's default character set: utf8
  sakila.actor.last_name - column's default character set: utf8
  sakila.actor_info.first_name - column's default character set: utf8
  sakila.actor_info.last_name - column's default character set: utf8
  sakila.actor_info.film_info - column's default character set: utf8
  sakila.address.address - column's default character set: utf8
  sakila.address.address2 - column's default character set: utf8
  sakila.address.district - column's default character set: utf8
  sakila.address.postal_code - column's default character set: utf8
  sakila.address.phone - column's default character set: utf8
  sakila.category.name - column's default character set: utf8
  sakila.city.city - column's default character set: utf8
  sakila.country.country - column's default character set: utf8
  sakila.customer.first_name - column's default character set: utf8
  sakila.customer.last_name - column's default character set: utf8
  sakila.customer.email - column's default character set: utf8
  sakila.customer_list.name - column's default character set: utf8
  sakila.customer_list.address - column's default character set: utf8
  sakila.customer_list.zip code - column's default character set: utf8
  sakila.customer_list.phone - column's default character set: utf8
  sakila.customer_list.city - column's default character set: utf8
  sakila.customer_list.country - column's default character set: utf8
  sakila.customer_list.notes - column's default character set: utf8
  sakila.film.title - column's default character set: utf8
  sakila.film.description - column's default character set: utf8
  sakila.film.rating - column's default character set: utf8
  sakila.film.special_features - column's default character set: utf8
  sakila.film_list.title - column's default character set: utf8
  sakila.film_list.description - column's default character set: utf8
  sakila.film_list.category - column's default character set: utf8
  sakila.film_list.rating - column's default character set: utf8
  sakila.film_list.actors - column's default character set: utf8
  sakila.film_text.title - column's default character set: utf8
  sakila.film_text.description - column's default character set: utf8
  sakila.language.name - column's default character set: utf8
  sakila.nicer_but_slower_film_list.title - column's default character set: utf8
  sakila.nicer_but_slower_film_list.description - column's default character set: utf8
  sakila.nicer_but_slower_film_list.category - column's default character set: utf8
  sakila.nicer_but_slower_film_list.rating - column's default character set: utf8
  sakila.nicer_but_slower_film_list.actors - column's default character set: utf8
  sakila.sales_by_film_category.category - column's default character set: utf8
  sakila.sales_by_store.store - column's default character set: utf8
  sakila.sales_by_store.manager - column's default character set: utf8
  sakila.staff.first_name - column's default character set: utf8
  sakila.staff.last_name - column's default character set: utf8
  sakila.staff.email - column's default character set: utf8
  sakila.staff.username - column's default character set: utf8
  sakila.staff.password - column's default character set: utf8
  sakila.staff_list.name - column's default character set: utf8
  sakila.staff_list.address - column's default character set: utf8
  sakila.staff_list.zip code - column's default character set: utf8
  sakila.staff_list.phone - column's default character set: utf8
  sakila.staff_list.city - column's default character set: utf8
  sakila.staff_list.country - column's default character set: utf8

3) Usage of use ZEROFILL/display length type attributes
  Notice: The following table columns specify a ZEROFILL/display length attributes. Please be aware that they will be ignored in MySQL 8.0

  sakila.customer.active - tinyint(1)
  sakila.staff.active - tinyint(1)

4) Issues reported by 'check table x for upgrade' command
  No issues found

5) Table names in the mysql schema conflicting with new tables in 8.0
  No issues found

6) Usage of old temporal type
  No issues found

7) Foreign key constraint names longer than 64 characters
  No issues found

8) Usage of obsolete MAXDB sql_mode flag
  No issues found

9) Usage of obsolete sql_mode flags
  No issues found

10) Usage of partitioned tables in shared tablespaces
  No issues found

11) Usage of removed functions
  No issues found

No fatal errors were found that would prevent a MySQL 8 upgrade, but some potential issues were detected. Please ensure that the reported issues are not significant before upgrading.
1

At the end,  “upgrade checker” prints a summary and returns an integer value describing he severity of the issues found:

• 0 – no issues or only ones categorized as notice,
• 1 – No fatal errors were found, but some potential issues were detected,
• 2 – UC found errors that must be fixed before upgrading to 8.0.

Upgrade from MySQL 5.7 to MySQL 8.0

Step 1 – Uninstall MySQL 5.7 

[root@localhost ~]# systemctl stop mysqld 
[root@localhost ~]# yum remove mysql-community-client.x86_64 mysql-community-common.x86_64 mysql-community-devel.x86_64 mysql-community-libs.x86_64 mysql-community-libs-compat.x86_64 mysql-community-server.x86_64 
Loaded plugins: fastestmirror
Resolving Dependencies
--> Running transaction check
---> Package mysql-community-client.x86_64 0:5.7.22-1.el7 will be erased
---> Package mysql-community-common.x86_64 0:5.7.22-1.el7 will be erased
---> Package mysql-community-devel.x86_64 0:5.7.22-1.el7 will be erased
---> Package mysql-community-libs.x86_64 0:5.7.22-1.el7 will be erased
---> Package mysql-community-libs-compat.x86_64 0:5.7.22-1.el7 will be erased
---> Package mysql-community-server.x86_64 0:5.7.22-1.el7 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================
 Package                        Arch      Version          Repository             Size
=======================================================================================
Removing:
 mysql-community-client         x86_64    5.7.22-1.el7     @mysql57-community    106 M
 mysql-community-common         x86_64    5.7.22-1.el7     @mysql57-community    2.5 M
 mysql-community-devel          x86_64    5.7.22-1.el7     @mysql57-community     21 M
 mysql-community-libs           x86_64    5.7.22-1.el7     @mysql57-community    9.4 M
 mysql-community-libs-compat    x86_64    5.7.22-1.el7     @mysql57-community    9.2 M
 mysql-community-server         x86_64    5.7.22-1.el7     @mysql57-community    743 M

Transaction Summary
=======================================================================================
Remove  6 Packages

Installed size: 892 M
Is this ok [y/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Erasing    : mysql-community-devel-5.7.22-1.el7.x86_64                           1/6 
  Erasing    : mysql-community-server-5.7.22-1.el7.x86_64                          2/6 
warning: /etc/my.cnf saved as /etc/my.cnf.rpmsave
  Erasing    : mysql-community-client-5.7.22-1.el7.x86_64                          3/6 
  Erasing    : mysql-community-libs-compat-5.7.22-1.el7.x86_64                     4/6 
  Erasing    : mysql-community-libs-5.7.22-1.el7.x86_64                            5/6 
  Erasing    : mysql-community-common-5.7.22-1.el7.x86_64                          6/6 
  Verifying  : mysql-community-libs-compat-5.7.22-1.el7.x86_64                     1/6 
  Verifying  : mysql-community-common-5.7.22-1.el7.x86_64                          2/6 
  Verifying  : mysql-community-devel-5.7.22-1.el7.x86_64                           3/6 
  Verifying  : mysql-community-server-5.7.22-1.el7.x86_64                          4/6 
  Verifying  : mysql-community-client-5.7.22-1.el7.x86_64                          5/6 
  Verifying  : mysql-community-libs-5.7.22-1.el7.x86_64                            6/6 

Removed:
  mysql-community-client.x86_64 0:5.7.22-1.el7                                         
  mysql-community-common.x86_64 0:5.7.22-1.el7                                         
  mysql-community-devel.x86_64 0:5.7.22-1.el7                                          
  mysql-community-libs.x86_64 0:5.7.22-1.el7                                           
  mysql-community-libs-compat.x86_64 0:5.7.22-1.el7                                    
  mysql-community-server.x86_64 0:5.7.22-1.el7                                         

Complete!
[root@localhost ~]# 

Step 2 – Install MySQL 8.0

[root@localhost MySQL8-Community-Edition]# rpm -ivh mysql-community-server-8.0.11-1.el7.x86_64.rpm mysql-community-client-8.0.11-1.el7.x86_64.rpm mysql-community-common-8.0.11-1.el7.x86_64.rpm mysql-community-devel-8.0.11-1.el7.x86_64.rpm mysql-community-libs-8.0.11-1.el7.x86_64.rpm mysql-community-libs-compat-8.0.11-1.el7.x86_64.rpm 
warning: mysql-community-server-8.0.11-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-common-8.0.11-1.e################################# [ 17%]
   2:mysql-community-libs-8.0.11-1.el7################################# [ 33%]
   3:mysql-community-client-8.0.11-1.e################################# [ 50%]
   4:mysql-community-server-8.0.11-1.e################################# [ 67%]
   5:mysql-community-devel-8.0.11-1.el################################# [ 83%]
   6:mysql-community-libs-compat-8.0.1################################# [100%]
[root@localhost MySQL8-Community-Edition]# 

Step 3- Start MySQL 8.0

[root@localhost MySQL8-Community-Edition]# systemctl start mysqld 
[root@localhost MySQL8-Community-Edition]# mysql -u root -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 10
Server version: 8.0.11 MySQL Community Server - GPL

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> 

Step 4 – Run “mysql_upgrade” , mysql_upgrade checks for all tables in all databases for incompatibilities with the current version of MySQL Server, it also upgrades the system tables so that you can take advantage of new privileges or capabilities that might have been added.

[root@localhost MySQL8-Community-Edition]# mysql_upgrade -u root -p 
Enter password: 
Checking if update is needed.
Checking server version.
Running queries to upgrade MySQL server.
Upgrading system table data.
Checking system database.
mysql.columns_priv                                 OK
mysql.component                                    OK
mysql.db                                           OK
mysql.default_roles                                OK
mysql.engine_cost                                  OK
mysql.func                                         OK
mysql.general_log                                  OK
mysql.global_grants                                OK
mysql.gtid_executed                                OK
mysql.help_category                                OK
mysql.help_keyword                                 OK
mysql.help_relation                                OK
mysql.help_topic                                   OK
mysql.innodb_index_stats                           OK
mysql.innodb_table_stats                           OK
mysql.ndb_binlog_index                             OK
mysql.password_history                             OK
mysql.plugin                                       OK
mysql.procs_priv                                   OK
mysql.proxies_priv                                 OK
mysql.role_edges                                   OK
mysql.server_cost                                  OK
mysql.servers                                      OK
mysql.slave_master_info                            OK
mysql.slave_relay_log_info                         OK
mysql.slave_worker_info                            OK
mysql.slow_log                                     OK
mysql.tables_priv                                  OK
mysql.time_zone                                    OK
mysql.time_zone_leap_second                        OK
mysql.time_zone_name                               OK
mysql.time_zone_transition                         OK
mysql.time_zone_transition_type                    OK
mysql.user                                         OK
Found outdated sys schema version 1.5.1.
Upgrading the sys schema.
Checking databases.
employees.departments                              OK
employees.dept_emp                                 OK
employees.dept_manager                             OK
employees.employees                                OK
employees.salaries                                 OK
employees.tab1                                     OK
employees.titles                                   OK
sakila.actor                                       OK
sakila.address                                     OK
sakila.category                                    OK
sakila.city                                        OK
sakila.country                                     OK
sakila.customer                                    OK
sakila.film                                        OK
sakila.film_actor                                  OK
sakila.film_category                               OK
sakila.film_text                                   OK
sakila.inventory                                   OK
sakila.language                                    OK
sakila.payment                                     OK
sakila.rental                                      OK
sakila.staff                                       OK
sakila.store                                       OK
sys.sys_config                                     OK
Upgrade process completed successfully.
Checking if update is needed.

Success  , You have successfully completed upgrade from MySQL 5.7 to MySQL 8.0 .

The post Using MySQL Shell 8.0.11 “upgrade checker” to upgrade from MySQL 5.7 to MySQL 8.0 successfully appeared first on The WebScale Database Infrastructure Operations Experts.

MySQL Group Replication can operate in both single-primary mode with automatic primary election, where only one server accepts updates at a time and multi-primary mode, where all servers can accept updates, even if they are issued concurrently. The built-in group membership service retains view of the group consistent and available for all servers at any given point in time, Servers can leave and join the group and the view is updated accordingly. If servers leave the group unexpectedly, the failure detection mechanism detects this and notifies the group that the view has changed, All this happens automatically !

For any transaction to commit, the majority of group members have to agree on the order of a given transaction in the global sequence of transactions. Decision to commit or abort a transaction is taken by respective servers, but all servers make same decision. In the case of systems built on network partition, When the members of the group are unable to reach agreement on a transaction, the system halt till the issue is resolved, This guarantees split-brain protection mechanism.

All these fault tolerance mechanisms are powered by Group Communication System (GCS) protocols, a group membership service, and completely safe ordered message delivery system. Group Replication is powered by Paxos algorithm (https://en.wikipedia.org/wiki/Paxos_(computer_science)), which acts as the group communication engine.

Till now, we have just posted about the capabilities of Group Replication, Now we have mentioned below (in bullets) about all the limitations of Group Replication:

• set –binlog-checksum=NONE – Group replication cannot benefit from –binlog-checksum due to the design limitation of replication event checksums.
• Gap Locks not supported – The information about gap locks are not available outside InnoDB so certification process cannot acknowledge gap locks  .
• Table Locks and Named Locks not supported – Certification process will not acknowledge table / named locks.
• Concurrent DDL versus DML Operations – Concurrent data definition statements and data manipulation statements executing against the same object but on different servers is not supported when using multi-primary mode. During execution of Data Definition Language (DDL) statements on an object, executing concurrent Data Manipulation Language (DML) on the same object but on a different server instance has the risk of conflicting DDL executing on different instances not being detected.
• Very Large Transactions.  Individual transactions that result in GTID contents which are large enough that it cannot be copied between group members over the network within a 5 second window can cause failures in the group communication. To avoid this issue try and limit the size of your transactions as much as possible. For example, split up files used with LOAD DATA INFILE into smaller chunks.
• Multi-primary Mode Deadlock.  When a group is operating in multi-primary mode, SELECT … FOR UPDATE statements can result in a deadlock. This is because the lock is not shared across the members of the group, therefore the expectation for such a statement might not be reached.

The post MySQL 8.0 Group Replication Limitations appeared first on The WebScale Database Infrastructure Operations Experts.

How file based metadata management used to work in the past (before MySQL 8.0) ? 

• Every table in MySQL will have corresponding .frm file, This .frm file stores information like column names and data-types in the binary format, In addition to the .frm file, there are .trn, .trg and .par files to support triggers, trigger namespace and partitioning .

What are major bottlenecks faced due to the usage of file based metadata management ? 

• Operationally it always appeared very irrational, Why we need to have an separate mechanism to track the schema information ? Originally this was the idea from Drizzle –  Drizzle made it very clear (almost ) that it should get out of the way and let the storage engines be the storage engines and not try to second guess them or keep track of things behind their back.
• Dictionaries out of synch.– Before MySQL 8.0, the data dictionary is a  “split brain”, where  the “server” and InnoDB have their own separate data dictionary, where some information duplicated. Information that is duplicated in the MySQL server dictionary and the InnoDB dictionary might get out of synch, and we need one common “source of truth”  for dictionary information.
• INFORMATION_SCHEMA is the bottleneck– The main reason behind these performance issues in the INFORMATION_SCHEMA (before MySQL 8.0) implementation is that INFORMATION_SCHEMA tables are implemented as temporary tables that are created on-the-fly during query execution. For a MySQL server having hundreds of databases, each with hundreds of tables within them, the INFORMATION_SCHEMA query would end-up doing lot of I/O reading each individual FRM files from the file system. And it would also end-up using more CPU cycles in effort to open the table and prepare related in-memory data structures. It does attempt to use the MySQL server table cache (the system variable ‘table_definition_cache‘), however in large server instances it’s very rare to have a table cache that is large enough to accommodate all of these tables.
• No atomic DDL– Storing the data dictionary in non-transactional tables and files, means that DDLs are unsafe for replication (they are not transactional, not even atomic). If a compound DDL fails we still need to replicate it and hope that it fails with the same error. This is a best effort approach and there is a lot of logic coded to handle this . It is hard to maintain, slows down progress and bloats the replication codebase. The data dictionary is stored partly in non-transactional tables. These are not safe for replication building resilient HA systems on top of MySQL. For instance, some dictionary tables need to be manipulated using regular DML, which causes problems for GTIDs.
• Crash recovery. Since the DDL statements are not atomic, it is challenging to recover after crashing in the middle of a DDL execution, and is especially problematic for replication.

How things are changed with MySQL 8.0  ? 

MySQL 8.0 introduced a native data dictionary based on InnoDB.  This change has enabled us to get rid of file-based metadata store (FRM files) and also help MySQL to move towards supporting transactional DDL. We have now the metadata of all database tables stored in transactional data dictionary tables, it enables us to design an INFORMATION_SCHEMA table as a database VIEW over the data dictionary tables. This eliminates costs such as the creation of temporary tables for each INFORMATION_SCHEMA query during execution on-the-fly, and also scanning file-system directories to find FRM files. It is also now possible to utilize the full power of the MySQL optimizer to prepare better query execution plans using indexes on data dictionary tables. INFORMATION SCHEMA is now implemented as views over dictionary tables, requires no extra disc accesses, no creation of temporary tables, and is subject to similar handling of character sets and collations as user tables.

The following diagram (Source: MySQL server team blog) explains the difference in design in MySQL 5.7 and 8.0 :

The post MySQL 8.0 Data Dictionary appeared first on The WebScale Database Infrastructure Operations Experts.

What was our first reaction, when we first read about innodb_dedicated_server ?

Wow, That will be awesome … Indeed, When we manage several hundreds of MySQL instances, This feature will really improve efficiency and DBA Ops. governance.

Now, Let us explain what we have found:

How does innodb_dedicated_server system variable in MySQL 8.0 size the following variables:

• innodb_buffer_pool_size:
  • <1G – 128M (default value if innodb_dedicated_server is disabled / OFF)
  • <=4G = Detected Physical RAM * 0.5
  • >4G : Detected Physical RAM *0.75
• innodb_log_file_size: 
  • <1G: 48M(default value if innodb_dedicated_server is OFF)
  • <=4G: 128M
  • <=8G: 512M
  • <=16G: 1024M
  • >16G: 2G
• innodb_flush_method 
  • Set to O_DIRECT_NO_FSYNC if the setting is available on the system. If not, set it to the default InnoDB flush method

The first impression of innodb_dedicated_server system variable in MySQL 8.0 is impressive, Definitely will deliver much better performance than default value. This new feature will configure the MySQL system variable mentioned above more intuitively to improve DBA productivity. Till MySQL 5.7 it was always presumed 512M RAM with the default settings.

Are we going to follow this in our daily DBA checklists  ?

Not really, We are an very conservative team about implementing the new features immediately in the critical database infrastructure of our customers, Also we are afraid about the isolated issues due to auto sizing of MySQL / InnoDB memory structures, Let’s explain why we will not be using this feature immediately for our MySQL 8.0 customers:

• We carefully size InnoDB memory parameters on various factors like database size, transaction complexity, archiving policies etc.  So we want to be hands-on or follow manual sizing of system variables innodb_buffer_pool_size, innodb_log_file_size and innodb_flush_method.
• Capacity planning and sizing – We are always afraid of over / undersizing of our database infrastructure operations. Database infrastructure operations reliability is very critical for us, We have dedicated team with-in to monitor and trend database infrastructure operations and system resource usage consumption.

P.S – innodb_dedicated_server system variable is a relatively new feature, We are confident MySQL engineering team will be improving this component in coming days so our perspective will also change, We will never forget then to blog about this feature and why we are seriously thinking about implementing it for our customer production infrastructure.. Technology keeps changing for good, We are adaptive for the change !

The post The first impression of MySQL 8 system variable innodb_dedicated_server appeared first on The WebScale Database Infrastructure Operations Experts.