How to interpret “MySQL error 1040 – Too many connections ! ” ?

When a client tries to log into MySQL it may sometimes be rejected and receive an error message saying that there are “too many connections“. This means that the maximum number of clients that may be connected to the server has been reached. Either the client will have to wait for another client to log off, or the administrator will have to increase the maximum number of connections allowed.

Information about connections to a server can be found using the SHOW STATUS statement:

SHOW STATUS LIKE 'max_used_connections';

Prerequisite – Few points to remember before working or troubleshooting MySQL ” Too many connections ! ” error

1. MySQL does not have it’s own thread handling mechanism / implementation and it completely relies on the thread handling implementation of the underlying operating system.
2. MySQL system variable max_connections control the maximum number of clients the server permits to connect simultaneously,   You may have to increase max_connections if more clients attempt to connect simultaneously then the server is configured to handle (Explained more in detail –  “Too many connections”).
3. How MySQL connection handling (both connects and disconnects) works  ?
   1. MySQL Clients connects to MySQL Server via a simple and regular TCP-IP connect message though port 3306 on the Server instance
   2. The incoming connection requests are queued and then processed by the receiver thread one by one, All that receiver thread does is create user thread.  It’s actually user thread which handles the client-server protocol for connection management, Which involves allocate and initialize the corresponding THD for user authentication based on credentials stored on THD’s security policies / directories and finally if successfully authorized in the connection phase, the user thread will enter command phase
   3. The  receiver  thread will either create a new OS thread or reuse and existing “free” OS thread if available in the  thread cache. So we strongly recommend increasing  thread cache in  cases where number of  connections fluctuates  between ver few connections  and having  many connections. But there are three things which a thread might need to wait for:  A mutex, a database lock, or IO.
   4. THD basically is a large data structure used for several purposes like connection management, authorization and even unto to query execution, So how  much THD consumes memory is directly proportionate to the query execution complexities and connection traffic.

MySQL error – Too many connections, How to fix ?

Recently one of customers ( among the top 5 largest e-commerce companies in the world ) called us to check how graceful their connection handling works during peak hours of business, They had issues in  the past with ” ERROR 1040: Too many connections “ and that clearly explains the maximum number of clients that may be connected to the server has been reached so either the client will have to wait for another client to log off, or the administrator will have to increase the maximum number of connections allowed. so wanted us to do a detailed health-check on MySQL connection management and address “Too many connections” error proactively, We have explained below on how we could successfully reproduce this issue and recommended the fix:

Goal: Manage  50,000 connections on MySQL 8.0 (Ubuntu)

The default setting for system variable max_connections is “151”and we are benchmarking 50K connections so the first step before benchmarking  is to increase max_connections to 50000. we increased max_connections to 50000 dynamically and what happened after that was not expected, We have copied the results below:

root@MDB1:~# mysql -uroot -pMDB@PassWd2020 -se "select @@max_connections"
@@max_connections
697

We got only 697 connections, Let’s interpret MySQL error log before proceeding to next steps.. We have copied the same below:

2020-01-30T19:52:35.136192Z0 [Warning] Changed limits: max_open_files: 5129 (requested 10000)
2020-01-30T19:54:13.241937Z0 [Warning] Changed limits: max_connections: 4152 (requested 10000)
2020-01-30T19:57:47.51617Z0 [Warning] Changed limits: table_open_cache: 533 (requested 15000)

This is due to open files limitations for MySQL so let’s increase now the number of allowed open files for MySQL, The following steps we did to fix this resource limit issue:

• Option  1 – Locate the systemd configuration folder for MySQL and create file /etc/systemd/system/mysqld.service.d/override.conf (file can be called anything ending with .conf).
  • Add LimitNOFILE=55000 in the file override.conf
  • Add TasksMax=55000 in the file override.conf
  • Add LimitNPROC=55000 in the file override.conf
• Option 2 – We can also create/modify the override file by using native systemctl command like: systemctl edit mysql

root@MDB1:~# cat /etc/systemd/system/mysql.service.d/override.conf
[Service]
LimitNOFILE=55000
TasksMax=55000
LimitNPROC=55000

** MySQL uses some files for additional work and we need to set LimitNOFILE, TasksMax and LimitMPROC higher to get 50000 connections, lets set it to 55000 and reload the systemd daemon and restart the MySQL service.

Reload the systmed daemon and restart the MySQL service:

root@MDB1:~# systemctl daemon-reload
root@MDB1:~# systemctl restart mysql

Now let’s check max_connections to confirm the change applied:

root@MDB1:~# mysql -uroot -pMDB@PassWd2020 -se "select @@max_connections"
mysql: [Warning] Using a password on the command line interface can be insecure.
@@max_connections
50000

Conclusion

We have no fixed value recommendations for system variable max_connections, It completely depends on your application load and how your application does connection handling. We advice our customers to avoid too many connections opened concurrently because each thread connected needs memory and there is also resource intensive context switching causing overall performance degradation, Thanks for reading and comments are welcome !

References

• https://mysqlserverteam.com/mysql-connection-handling-and-scaling/ 
• http://mysql-nordic.blogspot.com/2019/04/mysql-error-too-many-connections.html 
• https://dev.mysql.com/doc/refman/5.7/en/using-systemd.html 

Book your appointment for 30 minutes ( absolutely free ) MinervaDB consulting 

The post How MySQL handles connection ? Troubleshooting MySQL ERROR 1040, Too many connections! appeared first on The WebScale Database Infrastructure Operations Experts.

Happy  New Year

MinervaDB wishing you all a very happy and prosperous new year – 2020 ! When we talk to our customers (both existing and prospective) and other community stakeholders during the month January, We get to know their resolutions (specifically on Business and Database Infrastructure Operations) and every company is so very passionate to share their goals for  2020, They actually spend so much  energy is  articulating this very well, loud and clear so that the true spirit never get diluted. In this blog we would like share with you few interesting points which really can influence all of the readers (most of our expected audiences are geeks and technology business owners) too. This post is mostly about how our resolutions are very well aligned to our customers ambitions / goals, As we often say – We are not an yet another DBA outsourcing company, We are always an extended team of customers committed to building optimal, scalable, highly reliable, fault-tolerant and secured Database Infrastructure Operations, Our accomplishments totally depend on the customer success.

Growing older by another year –  Growth is awesome for every business

Every successful business build loyalty and customer satisfaction year-on-year, They solve more customer problems faster and scale the growth ( with much deeper technology commitments ) to create an impact, so the following are our customer goals for 2020:

• Every year the internet penetration rate is going higher (especially in the emerging markets) and that means the companies (majorly consumer internet properties) should scale performance for increased data volume and transactions, So where are they investing ? Please read the bullets below to know more:
  • More responsive observability and monitoring infrastructure – Proactive health-check and diagnostics  / forensics of Database Infrastructure Operations
  • Capacity Planning and Sizing cost efficiently, Spending more or less for new infrastructure is equally bad, The oversized  capacity planning / sizing is just a waste of money and when undersized, It will be a serious hit for both business and other respective stakeholders (customers, investors and definitely employees)
  • Database Systems cannot be the single-point-of-failures ! Yes, All our customers has this one common requirement: Database Systems should be high reliable / available, redundant and self-healing.
  • We are definitely going to generate more data in 2020 than what was accomplished on 2019, So we have now bigger performance challenges to address –  Database Systems Performance is the most valued enabler in Data Powered Commerce Platforms
• Technology Roadmap – Is it all going to cloud or bare-metal also expected ?
  • None of our 200 odd customers are on 100% cloud though we have majority of customers on bare-metal servers (for both strategic and law-of-the-land reasons)
  • More than 50% of  customers are now more serious than ever before  on data privacy and security, Interesting time ahead of us in MinervaDB as we always  love to get challenged technically.
  • More investments for automation and efficiency management – The database infrastructure is growing aggressively for most of our customers, This  means more data to manage for  us and we at MinervaDB love “DevOps. for Database Ops.” so enterprise-wide we  are automating from incident reporting to workflow operations.
  • Technology adoption strategy – Vendor neutral and independent, Our customers are so passionate about open source Database Systems, They are exploring equally now MySQL, MariaDB, MyRocks and PostgreSQL
• Business Strategy – Customer Success and Employee Satisfaction
  • Customer Success:
    • High performance transaction performance to  process orders and complete transactions faster
    • Building systems for scale
    • Available 24*7*365 – Database Systems are available 99.999% for solving customer problems
    • Secured Database Infrastructure Operations – Data Privacy and Transaction Security
  • Employee Satisfaction:
    • Improve work-life balance – Investing to build scalable, distributed, fault-tolerant and reliable database infrastructure
    • Do not disturb your resident DBAs on vacation – Plan and subscribe MinervaDB Vacation DBA Services proactively
    • Learning and professional development program – Investing for competency development and improvement
    • Automation and tools  – Automating redundant & effort intensive tasks for cost efficiency and quality management

Twenty Reasons to engage with MinervaDB on 2020

1. MinervaDB launching PostgreSQL Consulting and Support from Q2 2020, Great value proposition for customers – Single vendor engagement for consulting and 24*7 support for MySQL, MariaDB, MyRocks, ClickHouse and PostgreSQL
2. Vendor neutral and independent Open Source Database Technologies consulting and 24*7 support  provider – Consulting, Support and Remote DBA Services for MySQL, MariaDB, MyRocks, PostgreSQL  and ClickHouse
3. MinervaDB for startups to large corporations – You can engage us for as-low-as 4 hours / month to 24*7*365, To know more please refer to MinervaDB Remote DBA subscription plans 
4. Build secured Database Infrastructure Operations with MinervaDB:
   • GDPR: General Data Protection Directive
   • PCI DSS: Payment Card Industry Data Security Standard
   • HIPAA: Health Insurance Portability and Accountability Act
   • HITECH: Health Information Technology for Economic and Clinical Health Act
   • Data Protection Act: United Kingdom
   • SOX: Sarbanes Oxley
   • FERPA: Family Educational Rights and Privacy Act
   • And many more
5. MinervaDB delivers 24*7 consultative support for MySQL, MariaDB, MyRocks, PostgreSQL and ClickHouse
6. Emergency Support available 24*7 – Not a MinervaDB customer ? No problem, We can deliver 24*7 emergency support on MySQL, MariaDB, MyRocks, ClickHouse and PostgreSQL for you
7. Flexible pricing – Pay us only for the hours engaged
8. Consulting and professional services for MySQL, MariaDB, MyRocks, ClickHouse and PostgreSQL available both onsite and remote
9. Competitive pricing – We are an virtual corporation so we don’t charge the customers for our infrastructure cost, What you pay us goes purely for our unmatched technology team
10. Pay per incident option available – You need our support in just fixing a single incident ? No problem, We have that option available.
11. Big Data Analytics and Columnar Store expertise – MariaDB ColumnStore and ClickHouse
12. Automation and DevOps. for building WebScale Database Infrastructure Operations
13. We operate 24*7 – Our team operates from multiple locations worldwide so we are available 24*7
14. Transparent billing  – We publish our consulting (https://minervadb.com/index.php/mysql-consulting/), support (https://minervadb.com/index.php/mysql-support/) and remote DBA services (https://minervadb.com/index.php/mysql-remote-dba-services-by-minervadb/) rates in the website. YOU WILL NEVER EVER PAY EVEN A SINGLE DIME MORE THAN WHAT IS MENTIONED IN THIS WEBSITE FOR OUR SERVICES, There are no hidden charges in our contracts !
15. Full-stack DBA operations – We are experts in performance, scalability, high availability, disaster recovery, database security and database infrastructure upgrades / migrations. You can stop worrying about data and focus more on business
16. Customer centric and boot strapped – Our business is 100% customer centric and we earn only from consulting and remote DBA services so customer success is our primary goal. We are not yet another DBA outsourcing company rather an extended team of our customers committed to optimal, scalable and highly available database infrastructure operations
17. Web-scale DBA expertise – Our consultants have worked for several planet-scale internet properties and we know what it takes to build an optimal, scalable and highly available web-scale database infrastructure operations
18. Vacation DBA Service – We can support your database infrastructure operations when the resident DBA is on a holiday / vacation so you can guarantee an optimal work-life balance for your DBA
19. Flexible payment options – Wire, PayPal, Credit/Debit Card, Cheque and Cash
20. Transparent ticketing system – We share with you the detailed work report of what we have done for your database infrastructure, This also includes how you will get benefitted with change we have done. We love absolute transparency and detailed documentation

Conclusion

We are living on an Datanomy world, The data powers business and Database Infrastructure going unavailable will directly impact customers, employees and investors, We at MinervaDB is committed to building and delivering optimal, scalable, highly available,  fault-tolerant, reliable and secured database infrastructure operations for our customers globally. To know more on how MinervaDB creates value for your Database Infrastructure Operations, Please download our corporate flyer PDF here.

The post 2020 New Year Resolution with MinervaDB appeared first on The WebScale Database Infrastructure Operations Experts.

The post Why engage MinervaDB for web-scale DBA consulting and remote DBA services ? appeared first on The WebScale Database Infrastructure Operations Experts.