MySQL Consulting

Database Replication from MySQL to ClickHouse for High Performance WebScale Analytics

Shiv Iyer — Tue, 01 Oct 2019 19:12:51 +0000

MySQL to ClickHouse Replication

MySQL works great for Online Transaction Processing (OLTP) systems, MySQL performance degrades with analytical queries on very large database infrastructure, I agree you can optimize MySQL query performance with InnoDB compressions but why then combine OLTP and OLAP (Online Analytics Processing Systems) when you have columnar stores which can deliver high performance analytical queries more efficiently? I have seen several companies building dedicated MySQL servers for Analytics but over the period of time they end spending more money in fine tuning MySQL for Analytics with no significant improvements, There is no point in blaming MySQL for what it is not built for, MySQL / MariaDB is any day a bad choice for columnar analytics / big data solutions. Columnar database systems are best suited for handling large quantities of data: data stored in columns typically is easier to compress, it is also easier to access on per column basis – typically you ask for some data stored in a couple of columns – an ability to retrieve just those columns instead of reading all of the rows and filter out unneeded data makes the data accessed faster. So how can you combine the best of both ? Using MySQL purely for Transaction Processing Systems and Archiving MySQL Transactional Data for Data Analytics on a columnar store like ClickHouse, This post is about archiving and replicating data from MySQL to ClickHouse, you can continue reading from here.

Recommended Reading for ClickHouse:

Why we recommend ClickHouse over many other columnar database systems ? https://minervadb.com/index.php/2018/03/06/why-we-recommend-clickhouse-over-many-other-columnar-database-systems/
Benchmarking ClickHouse on my MacBook Pro (Super impressive performance) – https://minervadb.com/index.php/2018/01/23/benchmarking-clickhouse-2/
ClickHouse GitHub – https://github.com/ClickHouse
Altinity GitHub – https://github.com/Altinity
What you should know about RAID to improve Database Systems Performance – https://minervadb.com/index.php/2019/08/04/raid-redundant-storage-for-database-reliability/

The post Database Replication from MySQL to ClickHouse for High Performance WebScale Analytics appeared first on The WebScale Database Infrastructure Operations Experts.

Why we recommend ClickHouse over many other columnar database systems ?

MinervaDB Corporation — Tue, 06 Mar 2018 04:42:38 +0000

Compact data storage – Ten billions UInt8-type values should exactly consume 10GB uncompressed to efficiently use available CPU . Optimal storage even when uncompressed benefit performance and resource management . ClickHouse is built is store data efficiently without any garbage .
CPU efficient – Whenever possible, ClickHouse operations are dispatched on arrays, rather than on individual values. This is called “vectorized query execution,” and it helps lower the cost of actual data processing.
Data compression – ClickHouse supports two kinds of compression LZ4 and ZSTD . LZ4 is faster than ZSTD but compression ratio is smaller .ZSTD is faster and compress better than traditional Zlib but slower than LZ4 . We recommend customers LZ4 , when I/O is fast enough so decompression speed will become a bottleneck . When using super ultra fast disk subsystems you have an option to specify “none” compression . ZSTD is recommended when I/O is the bottleneck in queries with large range scans .
Can store data in disk – The columnar database systems like SAP HANA and Google PowerDrill can only work in the RAM .
Massively Parallel Processing – ClickHouse is capable of Massively Parallel Processing very large / complex SQL(s) optimally and cost efficiently
Built for web-scale data analytics – ClickHouse support sharding and distributed processing, This makes ClickHouse most preferred columnar database system for web-scale . Each shard in ClickHouse can be a group of replicas addressing maximum reliability and fault tolerance .
ClickHouse support Primary Key – ClickHouse permits real-time data updates with primary key (there will be no locking when adding data) . Data is sorted incrementally using the merge tree to perform queries on the range of primary key values .
Built for statistical analysis and support partial aggregation – ClickHouse is statistical query analysis ready columnar database store supporting aggregate functions for approximated calculation of the number of various values, medians, and quantiles. ClickHouse support aggregation for a limited number of random keys, instead for all the keys . You can query on a part (sample) of data and generate approximate result reducing disk I/O operations considerably .
Supports SQL – ClickHouse supports SQL, Subqueries are supported in FROM, IN, and JOIN clauses, as well as scalar subqueries. Dependent subqueries are not supported.
Supports data replication – ClickHouse supports asynchronous multi-master and master-slave replication .

The post Why we recommend ClickHouse over many other columnar database systems ? appeared first on The WebScale Database Infrastructure Operations Experts.

Join ClickHouse India Users Group

Shiv Iyer — Sun, 04 Feb 2018 17:19:03 +0000

We have created ClickHouse India Users Group for passionate ClickHouse professionals to discus about their favourite open source columnar database system . We have started ClickHouse India Users Group as an Facebook group page (https://www.facebook.com/groups/AICUG/) to connect with ClickHouse professionals worldwide better . As we grow, we will have an dedicated web property for ClickHouse India Users Group . This will be an open group forever and ClickHouse users can use this group for conferences, meet-ups, knowledge sharing, professional development and networking , Thanks for your support and welcome to ClickHouse India Users Group !

ClickHouse repositories and downloads

Github:

Server Packages:

Debian/Ubuntu: http://repo.yandex.ru/clickhouse/
RPMs: https://packagecloud.io/altinity/clickhouse
- Source: https://github.com/Altinity/clickhouse-rpm
Docker: https://hub.docker.com/r/yandex/clickhouse-server/

Client Packages:

The post Join ClickHouse India Users Group appeared first on The WebScale Database Infrastructure Operations Experts.