Is it necessary to configure large write buffers for good performance of X-Engine? about polardbx-engine HOT 4 OPEN

In the end, it's up to the users to configure these numbers according to their workloads. For the commercial version of X-Engine, there are recommended parameters available in the Alibaba Cloud Database Console, which are selected to fit the environments in Alibaba Cloud.

Using small memtables does not prevent your X-Engine (or RocksDB) from being bounded by compactions if your workload is very write-intensive over a long period of time. A very large main memory may delay flushes. But, in the end, all new data needs to be flushed and compacted.

If you want to compare X-Engine with RocksDB using a benchmark, you should consider setting their parameters to the same values for a fair comparison.

A rich literature on configuring LSM-tree systems could be found here: https://disc.bu.edu/publications.

Recovery time is only an issue when the system does crash. X-Engine has parallel recovery in place to accelerate this process. If you opt for fast recoveries, you could consider configuring the system with more aggressive flushes to reduce the scale of volatile states in the main memory. With X-Engine, we also have a persistent memory system that achieves very fast recoveries (paper: http://www.vldb.org/pvldb/vol14/p1872-yan.pdf).

from polardbx-engine.

this document will tell you how to run x-engine with correct configuration.

[X-Engin Configuration](https://github.com/ApsaraDB/galaxyengine/wiki/2.1-MySQL-X-Engine%E5%BC%95%E6%93%8E%E5%8F%82%E6%95%B0%E9%85%8D%E7%BD%AE%E5%BB%BA%E8%AE%AE#x-engine%E5%86%85%E5%AD%98%E5%8F%8A%E7%BA%BF%E7%A8%8B%E5%8F%82%E6%95%B0）

i usually run pt_beilou.sh on machine with 96cpu / 768g memory. so it may not be a suitable configuration for you

from polardbx-engine.

Thanks a lot @chengxuntao-ntu, @luckywhu. Your information is really helpful!

I studied your paper before. My work at Intel is related to Persistent Memory too.

It's a great idea to use Persistent Memory in X-Engine, achieving better performance, faster recovery and less DRAM cost.

from polardbx-engine.

Gald to know we have readers out there! With PM buffering or caching most of the main-memory data, the recovery could be made lightning fast. Faster than high-availability switches (switch the hot backup to be the new master, when the old master crashes). This could potentially remove the number of hot backup nodes in database clusters, by up to 50%. But this kind of new design is a bit too aggressive. We are not expecting any real deployment soon.

Regarding memtable sizes, ideally, one could dynamically adjust them in response to changing workload pressures and types. We have observed in real production environments in the cloud that there are only so few hours within a day that database clusters have to process significant transactions (writes). Most of the time, there aren't many writes. So, caches would be more useful than memtables.

from polardbx-engine.