Adaptive code learning for spark configuration tuning

C Lin, J Zhuang, J Feng, H Li… - 2022 IEEE 38th …, 2022 - ieeexplore.ieee.org
Configuration tuning is vital to optimize the performance of big data analysis platforms like
Spark. Existing methods (eg auto-tuning relational databases) are not effective for tuning …

Locat: Low-overhead online configuration auto-tuning of spark sql applications

J Xin, K Hwang, Z Yu - … of the 2022 International Conference on …, 2022 - dl.acm.org
Spark SQL has been widely deployed in industry but it is challenging to tune its
performance. Recent studies try to employ machine learning (ML) to solve this problem, but …

Rover: An online Spark SQL tuning service via generalized transfer learning

Y Shen, X Ren, Y Lu, H Jiang, H Xu, D Peng… - Proceedings of the 29th …, 2023 - dl.acm.org
Distributed data analytic engines like Spark are common choices to process massive data in
industry. However, the performance of Spark SQL highly depends on the choice of …

A novel reinforcement learning approach for spark configuration parameter optimization

X Huang, H Zhang, X Zhai - Sensors, 2022 - mdpi.com
Apache Spark is a popular open-source distributed data processing framework that can
efficiently process massive amounts of data. It provides more than 180 configuration …

Towards general and efficient online tuning for spark

Y Li, H Jiang, Y Shen, Y Fang, X Yang, D Huang… - arXiv preprint arXiv …, 2023 - arxiv.org
The distributed data analytic system--Spark is a common choice for processing massive
volumes of heterogeneous data, while it is challenging to tune its parameters to achieve …

Pets: Bottleneck-aware spark tuning with parameter ensembles

TBG Perez, W Chen, R Ji, L Liu… - 2018 27th International …, 2018 - ieeexplore.ieee.org
Spark tuning with its dozens of parameters for performance improvement is both a challenge
and time consuming effort. Current techniques rely on trial-and-error or best guess utilizing …

Dremel: Adaptive configuration tuning of rocksdb kv-store

C Zhao, T Chugh, J Min, M Liu… - Proceedings of the ACM …, 2022 - dl.acm.org
LSM-tree-based key-value stores like RocksDB are widely used to support many
applications. However, configuring a RocksDB instance is challenging for the following …

Speedup your analytics: Automatic parameter tuning for databases and big data systems

J Lu, Y Chen, H Herodotou, S Babu - Proceedings of the VLDB …, 2019 - dl.acm.org
Database and big data analytics systems such as Hadoop and Spark have a large number
of configuration parameters that control memory distribution, I/O optimization, parallelism …

Tuneful: An online significance-aware configuration tuner for big data analytics

A Fekry, L Carata, T Pasquier, A Rice… - arXiv preprint arXiv …, 2020 - arxiv.org
Distributed analytics engines such as Spark are a common choice for processing extremely
large datasets. However, finding good configurations for these systems remains challenging …

Towards automatic tuning of apache spark configuration

N Nguyen, MMH Khan, K Wang - 2018 IEEE 11th International …, 2018 - ieeexplore.ieee.org
Apache Spark provides a large number of configuration settings that may be tuned to
improve the performance of specific applications running on the platform. However, it is non …