Adaptive code learning for spark configuration tuning
Configuration tuning is vital to optimize the performance of big data analysis platforms like
Spark. Existing methods (eg auto-tuning relational databases) are not effective for tuning …
Spark. Existing methods (eg auto-tuning relational databases) are not effective for tuning …
Locat: Low-overhead online configuration auto-tuning of spark sql applications
J Xin, K Hwang, Z Yu - … of the 2022 International Conference on …, 2022 - dl.acm.org
Spark SQL has been widely deployed in industry but it is challenging to tune its
performance. Recent studies try to employ machine learning (ML) to solve this problem, but …
performance. Recent studies try to employ machine learning (ML) to solve this problem, but …
Rover: An online Spark SQL tuning service via generalized transfer learning
Distributed data analytic engines like Spark are common choices to process massive data in
industry. However, the performance of Spark SQL highly depends on the choice of …
industry. However, the performance of Spark SQL highly depends on the choice of …
A novel reinforcement learning approach for spark configuration parameter optimization
X Huang, H Zhang, X Zhai - Sensors, 2022 - mdpi.com
Apache Spark is a popular open-source distributed data processing framework that can
efficiently process massive amounts of data. It provides more than 180 configuration …
efficiently process massive amounts of data. It provides more than 180 configuration …
Towards general and efficient online tuning for spark
The distributed data analytic system--Spark is a common choice for processing massive
volumes of heterogeneous data, while it is challenging to tune its parameters to achieve …
volumes of heterogeneous data, while it is challenging to tune its parameters to achieve …
Pets: Bottleneck-aware spark tuning with parameter ensembles
Spark tuning with its dozens of parameters for performance improvement is both a challenge
and time consuming effort. Current techniques rely on trial-and-error or best guess utilizing …
and time consuming effort. Current techniques rely on trial-and-error or best guess utilizing …
Dremel: Adaptive configuration tuning of rocksdb kv-store
LSM-tree-based key-value stores like RocksDB are widely used to support many
applications. However, configuring a RocksDB instance is challenging for the following …
applications. However, configuring a RocksDB instance is challenging for the following …
Speedup your analytics: Automatic parameter tuning for databases and big data systems
Database and big data analytics systems such as Hadoop and Spark have a large number
of configuration parameters that control memory distribution, I/O optimization, parallelism …
of configuration parameters that control memory distribution, I/O optimization, parallelism …
Tuneful: An online significance-aware configuration tuner for big data analytics
Distributed analytics engines such as Spark are a common choice for processing extremely
large datasets. However, finding good configurations for these systems remains challenging …
large datasets. However, finding good configurations for these systems remains challenging …
Towards automatic tuning of apache spark configuration
Apache Spark provides a large number of configuration settings that may be tuned to
improve the performance of specific applications running on the platform. However, it is non …
improve the performance of specific applications running on the platform. However, it is non …