We explore the trade-offs of performing linear algebra using Apache Spark, compared to
traditional C and MPI implementations on HPC platforms. Spark is designed for data …

User-Defined Functions (UDF) allow application programmers to specify analysis operations
on data, while leaving the data management tasks to the system. This general approach …

Climate observations and model simulations produce vast amounts of data. The
unprecedented data volume and the complexity of geospatial statistics and analysis requires …

The WARC file format is widely used by web archives to preserve collected web content for
future use. With the rapid growth of web archives and the increasing interest to reuse these …

A full High Energy Physics (HEP) data analysis is divided into multiple data reduction
phases. Processing within these phases is extremely time consuming, therefore …

Distributed data processing ecosystems are widespread and their components are highly
specialized, such that efficient interoperability is urgent. Recently, Apache Arrow was …

Modern High Performance Computing (HPC) applications, such as Earth science
simulations, produce large amounts of data due to the surging of computing power, while big …

With the advent of advances in imaging and computing technologies, large-scale data
acquisition and processing have become commonplace in many science and engineering …

We investigate the performance of Apache Spark, a cluster computing framework, for
analyzing data from future LSST-like galaxy surveys. Apache Spark attempts to address big …

Processor architectures have reached a physical boundary that prevents scaling
performance with the number of transistors. Effectively, this means that the sequential …