OctopusFS: A distributed file system with tiered storage management
E Kakoulli, H Herodotou - Proceedings of the 2017 acm international …, 2017 - dl.acm.org
Proceedings of the 2017 acm international conference on management of data, 2017•dl.acm.org
The ever-growing data storage and I/O demands of modern large-scale data analytics are
challenging the current distributed storage systems. A promising trend is to exploit the recent
improvements in memory, storage media, and networks for sustaining high performance and
low cost. While past work explores using memory or SSDs as local storage or combine local
with network-attached storage in cluster computing, this work focuses on managing multiple
storage tiers in a distributed setting. We present OctopusFS, a novel distributed file system …
challenging the current distributed storage systems. A promising trend is to exploit the recent
improvements in memory, storage media, and networks for sustaining high performance and
low cost. While past work explores using memory or SSDs as local storage or combine local
with network-attached storage in cluster computing, this work focuses on managing multiple
storage tiers in a distributed setting. We present OctopusFS, a novel distributed file system …
The ever-growing data storage and I/O demands of modern large-scale data analytics are challenging the current distributed storage systems. A promising trend is to exploit the recent improvements in memory, storage media, and networks for sustaining high performance and low cost. While past work explores using memory or SSDs as local storage or combine local with network-attached storage in cluster computing, this work focuses on managing multiple storage tiers in a distributed setting. We present OctopusFS, a novel distributed file system that is aware of heterogeneous storage media (e.g., memory, SSDs, HDDs, NAS) with different capacities and performance characteristics. The system offers a variety of pluggable policies for automating data management across the storage tiers and cluster nodes. The policies employ multi-objective optimization techniques for making intelligent data management decisions based on the requirements of fault tolerance, data and load balancing, and throughput maximization. At the same time, the storage media are explicitly exposed to users and applications, allowing them to choose the distribution and placement of replicas in the cluster based on their own performance and fault tolerance requirements. Our extensive evaluation shows the immediate benefits of using OctopusFS with data-intensive processing systems, such as Hadoop and Spark, in terms of both increased performance and better cluster utilization.
ACM Digital Library
以上显示的是最相近的搜索结果。 查看全部搜索结果