Submarine: A subscription‐based data streaming framework for integrating large facilities and advanced cyberinfrastructure
AR Zamani, M AbdelBaky… - Concurrency and …, 2020 - Wiley Online Library
Concurrency and Computation: Practice and Experience, 2020•Wiley Online Library
Large scientific facilities provide researchers with instrumentation, data, and data products
that can accelerate scientific discovery. However, increasing data volumes coupled with
limited local computational power prevents researchers from taking full advantage of what
these facilities can offer. Many researchers looked into using commercial and academic
cyberinfrastructure (CI) to process these data. Nevertheless, there remains a disconnect
between large facilities and CI that requires researchers to be actively part of the data …
that can accelerate scientific discovery. However, increasing data volumes coupled with
limited local computational power prevents researchers from taking full advantage of what
these facilities can offer. Many researchers looked into using commercial and academic
cyberinfrastructure (CI) to process these data. Nevertheless, there remains a disconnect
between large facilities and CI that requires researchers to be actively part of the data …
Summary
Large scientific facilities provide researchers with instrumentation, data, and data products that can accelerate scientific discovery. However, increasing data volumes coupled with limited local computational power prevents researchers from taking full advantage of what these facilities can offer. Many researchers looked into using commercial and academic cyberinfrastructure (CI) to process these data. Nevertheless, there remains a disconnect between large facilities and CI that requires researchers to be actively part of the data processing cycle. The increasing complexity of CI and data scale necessitates new data delivery models, those that can autonomously integrate large‐scale scientific facilities and CI to deliver real‐time data and insights. In this paper, we present our initial efforts using the Ocean Observatories Initiative project as a use case. In particular, we present a subscription‐based data streaming service for data delivery that leverages the Apache Kafka data streaming platform. We also show how our solution can automatically integrate large‐scale facilities with CI services for automated data processing.
Wiley Online Library
以上显示的是最相近的搜索结果。 查看全部搜索结果