The lack of reliable and readily accessible climate-related data has been a hindrance for financial sector stakeholders to properly assess financial stability and manage climate-related risks. This data gap substantially raises the barrier to channeling global capital flows towards climate change mitigation and resilience, by forcing businesses to engage in costly ad-hoc ingestion and curation efforts that cannot benefit from shared data or open protocols.
At the Open Source Climate (OS-Climate) initiative, we are building an open data science platform that supports complex data ingestion, processing and quality management requirements. We take advantage of the latest advances in open source data platform tools, machine learning, and the development of scenario-based predictive analytics by OS-Climate community members.
In order to build a data platform that is open, auditable, and supports durable and repeatable deployments, we leveraged the Operate First program. Operate First builds on the foundation of git-ops to extend open source principles with open deployments and operational knowledge. At OS-Climate, we manage the configuration of our data platform cluster and all of its deployments with GitHub issues and pull requests against the Operate First git repository.
In this talk, we will present the journey of the OS-Climate data commons, from unmanaged prototype cluster to a platform that is rigorously managed by Operate First. You will learn how our data platform is configured by GitHub issues and pull requests, at both the cluster level and at the level of individual application deployments. We will explain how the OS-Climate platform leverages open source tools including KubeFlow, Trino, Jupyter, Elyra pipelines and OpenShift to build maintainable and collaborative pipelines, and federate heterogeneous data sources into a controlled, common data resource for our community of climate data scientists and financial sector stakeholders. And finally, how the architecture relates to the rigorous Data Governance requirements of the financial sector and regulators on one side, Corporate CSOs who seek to operationalize transitional strategies to greater sustainability, while still inviting participation and innovation from the open source community.
Register to attend the conference:
https://hopin.com/events/devconf-cz-2022/
Video Stream