Did you try storing the data in specialized time series databases such as VictoriaMetrics or ClickHouse? These databases may provide higher compression ratio for the weather data stored on disk compared to custom schemes. They also provide query languages optimized for typical queries over time series data such as MetricsQL - https://docs.victoriametrics.com/metricsql/
Keep up the good work
Did you try storing the data in specialized time series databases such as VictoriaMetrics or ClickHouse? These databases may provide higher compression ratio for the weather data stored on disk compared to custom schemes. They also provide query languages optimized for typical queries over time series data such as MetricsQL - https://docs.victoriametrics.com/metricsql/
See, for example, a benchmark for ingesting 500 billion of samples into VictoriaMetrics - https://valyala.medium.com/billy-how-victoriametrics-deals-with-more-than-500-billion-rows-e82ff8f725da
Hi,
Truly good work !
A question : the data we obtain from 2017 onwards are from ECMWF IFS, ERA5 datasets or a combination of both ? Thank you