💽 Archive Long Dataset#

SynopticPy doesn’t let you request more than a month of data. But what if we want all of WBB’s data for a year?

Make multiple API requests.
Write each month of data into it’s own Parquet file.
Read from Parquet files for fast loading!

[1]:

from datetime import datetime, timedelta
from synoptic import TimeSeries

import polars as pl

[2]:

# Can't request more than a month of data at a time...

df = TimeSeries(stid="wbb", recent=timedelta(days=365)).df()

🚚💨 Speedy delivery from Synoptic timeseries service.

---------------------------------------------------------------------------
SynopticAPIError                          Traceback (most recent call last)
Cell In[2], line 3
      1 # Can't request more than a month of data at a time...
----> 3 df = TimeSeries(stid="wbb", recent=timedelta(days=365)).df()

File ~/GITHUB/SynopticPy/synoptic/services.py:399, in TimeSeries.__init__(self, **params)
    398 def __init__(self, **params):
--> 399     super().__init__("timeseries", **params)

File ~/GITHUB/SynopticPy/synoptic/services.py:347, in SynopticAPI.__init__(self, service, token, verbose, **params)
    343 # -------------------
    344 # Check returned data
    345 # Note: SUMMARY is always returned in the JSON.
    346 if self.SUMMARY["RESPONSE_CODE"] != 1:
--> 347     raise SynopticAPIError(
    348         "\n"
    349         f"🛑 FATAL: Not a valid Synoptic API request.\n"
    350         f"  ├─ message: {self.SUMMARY['RESPONSE_MESSAGE']}\n"
    351         f"  └─ url: {self.response.url}\n"
    352         f"See {self.help_url} for help."
    353     )
    355 if self.verbose:
    356     print(
    357         f"📦 Received data from {self.SUMMARY.get('NUMBER_OF_OBJECTS'):,} stations."
    358     )

SynopticAPIError:
🛑 FATAL: Not a valid Synoptic API request.
  ├─ message: RECENT must be less than or equal to a month (43200 minutes).
  └─ url: https://api.synopticdata.com/v2/stations/timeseries?stid=wbb&recent=525600&token=0bbe0e9fda7945a68951cc1bdebb2b0d
See https://docs.synopticdata.com/services/weather-data-api for help.

[3]:

# Create month intervals for a year datetime range

dates = pl.datetime_range(
    datetime(2023, 1, 1),
    datetime(2024, 1, 1),
    interval="1mo",
    eager=True,
)
dates

[3]:

shape: (13,)

literal
datetime[μs]
2023-01-01 00:00:00
2023-02-01 00:00:00
2023-03-01 00:00:00
2023-04-01 00:00:00
2023-05-01 00:00:00
…
2023-09-01 00:00:00
2023-10-01 00:00:00
2023-11-01 00:00:00
2023-12-01 00:00:00
2024-01-01 00:00:00

[4]:

%%time

for i, (start, end) in enumerate(zip(dates, dates[1:]), start=1):
    print(f"Working on loop {i}: {start=}, {end=}", end="\r")
    df = TimeSeries(
        stid="wbb",
        vars="air_temp,relative_humidity,wind_speed,wind_direction",
        start=start,
        end=end - timedelta(microseconds=1),
        verbose=False,
    ).df()
    df.write_parquet(
        f"sample_long_timeseries_{start:%Y%m%d%H%M%S}_{end:%Y%m%d%H%M%S}.parquet"
    )

CPU times: user 7.7 s, sys: 909 ms, total: 8.61 s 12, 1, 0, 0), end=datetime.datetime(2024, 1, 1, 0, 0))
Wall time: 40.7 s

[5]:

%%time

# Read all parquet file; look how fast this is
wbb_2023 = pl.read_parquet("sample_long_timeseries_*.parquet")
wbb_2023

CPU times: user 799 ms, sys: 881 ms, total: 1.68 s
Wall time: 564 ms

[5]:

shape: (2_100_848, 20)

date_time	variable	sensor_index	is_derived	value	units	id	stid	name	elevation	latitude	longitude	mnet_id	state	timezone	elev_dem	period_of_record_start	period_of_record_end	is_restricted	is_active
datetime[μs, UTC]	str	u32	bool	f64	str	u32	str	str	f64	f64	f64	u32	str	str	f64	datetime[μs, UTC]	datetime[μs, UTC]	bool	bool
2023-01-01 00:00:00 UTC	"wind_direction"	1	false	56.97	"Degrees"	1	"WBB"	"U of U William Browning Buildi…	4806.0	40.76623	-111.84755	153	"UT"	"America/Denver"	4727.7	1997-01-01 00:00:00 UTC	2024-10-17 05:50:00 UTC	false	true
2023-01-01 00:01:00 UTC	"wind_direction"	1	false	58.74	"Degrees"	1	"WBB"	"U of U William Browning Buildi…	4806.0	40.76623	-111.84755	153	"UT"	"America/Denver"	4727.7	1997-01-01 00:00:00 UTC	2024-10-17 05:50:00 UTC	false	true
2023-01-01 00:02:00 UTC	"wind_direction"	1	false	60.34	"Degrees"	1	"WBB"	"U of U William Browning Buildi…	4806.0	40.76623	-111.84755	153	"UT"	"America/Denver"	4727.7	1997-01-01 00:00:00 UTC	2024-10-17 05:50:00 UTC	false	true
2023-01-01 00:03:00 UTC	"wind_direction"	1	false	57.05	"Degrees"	1	"WBB"	"U of U William Browning Buildi…	4806.0	40.76623	-111.84755	153	"UT"	"America/Denver"	4727.7	1997-01-01 00:00:00 UTC	2024-10-17 05:50:00 UTC	false	true
2023-01-01 00:04:00 UTC	"wind_direction"	1	false	61.8	"Degrees"	1	"WBB"	"U of U William Browning Buildi…	4806.0	40.76623	-111.84755	153	"UT"	"America/Denver"	4727.7	1997-01-01 00:00:00 UTC	2024-10-17 05:50:00 UTC	false	true
…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…
2023-12-31 23:55:00 UTC	"wind_speed"	1	false	0.761	"m/s"	1	"WBB"	"U of U William Browning Buildi…	4806.0	40.76623	-111.84755	153	"UT"	"America/Denver"	4727.7	1997-01-01 00:00:00 UTC	2024-10-17 05:50:00 UTC	false	true
2023-12-31 23:56:00 UTC	"wind_speed"	1	false	0.689	"m/s"	1	"WBB"	"U of U William Browning Buildi…	4806.0	40.76623	-111.84755	153	"UT"	"America/Denver"	4727.7	1997-01-01 00:00:00 UTC	2024-10-17 05:50:00 UTC	false	true
2023-12-31 23:57:00 UTC	"wind_speed"	1	false	0.628	"m/s"	1	"WBB"	"U of U William Browning Buildi…	4806.0	40.76623	-111.84755	153	"UT"	"America/Denver"	4727.7	1997-01-01 00:00:00 UTC	2024-10-17 05:50:00 UTC	false	true
2023-12-31 23:58:00 UTC	"wind_speed"	1	false	0.422	"m/s"	1	"WBB"	"U of U William Browning Buildi…	4806.0	40.76623	-111.84755	153	"UT"	"America/Denver"	4727.7	1997-01-01 00:00:00 UTC	2024-10-17 05:50:00 UTC	false	true
2023-12-31 23:59:00 UTC	"wind_speed"	1	false	0.54	"m/s"	1	"WBB"	"U of U William Browning Buildi…	4806.0	40.76623	-111.84755	153	"UT"	"America/Denver"	4727.7	1997-01-01 00:00:00 UTC	2024-10-17 05:50:00 UTC	false	true