🌐 MET ASCII2NC#

Demonstrates the following:

Get a timeseries of data

Modify and writing DataFrame contents using Polars to a specific format required by MET.

The Model Evaluation Tools (MET) software is used by the NOAA, NRL, BoM, and other meteorological institutions to verify numerical weather prediction forecasts. You can use your own data to verify forecast grids, but you need to convert your data to an ASCII file. Then, MET’s ASCII2NC tool can convert that file into a NetCDF that MET can read in.

This notebook demonstrates how to use Polars to convert a SynopticPy DataFrame to a file that MET can read.

WARNING: I haven’t actually tested that the file it writes can be used by MET’s ASCII2NC tool. This is primarily a proof of concept. Please open a PR if you want to see this feature improved and tested.

[9]:

import synoptic
import polars as pl

SynopticPy comes with an easy function to write a DataFrame to MET’s ASCII2NC format for you

[10]:

df = synoptic.TimeSeries(stid="ubkbk,wbb", recent=60).df()
df.synoptic.write_met("sample_ascii_for_met.txt")

🚚💨 Speedy delivery from Synoptic's timeseries service.
📦 Received data from 1 stations (0.31 seconds).

/home/blaylock/GITHUB/SynopticPy/src/synoptic/polars_namespace.py:37: UserWarning: `write_met` is experimental and proof of concept. NEEDS TESTING WIT MET's ASCII2NC tool.
  warnings.warn(

Now let’s look at what that file we wrote looks like…

[11]:

%%bash
head sample_ascii_for_met.txt

MESONET WBB 20241222_033100 40.76623 -111.84755 1464.8688 TMP NA NA passed 1.978
MESONET WBB 20241222_033200 40.76623 -111.84755 1464.8688 TMP NA NA passed 1.972
MESONET WBB 20241222_033300 40.76623 -111.84755 1464.8688 TMP NA NA passed 2.028
MESONET WBB 20241222_033400 40.76623 -111.84755 1464.8688 TMP NA NA passed 2.139
MESONET WBB 20241222_033500 40.76623 -111.84755 1464.8688 TMP NA NA passed 2.222
MESONET WBB 20241222_033600 40.76623 -111.84755 1464.8688 TMP NA NA passed 2.283
MESONET WBB 20241222_033700 40.76623 -111.84755 1464.8688 TMP NA NA passed 2.289
MESONET WBB 20241222_033800 40.76623 -111.84755 1464.8688 TMP NA NA passed 2.344
MESONET WBB 20241222_033900 40.76623 -111.84755 1464.8688 TMP NA NA passed 2.361
MESONET WBB 20241222_034000 40.76623 -111.84755 1464.8688 TMP NA NA passed 2.333

That’s nice, but you might wonder how it works. You can look at the source code, or I’ll just show you below the most important parts…

[12]:

# Get DataFrom Synoptic
df = synoptic.TimeSeries(stid="ubkbk,wbb", recent=60).df()
df

🚚💨 Speedy delivery from Synoptic's timeseries service.
📦 Received data from 1 stations (0.22 seconds).

[12]:

shape: (1_155, 23)

stid	date_time	variable	sensor_index	is_derived	value	value_sting	units	id	name	elevation	latitude	longitude	mnet_id	state	timezone	elev_dem	period_of_record_start	period_of_record_end	qc_flagged	is_restricted	restricted_metadata	is_active
str	datetime[μs, UTC]	str	u32	bool	f64	str	str	u32	str	f64	f64	f64	u32	str	str	f64	datetime[μs, UTC]	datetime[μs, UTC]	bool	bool	bool	bool
"WBB"	2024-12-22 03:31:00 UTC	"air_temp"	1	false	1.978	null	"Celsius"	1	"U of U William Browning Buildi…	4806.0	40.76623	-111.84755	153	"UT"	"America/Denver"	4727.7	1997-01-01 00:00:00 UTC	2024-12-22 04:05:00 UTC	false	false	false	true
"WBB"	2024-12-22 03:32:00 UTC	"air_temp"	1	false	1.972	null	"Celsius"	1	"U of U William Browning Buildi…	4806.0	40.76623	-111.84755	153	"UT"	"America/Denver"	4727.7	1997-01-01 00:00:00 UTC	2024-12-22 04:05:00 UTC	false	false	false	true
"WBB"	2024-12-22 03:33:00 UTC	"air_temp"	1	false	2.028	null	"Celsius"	1	"U of U William Browning Buildi…	4806.0	40.76623	-111.84755	153	"UT"	"America/Denver"	4727.7	1997-01-01 00:00:00 UTC	2024-12-22 04:05:00 UTC	false	false	false	true
"WBB"	2024-12-22 03:34:00 UTC	"air_temp"	1	false	2.139	null	"Celsius"	1	"U of U William Browning Buildi…	4806.0	40.76623	-111.84755	153	"UT"	"America/Denver"	4727.7	1997-01-01 00:00:00 UTC	2024-12-22 04:05:00 UTC	false	false	false	true
"WBB"	2024-12-22 03:35:00 UTC	"air_temp"	1	false	2.222	null	"Celsius"	1	"U of U William Browning Buildi…	4806.0	40.76623	-111.84755	153	"UT"	"America/Denver"	4727.7	1997-01-01 00:00:00 UTC	2024-12-22 04:05:00 UTC	false	false	false	true
…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…
"WBB"	2024-12-22 04:21:00 UTC	"wind_cardinal_direction"	1	true	null	"N"	"wind_cardinal_direction"	1	"U of U William Browning Buildi…	4806.0	40.76623	-111.84755	153	"UT"	"America/Denver"	4727.7	1997-01-01 00:00:00 UTC	2024-12-22 04:05:00 UTC	false	false	false	true
"WBB"	2024-12-22 04:22:00 UTC	"wind_cardinal_direction"	1	true	null	"NNE"	"wind_cardinal_direction"	1	"U of U William Browning Buildi…	4806.0	40.76623	-111.84755	153	"UT"	"America/Denver"	4727.7	1997-01-01 00:00:00 UTC	2024-12-22 04:05:00 UTC	false	false	false	true
"WBB"	2024-12-22 04:23:00 UTC	"wind_cardinal_direction"	1	true	null	"NE"	"wind_cardinal_direction"	1	"U of U William Browning Buildi…	4806.0	40.76623	-111.84755	153	"UT"	"America/Denver"	4727.7	1997-01-01 00:00:00 UTC	2024-12-22 04:05:00 UTC	false	false	false	true
"WBB"	2024-12-22 04:24:00 UTC	"wind_cardinal_direction"	1	true	null	"NE"	"wind_cardinal_direction"	1	"U of U William Browning Buildi…	4806.0	40.76623	-111.84755	153	"UT"	"America/Denver"	4727.7	1997-01-01 00:00:00 UTC	2024-12-22 04:05:00 UTC	false	false	false	true
"WBB"	2024-12-22 04:25:00 UTC	"wind_cardinal_direction"	1	true	null	"NE"	"wind_cardinal_direction"	1	"U of U William Browning Buildi…	4806.0	40.76623	-111.84755	153	"UT"	"America/Denver"	4727.7	1997-01-01 00:00:00 UTC	2024-12-22 04:05:00 UTC	false	false	false	true

[13]:

# Let's not consider any rows where the observed value is None
met = df.filter(~pl.col("value").is_null())

# MET expects the data to be in 11 columns. This just requires some renaming.
# Reference: https://met.readthedocs.io/en/latest/Users_Guide/reformat_point.html#ascii2nc-tool
met = met.select(
    pl.lit("MESONET").alias("Message_Type"),
    pl.col("stid").alias("Station_ID"),
    pl.col("date_time").dt.strftime("%Y%m%d_%H%M%S").alias("Valid_Time"),
    pl.col("latitude").alias("Lat"),
    pl.col("longitude").alias("Lon"),
    pl.col("elevation").alias("Elevation") * 0.3048,  # feet to meters
    pl.col("variable").alias("Variable_Name"),
    pl.lit(None).alias("Level"),
    pl.lit(None).alias("Height"),
    pl.when(pl.col("qc_flagged"))
    .then(pl.lit("flagged"))
    .otherwise(pl.lit("passed"))
    .alias("QC_String"),
    pl.col("value").alias("Observation_Value"),
)

# Now let's replace Synoptic's variable name with the GRIB short name
# TODO: List is incomplete
met = met.with_columns(
    pl.col("Variable_Name").replace(
        {
            "air_temp": "TMP",
            "relative_humidity": "RH",
            "dew_point_temperature": "DPT",
            "wind_speed": "WIND",
            "wind_direction": "WDIR",
            "sea_level_pressure": "PRMSL",
            "pressure": "PRES",
        }
    )
)

# Let's see what we have now
met

[13]:

shape: (790, 11)

Message_Type	Station_ID	Valid_Time	Lat	Lon	Elevation	Variable_Name	Level	Height	QC_String	Observation_Value
str	str	str	f64	f64	f64	str	null	null	str	f64
"MESONET"	"WBB"	"20241222_033100"	40.76623	-111.84755	1464.8688	"TMP"	null	null	"passed"	1.978
"MESONET"	"WBB"	"20241222_033200"	40.76623	-111.84755	1464.8688	"TMP"	null	null	"passed"	1.972
"MESONET"	"WBB"	"20241222_033300"	40.76623	-111.84755	1464.8688	"TMP"	null	null	"passed"	2.028
"MESONET"	"WBB"	"20241222_033400"	40.76623	-111.84755	1464.8688	"TMP"	null	null	"passed"	2.139
"MESONET"	"WBB"	"20241222_033500"	40.76623	-111.84755	1464.8688	"TMP"	null	null	"passed"	2.222
…	…	…	…	…	…	…	…	…	…	…
"MESONET"	"WBB"	"20241222_042100"	40.76623	-111.84755	1464.8688	"DPT"	null	null	"passed"	-1.48
"MESONET"	"WBB"	"20241222_042200"	40.76623	-111.84755	1464.8688	"DPT"	null	null	"passed"	-1.53
"MESONET"	"WBB"	"20241222_042300"	40.76623	-111.84755	1464.8688	"DPT"	null	null	"passed"	-1.54
"MESONET"	"WBB"	"20241222_042400"	40.76623	-111.84755	1464.8688	"DPT"	null	null	"passed"	-1.47
"MESONET"	"WBB"	"20241222_042500"	40.76623	-111.84755	1464.8688	"DPT"	null	null	"passed"	-1.38

[15]:

# And finally, rrite this to an ASCII file
# TODO: The file written is space-delimitated, not fixed with.
# TODO: Is that OK for MET? If not, need to use formatted np.savetxt.
met.with_columns(pl.all().cast(str)).fill_null("NA").write_csv(
    "sample_ascii_for_met2.txt",
    separator=" ",
    include_header=False,
)

🌐 MET ASCII2NC#

This Page