sun, 03-dec-2023, 11:22

Introduction

Yesterday Richard James posted about “hythergraphs”, which he’d seen on Toolik Field Station’s web site.

Hythergraphs show monthly weather parameters for an entire year, plotting temperature against precipitation (or other paired climate variables) against each other for each month of the year, drawing a line from month to month. When contrasting one climate record against another (historic vs. contemporary, one station against another), the differences stand out.

I was curious to see how easy it would be to produce one with R and ggplot.

Data

I’ll produce hythergraphs, one that compares Fairbanks Airport data against the data collected at our station on Goldstream Creek for the period of record for our station (2011‒2022) and one that compares the Fairbanks Airport station data from 1951‒2000 against data from 2001‒2022 (similar to what Richard did).

I’m using the following R packages:

library(tidyverse)
library(RPostgres)
library(lubridate)
library(scales)

I’ll skip the part where I pull the data from the GHCND database. What we need is a table of observations that look like this. We’ve got a categorical column (station_name), a date column, and the two climate variables we’re going to plot:

# A tibble: 30,072 × 4
   station_name     dte         PRCP   TAVG
   <chr>            <date>     <dbl>  <dbl>
   1 GOLDSTREAM CREEK 2011-04-01   0   -17.5
   2 GOLDSTREAM CREEK 2011-04-02   0   -15.6
   3 GOLDSTREAM CREEK 2011-04-03   0    -8.1
   4 GOLDSTREAM CREEK 2011-04-04   0    -5
   5 GOLDSTREAM CREEK 2011-04-05   0    -5
   6 GOLDSTREAM CREEK 2011-04-06   0.5  -3.9
   7 GOLDSTREAM CREEK 2011-04-07   0    -8.3
   8 GOLDSTREAM CREEK 2011-04-08   2    -5.85
   9 GOLDSTREAM CREEK 2011-04-09   0.5  -1.65
  10 GOLDSTREAM CREEK 2011-04-10   0    -4.45
# ℹ 30,062 more rows

From that raw data, we’ll aggregate to year and month, calculating the montly precipitation sum and mean average temperature, then aggregate to station and month, calculating the mean monthly precipitation and temperature.

The final step adds the necessary aesthetics to produce the plot using ggplot. We’ll draw the monthly scatterplot values using the first letter of the month, calculated using month_label = substring(month.name[month], 1, 1) below. To draw the lines from one month to the next we use geom_segement and calculate the ends of each segment by setting xend and yend to the next row’s value from the table.

One flaw in this approach is that there’s no line between December and January because there is no “next” value in the data frame. This could be fixed by seperately finding the January positions, then passing those to lead() as the default value (which is normally NA).

airport_goldstream <- pivot |>
   filter(dte >= "2010-04-01") |>
   # get monthly precip total, mean temp
   mutate(
      year = year(dte),
      month = month(dte)
   ) |>
   group_by(station_name, year, month) |>
   summarize(
      sum_prcp_in = sum(PRCP, na.rm = TRUE) / 25.4,
      mean_tavg_f = mean(TAVG, na.rm = TRUE) * 9 / 5.0 + 32,
      .groups = "drop"
   ) |>
   # get monthy means for each station
   group_by(station_name, month) |>
   summarize(
      mean_prcp_in = mean(sum_prcp_in),
      mean_tavg_f = mean(mean_tavg_f),
      .groups = "drop"
   ) |>
   # add month label, line segment ends
   arrange(station_name, month) |>
   group_by(station_name) |>
   mutate(
      month_label = substring(month.name[month], 1, 1),
      xend = lead(mean_prcp_in),
      yend = lead(mean_tavg_f)
   )

Here’s what that data frame looks like:

# A tibble: 24 × 7
# Groups:   station_name [2]
   station_name      month mean_prcp_in mean_tavg_f month_label  xend   yend
   <chr>             <dbl>        <dbl>       <dbl> <chr>       <dbl>  <dbl>
   1 FAIRBANKS INTL AP     1        0.635      -6.84  J           0.988 -0.213
   2 FAIRBANKS INTL AP     2        0.988      -0.213 F           0.635 11.5
   3 FAIRBANKS INTL AP     3        0.635      11.5   M           0.498 33.1
   4 FAIRBANKS INTL AP     4        0.498      33.1   A           0.670 51.2
   5 FAIRBANKS INTL AP     5        0.670      51.2   M           1.79  61.3
   6 FAIRBANKS INTL AP     6        1.79       61.3   J           2.41  63.1
   7 FAIRBANKS INTL AP     7        2.41       63.1   J           2.59  57.9
   8 FAIRBANKS INTL AP     8        2.59       57.9   A           1.66  46.5
   9 FAIRBANKS INTL AP     9        1.66       46.5   S           1.04  29.5
  10 FAIRBANKS INTL AP    10        1.04       29.5   O           1.16   5.21
# ℹ 14 more rows

Plots

Here’s the code to produce the plot. The month labels are displayed using geom_label, and the lines between months are generated from geom_segment.

airport_v_gsc <- ggplot(
   data = airport_goldstream,
   aes(x = mean_prcp_in, y = mean_tavg_f, color = station_name)
) +
   theme_bw() +
   geom_segment(aes(xend = xend, yend = yend, color = station_name)) +
   geom_label(aes(label = month_label), show.legend = FALSE) +
   scale_x_continuous(
      name = "Monthly Average Precipitation (inches liquid)",
      breaks = pretty_breaks(n = 10)
   ) +
   scale_y_continuous(
      name = "Monthly Average Tempearature (°F)",
      breaks = pretty_breaks(n = 10)
   ) +
   scale_color_manual(
      name = "Station",
      values = c("darkorange", "darkcyan")
   ) +
   theme(
      legend.position = c(0.8, 0.2),
      legend.background = element_rect(
      fill = "white", linetype = "solid", color = "grey80", size = 0.5
      )
   ) +
   labs(
      title = "Monthly temperature and precipitation",
      subtitle = "Fairbanks Airport and Goldstream Creek Stations, 2011‒2022"
   )
Fairbanks Airport, Goldstream Creek Hythergraph

You can see from the plot that we are consistently colder than the airport, curiously more dramatically in the summer than winter. The airport gets slighly more precipitation in winter, but our summer precipitation is significantly higher, especially in August.

The standard plot to display this information would be two bar charts with one plot showing the monthly mean temperature for each station, and a second plot showing precipitation. The advantage of such a display is that the differences would be more clear, and the bars could include standard errors (or standard deviation) that would help provide an idea of whether the differences between stations are statistically significant or not.

For example (the lines above the bars are one standard deviation above or below the mean):

Fairbanks Airport, Goldstream Creek Bar Chart

In this plot of the same data, you can tell from the standard deviation lines that the precipitation differences between stations are probably not significant, but the cooler summer temperatures at Goldstrem Creek may be.

If we calculate the standard deviations of the monthly means, we can use geom_tile to draw significance boxes around each monthly value in the hytherplot as Richard suggests in his post. Here’s the ggplot geom to do that:

geom_tile(
  aes(width = 2*sd_prcp_in, height = 2*sd_tavg_f, fill = station_name),
  show.legend = FALSE, alpha = 0.25
) +

And the updated plot:

Fairbanks Airport, Goldstream Creek

This clearly shows the large variation in precipitation, and if you carefully compare the boxes for a particular month, you can draw concusions similar to what is made fairly clear in the bar charts. For example, if we focus on August, you can see that the Goldstream Creek precipitation box clearly overlaps that of the airport station, but the temperature ranges do not overlap, suggesting that August temperatures are cooler at Goldstream Creek but that while precipitation is much higher, it’s not statistically significant.

Airport station, different time periods

Here’s the plot for the airport station that is similar to the plot Richard created (I used different time periods).

Fairbanks Airport Hythergraph

This plot demonstrates that while temperatures have increased in the last two decades, it’s the differences in the pattern of precipitation that stands out, with July and August precipitation much larger in the last 20 years. It’s also curious that February and April precipitation is higher, but the differences are smaller in the other winter months. This is a case where some sense of the distribution of the values would be useful.

tags: R  weather  ggplot 
sat, 25-apr-2015, 10:21

Introduction

One of the best sources of weather data in the United States comes from the National Weather Service's Cooperative Observer Network (COOP), which is available from NCDC. It's daily data, collected by volunteers at more than 10,000 locations. We participate in this program at our house (station id DW1454 / GHCND:USC00503368), collecting daily minimum and maximum temperature, liquid precipitation, snowfall and snow depth. We also collect river heights for Goldstream Creek as part of the Alaska Pacific River Forecast Center (station GSCA2). Traditionally, daily temperature measurements were collecting using a minimum maximum thermometer, which meant that the only way to calculate average daily temperature was by averaging the minimum and maximum temperature. Even though COOP observers typically have an electronic instrument that could calculate average daily temperature from continuous observations, the daily minimum and maximum data is still what is reported.

In an earlier post we looked at methods used to calculate average daily temperature, and if there are any biases present in the way the National Weather Service calculates this using the average of the minimum and maximum daily temperature. We looked at five years of data collected at my house every five minutes, comparing the average of these temperatures against the average of the daily minimum and maximum. Here, we will be repeating this analysis using data from the Climate Reference Network stations in the United States.

The US Climate Reference Network is a collection of 132 weather stations that are properly sited, maintained, and include multiple redundant measures of temperature and precipitation. Data is available from http://www1.ncdc.noaa.gov/pub/data/uscrn/products/ and includes monthly, daily, and hourly statistics, and sub-hourly (5-minute) observations. We’ll be focusing on the sub-hourly data, since it closely matches the data collected at my weather station.

A similar analysis using daily and hourly CRN data appears here.

Getting the raw data

I downloaded all the data using the following Unix commands:

$ wget http://www1.ncdc.noaa.gov/pub/data/uscrn/products/stations.tsv
$ wget -np -m http://www1.ncdc.noaa.gov/pub/data/uscrn/products/subhourly01/
$ find www1.ncdc.noaa.gov/ -type f -name 'CRN*.txt' -exec gzip {} \;

The code to insert all of this data into a database can be found here. Once inserted, I have a table named crn_stations that has the station data, and one named crn_subhourly with the five minute observation data.

Methods

Once again, we’ll use R to read the data, process it, and produce plots.

Libraries

Load the libraries we need:

library(dplyr)
library(lubridate)
library(ggplot2)
library(scales)
library(grid)

Connect to the database and load the data tables.

noaa_db <- src_postgres(dbname="noaa", host="mason")

crn_stations <- tbl(noaa_db, "crn_stations") %>%
    collect()

crn_subhourly <- tbl(noaa_db, "crn_subhourly")

Remove observations without temperature data, group by station and date, calculate average daily temperature using the two methods, remove any daily data without a full set of data, and collect the results into an R data frame. This looks very similar to the code used to analyze the data from my weather station.

crn_daily <-
    crn_subhourly %>%
        filter(!is.na(air_temperature)) %>%
        mutate(date=date(timestamp)) %>%
        group_by(wbanno, date) %>%
        summarize(t_mean=mean(air_temperature),
                  t_minmax_avg=(min(air_temperature)+
                                max(air_temperature))/2.0,
                  n=n()) %>%
        filter(n==24*12) %>%
        mutate(anomaly=t_minmax_avg-t_mean) %>%
        select(wbanno, date, t_mean, t_minmax_avg, anomaly) %>%
        collect()

The two types of daily average temperatures are calculated in this step:

summarize(t_mean=mean(air_temperature),
            t_minmax_avg=(min(air_temperature)+
                        max(air_temperature))/2.0)

Where t_mean is the value calculated from all 288 five minute observations, and t_minmax_avg is the value from the daily minimum and maximum.

Now we join the observation data with the station data. This attaches station information such as the name and latitude of the station to each record.

crn_daily_stations <-
    crn_daily %>%
        inner_join(crn_stations, by="wbanno") %>%
        select(wbanno, date, state, location, latitude, longitude,
               t_mean, t_minmax_avg, anomaly)

Finally, save the data so we don’t have to do these steps again.

save(crn_daily_stations, file="crn_daily_averages.rdata")

Results

Here are the overall results of the analysis.

summary(crn_daily_stations$anomaly)
##     Min.  1st Qu.   Median     Mean  3rd Qu.     Max.
## -11.9000  -0.1028   0.4441   0.4641   1.0190  10.7900

The average anomaly across all stations and all dates is 0.44 degrees Celsius (0.79 degrees Farenheit). That’s a pretty significant error. Half the data is between −0.1 and 1.0°C (−0.23 and +1.8°F) and the full range is −11.9 to +10.8°C (−21.4 to +19.4°F).

Plots

Let’s look at some plots.

Raw data by latitude

To start, we’ll look at all the anomalies by station latitude. The plot only shows one percent of the actual anomalies because plotting 512,460 points would take a long time and the general pattern is clear from the reduced data set.

set.seed(43)
p <- ggplot(data=crn_daily_stations %>% sample_frac(0.01),
            aes(x=latitude, y=anomaly)) +
    geom_point(position="jitter", alpha="0.2") +
    geom_smooth(method="lm", se=FALSE) +
    theme_bw() +
    scale_x_continuous(name="Station latitude", breaks=pretty_breaks(n=10)) +
    scale_y_continuous(name="Temperature anomaly (degrees C)",
                       breaks=pretty_breaks(n=10))

print(p)
//media.swingleydev.com/img/blog/2015/04/crn_minmax_anomaly_scatterplot.svg

The clouds of points show the differences between the min/max daily average and the actual daily average temperature, where numbers above zero represent cases where the min/max calculation overestimates daily average temperature. The blue line is the fit of a linear model relating latitude with temperature anomaly. We can see that the anomaly is always positive, averaging around half a degree at lower latitudes and drops somewhat as we proceed northward. You also get a sense from the actual data of how variable the anomaly is, and at what latitudes most of the stations are found.

Here are the regression results:

summary(lm(anomaly ~ latitude, data=crn_daily_stations))
##
## Call:
## lm(formula = anomaly ~ latitude, data = crn_daily_stations)
##
## Residuals:
##      Min       1Q   Median       3Q      Max
## -12.3738  -0.5625  -0.0199   0.5499  10.3485
##
## Coefficients:
##               Estimate Std. Error t value Pr(>|t|)
## (Intercept)  0.7403021  0.0070381  105.19   <2e-16 ***
## latitude    -0.0071276  0.0001783  -39.98   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.9632 on 512458 degrees of freedom
## Multiple R-squared:  0.00311,    Adjusted R-squared:  0.003108
## F-statistic:  1599 on 1 and 512458 DF,  p-value: < 2.2e-16

The overall model and coefficients are highly significant, and show a slight decrease in the positive anomaly as we move farther north. Perhaps this is part of the reason why the analysis of my station (at a latitude of 64.89) showed an average anomaly close to zero (−0.07°C / −0.13°F).

Anomalies by month and latitude

One of the results of our earlier analysis was a seasonal pattern in the anomalies at our station. Since we also know there is a latitudinal pattern, in the data, let’s combine the two, plotting anomaly by month, and faceting by latitude.

Station latitude are binned into groups for plotting, and the plots themselves show the range that cover half of all anomalies for that latitude category × month. Including the full range of anomalies in each group tends to obscure the overall pattern, and the plot of the raw data didn’t show an obvious skew to the rarer anomalies.

Here’s how we set up the data frames for the plot.

crn_daily_by_month <-
    crn_daily_stations %>%
        mutate(month=month(date),
               lat_bin=factor(ifelse(latitude<30, '<30',
                                     ifelse(latitude>60, '>60',
                                            paste(floor(latitude/10)*10,
                                                  (floor(latitude/10)+1)*10,
                                                  sep='-'))),
                              levels=c('<30', '30-40', '40-50',
                                       '50-60', '>60')))

summary_stats <- function(l) {
    s <- summary(l)
    data.frame(min=s['Min.'],
               first=s['1st Qu.'],
               median=s['Median'],
               mean=s['Mean'],
               third=s['3rd Qu.'],
               max=s['Max.'])
}

crn_by_month_lat_bin <-
    crn_daily_by_month %>%
        group_by(month, lat_bin) %>%
        do(summary_stats(.$anomaly)) %>%
        ungroup()

station_years <-
    crn_daily_by_month %>%
        mutate(year=year(date)) %>%
        group_by(wbanno, lat_bin) %>%
        summarize() %>%
        group_by(lat_bin) %>%
        summarize(station_years=n())

And the plot itself. At the end, we’re using a function called facet_adjust, which adds x-axis tick labels to the facet on the right that wouldn't ordinarily have them. The code comes from this stack overflow post.

p <- ggplot(data=crn_by_month_lat_bin,
            aes(x=month, ymin=first, ymax=third, y=mean)) +
    geom_hline(yintercept=0, alpha=0.2) +
    geom_hline(data=crn_by_month_lat_bin %>%
                        group_by(lat_bin) %>%
                        summarize(mean=mean(mean)),
               aes(yintercept=mean), colour="darkorange", alpha=0.5) +
    geom_pointrange() +
    facet_wrap(~ lat_bin, ncol=3) +
    geom_text(data=station_years, size=4,
              aes(x=2.25, y=-0.5, ymin=0, ymax=0,
                  label=paste('n =', station_years))) +
    scale_y_continuous(name="Range including 50% of temperature anomalies") +
    scale_x_discrete(breaks=1:12,
                     labels=c('Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun',
                              'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec')) +
    theme_bw() +
    theme(axis.text.x=element_text(angle=45, hjust=1, vjust=1.25),
          axis.title.x=element_blank())
facet_adjust(p)
//media.swingleydev.com/img/blog/2015/04/crn_minmax_anomalies_by_month_lat.svg

Each plot shows the range of anomalies from the first to the third quartile (50% of the observed anomalies) by month, with the dot near the middle of the line at the mean anomaly. The orange horizontal line shows the overall mean anomaly for that latitude category, and the count at the bottom of the plot indicates the number of “station years” for that latitude category.

It’s clear that there are seasonal patterns in the differences between the mean daily temperature and the min/max estimate. But each plot looks so different from the next that it’s not clear if the patterns we are seeing in each latitude category are real or artificial. It is also problematic that three of our latitude categories have very little data compared with the other two. It may be worth performing this analysis in a few years when the lower and higher latitude stations have a bit more data.

Conclusion

This analysis shows that there is a clear bias in using the average of minimum and maximum daily temperature to estimate average daily temperature. Across all of the CRN stations, the min/max estimator overestimates daily average temperature by almost a half a degree Celsius (0.8°F).

We also found that this error is larger at lower latitudes, and that there are seasonal patterns to the anomalies, although the seasonal patterns don’t seem to have clear transitions moving from lower to higher latitudes.

The current length of the CRN record is quite short, especially for the sub-hourly data used here, so the patterns may not be representative of the true situation.

tags: R  temperature  weather  climate  CRN  COOP  ggplot 
Meta Photolog Archives