Introduction

Often when I’m watching Major League Baseball games a player will come up to bat or pitch and I’ll comment “former Oakland Athletic” and the player’s name. It seems like there’s always one or two players on the roster of every team that used to be an Athletic.

Let’s find out. We’ll use the Retrosheet database again, this time using the roster lists from 1990 through 2014 and comparing it against the 40-man rosters of current teams. That data will have to be scraped off the web, since Retrosheet data doesn’t exist for the current season and rosters change frequently during the season.

Methods

As usual, I’ll use R for the analysis, and rmarkdown to produce this post.

library(plyr)
library(dplyr)
library(rvest)
options(stringsAsFactors=FALSE)

We’re using plyr and dplyr for most of the data manipulation and rvest to grab the 40-man rosters for each team from the MLB website. Setting stringsAsFactors to false prevents various base R packages from converting everything to factors. We're not doing any statistics with this data, so factors aren't necessary and make comparisons and joins between data frames more difficult.

retrosheet_db <- src_postgres(host="localhost", port=5438,
                              dbname="retrosheet", user="cswingley")

rosters <- tbl(retrosheet_db, "rosters")

all_recent_players <-
     rosters %>%
     filter(year>1989) %>%
     collect() %>%
     mutate(player=paste(first_name, last_name),
            team=team_id) %>%
     select(player, team, year)

save(all_recent_players, file="all_recent_players.rdata", compress="gzip")

oakland_players <- all_recent_players %>%
    filter(team=='OAK') %>%
    group_by(player) %>%
    summarise(years=paste(year, collapse=', '))

kable(head(oakland_players))

teams=c("ana", "ari", "atl", "bal", "bos", "cws", "chc", "cin", "cle", "col",
        "det", "mia", "hou", "kc", "la", "mil", "min", "nyy", "nym", "oak",
        "phi", "pit", "sd", "sea", "sf", "stl", "tb", "tex", "tor", "was")

team_names = c("Angels", "Diamondbacks", "Braves", "Orioles", "Red Sox",
               "White Sox", "Cubs", "Reds", "Indians", "Rockies", "Tigers",
               "Marlins", "Astros", "Royals", "Dodgers", "Brewers", "Twins",
               "Yankees", "Mets", "Athletics", "Phillies", "Pirates",
               "Padres", "Mariners", "Giants", "Cardinals", "Rays", "Rangers",
               "Blue Jays", "Nationals")

get_players <- function(team) {
    # reads the 40-man roster data for a team, returns a data frame

    roster_html <- html(paste("http://www.mlb.com/team/roster_40man.jsp?c_id=",
                              team,
                              sep=''))

    players <- roster_html %>%
        html_nodes("#roster_40_man a") %>%
        html_text()

    data.frame(team=team, player=players)
}

current_rosters <- ldply(teams, get_players)

save(current_rosters, file="current_rosters.rdata", compress="gzip")

kable(head(current_rosters))

Combine the data

To find out how many players on each Major League team used to play for the A’s we combine the former A’s players with the current rosters using player name. This may not be perfect due to differences in spelling (accented characters being the most likely issue), but the results look pretty good.

roster_with_oakland_time <- current_rosters %>%
    left_join(oakland_players, by="player") %>%
    filter(!is.na(years))

kable(head(roster_with_oakland_time))

You can see from this table (just the first six rows of the results) that the Angels have three players that were Athletics.

Let’s do the math and find out how many former A’s are on each team’s roster.

n_former_players_by_team <-
    roster_with_oakland_time %>%
        group_by(team) %>%
        arrange(player) %>%
        summarise(number_of_players=n(),
                  players=paste(player, collapse=", ")) %>%
        arrange(desc(number_of_players)) %>%
        inner_join(data.frame(team=teams, team_name=team_names),
                   by="team") %>%
        select(team_name, number_of_players, players)

names(n_former_players_by_team) <- c('Team', 'Number',
                                     'Former Oakland Athletics')
kable(n_former_players_by_team,
      align=c('l', 'r', 'r'))

Pretty cool. I do notice one problem: there are actually two Chris Young’s playing in baseball today. Chris Young the outfielder played for the A’s in 2013 and now plays for the Yankees. There’s also a pitcher named Chris Young who shows up on our list as a former A’s player who now plays for the Royals. This Chris Young never actually played for the A’s. The Retrosheet roster data includes which hand (left and/or right) a player bats and throws with, and it’s possible this could be used with the MLB 40-man roster data to eliminate incorrect joins like this, but even with that enhancement, we still have the problem that we’re joining on things that aren’t guaranteed to uniquely identify a player. That’s the nature of attempting to combine data from different sources.

One other interesting thing. I kept the A’s in the list because the number of former A’s currently playing for the A’s is a measure of how much turnover there is within an organization. Of the 40 players on the current A’s roster, only 22 of them have ever played for the A’s. That means that 18 came from other teams or are promotions from the minors that haven’t played for any Major League teams yet.

All teams

teams <- c("ANA", "ARI", "ATL", "BAL", "BOS", "CAL", "CHA", "CHN", "CIN",
           "CLE", "COL", "DET", "FLO", "HOU", "KCA", "LAN", "MIA", "MIL",
           "MIN", "MON", "NYA", "NYN", "OAK", "PHI", "PIT", "SDN", "SEA",
           "SFN", "SLN", "TBA", "TEX", "TOR", "WAS")

team_names <- c("Angels", "Diamondbacks", "Braves", "Orioles", "Red Sox",
                "Angels", "White Sox", "Cubs", "Reds", "Indians", "Rockies",
                "Tigers", "Marlins", "Astros", "Royals", "Dodgers", "Marlins",
                "Brewers", "Twins", "Expos", "Yankees", "Mets", "Athletics",
                "Phillies", "Pirates", "Padres", "Mariners", "Giants",
                "Cardinals", "Rays", "Rangers", "Blue Jays", "Nationals")

players_on_other_teams <- all_recent_players %>%
    group_by(player, team) %>%
    summarise(years=paste(year, collapse=", ")) %>%
    inner_join(current_rosters, by="player") %>%
    mutate(current_team=team.y, former_team=team.x) %>%
    select(player, current_team, former_team, years) %>%
    inner_join(data.frame(former_team=teams, former_team_name=team_names),
                by="former_team") %>%
    group_by(former_team_name, current_team) %>%
    summarise(n=n()) %>%
    group_by(former_team_name) %>%
    arrange(desc(n)) %>%
    mutate(rank=row_number()) %>%
    filter(rank!=1) %>%
    summarise(n=sum(n)) %>%
    arrange(desc(n))

names(players_on_other_teams) <- c('Former Team', 'Number of players')

kable(players_on_other_teams)

players_on_same_team <- all_recent_players %>%
    group_by(player, team) %>%
    summarise(years=paste(year, collapse=", ")) %>%
    inner_join(current_rosters, by="player") %>%
    mutate(current_team=team.y, former_team=team.x) %>%
    select(player, current_team, former_team, years) %>%
    inner_join(data.frame(former_team=teams, former_team_name=team_names),
                by="former_team") %>%
    group_by(former_team_name, current_team) %>%
    summarise(n=n()) %>%
    group_by(former_team_name) %>%
    arrange(desc(n)) %>%
    mutate(rank=row_number()) %>%
    filter(rank==1,
           former_team_name!="Expos") %>%
    summarise(n=sum(n)) %>%
    arrange(desc(n))

names(players_on_same_team) <- c('Team', 'Number of players')

kable(players_on_same_team)

Game Length

Game length in minutes is in column 19. Let’s read it and analyze it with Pandas.

import pandas as pd
import matplotlib.pyplot as plt
import datetime

def fix_df(input_df, cols):
    """ Pulls out the columns from cols, labels them and returns
        a new DataFrame """
    df = pd.DataFrame(index=range(len(input_df)))
    for k, v in cols.items():
        df[v] = input_df[k]

    return df

cols = {0:'dt_str', 2:'day_of_week', 3:'visiting_team',
        6:'home_team', 9:'visitor_score', 10:'home_score',
       11:'game_outs', 12:'day_night', 13:'complete',
       16:'park_id', 17:'attendance', 18:'game_time_min'}
raw_gamelogs = []
for i in range(1871, 2013):
    fn = "GL{0}.TXT".format(i)
    raw_gamelogs.append(pd.read_csv(fn, header=None, index_col=None))
raw_gamelog = pd.concat(raw_gamelogs, ignore_index=True)
gamelog = fix_df(raw_gamelog, cols)
gamelog['dte'] = gamelog.apply(
    lambda x: datetime.datetime.strptime(str(x['dt_str']), '%Y%m%d').date(),
    axis=1)
gamelog.ix[0:5, 1:] # .head() but without the dt_str col

(Note that I’ve abbreviated the column names so they fit)

The data looks reasonable, although I don’t know quite what to make of the team names from back in 1871. Now I’ll take a look at the game time data for the whole data set:

times = gamelog['game_time_min']
times.describe()

count    162259.000000
mean        153.886145
std          74.850459
min          21.000000
25%         131.000000
50%         152.000000
75%         173.000000
max       16000.000000

The statistics look reasonable, except that it’s unlikley that there was a completed game in 21 minutes, or that a game took 11 days, so we have some outliers in the data. Let’s see if something else in the data might help us filter out these records.

First the games longer than 24 hours:

print(gamelog[gamelog.game_time_min > 60 * 24][['visiting_team',
                    'home_team', 'game_outs', 'game_time_min']])
print("Removing all NaN game_outs: {0}".format(
        len(gamelog[np.isnan(gamelog.game_outs)])))
print("Max date removed: {0}".format(
        max(gamelog[np.isnan(gamelog.game_outs)].dte)))

      visiting_team home_team  game_outs  game_time_min
26664           PHA       WS1        NaN          12963
26679           PHA       WS1        NaN           4137
26685           WS1       PHA        NaN          16000
26707           NYA       PHA        NaN          15115
26716           CLE       CHA        NaN           3478
26717           DET       SLA        NaN           1800
26787           PHA       NYA        NaN           3000
26801           PHA       NYA        NaN           6000
26880           CHA       WS1        NaN           2528
26914           SLA       WS1        NaN           2245
26921           SLA       WS1        NaN           1845
26929           CLE       WS1        NaN           3215
Removing all NaN game_outs: 37890
Max date removed: 1915-10-03

There’s no value for game_outs, so there isn’t data for how long the game actually was. We remove 37,890 records by eliminating this data, but these are all games from prior to the 1916 season, so it seems reasonable:

gamelog = gamelog[pd.notnull(gamelog.game_outs)]
gamelog.game_time_min.describe()

count    156639.000000
mean        154.808975
std          32.534916
min          21.000000
25%         133.000000
50%         154.000000
75%         174.000000
max        1150.000000

What about the really short games?

gamelog[gamelog.game_time_min < 60][[
        'dte', 'game_time_min', 'game_outs']].tail()

               dte  game_time_min  game_outs
79976   1948-07-02             24         54
80138   1948-07-22             59         30
80982   1949-05-29             21         42
113455  1971-07-30             48         27
123502  1976-09-10             57         30

Many of these aren’t nine inning games because there are less than 51 outs (8 innings for a home team, 9 for the visitor in a home team win ✕ 3 innings = 51). At the moment, I’m interested in looking at how long a game is likely to be, rather than the pattern of game length over time, so we can leave these records in the data.

Now we filter the data to just the games played since 2000.

twenty_first = gamelog[gamelog['dte'] > datetime.date(1999, 12, 31)]
times_21 = twenty_first['game_time_min']
times_21.describe()

count    31580.000000
mean       175.084421
std         26.621124
min         79.000000
25%        157.000000
50%        172.000000
75%        189.000000
max        413.000000

The average game length between 2000 and 2012 is 175 minutes, or just under three hours. And three quarters of all the games played are under three hours and ten minutes.

Here’s the code to look at the distribution of these times. The top plot shows a histogram (the count of the games in each bin) and a density estimation of the same data.

The second plot is a cumulative density plot, which makes it easier to see what percentage of games are played within a certain time.

from scipy import stats
# Calculate a kernel density function
density = stats.kde.gaussian_kde(times_21)
x = range(90, 300)
rc('grid', linestyle='-', alpha=0.25)
fig, axes = plt.subplots(ncols=1, nrows=2, figsize=(8, 6))

# First plot (histogram / density)
ax = axes[0]
ax2 = ax.twinx()
h = ax.hist(times_21, bins=50, facecolor="lightgrey", histtype="stepfilled")
d = ax2.plot(x, density(x))
ax2.set_ylabel("Density")
ax.set_ylabel("Count")
plt.title("Distribution of MLB game times, 2000-2012")
ax.grid(True)
ax.set_xlim(90, 300)
ax.set_xticks(range(90, 301, 15))

# Second plot (cumulative histogram)
ax1 = axes[1]
n, bins, patches = ax1.hist(times_21, bins=50, normed=True, cumulative=True,
        facecolor="lightgrey", histtype="stepfilled")
y = n / float(n[-1]) # Convert counts to percentage of total
y = np.concatenate(([0], y))
bins = bins - ((bins[1] - bins[0]) / 2.0) # Center curve on bars
ax1.plot(bins, y, color="blue")
ax1.set_ylabel("Cumulative percentage")
ax1.set_xlabel("Game time (minutes)")
ax1.grid(True)
ax1.set_xlim(90, 300)
ax1.set_xticks(range(90, 301, 15))
ax1.set_yticks(np.array(range(0, 101, 10)) / 100.0)
plt.savefig('game_time.png', dpi=87.5)
plt.savefig('game_time.svg')
plt.savefig('game_time.pdf')

The two plots show what the descriptive statistics did: 70% of games are under three hours but it’s not uncommon for games to last three hours and fifteen minutes. Beyond that, it’s pretty uncommon.

Change in game times in the last 100 years

Now let’s look at how game times have changed over the years. First we eliminate all the games that weren’t complete in 51 or 54 innings to standardize the “game” were’re evaluating.

nine_innings = gamelog[gamelog.game_outs.isin([51, 54])]
nine_innings['year'] = nine_innings.apply(lambda x: x['dte'].year, axis=1)
nine_groupby_year = nine_innings.groupby('year')
mean_time_by_year = nine_groupby_year['game_time_min'].aggregate(np.mean)
fig = plt.figure()
p = mean_time_by_year.plot(figsize=(8, 4))
p.set_xlim(1915, 2015)
p.set_xlabel('Year')
p.set_ylabel('Game Time (minutes)')
p.set_title('Average MLB nine-inning game time since 1915')
p.set_xticks(range(1915, 2016, 10))
plt.savefig('game_time_by_year.png', dpi=87.5)
plt.savefig('game_time_by_year.svg')
plt.savefig('game_time_by_year.pdf')

That shows a pretty clear upward trend interrupted by a slight decline between 1960 and 1975 (when offense was down across baseball). Since the mid-90s, game times have hovered around 2:50, so maybe MLB’s efforts at increasing the pace of the game have at least stopped what had been an almost continual rise in game times.

Attendance

I’ll be seeing a game at Wrigley Field, so let’s examine attendance at Wrigley.

Attendance is field 17, “Park ID” is field 16, but we can probably use Home team (field 6) = “CHN” for seasons after 1914 when it opened.

twenty_first[twenty_first['home_team'] == 'CHN']['attendance'].describe()

count     1053.000000
mean     37326.561254
std       5544.121773
min          0.000000
25%      36797.000000
50%      38938.000000
75%      40163.000000
max      55000.000000

We see the minimum is zero, which may indicate bad or missing data. Let’s look at all the games at Wrigley with less than 10,000 fans:

twenty_first[(twenty_first.home_team == 'CHN') &
             (twenty_first.attendance < 10000)][[
            'dte', 'visiting_team', 'home_team', 'visitor_score',
            'home_score', 'game_outs', 'day_night', 'attendance',
            'game_time_min']]

Looks like the zeros are just missing data because these games have relevant data for the other fields and it’s impossible to believe that not a single person was in the stands. We’ll get rid of them for the attendance analysis.

Now we’ll filter the games so we’re only looking at day games played at Wrigley where the attendance value is above zero, and group the data by day of the week.

groupby_dow = twenty_first[(twenty_first['home_team'] == 'CHN') &
           (twenty_first['day_night'] == 'D') &
           (twenty_first['attendance'] > 0)].groupby('day_of_week')

groupby_dow['attendance'].aggregate(np.mean).order()

day_of_week
Tue            34492.428571
Wed            35312.265957
Thu            35684.737864
Mon            37938.757576
Fri            38066.060976
Sun            38583.833333
Sat            39737.428571
Name: attendance

And plot it:

filtered = twenty_first[(twenty_first['home_team'] == 'CHN') &
                           (twenty_first['day_night'] == 'D') &
                           (twenty_first['attendance'] > 0)]
dows = {'Sun':0, 'Mon':1, 'Tue':2, 'Wed':3, 'Thu':4, 'Fri':5, 'Sat':6}
filtered['dow'] = filtered.apply(lambda x: dows[x['day_of_week']], axis=1)
filtered['attendance'] = filtered['attendance'] / 1000.0
fig = plt.figure()
d = filtered.boxplot(column='attendance', by='dow', figsize=(8, 4))
plt.suptitle('')
d.set_xlabel('Day of the week')
d.set_ylabel('Attendance (thousands)')
d.set_title('Wrigley Field attendance by day of the week, 2000-2012')
labels = ['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat']
l = d.set_xticklabels(labels)
d.set_ylim(15, 45)
plt.savefig('attendance.png', dpi=87.5)
plt.savefig('attendance.svg') # lines don't show up?
plt.savefig('attendance.pdf')

There’s a pretty clear pattern here, with increasing attendance from Tuesday through Monday, and larger variances in attendance during the week.

It’s a bit surprising that Monday games are as popular as Saturday games, especially since we’re only looking at day games. On any given Monday when the Cubs are at home, there’s a good chance that there will be fourty thousand people not showing up to work!