wed, 08-oct-2008, 17:51

ipfw

os x firewall tool

One of the more annoying things about Apple’s wireless routers is that there’s no way to shape the bandwidth. With two of us in the house, commonly using the Internet at the same time, and a limited 43 KB/s bandwidth, we wind up stepping on each other’s use fairly often. One bandwidth limiting tool is the Unix command trickle which allows you to control bandwidth on individual, command line programs. Something like:

    trickle -u 20 -d 20 wget http://bigfiles.com/bigfile.mp3
would limit the file download to 20 KB/second, about half our our bandwidth. Many commands like wget and rsync have bandwidth limiting built in, making trickle unnecessary for those programs.

These techniques don’t work when the programs don’t include limiting internally, and when you can’t run them from the command line. The program I use to download music from eMusic (eMusicJ) is an example. With my downloads refreshing in a couple days, I wanted to find a way to get my downloads in, without ruining the network for the next day and a half.

Since OS X is built on BSD, it comes with a super-sophisticated firewall, ipfw, that has traffic shaping built in. So here’s how I was able to consume only half of our bandwidth downloading music:

Start the download and use netstat -an to find the IP address of the download site (or do netstat -an before and after you’ve started the download to identify the new download IP Address):

    $ netstat -an | less
    Active Internet connections (including servers)
    Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)
    tcp4       0      0  10.0.1.198.54176       38.98.87.100.80        ESTABLISHED
    tcp4       0      0  10.0.1.198.54175       38.98.87.100.80        ESTABLISHED
    tcp4       0      0  10.0.1.198.54142       38.98.87.100.80        ESTABLISHED
    tcp4       0      0  10.0.1.198.54136       38.98.87.100.80        ESTABLISHED

Set up a pipe for data coming from that site:

    $ sudo ipfw add pipe 10 ip from 38.98.87.100 to any
    00100 pipe 10 ip from 38.98.87.100 to any

Configure the pipe to limit bandwidth:

    $ sudo ipfw pipe 10 config bw 20KBytes/s queue 10KBytes

After you're done, delete the pipe:

    $ sudo ipfw list
    00100 pipe 10 ip from 38.98.87.100 to any
    65535 allow ip from any to any

    $ sudo ipfw del 00100
wed, 13-jun-2007, 05:37

Glass Album Cover

Philip Glass Cover

Yesterday I found myself with 43 eMusic downloads available and my refresh date approaching quickly. I’ve got quite a few records in my queue, and choosing among them to exactly consume my available downloads is difficult to do by hand. So I wrote a program to do it.

It’s a Python script, so it’ll run any any platform. Click this link to download it: choose_albums.py

To use it, you’ll need to create a separate file that contains a list of the albums you’re interested in and the number of tracks on each album. Here’s the file I was working with yesterday, called queue:

clientele 14
rosebuds 9
okkervil river 11
stravinsky 19
saint-saens violin 3 8
mapmaker 12
of montreal 5
long blondes 14
glass #4 7
widor #5&9 9
bonnie billy 13

Each line contains an album name, a space, and the lines end with the number of tracks on the album.

To run the program, call it and pass the name of your file and the number of downloads you’ve got left:

$ ./choose_albums.py queue 43
glass #4, saint-saens violin 3, stravinsky, widor #5&9: 43

This is one (of many) ways to use up my 43 downloads. The script chooses albums randomly, so if you want to see all the possibilities, you’ll need to run it a lot. I wrote a very simple shell script to do that:

#! /bin/sh

tracks=$@

for i in `seq 1 100`; do ./choose_albums.py queue $tracks; done | sort | uniq

You can download it here: doit.sh

Depending on how large your queue is, you may need to increase the number of times it runs the script. Because it’s random and not deterministic, it can take a lot of runs to find all possible options (in fact, with 25 albums in the queue and 90 tracks available, there are more than 40,000 possible combinations, so this script is best at choosing from a small set of options, unless a random choice is what you're after). You’ll also need to change the name of your queue file if it’s not called queue.

Here’s what I did yesterday:

$ ./doit.sh 43 | grep blonde | grep glass
bonnie billy, glass #4, long blondes, rosebuds: 43
bonnie billy, glass #4, long blondes, widor #5&9: 43
clientele, glass #4, long blondes, saint-saens violin 3: 43
glass #4, long blondes, of montreal, rosebuds, saint-saens violin 3: 43
glass #4, long blondes, of montreal, saint-saens violin 3, widor #5&9: 43

The two grep commands were included because I knew I wanted to include the new Long Blondes album and Philip Glass’s Fourth Symphony in my selections. I wound up going with the second choice, adding Bonnie “Prince” Billy’s The Letting Go and Widor’s Fifth and Ninth Symphonies.

One final note: if after 1,000 tries, the script doesn’t find a set of choices that uses up all your downloads, it’ll report the last set of albums it found and the number of tracks used up. Be sure that the final number reported matches the number you passed in or you won’t be using all your downloads for the month. The script isn’t smart enough to find the “best” solution in this situation, so if this happens, you’ll need to run it a bunch of times to maximize the number you’re downloading (or better, add more items to the queue file and run it again).

tags: eMusic  music  programming  python 
Meta Photolog Archives