Given a latitude and longitude, I need to pull the nearest neighbors out of the database and return them in proximity order.  To get this to work we use:

• geohash – every entry in the database records its geohash code
• law of cosines - fast trig based solution to the distance problem
• python sorting

To do the query, we need to compute a bounding box:

swhash = Geohash.encode(latitude-delta, longitude-delta)
nehash = Geohash.encode(latitude+delta, longitude+delta)

This produced two hashcodes, one to the southwest of the orig, and one the northeast.  A delta of 0.5 is about 138 KM, in North America, at least.  Geohash codes are great because they can be sorted in lexical order, so my boto query now looks like this:

sdb = boto.connect_sdb(AWS_ACCESS_KEY, AWS_SECRET_KEY])
domain = sdb.get_domain(PLACES_DOMAIN)
places = domain.select("select * from `Places` where
    `geohash` >= '%s' and `geohash` <= '%s' LIMIT 10"
    %(swhash, nehash), max_items=10)

The places object is actually an iterator much thanks to Chris Moyer for documenting this. We want the array.  NOTE, the query is lazily performed when you interpret the first token from the iterator.

results = [place for place in places]

The distance function is very simple:

def distance(a,b):
    R = 6371 # radius of Earth in KM
    lat1 = radians(a[0])
    lon1 = radians(a[1])
    lat2 = radians(b[0])
    lon2 = radians(b[1])
    d = math.acos(math.sin(lat1)*math.sin(lat2) +
        math.cos(lat1)*math.cos(lat2) * math.cos(lon2-lon1)) * R;
    return d

Finally, python sorts the result:

results = sorted(results, key=lambda place: distance(orig,
    (float(place['latitude']), float(place['longitude']))))

This actually worked on the second try.

Notes: there are better distance functions, but this one works well and is quite fast.  I have not put in the error handling, or the back off code if you get too many or too few results.

My app gets images from various photo sharing sites.  After checking for copyright correctness, I download images onto my own machine for review and Photoshopping. Then I upload them into my S3/SimpleDB cloud storage.

The metadata I get back from the photo sharing sites is generally in JSON format, but I frequently want to edit this metadata and even though JSON is the bee’s knees, it’s not that great to edit, plus grepping for metadata kinda sucks too.

Enter Yaml:

city: ''
country: ''
filtered: 0
geohash: dhwfq5vbb02h
id: '3077164973'
kind: scene
latitude: 25.728738
license: '4'
longitude: -80.236244
owner: 11211909@N00
ownername: corsi photo
state: ''
tags: miami floridawaterbeachskylineshorelineoceanbuildings
title: Miami shoreline
url_z: http://farm4.staticflickr.com/3149/30771649c6a6c_z.jpg
vp_id: flickr.scene.11211909@N00-3077164973

Exit JSON:

{'tags': 'miami floridawaterbeachskylineshorelineoceanbuildings', 'vp_id': 'flickr.scene.11211909@N00-3077164973', 'owner': '11211909@N00', 'id': '3077164973', 'city': '', 'kind': 'scene', 'url_z': 'http://farm4.staticflickr.com/3149/3077164973_a31b4c6a6c_z.jpg', 'license': '4', 'title': 'Miami shoreline', 'country': '', 'longitude': -80.236244, 'state': '', 'geohash': 'dhwfq5vbb02h', 'ownername': 'corsi photo', 'latitude': 25.728738, 'filtered': 0}

PyYAML even has type inferencing, so when I reload the data, it seems to do just what I want.

It all started with a test…

I wanted to add image renditions to my app server (who doesn’t?), so I thought I would use PIL.

I wrote a unit test:

def testRenditions(self): 
    rv1 = self.app.get('/images/%s/120' % self.test_scene_name)
    assert rv1.mimetype == 'image/jpeg'
    assert len(rv1.data) < len(open(self.test_scene_path).read())
    assert len(rv1.data) > 0

And then I tried to get the test to pass by implementing the server method.  Renditions are trivial in PIL, but getting PIL to work on my Mac, not so much…

I vanished down the rabbit hole of problems with Lion, Xcode, PIL, virtualenv, jpeglib, arghh.  Lots and lots of people have had trouble getting this to work, but seemingly they are all smarter / luckier than me, as most people eventually got it to work.  Every single time, the problem boiled down to my inability to import the _imaging module:

Symbol not found: _jpeg_resync_to_restart

There is a mysterious binary incompatibility that causes the _imaging module (which builds fine) to not load. Even more maddening, this worked in my downstairs computer, and I cannot spot a difference. It also worked great on the server, but I really didn’t want to have untested methods, and I am (so far) committed to running the tests in my local server first.

In the end, I have given up, theorizing that someday someone awesome like the Pillow maintainers will fix the problem.

The solution: use ImageMagick and two lines of Python:

p = subprocess.Popen(["convert", "-size", dimension, "-resize",     dimension, "-", "-"], stdin=subprocess.PIPE, 
    stdout=subprocess.PIPE) 
dst = p.communicate(src)[0]

This might actually be a better solution, but at any rate I’m unblocked.

In a very bad bout of mismanagement, I’ve spent more time on this single issue than literally any other aspect of the project.