Origins
So, here’s a quick rundown of how this bot came about, for people who somehow find it in the future, after the human race is wiped out and all that is left is this blog and a pile of cheap but varied knick-knacks.
Things started some time ago, around May 1st, 2009, when I first saw the following xkcd comic:
I don’t think I saw it the day of, but it wasn’t very long after. I thought it was a neat idea, and honestly I waited a bit for someone to create it, so I could live vicariously through that person. I apparently gave up waiting after a few months; based on my timestamps I originally started coding this thing in September of 2009.
The first bits of progress were pretty basic: I had a page that would search for auctions ending in the next hour, for less than a dollar, with free shipping, based on a set of keywords that I figured would get me the most results. It was a simple search of the whole site for things like “the, and, a, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9” in the fulltext of all auctions. There are two issues with this, the first being that it obviously misses a ton of auctions, and the second being that you end up with a lot of auctions for things that are virtual goods, or things like phone contracts. I didn’t want any of this stuff, because it seemed to go against the whole “get a random package” spirit, so I added some keywords to the search that should be filtered, like “-ebook, -contract”. I eventually had a really really long search string that, half the time, eBay would choke on and die, returning an error. What I had now was a page that would list all the auctions ending in a few hours that matched this search, except for the times it choked, which was pretty damn often. After tweaking it a bit to allow you to display either US based or Worldwide auctions, I set it aside for a few days.
Nearly three years later, I decided to pick it back up. I figured in this time, someone would have created the bot already, and sure enough, someone had! Sort of. Someone had created a bot like this for a site in New Zealand, and although the site is down now you may be able to read about it here, on the Internet Archive here (found a working link). Although it was no longer active, it inspired me to dust off my old code and see if I could get it up and running.
The first thing I noticed was that the search I had seemed to return 100% of the time now. I guess the eBay API got snappier since I started. I decided to do more digging in the API docs, and found something interesting; I could search by category, for up to 3 categories, without needing to specify a keyword at all! This change meant I had a lot more results being returned, but also I was now unable to filter out easily based on keywords. I semi-solved this by excluding results based on their category. If they are, for example, in a category for “Digital Photographs”, I drop them. I haven’t gone through every category, eBay US has over 20,000 of them, but I got many of the ones that showed up often in my searches.
I added a bit more filtering based on the titles again, to try to catch miscategorized items, then filtered out items that were local pickup only. I also added a full description filter, but currently don’t use it, as the first two seem to work pretty well. I now had a good way to get a bunch of random ending soon auctions, filter them, and then randomly pick one out. The problem I now had was making the bot actually bid on items. There is a bidding API, but from what I could tell, it needed me to apply and jump through some hoops to use. I really didn’t want to do this, because I’m easily distracted, and I figured if I dropped the project again I may not pick it up for another 3 years. I went looking for premade bid coding, and managed to find a simple bot that will do bidding for you. I also found a PHP frontend for this bot, and after tweaking the frontend’s API to allow me to specify bids from my script, I had the bot up and bidding in nearly no time at all.
I didn’t let it actually buy anything during testing, but once I was happy that everything looked good, I started running it manually a few days in a row. This let me get good data that I could then use to figure out how the auction had gone. I was now able to tell if I had won or lost, and the price I ultimately had paid. Pop all this in a database, and now you have all the data you need to let the bot figure out for itself when it needs to bid, how much money it has, etc. There’s a bit more to it, but I’m going to save more detailed info for the next post. I think this is a big enough wall of text for now!