legally superfluous

Hey there!

I'm Martin Jansen and this is my tumblelog. Most cool kids seem to have one these days. Do you?

The things I publish here are replicated on my other home page and there is more stuff there, too.

Jan 5

You can’t build a full web crawl in one day. You’d be blocked instantly. Crawlers have to be polite.

We spend a TON of time working with blog hosting companies to make sure our crawlers yield to their policies. For example, LiveJournal won’t allow us to crawl with more than 10 threads.

Kevin Burton