Saturday, June 02, 2007

crazy ideas

around 500 days ago i had a crazy idea: mapping the net in a database. all domains, all hostnames, all relations of ns- and mx-servers.

i knew a few sites who should have this data but would not really let you look it up - whois.sc and netcraft.com were amongst them. that was all i knew. oh and yes, i knew mysql, i worked with sqlite and microsofts sql-server for years.

i expected this to be an adventure. a textadventure, fun.

and hence, it was fun. the database crashed, servers got blocked, i had errors in my harvesting scripts and i was overwhelmed when i got around 100 million domains with even more hosts at once.

and now i'm sitting on a bunch of data that gets older. my time is limited, my hardware-ressources are as well. data is getting old. i started updating the hostname/IP-entries these days. I should have done this earlyier, i know - but i didn't. time is limited - remember?

a bit frustrating: the "update-lag", the time between the last update of a hostentry is currently exactly at 427,167 days. most frustrating: it's still increasing.


hmpf.

tom

1 comment:

susane said...

That's the first I just visited your website and I find it really interesting! Congratulations!

voyance gratuite en ligne