README

ResolveUrls.py: A small threaded url resolution utility, originally written for the 
University of Illinois' Graduate School of Library and Information Science's TREC 2011 
Microblog team.


Usage:
from ResolveUrls import UrlResolver

# Optional params and defaults: num_threads = 400, mysql_host = 'localhost', http_redirects = 10
resolver = UrlResolver(url_file = '~/urls.txt', mysql_user = 'root', 
                       mysql_password = 'password', mysql_db = 'my_db')

resolver.run()


Alternate usage:
from ResolveUrls import ResolverThread

myQueue = Queue.Queue(maxsize=100)
for i in range(100):
    # Optional params and defaults: mysql_host = 'localhost', http_redirects = 10
    t = ResolverThread(myQueue, mysql_user = 'root', mysql_password = 'password', 
                  mysql_db = 'my_db')
    t.setDaemon(True)
    t.start()

# Add your own logic here to add urls to the queue
...
    urlQueue.put((url), block = True, timeout = None)
...

myQueue.join()


MySQL table structure:
resolved_urls
+---------------+---------------------+------+-----+---------+----------------+
| Field         | Type                | Null | Key | Default | Extra          |
+---------------+---------------------+------+-----+---------+----------------+
| id            | int(10) unsigned    | NO   | PRI | NULL    | auto_increment |
| short_url     | text                | NO   |     | NULL    |                |
| response_code | int(11)             | NO   |     | NULL    |                |
| long_url      | text                | YES  |     | NULL    |                |
| num_redirects | int(10) unsigned    | YES  |     | NULL    |                |
+---------------+---------------------+------+-----+---------+----------------+

url_headers
+---------+---------+------+-----+---------+-------+
| Field   | Type    | Null | Key | Default | Extra |
+---------+---------+------+-----+---------+-------+
| id      | int(11) | NO   | MUL | NULL    |       |
| headers | text    | NO   |     | NULL    |       |
+---------+---------+------+-----+---------+-------+