readme

As part of the Open Government Directive, each of the major federal
agencies is required to seek input from the public for an Open
Government Plan. This consultation process is taking place February 6
- March 19, 2010, using a platform called IdeaScale. Each agency has
it's own, isolated IdeaScale site.

Basic stats are available for each agency separately, but there is
nowhere to easily compare participation across the agencies, nor for
those interested in participating to get a broad overview of the
activities. 

This is a stateless (for now) dashboard showing stats and comparisons
regarding activity on each agency's brainstorming site, using the
basic API made available by IdeaScale (thanks!).

Dependencies:
* tornado (http://www.tornadoweb.org/)
* mongodb
* pymongo (python module to interact with mongodb)
* json or simplejson

To Run:
* add a line like so in /etc/crontab
  00,30 * * * * username /path/to/cron.py
  which updates the data in the database every 30 minutes.
* run dashboard.py

cron.py populates the mongo database. Each time the data is updated, a
new mongo collection (which is like a table) is created. The name of
each collection is a timestamp representing the time of the
update. Each collection/update stores 1 document for each idea, plus a
document representing aggregate stats per agency, and a document
representing top ideas per agency, at the time of that update. Thus if
there are n total ideas across all agencies, the collection will have
n+2 documents. Initial population can take a few minutes.

each idea object looks roughly like this:
{
 '_id':   	 ideascale idea ID,
 'category': 	 nominal category name,
 'agency' : 	 agency,
 'idea':  	 ideascale idea object
}

mongodb does not allow keys to have periods or dollar signs in
them. consequently, the cronjob encodes all periods as four percent
signs, and should decode them before display in the application (but
they are not being displayed anywhere at the moment).

dashboard.py, the main application, pulls its data from the database.

todo: add information about twitter, govtrackermeta db, data
collection.

Differences between opengovtracker.com and Ideascale's summaries:

- idea count differences-- uncertain

- vote count differences-- current theory is that ideascale is showing
  the total votes, while the API is returning net votes (of up and
  down). we could attempt to retrieve total votes from the author
  objects, but the author API call is currently truncated after 10
  authors are returned (It's possible IdeaScale would be willing to
  lift that if we asked).

- comment count differences - perhaps comments on ideas that have been
  moved to off topic are still being counted by the IdeaScale
  summaries, but not accounted for in the information returned by the
  API?