automate rsphider indexing of content

[usernamenumber]

_Note:_ See this comment for current details.

rsphider is a system for making local web-based content searchable. By its self it might not be a requirement for our initial deployment, but several RACHEL modules depend on it, and the branch that implements RACHEL resolves multiple issues that block the initial deployment milestone: #33, #35, #38.

Automating deployment and configuration of rsphider should be do-able. I've take some notes on what will need to be done on this wiki page

[usernamenumber]

A summary for new people who want to help with this issue...

_Background_

• x2go incorporates several offline resources provided by WorldPossible's RACHEL project.
• RACHEL content is made searchable by a php-based crawler called rsphider
  • Link goes to a tunapanda fork I made so we can make internal modifications if needed
• rsphider is WorldPossible's fork of what appears to be a much older tool, sphider-plus.There is also the possibility that it would be better to use a more established crawler, like Apache Solr and adapt the RACHEL modules to use it, rather than the other way around.
• As far as I know, the sphider-plus docs apply to rsphider

_What needs to be done_

1. Figure out the best way to auto-index the RACHEL content designed to work with rsphider.
   • I've made some notes on this here
2. Write ansible plays to do it for any given site.
   • Maybe create a sphider_index role that takes the information it needs as arguments when called as a dependency by roles that create indexable content, like when the nginx_vhosts role is called by the learninglocker role
     • technically it calls web_server_vhosts, but that just hands off to nginx_vhosts
3. Figure out what would need to be done to make content (edX courses?) also searchable, and included in the global search bar on the portal page.

[usernamenumber]

This should be done now, though the code is currently in the default sync script for rsync'd content, which is really not the best place. Moving it will require indexed content to either use its own sync script, or a way to ensure that indexing doesn't start until syncing the data down is complete.