diff --git a/db/migrate/20101028165208_install_faceting.rb b/db/migrate/20101028165208_install_faceting.rb index 6aba470..db4a83d 100644 --- a/db/migrate/20101028165208_install_faceting.rb +++ b/db/migrate/20101028165208_install_faceting.rb @@ -27,7 +27,7 @@ class InstallFaceting < ActiveRecord::Migration def self.up if Rails.env.production? execute('CREATE SCHEMA facet') - execute( faceting_api_sql(:bytea, 'facet') ) + execute( faceting_api_sql(:varbit, 'facet') ) else execute('CREATE EXTENSION faceting') end diff --git a/vendor/gems/repertoire-faceting-0.6.0/.gitignore b/vendor/gems/repertoire-faceting-0.6.0/.gitignore new file mode 100644 index 0000000..937dbf0 --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/.gitignore @@ -0,0 +1,22 @@ +test_log +pkg +pkg/* +*.log +log +!log*.rb +*/log +log/* +*/log/* +coverage +.DS_Store +*.pid +*.gem +TAGS +*.rbc +*.tmproj +.#* +.bundle +doc/* +ext/**/*.o +ext/**/*.so +ext/*.sql diff --git a/vendor/gems/repertoire-faceting-0.6.0/FAQ b/vendor/gems/repertoire-faceting-0.6.0/FAQ new file mode 100644 index 0000000..c0c08b0 --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/FAQ @@ -0,0 +1,187 @@ += Repertoire Faceting FAQ + +== General questions + +*Q* Can I use Rails migrations with Repertoire Faceting? + +*A* In general, yes. However, Rails' developers recommend you use the database's native dump format rather than schema.rb. Put the following line in environment.rb: + + config.active_record.schema_format = :sql + + You can also use migrations to install the API in your database. A simple example: + + def self.change + reversible do |dir + dir.up { execute('CREATE EXTENSION faceting') } + dir.down { execute('DROP EXTENSION faceting CASCADE') } + end + end + + In cases where you do not have superuser access to the deployment host (e.g. Heroku) and so cannot run + "rake db:faceting:extensions:install", you can get use the connection's "faceting_api_sql" method to load the API + by hand. See the repertoire-faceting-example application's migrations for a concrete example of usage. + + +== About facet indexing and the signature SQL type + +*Q* What's the scalability of this thing? + +*A* Up to about 500,000 items, supposing 6-8 simultaneous facets with domains anywhere from 2-100 values. In other words, beyond the size of most commonly available datasets. See the citizens example in the specs directory & example faceting app. It has been tested with up to 1,000,000 items, but this requires unix configuration to give Postgresql lots of shared memory. + + +*Q* My facets are empty. + +*A* Make sure the facet indices aren't empty. Running '.index_facets([])' from the console will drop them all. + + +*Q* Can I facet over multiple models? + +*A* Not currently. However, this may be possible using an ActiveRecord polymorphic relation on the main model. + + +*Q* Why a new native PostgreSQL type? + +*A* As of PostgreSQL 9.3, there is a binding of the Repertoire in-database faceting functions based on VARBIT bit strings. +However, it is many times slower than using the C-language signature type above. + + +== About the ajax faceting widgets + + +*Q* Rails is sending JSON data in a format that my javascript widgets don't understand. + +*A* Put the following line in config/application.rb: + + config.active_record.include_root_in_json = false + + +*Q* A web page from the default Rails app refuses to load the faceting widgets. + +*A* Repertoire Faceting widgets are based on JQuery, which is incompatible with Prototype. You should remove prototype.js and rails.js from the javascripts directory, and delete the <%= javascript_include_tag :defaults %> line from application.html.erb. + + +*Q* How do I send page-specific data (for example, a search field) to the webservice with the facet widgets' data? + +*A* If you provide a function to the facet_context plugin, it will merge the params you return before dispatching to the webservice, e.g. + + $('#invoices').facet_context(function() { + return { + search: $("#search_field").val() + }; + }); + + +*Q* I want to change the default options for all widgets of a given class. + +*A* See the syntax for defining jquery plugins - you can alter the defaults for all widgets by reassigning them in your view code. + +*Q* How do I make one-time, minor changes to the behaviour of a widget? For example, I want to add a control. + +*A* Use the inject option, which is part of the base functionality. Your injector function receives a reference to the widget's jquery element and to the widget javascript object. Use jquery to add your control's markup, then register an event handler to add its behaviour. For example, this injector adds a clear-all button in the title: + + $('#genre').facet({ + injectors: { + '.title .controls' : function(self, data) { $(this).append('[x]'); } + }, + handlers: { + 'click!.clear_control' : function(self) { + self.refinements('genre').length = 0; + self.state_changed(); + return false; + } + } + }); + +The injector adds markup for the control at the specific jquery selector, and the handler receives events on that markup. Both receive a single argument 'self' for the widget object, and 'this' for the matched DOM element. + +Note the syntax used to identify a handler's event and dom element: '!'. Both event and namespace are optional - leave them out to register a click handler with a unique namespace. + +In injectors and handlers, you have access to the complete faceting widget API (state, refinements, toggle, is_selected, etc.). You can basically build a new widget, if you need to. See the documentation for the faceting_widget class for details. + + +*Q* My additonal control needs to send data back to the webservice too. + +*A* You can pre-process the entire context's state before it's sent to the webservice by update(): + + var min = 5; + $('#genre').facet({ + injectors: { ... }, + handlers: { ... }, + pre_update: function(state) { state.minimum = genre_min; } + } + + +*Q* How do I subclass an existing widget, so I can reuse my changes repeatedly? + +*A* Basically you define a new widget class and move your injectors and handlers (above) into the appropriate places. See the results widget for the simplest possible example, and nested_facet for a real-world example that extends the default facet widget. At a bare minimum, you will over-ride the render() method, and possibly the update() method too. Here is a 'hello world' that extends the default facet count widget: + + var hello_world = function($elem, options) { + /* declare superclass */ + var self = repertoire.facet($elem, options); + + /* handlers */ + handler('.hello', function() { + alert('hello, world!'); + }); + + /* injectors */ + var $template_fn = self.render; + self.render = function(data) { + var $markup = $template_fn(data); + $markup.find('.title .controls').append('
click me!= 0.11, < 0.18) + rails (>= 3.2.11, < 4.1) + +GEM + remote: https://rubygems.org/ + specs: + actionmailer (4.0.3) + actionpack (= 4.0.3) + mail (~> 2.5.4) + actionpack (4.0.3) + activesupport (= 4.0.3) + builder (~> 3.1.0) + erubis (~> 2.7.0) + rack (~> 1.5.2) + rack-test (~> 0.6.2) + activemodel (4.0.3) + activesupport (= 4.0.3) + builder (~> 3.1.0) + activerecord (4.0.3) + activemodel (= 4.0.3) + activerecord-deprecated_finders (~> 1.0.2) + activesupport (= 4.0.3) + arel (~> 4.0.0) + activerecord-deprecated_finders (1.0.3) + activesupport (4.0.3) + i18n (~> 0.6, >= 0.6.4) + minitest (~> 4.2) + multi_json (~> 1.3) + thread_safe (~> 0.1) + tzinfo (~> 0.3.37) + arel (4.0.2) + atomic (1.1.14) + builder (3.1.4) + erubis (2.7.0) + hike (1.2.3) + i18n (0.6.9) + jquery-rails (3.1.0) + railties (>= 3.0, < 5.0) + thor (>= 0.14, < 2.0) + mail (2.5.4) + mime-types (~> 1.16) + treetop (~> 1.4.8) + mime-types (1.25.1) + minitest (4.7.5) + multi_json (1.8.4) + pg (0.17.1) + polyglot (0.3.4) + rack (1.5.2) + rack-test (0.6.2) + rack (>= 1.0) + rails (4.0.3) + actionmailer (= 4.0.3) + actionpack (= 4.0.3) + activerecord (= 4.0.3) + activesupport (= 4.0.3) + bundler (>= 1.3.0, < 2.0) + railties (= 4.0.3) + sprockets-rails (~> 2.0.0) + railties (4.0.3) + actionpack (= 4.0.3) + activesupport (= 4.0.3) + rake (>= 0.8.7) + thor (>= 0.18.1, < 2.0) + rake (10.1.1) + sprockets (2.11.0) + hike (~> 1.2) + multi_json (~> 1.0) + rack (~> 1.0) + tilt (~> 1.1, != 1.3.0) + sprockets-rails (2.0.1) + actionpack (>= 3.0) + activesupport (>= 3.0) + sprockets (~> 2.8) + thor (0.18.1) + thread_safe (0.1.3) + atomic + tilt (1.4.1) + treetop (1.4.15) + polyglot + polyglot (>= 0.3.1) + tzinfo (0.3.38) + +PLATFORMS + ruby + +DEPENDENCIES + repertoire-faceting! diff --git a/vendor/gems/repertoire-faceting-0.6.0/INSTALL b/vendor/gems/repertoire-faceting-0.6.0/INSTALL new file mode 100644 index 0000000..d97cec9 --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/INSTALL @@ -0,0 +1,211 @@ +=== Installing Repertoire Faceting + +N.B. Repertoire Faceting requires Rails 3.2+, Ruby 2.0.0+, and JQuery 1.3.2+. + +== Short version. + +(You need a working Rails app, with a model, controller, and a partial to show the model) + + { in Gemfile } + gem 'repertoire-faceting' // 1 + + { install native bitset extensions } + $ bundle install + $ rake db:faceting:extensions:build + $ rake db:faceting:extensions:install { provide sudo your password } + + { in ./app/models/some_model.rb } + class SomeModel + include Repertoire::Faceting::Model // 2 + facet :some_column // 3 + end + + { in ./app/controllers/some_controller] + class SomeController + include Repertoire::Faceting::Controller // 4 + def base; return SomeModel; end // 5 + end + + { in ./config/routes.rb } + SomeApp::Application.routes.draw do + faceting_for :some_model // 6 + end + + { in ./public/javascripts/application.js } + //= require rep.faceting // 7 + + { in ./app/views/some_controller/index.html.erb } + +
// 11 +
// 12 +
// 13 +
+ + ... that's a complete faceted browser in only 13 new lines of code in your app! + + Additionally, you may wish to index the facets. At the console or in a migration: + + SomeModel.index_facets([:some_column]) + + The faceting subsystem automatically detects available facet indices and uses + them when appropriate. + + +== Detailed version. + +Start with a working Rails application with a PostgreSQL database. + + * Require repertoire-faceting in Gemfile. + + gem 'repertoire-faceting' + + * Make sure you use Rails version 3.0.2+, which adopted Arel 2.0.1. If necessary + you can pull rails, arel, and rack from git. + + gem 'rails', :git => 'git://github.com/rails/rails.git' + gem 'arel', :git => 'git://github.com/rails/arel.git' + gem "rack", :git => "git://github.com/rack/rack.git" + + * At the command line, bundle everything into your application: + + $ bundle install + + * From your application root, build and install the repertoire-faceting native + extensions to PostgreSQL. These provide a bitwise signature type used to + index and count facets. + + $ rake db:faceting:build { sudo will prompt you for your password } + + * Load the extension into your local application database. This ensures the + plpgsql language is installed, and loads (or re-loads) the new bitset signature + type. + + $ createlang plpgsql -U + $ rake db:faceting:load + + Or, if you prefer to use migrations create one with the following contents: + + .... + def self.up + load_faceting # unload_faceting is the reverse + end + + Before proceeding, you can confirm the module is installed as follows. + + $ psql -c "SELECT count('101010101'::signature);" -U + + * Install the faceting mixin in your Rails model and declare a facet on an + existing database column. (See the README for complete configuration options + for facets.) + + { ./app/models/painting.rb } + class Painting + include Repertoire::Faceting::Model + facet :genre + end + + * Test doing facet count and result queries: + + $ rails c + > Painting.count(:genre) + => {"Impressionist"=>2, "Medieval"=>2} + > Painting.refine(:genre => 'Impressionist') + => [#, + #] + + Or, with a base query as well: + + > Painting.where(["title like ?", 'Moon%']).count(:genre) + => {"Impressionist"=>1} + + * Add faceting webservices to your controller and define base() to indicate which model to base queries on + + { ./app/controllers/paintings_controller } + class PaintingsController + include Repertoire::Faceting::Controller + + def base + search = "%#{params[:search]}%" + Painting.where(["title like ?", search]) + end + end + + * Add faceting routes to your application. + + { ./config/routes.rb } + + PaintingsApp::Application.routes.draw do + faceting_for :paintings # NB must be BEFORE any resources! + ... + end + + Confirm they load: + + $ rake routes + ... + /paintings/counts/:facet(.:format) {:controller=>"paintings", :action=>"counts"} + paintings_results /paintings/results(.:format) {:controller=>"paintings", :action=>"results"} + ... + + * Add facet count and result widgets to your HTML page. The facet context div + collects widgets that affect the same query together. (For complete options, + see the README ) + + { ./app/views/paintings/index.html.erb } + + +
+
+
+
+ + * If you don't already have one, create a partial for displaying your model in + results lists. + + { ./app/views/paintings/_painting.html.erb } + +
+
Title: <%= painting.title %>
+
Painter: <%= painting.painter %>
+
Genre: <%= painting.genre %>
+
+ + * [ Optional ] Add bitset indexes to some facets on your model. The module will + automatically use facet indexes when they are available. Facet indexes scale + out of the box to over a million model items, and requires no additional + configuration. + + $ rails generate migration AddFacetIndex + + { ./db/migrate/add_facet_index.rb } + class AddFacetIndex < ActiveRecord::Migration + def self.up + Painting.index_facets([:genre]) + end + + def self.down + Painting.index_facets + end + end + + * [ Optional ] Periodically update indexes via a crontab task. + + { ./lib/tasks/update_facets.rake } + + task :reindex => :environment do + Painting.index_facets + end + + ... And then configure crontab to execute 'rake reindex' at appropriate intervals. \ No newline at end of file diff --git a/vendor/gems/repertoire-faceting-0.6.0/LICENSE b/vendor/gems/repertoire-faceting-0.6.0/LICENSE new file mode 100644 index 0000000..527dd56 --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/LICENSE @@ -0,0 +1,22 @@ +Copyright (c) 2009-2012 MIT Hyperstudio + +Permission is hereby granted, free of charge, to any person +obtaining a copy of this software and associated documentation +files (the "Software"), to deal in the Software without +restriction, including without limitation the rights to use, +copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the +Software is furnished to do so, subject to the following +conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +OTHER DEALINGS IN THE SOFTWARE. diff --git a/vendor/gems/repertoire-faceting-0.6.0/README b/vendor/gems/repertoire-faceting-0.6.0/README new file mode 100644 index 0000000..d694504 --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/README @@ -0,0 +1,223 @@ +=== Repertoire Faceting README + +Repertoire Faceting is highly scalable and extensible module for creating database-driven faceted browsers in Rails 3 & 4. It consists of three components: (1) a native PostgreSQL data type for constructing fast bitset indices over controlled vocabularies; (2) Rails model and controller mixins that add a faceting API to your existing application; and (3) a set of extensible javascript widgets for building user-interfaces. In only 10-15 lines of new code you can implement a fully-functional faceted browser for your existing Rails data models, with scalability out of the box to over 1,000,000 items. + +== Features + +Several features distinguish Repertoire Faceting from other faceting systems such as Simile Exhibit, Endeca, and Solr. + +Repertoire Faceting is an end-to-end solution that works with your existing database schema and Rails models. There's no need to munge your data into a proprietary format, run a separate facet index server, or construct your own user-interface widgets. (Conversely, however, your project needs to use Rails, PostgreSQL, and JQuery.) + +The module works equally well on small and large data sets, which means there's a low barrier to entry but existing projects can grow easily. In 'training wheels' mode, the module produces SQL queries for facet counts directly from declarations in your model so you can get a project up and running quickly. Then, after your dataset grows beyond a thousand items or so, just add indices as necessary. The module detects and uses these automatically, with no changes to your code or additional SQL necessary. + +Unlike some faceting systems, hierarchical vocabularies are supported out of the box. Using familiar SQL expressions you can decompose a date field into a drillable year / month / day facet. Or you can combine several columns into a single nested facet, for example from countries to states to cities. + +Both facet widgets and indexes are pluggable and extensible. You can subclass the javascript widgets to build drillable data visualizations, for example using bar graphs, donut and scatter charts or heat-maps to display the current search state and results. + +Similarly, you can write new facet implementations for novel data types, which automatically detect and index appropriate columns. For example, the module has been used to do facet value counts over GIS data points on a map, by drilling down through associated GIS layers using spatial logic relations. + +For an out-of-the box example using Repertoire Faceting, which demonstrates the module's visualization and scalability features, see the example application (http://github.com/yorkc/repertoire-faceting-example). + + +== Installation + +See the INSTALL document for a description of how to install the module and build a basic faceted browser for your existing Rails app. + + +== Running unit tests + +You can run the unit tests from the module's root directory. You will need a local PostgreSQL superuser role with your unix username (use 'createuser -Upostgres'). + + $ bundle install + $ rake db:faceting:build { sudo will prompt for your password } + $ rake db:create + $ rake test + + +== Generating documentation + +All API documentation, both ruby or javascript, is inline. To generate: + + $ rake doc + +For the javascript API documentation, please look in the source files. + + +== Faceting declarations (Model API) + +See Repertoire::Faceting::Model::ClassMethods + + +== Faceting webservices (Controller API) + +See Repertoire::Faceting::Controller + + +== Facet widgets / HTML (User Interface API) + +See rep.faceting.js inline documentation in the source tree + + +== Custom facet implementations + +See Repertoire::Faceting::Facets::AbstractFacet + + +== Updating Facet Indices + +It is very useful to create a rake task to update your application's indices. In the project's rake task file: + + task :reindex => :environment do + Painting.index_facets([:genre, :era]) + end + +Then run 'rake reindex' whenever you need to update indices manually. + +*static* If the facet data is unchanging, use a rake task like the one above to create indices manually while developing or deploying. + +*crontab* The easiest way to update indices periodically is to run a rake task like the one above via a UNIX tool such as launchd, periodic, or crontab. See the documentation for your tool of choice. + + +== Deployment + +Because repertoire-faceting depends on a native shared library loaded by the PostgreSQL server, the first time you deploy you will need to build and install the extension. + + $ bundle install --deployment + $ export RAILS_ENV=production + $ rake db:faceting:build + $ # ... from here, follow normal deployment procedure + + +== How the module works + +It is helpful to think of faceted data as a set of model items categorised by one or more controlled vocabularies, as this eliminates confusion from the start. (A faceted classification is neither object-oriented nor relational, though it can be represented in either.) For example, one might categorise Shakespeare's plays by a controlled vocabulary of genres -- comedy, history, tragedy, or romance. Counting the total number of plays for each vocabulary item in this "genre" facet, we see 13 comedies, 10 histories, 10 tragedies, and 4 romances. + +There are three direct implementations for faceted classifications like this in an SQL database. The controlled vocabulary can be listed explicitly in a separate table, or implicit in the range of values in a column on the central table (for single-valued facets) or on a join table (for multi-valued facets). Repertoire Faceting supports all of these configurations. + +*1:* Explicit controlled vocabulary, multiple valued facet + + genres plays_genres plays + ----+--------- ---------+---------- ----+------------------+--------- + id | name play_id | genre_id id | title | date ... + ----+--------- ---------+---------- ----+------------------|--------- + 1 | comedy 1 | 4 1 | The Tempest | + 2 | tragedy 2 | 3 2 | Henry 4, pt 1 | + 3 | history 3 | 3 3 | Henry 4, pt 2 | + 4 | romance 4 | 3 4 | Henry 5 | + 5 | 1 5 | As You Like It | + 6 | 1 6 | Comedy of Errors | + 7 | 2 7 | Macbeth | + 8 | 2 8 | Hamlet | + ... .... + +*2:* Implicit vocabulary, multiple valued facet + + plays_genres plays + ---------+---------- ----+------------------+--------- + play_id | genre_id id | title | date ... + ---------+---------- ----+------------------|--------- + 1 | romance 1 | The Tempest | + 2 | history 2 | Henry 4, pt 1 | + 3 | history 3 | Henry 4, pt 2 | + 4 | history 4 | Henry 5 | + 5 | comedy 5 | As You Like It | + 6 | comedy 6 | Comedy of Errors | + 7 | tragedy 7 | Macbeth | + 8 | tragedy 8 | Hamlet | + ... .... + +*3:* Implicit vocabulary, single valued facet + + plays + ----+-----------------+---------+--------- + id | title | genre | date ... + ----+-----------------|---------+--------- + 1 | The Tempest | romance | + 2 | Henry 4, pt 1 | history | + 3 | Henry 4, pt 2 | history | + 4 | Henry 5 | history | + 5 | As You Like It | comedy | + 6 | Comedy of Errors | comedy | + 7 | Macbeth | tragedy | + 8 | Hamlet | tragedy | + ... .... + +For all of these representations, Repertoire Faceting works by constructing an inverted bitset index from the controlled vocabulary to your central model. Each bit represents a distinct model row (plays.id in this example). 1 indicates the play is in the category, and 0 that it is not: + + _plays_genre_facet + ---------+----------- + genre | signature + ---------+----------- + comedy | 00001100 + history | 01110000 + romance | 10000000 + tragedy | 00000011 + +From these bitset "signatures", Repertoire Faceting can easily count the number of member plays for each category, even in combination with other facets and a base query. For example, the bitset signature for all plays whose title contains the search word "Henry" is 0110000. Masking this (via bitwise "and") with each signature in the genre index above, we see that there are 2 histories that match the base search - Henry 4 parts 1 & 2 - a none in the other categories: + + ---------+------------------ + genre | signature & base + ---------+------------------ + comedy | 00000000 + history | 01100000 + romance | 00000000 + tragedy | 00000000 + +Refinements on other facets are processed similarly, by looking up the relevant bitset signature for the refined value, and masking it against each potential value in the facet to be enumerated. + +As you may have noticed, this scheme depends on play ids being sequential. Otherwise many bits corresponding to no-existent ids are wasted in every signature. To address this issue, Repertoire Faceting examines the projected wastage in constructing bitset signatures from the primary key id of your model table. If more than a predefined amount (e.g. 15%) of the signature would be wasted, the module instead adds a new column of sequentially packed ids that are used only for faceted searches. When the model's facets are re-indexed, the ids are examined and repacked if too much space is wasted. + +References on faceted search: + +- http://flamenco.berkeley.edu/pubs.html +- http://en.wikipedia.org/wiki/Controlled_vocabulary + + +== Known issues + +- Running the unit tests issues warnings about a circular require. These can be ignored. + + +== PostgreSQL Faceting API + +Several bindings for the in-database faceting API are provided. In order of capability, they are: + +- signature C language, requires superuser permissions +- bytea Javascript language, requires plv8 extension +- varbit No language or superuser requirements + +In general, if you have superuser permissions you should build and install the C-language (signature) API, as it is more scalable than the others, at no cost. + +All the Repertoire Faceting APIs add functionality for bitwise operations and population counts to PostgreSQL. For API details, see the ext directory. + +Signature: an auto-sizing bitset with the following functions + +- count(a) => { count of 1s in a } +- contains(a, i) => { true if the ith bit of a set } +- members(a) => { set of integers corresponding to set bits } + +- sig_in, sig_out => { mandatory I/O functions } +- sig_and(a, b) => a & b +- sig_or(a, b) => a | b +- sig_xor(a) => ~a +- sig_length(a) => { number of bits in a } +- sig_min(a) => { lowest 1 in a, a.length } +- sig_get(a, i) => { ith bit of a, or 0 } +- sig_set(a, i, n) => { sets ith bit of a to n } +- sig_resize(a, n) => { resizes a to hold n bits } + +Bitwise signature operators: &, | + +Bitwise aggregates: + +- signature(int) => assemble ints into a signature +- collect(signature) => 'or' signature results together +- filter(signature) => 'and' signature results together + +Helper functions: + +- wastage(INT) -> REAL + +Aggregator that examines a table's primary key column, checking what proportion of signature bits constructed from the table would be wasted. If the proportion of wasted bits to valid bits is high, you should consider adding a new serial column. + +The Rails API introspects signature wastage before any facet indexing operation, and adds or removes a new serial column (called _packed_id) as necessary. \ No newline at end of file diff --git a/vendor/gems/repertoire-faceting-0.6.0/Rakefile b/vendor/gems/repertoire-faceting-0.6.0/Rakefile new file mode 100644 index 0000000..119f7df --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/Rakefile @@ -0,0 +1,104 @@ +# encoding: UTF-8 + +# +# Rake testing tasks for Repertoire Faceting +# +# N.B. Tasks mirror those of a standard Rails 4 app. +# + +require 'rdoc/task' +require 'pathname' + +require 'bundler' +Bundler.require(:default) + +# Setup + +dir = Pathname.new(__FILE__).dirname +load dir + 'lib/repertoire-faceting/tasks/all.rake' + +gemspec = eval(File.read("repertoire-faceting.gemspec")) + +$LOAD_PATH.unshift File.expand_path(dir + 'test') + +# Tasks + +desc 'Build the gem' +task :build => "#{gemspec.full_name}.gem" + +file "#{gemspec.full_name}.gem" => gemspec.files + ["repertoire-faceting.gemspec"] do + system "gem build repertoire-faceting.gemspec" + system "gem install repertoire-faceting-#{Repertoire::Faceting::VERSION}.gem" +end + +desc 'Default: run tests' +task :default => "test:psql:signature" + +namespace 'test:psql' do + [:signature, :varbit, :bytea].each do |api| + + desc "Run tests for PostgreSQL #{api} binding" + task api do + # find current PostgreSQL API binding name + current_api = %x( psql repertoire_testing -c'SELECT extname FROM pg_extension' ).split.grep( /(faceting\w*)/ )[0] + + # determine the extension name + api = "faceting_#{api}".sub(/_signature/, '') + + # switch into binding to test + %x( psql repertoire_testing -c'DROP EXTENSION #{current_api} CASCADE' ) if current_api + %x( psql repertoire_testing -c'CREATE EXTENSION #{api}' ) + + # run the tests + Dir.glob("test/cases/**/*_test.rb") do |file| + file.sub!(%r{test/}, '') + require file + end + end + end +end + +desc 'Generate documentation' +Rake::RDocTask.new(:doc) do |rdoc| + rdoc.rdoc_dir = 'doc' + rdoc.title = 'Repertoire Faceting' + rdoc.options << '--line-numbers' << '--inline-source' + rdoc.main = 'README' + rdoc.rdoc_files.include %w(FAQ INSTALL LICENSE README TODO) + rdoc.rdoc_files.include('lib/**/*.rb') +end + +namespace :db do + + desc 'Build the PostgreSQL test databases' + task :create do + %x( createdb -E UTF8 repertoire_testing ) + end + + desc 'Drop the PostgreSQL test databases' + task :drop do + %x( dropdb repertoire_testing ) + end + + namespace :schema do + desc "Create the text database schema" + task :load do + ENV["RAILS_ENV"] = "test" + require 'config' + require 'connection' + load "schema/schema.rb" + end + end + + namespace :fixtures do + desc "Load fixtures into the test database." + task :load do + require 'config' + %x( psql repertoire_testing -f #{FIXTURES_ROOT}/fixtures.sql ) + end + end + + desc "Create, load, and prepare database for test suite" + task :setup => [ 'db:faceting:extensions:install', + 'db:create', 'db:schema:load', 'db:fixtures:load' ] +end \ No newline at end of file diff --git a/vendor/gems/repertoire-faceting-0.6.0/SNIPPETS b/vendor/gems/repertoire-faceting-0.6.0/SNIPPETS new file mode 100644 index 0000000..b6682fc --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/SNIPPETS @@ -0,0 +1,425 @@ +======= potentially of use later: + +-- Utility function to expand nesting in facet indices +-- +-- Initially a acet index will include only leaves of +-- nesting tree. This function adds all interior nodes +-- with their respective aggregate signatures, and adds a +-- postgresql index to the nested facet value. +-- +-- e.g. given the facet values +-- '{USA,Florida}' '10' +-- '{USA,Iowa}' '01' +-- +-- the function inserts +-- '{USA}' '11' +-- +-- N.B. expand_nesting may only be called once on a table +-- it refuses to add internal node duplicates +-- +CREATE OR REPLACE FUNCTION expand_nesting(tbl TEXT, col TEXT) RETURNS VOID AS $$ +DECLARE + len INTEGER; +BEGIN + -- add unique index on facet value column + EXECUTE 'CREATE UNIQUE INDEX ' || tbl || '_' || col || '_ndx ON ' || quote_ident(tbl) || '(' || quote_ident(col) || ')'; + -- compute maximum nesting length for facet values + EXECUTE 'SELECT max(array_length(' || quote_ident(col) || ', 1)) FROM ' || quote_ident(tbl) + INTO len; + -- expand each level + FOR i IN REVERSE (len-1)..1 LOOP + EXECUTE 'INSERT INTO ' || quote_ident(tbl) || '(' || quote_ident(col) || ', signature)' + || ' SELECT ' || quote_ident(col) || '[1:' || i || '], collect(signature)' + || ' FROM ' || quote_ident(tbl) + || ' WHERE array_length(' || quote_ident(col) || ', 1) > ' || i + || ' GROUP BY ' || quote_ident(col) || '[1:' || i || ']'; + END LOOP; +END; +$$ LANGUAGE plpgsql; + +PG_FUNCTION_INFO_V1( sig_or ); + +Datum +sig_or( PG_FUNCTION_ARGS ) +{ + Signature *sig1, + *sig2, + *res; + int32 sig1bytes, + sig2bytes, + resbytes, + i; + uint8 c; + + sig1 = PG_GETARG_SIGNATURE_P(0); + sig1bytes = VARSIZE(sig1) - VARHDRSZ - SIGNATUREHDRSZ; + + sig2 = PG_GETARG_SIGNATURE_P(1); + sig2bytes = VARSIZE(sig2) - VARHDRSZ - SIGNATUREHDRSZ; + + resbytes = MAX(sig1bytes, sig2bytes); + + // if aggregate accumulator, don't allocate new memory + if (fcinfo->context && IsA(fcinfo->context, AggState) && resbytes == sig1bytes) { + res = sig1; + } else { + res = (Signature *) palloc0( resbytes + VARHDRSZ + SIGNATUREHDRSZ ); + SET_VARSIZE(res, resbytes + VARHDRSZ + SIGNATUREHDRSZ ); + } + res->len = MAX(sig1->len, sig2->len); + + for(i=0; idata[i]; + } + if (i < sig2bytes) { + c |= sig2->data[i]; + } + res->data[i] = c; + } + + PG_FREE_IF_COPY(sig1, 0); + PG_FREE_IF_COPY(sig2, 1); + + PG_RETURN_SIGNATURE_P( res ); +} + + +-- sql to see if packed_id argument was provided and add clause + + sql = 'UPDATE ' || facet_table_name(context, facet.name) + + IF (NOT like(sql, '%WHERE%')) THEN + sql = sql || ' WHERE _packed_id = ' || quote_literal(packed_id); + ELSE + sql = sql || ' AND _packed_id = ' || quote_literal(packed_id); + END IF; +END IF; + + + + +-- Facet declarations table + +CREATE TABLE _facets( + context TEXT NOT NULL, + name TEXT NOT NULL, + select_expr TEXT CHECK (select_expr IS NULL OR select_expr LIKE 'SELECT % FROM %'), + PRIMARY KEY (context, name) +); + +-- Utility functions for naming facet index tables and sequences + +CREATE OR REPLACE FUNCTION facet_table_name(context TEXT, name TEXT) RETURNS TEXT AS $$ +BEGIN + RETURN quote_ident('_' || context || '_' || name || '_facet'); +END; +$$ LANGUAGE plpgsql; + +CREATE OR REPLACE FUNCTION facet_seq_name(context TEXT) RETURNS TEXT AS $$ +BEGIN + RETURN quote_ident('_' || context || '_packed_id_seq'); +END; +$$ LANGUAGE plpgsql; + +-- Declare that a table will be used as faceting context [ provides for packed ids ] + +CREATE OR REPLACE FUNCTION declare_context(context TEXT) RETURNS VOID AS $$ +BEGIN + EXECUTE 'CREATE SEQUENCE ' || facet_seq_name(context); + EXECUTE 'ALTER TABLE ' || quote_ident(context) || ' ADD COLUMN _packed_id INT UNIQUE DEFAULT nextval( ''' || facet_seq_name(context) || ''' )'; +END; +$$ LANGUAGE plpgsql; + +-- Update all facet counts for the given context + +CREATE OR REPLACE FUNCTION reindex_facets(context TEXT) RETURNS VOID AS $$ +DECLARE + select_expr TEXT; +BEGIN + -- Pack index ids + EXECUTE 'ALTER SEQUENCE ' || facet_seq_name(context) || ' RESTART WITH 1'; + EXECUTE 'UPDATE production SET _packed_id = nextval( ''' || facet_seq_name(context) || ''' )'; + -- Update facets for context table + FOR facet IN SELECT * FROM _facets WHERE _facets.context = context LOOP + select_expr = facet.select_expr; + -- From expr defaults to context table and facet column + IF (select_expr IS NULL) THEN + select_expr = 'SELECT ' || facet.name || ' FROM ' || facet.context; + END IF; + -- Augment to collect signature + select_expr = replace(select_expr, 'FROM', ', sig_collect(' || context || '._packed_id) AS signature FROM'); + -- Remove old facet value table + EXECUTE 'DROP TABLE IF EXISTS ' || facet_table_name(context, facet.name); + -- Create facet value table, with signature of ids + EXECUTE 'CREATE TABLE ' || facet_table_name(context, facet.name) || ' AS ' || select_expr + || ' GROUP BY ' || facet.name; + END LOOP; +END; +$$ LANGUAGE plpgsql; + + +-- extra functions for hash operator class (but for some reason postgres' hash_any fn core dumps. params not right? +-- no big advantage, since will rarely be merging more than 10 signatures with UNION + +--CREATE OR REPLACE FUNCTION sig_hash( signature ) +-- RETURNS int4 +-- AS 'signature.so', 'sig_hash' +-- LANGUAGE C STRICT IMMUTABLE; + +--CREATE OPERATOR CLASS signature_ops +--DEFAULT FOR TYPE signature USING hash AS +-- OPERATOR 1 = , +-- FUNCTION 1 sig_hash(signature); + + +/* +PG_FUNCTION_INFO_V1(sig_hash); + +Datum +sig_hash(PG_FUNCTION_ARGS) +{ + Signature *sig = (Signature *) PG_GETARG_POINTER(0); + int32 sig_bytes, + sig_bits; + uint8 x; + Datum result; + + sig_bytes = sig->len / 8; + sig_bits = sig->len % 8; + + // clear unused bits to ensure hash equality + if (sig_bits > 0) { + x = 0xFF >> sig_bits; + sig->data[sig_bytes] &= ~x; + } + + result = hash_any((unsigned char *) sig->data, sig_bytes + 1); + + // Avoid leaking memory for toasted inputs + PG_FREE_IF_COPY(sig, 0); + + PG_RETURN_DATUM(result); +} + + + + + + +// end gis facet factory +return self; +} + + + + + + // if called before google earth ready, just return + if (!ge) + return; + + // determine how feature indices map onto quantiles + var category_size = counts.length / options.quantiles.categories; + console.log("category size: " + category_size); + + // create a placemark style for each quantile + var styles = []; + for (var i = 0; i < options.quantiles.categories; i += 1 ) { + var fraction = i / options.quantiles.categories; + + var style = gex.dom.buildStyle(options.style || {}); + var color = gex.util.blendColors(options.quantiles.low, options.quantiles.high, fraction); + //style.getPolyStyle().setColor(color); + console.log("category " + i + ": " + fraction); + + styles[i] = style; + } + + // create a map between feature ids and their choropleth styles + var quantile = {}; + $.each(counts, function(index, facet_value_count) { + var value = facet_value_count[0]; + var count = facet_value_count[1]; + var category = Math.floor( index / category_size ); + quantile[value] = styles[category]; + }) + + // walk the dom and update style on all matching placemarks + gex.dom.walk({ + rootObject: ge, + visitCallback: function() { + console.log(this.getType()); + if ('getType' in this && this.getType() == 'KmlPlacemark') { + var id = this.getId(); + var style = quantile[id]; + + console.log(id); + + //if (style) { + console.log('setting ' + id); + this.setStyleSelector(style); + this.setVisibility(true); + //} else { + // console.log('hiding ' + id); + // this.setVisibility(false); + //} + } + + return true; + } + }); + + + + '' + + '' + + '' + label + '' + + '' + + '' + + + + + -- + -- Aggregate for generating weighted sample data from tables + -- + -- Usage: + -- + -- Given a table of data and frequencies, generate an array of 45 + -- statistically-representative values: + -- + -- SELECT weighted_sample(surname, frequency, 45) FROM male_names; + -- + -- If you only want one: + -- + -- SELECT weighted_sample(surname, frequency) FROM male_names; + -- + -- The frequency can be any series of numbers representing relative + -- weights. It is not necessary that they sum to 1.0. The values them- + -- selves are cast to TEXT. + -- + -- You can turn the resulting values back into rows with unnest() and + -- join them to other sample data. See the Postgresql 8.4 documentation. + -- + + CREATE TYPE sample AS (vals TEXT[], freqs DOUBLE PRECISION[], sum DOUBLE PRECISION, size INTEGER); + + -- + -- Given X (a series of vals) and Y (a series of DOUBLE PRECISION values), return a random + -- X/id that conforms to the weighted sample of all values Y within the total. + -- + CREATE OR REPLACE FUNCTION sample_matrix(state sample) RETURNS TEXT[] AS $$ + DECLARE + running_sum DOUBLE PRECISION; + i INTEGER; + rand DOUBLE PRECISION; + samples TEXT[]; + BEGIN + FOR i IN 1..state.size LOOP + -- select a random value and loop through until hitting the corresponding item + rand := random(); + running_sum := 0.0; + i := 0; + WHILE rand >= running_sum LOOP + i := i + 1; + running_sum := running_sum + (state.freqs[i] / state.sum); + END LOOP; + samples := samples || state.vals[i]; + END LOOP; + RETURN samples; + END + $$ LANGUAGE plpgsql; + + CREATE OR REPLACE FUNCTION sample_matrix_single(state sample) RETURNS TEXT AS $$ + BEGIN + RETURN (sample_matrix(state))[1]; + END + $$ LANGUAGE plpgsql; + + CREATE OR REPLACE FUNCTION matrix_agg(state sample, id ANYELEMENT, val DOUBLE PRECISION, size INTEGER) RETURNS sample AS $$ + BEGIN + state.size := size; + state.vals := state.vals || id::TEXT; + state.freqs := state.freqs || val::DOUBLE PRECISION; + state.sum := state.sum + val; + RETURN state; + END + $$ LANGUAGE plpgsql; + + CREATE OR REPLACE FUNCTION matrix_agg_single(state sample, id ANYELEMENT, val DOUBLE PRECISION) RETURNS sample AS $$ + BEGIN + state.size := 1; + state.vals := state.vals || id::TEXT; + state.freqs := state.freqs || val::DOUBLE PRECISION; + state.sum := state.sum + val; + RETURN state; + END + $$ LANGUAGE plpgsql; + + CREATE AGGREGATE weighted_sample(ANYELEMENT, DOUBLE PRECISION) + ( + sfunc = matrix_agg_single, + stype = sample, + finalfunc = sample_matrix_single, + initcond = '({}, {}, 0.0, 0)' + ); + + CREATE AGGREGATE weighted_sample(ANYELEMENT, DOUBLE PRECISION, INTEGER) + ( + sfunc = matrix_agg, + stype = sample, + finalfunc = sample_matrix, + initcond = '({}, {}, 0.0, 0)' + ); + +CREATE EXTENSION faceting_bytea; + +CREATE OR REPLACE FUNCTION sample(size INT, max INT) RETURNS SETOF INT AS $$ +BEGIN + FOR i IN 1 .. size LOOP + RETURN NEXT floor(random() * max)::INT; + END LOOP; +END $$ LANGUAGE plpgsql IMMUTABLE STRICT; + +CREATE TEMPORARY TABLE data_ints AS SELECT DISTINCT sample(5000, 100000) as id; + +CREATE TEMPORARY TABLE data_sig AS SELECT signature(id) FROM data_ints; + +-- test 1: count + +SELECT di.count - ds.count AS difference FROM + (SELECT COUNT(id) FROM data_ints) AS di, + (SELECT COUNT(signature) FROM data_sig) AS ds; + +-- test 2: members + +SELECT di.count - combo.count AS difference FROM + (SELECT members(signature) AS id FROM data_sig + UNION + SELECT id FROM data_ints) AS combo, + (SELECT id FROM data_ints) AS di; + +-- test 3: sig_and + +SELECT + + + +# Register known in-database API bindings TODO - clean this up +API_BINDINGS = [:signature] +Dir.glob("#{Repertoire::Faceting::MODULE_PATH}/ext/*.sql") +.grep( %r{faceting_(?.*)--#{Repertoire::Faceting::VERSION}\.sql} ) do +API_BINDINGS << Regexp.last_match[:name].to_sym +end + + + +CREATE or replace FUNCTION sig_rpad1( sig BYTEA, bytes INT ) RETURNS BYTEA AS $PROC$ + /* the spirit line: use sql to right pad a bytea in SQL, then send it back to js */ + /* it seems there's no other way to call byteacat from plv8 */ + var result = plv8.execute("SELECT byteacat($1, ($$\\x$$ || repeat($$00$$, $2))::BYTEA) AS val", [ sig, bytes ])[0].val; + return result; +$PROC$ LANGUAGE plv8 STRICT IMMUTABLE; + + diff --git a/vendor/gems/repertoire-faceting-0.6.0/TODO b/vendor/gems/repertoire-faceting-0.6.0/TODO new file mode 100644 index 0000000..ce1f638 --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/TODO @@ -0,0 +1,121 @@ +DESIRED FEATURES / IMPROVEMENTS. + +-- mysql support + [ design: mysql has a count function and bitwise operators for blobs. + Adding support is a matter of defining the adapter and moving signature() + calls into the postgresql adapter ] + +-- modify widgets to multiplex ajax calls to work around 2 call limit + in many browsers + [ design: fetch() queues ajax calls; update() implementations request + queue and merge current webservice call with ones already in the queue. + on controller side, receive multiple facet names, iterate, and bundle. + fetch() then unbundles the results can dispatches them to appropriate + callbacks ] + + +TODO + +-- clean up population in postgresql adapter? +-- clean up throughout the facet definitions +-- test harness + - test all extensions in order, skipping if cannot load +-- simplify the in-database API + - use materialized views instead of dropping and adding tables + - move signature_wastage into an aggregate function + - remove the renumber function + - the only remaining function should be expand_nesting +-- reference to the ActiveRecord API in the README +-- revise the README to describe API layers & use +-- Makefile is verbose; use patterns + + + + +- deploy to menzinga, document process +- make sure works with postgresql-crontab +- check formatting of all docs DONE + +- make rake tasks smarter about detecting whether to run NOT NECESSARY +- make rake tasks automatically choose task dep on db type NOT TO DO + +- make sure example app can be set up and deployed via rake DONE + +- README + * recipe for running tests DONE + * deployment + * recipe for adding facets to new app DONE + * declaring facets DONE + * faceting db api DONE + * installing postgresql extensions DONE + * migrations for indexing DONE + * updating indices (a) postgresql-crontab (b) rake crontab DONE +- generate routes that don't conflict with resource routes + (rails thinks /nobelists/results is the nobelist named 'results') NOT TO DO +- FAQ + * "not grouped error" + * migrations and facet decls + * facet plugin registration (and multiple claims) + +- repertoire-assets loaded automatically DONE +- figure out the faceting context state default DONE + +- get rid of annoying load warnings on test KNOWN ISSUE +- cannot refine on null values in facets KNOWN ISSUE + +- integration with postgresql crontab +- push compiled gems to gemcutter DONE (EXCEPT FACETING) + +- check nested indexed facet queries DONE +- test harness: unindexed / indexed for same test DONE + +- optimize count query: (a) empty base query DONE + (b) join masks rather than looping NOT NECESSARY + +- working on rails 3.0.1? NOT POSSIBLE +- citizens example DONE +- final level of nested faceting ~1 value DONE +- birth_decade has a null value DONE +- sorting extension facet example not working DONE +- final level of nested faceting ~1 value DONE +- birth_decade has a null value DONE + + +- indexed nested facet widget not working DONE +- factor out facet functionality from relation DONE +- not obeying orders for synthesized facets DONE +- auto-generated groups scan all joined tables NOT NECESSARY +- allow facet defs in method bodies KNOWN ISSUE RAILS 3 + +- calculated relations conflict with table names DONE +- pluggable facet implementations DONE +- results from indexes DONE +- nested faceting not working DONE +- remove mysql references in test suite DONE +- run all tests indexed and unindexed DONE +- migrations for facet indexes DONE +- indexed counts DONE +- mix indexed and non-indexed DONE +- test cases running in 'training wheels' mode DONE +- controller mixin with counts/results DONE +- routes extensions DONE +- get ride of annoying load warnings DONE - MUST REDO +- get core examples working DONE +- 'training wheels' using plain-jane SQL DONE +- facets should inherit scope DONE +- install procedure for new app DONE + * arrange to load without a generator/config file DONE + * check rake tasks load when installed in an app DONE +- system for checking presence of indexes DONE +- adapter for running indexed counts/queries DONE +- sort mode option built in to widget NOT TO DO +- minimum option built in to widget NOT TO DO +- widgets multiplex calls to count NOT TO DO +- rake task to update all indices NOT TO DO +- rake task to verify all indices NOT TO DO +- mysql for 'training wheels' NOT TO DO +- mysql adapter for indexing NOT TO DO +- rake task to install postgresql extensions DONE +- new method: relation.facet[:name] ... finds the facet relation, merges with current one, and returns result DONE +- sorting defaults NOT TO DO +- prettier output for raw facet relations in to_s NOT TO DO [ db dependent ] diff --git a/vendor/gems/repertoire-faceting-0.6.0/bin/bundler b/vendor/gems/repertoire-faceting-0.6.0/bin/bundler new file mode 100755 index 0000000..72c62ec --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/bin/bundler @@ -0,0 +1,16 @@ +#!/usr/bin/env ruby +# +# This file was generated by Bundler. +# +# The application 'bundler' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require 'pathname' +ENV['BUNDLE_GEMFILE'] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +require 'rubygems' +require 'bundler/setup' + +load Gem.bin_path('bundler', 'bundler') diff --git a/vendor/gems/repertoire-faceting-0.6.0/bin/erubis b/vendor/gems/repertoire-faceting-0.6.0/bin/erubis new file mode 100755 index 0000000..2c7348b --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/bin/erubis @@ -0,0 +1,16 @@ +#!/usr/bin/env ruby +# +# This file was generated by Bundler. +# +# The application 'erubis' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require 'pathname' +ENV['BUNDLE_GEMFILE'] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +require 'rubygems' +require 'bundler/setup' + +load Gem.bin_path('erubis', 'erubis') diff --git a/vendor/gems/repertoire-faceting-0.6.0/bin/rackup b/vendor/gems/repertoire-faceting-0.6.0/bin/rackup new file mode 100755 index 0000000..8cc9953 --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/bin/rackup @@ -0,0 +1,16 @@ +#!/usr/bin/env ruby +# +# This file was generated by Bundler. +# +# The application 'rackup' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require 'pathname' +ENV['BUNDLE_GEMFILE'] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +require 'rubygems' +require 'bundler/setup' + +load Gem.bin_path('rack', 'rackup') diff --git a/vendor/gems/repertoire-faceting-0.6.0/bin/rails b/vendor/gems/repertoire-faceting-0.6.0/bin/rails new file mode 100755 index 0000000..657440d --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/bin/rails @@ -0,0 +1,16 @@ +#!/usr/bin/env ruby +# +# This file was generated by Bundler. +# +# The application 'rails' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require 'pathname' +ENV['BUNDLE_GEMFILE'] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +require 'rubygems' +require 'bundler/setup' + +load Gem.bin_path('railties', 'rails') diff --git a/vendor/gems/repertoire-faceting-0.6.0/bin/rake b/vendor/gems/repertoire-faceting-0.6.0/bin/rake new file mode 100755 index 0000000..26c7a2d --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/bin/rake @@ -0,0 +1,16 @@ +#!/usr/bin/env ruby +# +# This file was generated by Bundler. +# +# The application 'rake' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require 'pathname' +ENV['BUNDLE_GEMFILE'] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +require 'rubygems' +require 'bundler/setup' + +load Gem.bin_path('rake', 'rake') diff --git a/vendor/gems/repertoire-faceting-0.6.0/bin/sprockets b/vendor/gems/repertoire-faceting-0.6.0/bin/sprockets new file mode 100755 index 0000000..09a1ad1 --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/bin/sprockets @@ -0,0 +1,16 @@ +#!/usr/bin/env ruby +# +# This file was generated by Bundler. +# +# The application 'sprockets' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require 'pathname' +ENV['BUNDLE_GEMFILE'] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +require 'rubygems' +require 'bundler/setup' + +load Gem.bin_path('sprockets', 'sprockets') diff --git a/vendor/gems/repertoire-faceting-0.6.0/bin/thor b/vendor/gems/repertoire-faceting-0.6.0/bin/thor new file mode 100755 index 0000000..8421e00 --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/bin/thor @@ -0,0 +1,16 @@ +#!/usr/bin/env ruby +# +# This file was generated by Bundler. +# +# The application 'thor' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require 'pathname' +ENV['BUNDLE_GEMFILE'] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +require 'rubygems' +require 'bundler/setup' + +load Gem.bin_path('thor', 'thor') diff --git a/vendor/gems/repertoire-faceting-0.6.0/bin/tilt b/vendor/gems/repertoire-faceting-0.6.0/bin/tilt new file mode 100755 index 0000000..09fe73e --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/bin/tilt @@ -0,0 +1,16 @@ +#!/usr/bin/env ruby +# +# This file was generated by Bundler. +# +# The application 'tilt' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require 'pathname' +ENV['BUNDLE_GEMFILE'] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +require 'rubygems' +require 'bundler/setup' + +load Gem.bin_path('tilt', 'tilt') diff --git a/vendor/gems/repertoire-faceting-0.6.0/bin/tt b/vendor/gems/repertoire-faceting-0.6.0/bin/tt new file mode 100755 index 0000000..6e3920b --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/bin/tt @@ -0,0 +1,16 @@ +#!/usr/bin/env ruby +# +# This file was generated by Bundler. +# +# The application 'tt' is installed as part of a gem, and +# this file is here to facilitate running it. +# + +require 'pathname' +ENV['BUNDLE_GEMFILE'] ||= File.expand_path("../../Gemfile", + Pathname.new(__FILE__).realpath) + +require 'rubygems' +require 'bundler/setup' + +load Gem.bin_path('treetop', 'tt') diff --git a/vendor/gems/repertoire-faceting-0.6.0/ext/Makefile b/vendor/gems/repertoire-faceting-0.6.0/ext/Makefile new file mode 100644 index 0000000..fdc0e64 --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/ext/Makefile @@ -0,0 +1,37 @@ +#------------------------------------------------------------------------- +# +# Makefile-- +# Makefile for Repertoire faceting API +# +# By default, this builds against an existing PostgreSQL installation +# (the one identified by whichever pg_config is first in your path). +# +#------------------------------------------------------------------------- + +API_VERSION = 0.6.0 + +MODULES = signature/signature +EXTENSION = signature/faceting \ + bytea/faceting_bytea \ + varbit/faceting_varbit +DATA_built = faceting--$(API_VERSION).sql \ + faceting_bytea--$(API_VERSION).sql \ + faceting_varbit--$(API_VERSION).sql +DOCS = README.faceting + +PG_CONFIG = pg_config +PGXS := $(shell $(PG_CONFIG) --pgxs) + +-include $(PGXS) +# the dash above means loading pgxs may fail silently on Heroku, but +# the API bindings will still be built: +api_bindings : $(DATA_built) + +faceting--$(API_VERSION).sql: signature/signature.sql common/util.sql + cat signature/signature.sql common/util.sql > $@ + +faceting_bytea--$(API_VERSION).sql: bytea/bytea.sql common/util.sql + cat bytea/bytea.sql common/util.sql > $@ + +faceting_varbit--$(API_VERSION).sql: varbit/varbit.sql common/util.sql + cat varbit/varbit.sql common/util.sql > $@ diff --git a/vendor/gems/repertoire-faceting-0.6.0/ext/README.faceting b/vendor/gems/repertoire-faceting-0.6.0/ext/README.faceting new file mode 100644 index 0000000..766de1e --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/ext/README.faceting @@ -0,0 +1,48 @@ +-- +-- +-- In-database support for Repertoire faceting module. +-- +-- These libraries add scalable faceted indexing to the PostgreSQL database. +-- +-- Basic approach is similar to other faceted browsers (Solr, Exhibit): an inverted bitmap index +-- allows fast computation of facet value counts, given a base context and prior facet refinements. +-- Bitsets can also be used to compute the result set of items. +-- +-- There are three bindings for the API. The first extends PostgreSQL with a new bitset datatype +-- written in C (called 'signature'). This version provides scaleable faceting up to 1,000,000 items +-- and beyond, but requires control over the PostgreSQL server instance to build and load the C +-- extensions. +-- +-- The second is implemented using PostgreSQL's built-in VARBIT data type, and scales to a rough +-- limit of about 30,000 items. It works in exactly the same way as the 'signature' data type above, +-- but is about a factor of 5-10 slower. However, it does not require administrative control over +-- the database server to install or use and so is suited to shared host deployment. +-- +-- The third uses PostgreSQL's built-in BYTEA data type, processed via the Google Javascript +-- language binding plv8 (https://code.google.com/p/plv8js/wiki/PLV8). Scalability and performance +-- are unknown, but should be similar to the native C 'signature' type. However, the server needs +-- to have the PLV8 language extension installed. +-- +-- Only one binding of the API needs to be loaded at any time. Each consists of: +-- +-- (a) functions for accessing the bitset data types. These are used to store inverted indices from +-- facet values to item ids. Functions are provided for doing refinements and counts on items +-- with a given facet value. +-- +-- (b) facilities for adding a packed (continuous) id sequence to the main item table. Packed ids +-- are used in the facet value indexes. +-- +-- (c) utility functions for creating/updating packed ids and facet value index tables, e.g. in +-- a crontab task. +-- +-- The API bindings can each be built as a PostgreSQL extension, and then loaded and dropped using +-- CREATE EXTENSION and DROP EXTENSION ... +-- +-- For hosts without administrative access, the individual sql files can be sourced directly. +-- +-- Installation (in a Rails app) +-- +-- $ cd repertoire-faceting +-- $ rake db:faceting:extensions:build +-- $ rake db:faceting:extensions:install +-- \ No newline at end of file diff --git a/vendor/gems/repertoire-faceting-0.6.0/ext/bytea/bytea.sql b/vendor/gems/repertoire-faceting-0.6.0/ext/bytea/bytea.sql new file mode 100644 index 0000000..05a6e61 --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/ext/bytea/bytea.sql @@ -0,0 +1,173 @@ +-- ============================================================================ +-- Faceting API implementing bitmap indices using PostgreSQL's built-in BYTEA +-- type, processed using plv8 typed arrays. +-- +-- This API is suitable for deployment on Heroku, where plv8 is installed by +-- default. Performance is many times better than the VARBIT-based faceting +-- API, primarily because of optimisations in memory handling in the count +-- function. +-- +-- See https://code.google.com/p/plv8js/wiki/PLV8 +-- https://postgres.heroku.com/blog/past/2013/6/5/javascript_in_your_postgres/ +-- +-- Christopher York +-- MIT Hyperstudio +-- February 2014 +-- ============================================================================ + +CREATE EXTENSION IF NOT EXISTS plv8; + +SET bytea_output TO hex; + +-- these functions are in pl/pgsql, because they involve appending bytea values, +-- which is easier done with direct access to the || operator + +CREATE FUNCTION @extschema@.sig_resize( sig BYTEA, bits INT ) RETURNS BYTEA AS $$ +DECLARE + len INT; + bytes INT; +BEGIN + bytes := ceil(bits / 8.0)::INT; + len := length(sig); + IF bytes > len THEN + -- RAISE NOTICE 'Extending signature from % to % bytes', len, bytes; + RETURN sig || ('\x' || repeat('00', bytes - len))::BYTEA; + ELSIF bits < len THEN + -- no provision in PostgreSQL for truncating a BYTEA + RETURN sig; + END IF; + RETURN sig; +END $$ LANGUAGE plpgsql STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_set( sig BYTEA, pos INT, val INT) RETURNS BYTEA AS $$ +BEGIN + RETURN set_bit(sig_resize(sig, pos+1), pos, val); +END $$ LANGUAGE plpgsql STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_set( sig BYTEA, pos INT) RETURNS BYTEA AS $$ +BEGIN + RETURN @extschema@.sig_set(sig, pos, 1); +END $$ LANGUAGE plpgsql STRICT IMMUTABLE; + +-- these functions are in javascript, because (1) pl/pgsql is close +-- to the worst language in the world; (2) plv8's typed arrays make +-- the count function much faster + +CREATE FUNCTION @extschema@.sig_get( sig BYTEA, pos INT ) RETURNS INT AS $$ + if (pos <= sig.length * 8) { + return sig[ Math.floor(pos / 8) ] >> (pos % 8) & 1; + } else { + return 0; + } +$$ LANGUAGE plv8 STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_length( sig BYTEA ) RETURNS INT AS $$ + return sig.length; +$$ LANGUAGE plv8 STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_and(sig1 BYTEA, sig2 BYTEA) RETURNS BYTEA AS $$ + if (sig2.length < sig1.length) { + var tmp = sig1; + sig1 = sig2; + sig2 = tmp; + } + for (var i = 0; i < sig1.length; i++) { + sig1[i] = sig1[i] & sig2[i]; + } + return sig1; +$$ LANGUAGE plv8 STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_or(sig1 BYTEA, sig2 BYTEA) RETURNS BYTEA AS $$ + if (sig2.length > sig1.length) { + var tmp = sig1; + sig1 = sig2; + sig2 = tmp; + } + for (var i = 0; i < sig2.length; i++) { + sig1[i] = sig1[i] | sig2[i]; + } + return sig1; +$$ LANGUAGE plv8 STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.count(sig bytea) RETURNS int4 AS $$ + var count_table = [ + 0, 1, 1, 2, 1, 2, 2, 3, 1, 2, 2, 3, 2, 3, 3, 4, + 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5, + 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5, + 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, + 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5, + 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, + 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, + 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7, + 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5, + 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, + 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, + 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7, + 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, + 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7, + 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7, + 4, 5, 5, 6, 5, 6, 6, 7, 5, 6, 6, 7, 6, 7, 7, 8, + ]; + var count = 0; + for (var i = 0; i < sig.length; i++) { count += count_table[ sig[i] ]; } + return count; +$$ LANGUAGE plv8 STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.contains( sig BYTEA, pos INT ) RETURNS BOOL AS $$ + return (pos <= sig.length * 8) && (sig[ Math.floor(pos / 8) ] >> (pos % 8) & 1); +$$ LANGUAGE plv8 STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.members( sig BYTEA ) RETURNS SETOF INT AS $$ + for (var i = 0; i < sig.length; i++) { + for (var j = 0; j < 8; j++) { + if (sig[i] >> j & 1) { + plv8.return_next(i * 8 + j); + } + } + } +$$ LANGUAGE plv8 STRICT IMMUTABLE; + + +-- operators for faceting + +CREATE OPERATOR @extschema@.& ( + leftarg = BYTEA, + rightarg = BYTEA, + procedure = @extschema@.sig_and, + commutator = & +); + +CREATE OPERATOR @extschema@.| ( + leftarg = BYTEA, + rightarg = BYTEA, + procedure = @extschema@.sig_or, + commutator = | +); + +CREATE OPERATOR @extschema@.+ ( + leftarg = BYTEA, + rightarg = int, + procedure = @extschema@.sig_set +); + + +-- aggregate functions for faceting + +CREATE AGGREGATE @extschema@.collect( BYTEA ) +( + sfunc = @extschema@.sig_or, + stype = BYTEA +); + +CREATE AGGREGATE @extschema@.filter( BYTEA ) +( + sfunc = @extschema@.sig_and, + stype = BYTEA +); + +CREATE AGGREGATE @extschema@.signature( INT ) +( + sfunc = @extschema@.sig_set, + stype = BYTEA, + initcond = '' +); \ No newline at end of file diff --git a/vendor/gems/repertoire-faceting-0.6.0/ext/bytea/faceting_bytea.control b/vendor/gems/repertoire-faceting-0.6.0/ext/bytea/faceting_bytea.control new file mode 100644 index 0000000..0e5c4d2 --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/ext/bytea/faceting_bytea.control @@ -0,0 +1,6 @@ +# Faceting PostgreSQL extension module + +comment = 'API for faceted indexing and queries (based on plv8 + bytea bitmaps)' +requires = 'plv8, plpgsql' +default_version = '0.6.0' +schema = 'facet' \ No newline at end of file diff --git a/vendor/gems/repertoire-faceting-0.6.0/ext/common/util.sql b/vendor/gems/repertoire-faceting-0.6.0/ext/common/util.sql new file mode 100644 index 0000000..da21118 --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/ext/common/util.sql @@ -0,0 +1,29 @@ + +-- These functions are common to all bindings of the faceting API + +-- Aggregator to measure how many bits from a loosely-packed id column would be wasted, if +-- they were all collected into a bitset signature. Returns a float between 0 (no waste) +-- and 1.0 (all waste). An example of its use: +-- +-- SELECT wastage(id) FROM nobelists +-- =# 0.999999 +-- +-- ALTER TABLE nobelists ADD COLUMN _packed_id SERIAL +-- SELECT wastage(_packed_id) FROM nobelists +-- =# 0.015625 +-- +CREATE FUNCTION @extschema@.wastage_proportion(state INT[]) RETURNS double precision AS $$ + SELECT (1.0 - (state[1]::double precision / (COALESCE(state[2], 0.0) + 1.0))) +$$ LANGUAGE sql; + +CREATE FUNCTION @extschema@.wastage_accum(state INT[], val INT) RETURNS INT[] AS $$ + SELECT ARRAY[ state[1]+1, GREATEST(state[2], val) ]; +$$ LANGUAGE sql; + +CREATE AGGREGATE @extschema@.wastage( INT ) +( + sfunc = @extschema@.wastage_accum, + stype = INT[], + finalfunc = @extschema@.wastage_proportion, + initcond = '{0,0}' +); \ No newline at end of file diff --git a/vendor/gems/repertoire-faceting-0.6.0/ext/extconf.rb b/vendor/gems/repertoire-faceting-0.6.0/ext/extconf.rb new file mode 100644 index 0000000..2f521ee --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/ext/extconf.rb @@ -0,0 +1,3 @@ +require "mkmf" + +# nothing to do since we use a hand-coded Makefile diff --git a/vendor/gems/repertoire-faceting-0.6.0/ext/signature/faceting.control b/vendor/gems/repertoire-faceting-0.6.0/ext/signature/faceting.control new file mode 100644 index 0000000..932093c --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/ext/signature/faceting.control @@ -0,0 +1,6 @@ +# Faceting PostgreSQL extension module + +comment = 'API for faceted indexing and queries (based on custom C bitmap type)' +requires = plpgsql +default_version = '0.6.0' +schema = 'facet' \ No newline at end of file diff --git a/vendor/gems/repertoire-faceting-0.6.0/ext/signature/signature.c b/vendor/gems/repertoire-faceting-0.6.0/ext/signature/signature.c new file mode 100644 index 0000000..c836f08 --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/ext/signature/signature.c @@ -0,0 +1,740 @@ +#include "postgres.h" +#include "fmgr.h" +#include "funcapi.h" + +PG_MODULE_MAGIC; + +typedef struct +{ + int32 vl_len_; + uint32 len; + uint8 data[1]; +} Signature; + +#define MIN(X,Y) ((X) < (Y) ? (X) : (Y)) +#define MAX(X,Y) ((X) > (Y) ? (X) : (Y)) + +#define IN_AGGR (fcinfo->context && IsA(fcinfo->context, AggState)) +#define AGGR_GROW_SIZE 8192 + +/* + * fmgr interface macros + */ +#define DatumGetSignatureP(X) ((Signature *) PG_DETOAST_DATUM(X)) +#define DatumGetSignaturePCopy(X) ((Signature *) PG_DETOAST_DATUM_COPY(X)) +#define SignaturePGetDatum(X) PointerGetDatum(X) +#define PG_GETARG_SIGNATURE_P(n) DatumGetSignatureP(PG_GETARG_DATUM(n)) +#define PG_GETARG_SIGNATURE_P_COPY(n) DatumGetSignaturePCopy(PG_GETARG_DATUM(n)) +#define PG_RETURN_SIGNATURE_P(x) return SignaturePGetDatum(x) + +/* Header overhead *in addition to* VARHDRSZ */ +#define SIGNATUREHDRSZ sizeof(uint32) + + +Datum sig_in( PG_FUNCTION_ARGS ); +Datum sig_out( PG_FUNCTION_ARGS ); +Datum sig_resize( PG_FUNCTION_ARGS ); +Datum sig_set( PG_FUNCTION_ARGS ); +Datum sig_get( PG_FUNCTION_ARGS ); +Datum sig_length( PG_FUNCTION_ARGS ); +Datum sig_min( PG_FUNCTION_ARGS ); +Datum sig_and( PG_FUNCTION_ARGS ); +Datum sig_or( PG_FUNCTION_ARGS ); +Datum sig_xor( PG_FUNCTION_ARGS ); +Datum sig_on( PG_FUNCTION_ARGS ); + +Datum contains( PG_FUNCTION_ARGS ); +Datum members( PG_FUNCTION_ARGS ); +Datum count( PG_FUNCTION_ARGS ); + +Datum sig_cmp( PG_FUNCTION_ARGS ); +Datum sig_lt( PG_FUNCTION_ARGS ); +Datum sig_lte( PG_FUNCTION_ARGS ); +Datum sig_eq( PG_FUNCTION_ARGS ); +Datum sig_gt( PG_FUNCTION_ARGS ); +Datum sig_gte( PG_FUNCTION_ARGS ); + +Datum sig_hash( PG_FUNCTION_ARGS ); + +int COUNT_TABLE[] = { + 0, 1, 1, 2, 1, 2, 2, 3, 1, 2, 2, 3, 2, 3, 3, 4, + 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5, + 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5, + 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, + 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5, + 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, + 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, + 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7, + 1, 2, 2, 3, 2, 3, 3, 4, 2, 3, 3, 4, 3, 4, 4, 5, + 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, + 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, + 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7, + 2, 3, 3, 4, 3, 4, 4, 5, 3, 4, 4, 5, 4, 5, 5, 6, + 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7, + 3, 4, 4, 5, 4, 5, 5, 6, 4, 5, 5, 6, 5, 6, 6, 7, + 4, 5, 5, 6, 5, 6, 6, 7, 5, 6, 6, 7, 6, 7, 7, 8, +}; + + +PG_FUNCTION_INFO_V1( sig_in ); + +Datum +sig_in( PG_FUNCTION_ARGS ) +{ + char *arg = PG_GETARG_CSTRING(0); + int32 len, + bytes; + Signature *result; + + char *sptr; + uint8 *bptr; + uint8 x; + + len = strlen(arg); + bytes = (len + 7) / 8 + VARHDRSZ + SIGNATUREHDRSZ; + + result = (Signature *) palloc0(bytes); + SET_VARSIZE(result, bytes); + result->len = len; + + bptr = result->data; + x = 0x80; + for (sptr = arg; *sptr; sptr++) { + if (*sptr == '1') { + *bptr |= x; + } + x >>= 1; + if (x == 0) { + x = 0x80; + bptr++; + } + } + + PG_RETURN_SIGNATURE_P(result); +} + + +PG_FUNCTION_INFO_V1( sig_out ); + +Datum +sig_out( PG_FUNCTION_ARGS ) +{ + Signature *s = PG_GETARG_SIGNATURE_P(0); + char *result; + uint8 *bptr, + x; + char *sptr; + int32 len, + i, j, k; + + len = s->len; + result = (char *) palloc(len + 1); + bptr = s->data; + sptr = result; + + for (i = 0; i <= len - 8; i += 8, bptr++) { + x = *bptr; + for (j = 0; j < 8; j++) { + *sptr++ = (x & 0x80) ? '1' : '0'; + x <<= 1; + } + } + if (i < len) { + x = *bptr; + for (k = i; k < len; k++) { + *sptr++ = (x & 0x80) ? '1' : '0'; + x <<= 1; + } + } + *sptr = '\0'; + + PG_RETURN_CSTRING(result); +} + + +PG_FUNCTION_INFO_V1( sig_resize ); + +Datum +sig_resize( PG_FUNCTION_ARGS ) +{ + Signature *sig, + *res; + int32 sigbytes, + resbytes, + reslen; + + sig = PG_GETARG_SIGNATURE_P(0); + sigbytes = VARSIZE(sig) - VARHDRSZ - SIGNATUREHDRSZ; + + reslen = PG_GETARG_INT32(1); + resbytes = (reslen + 7) / 8; + + res = (Signature *) palloc0( resbytes + VARHDRSZ + SIGNATUREHDRSZ ); + SET_VARSIZE(res, resbytes + VARHDRSZ + SIGNATUREHDRSZ); + res->len = reslen; + + memcpy(res->data, sig->data, MIN(sigbytes, resbytes)); + + PG_FREE_IF_COPY(sig, 0); + + PG_RETURN_SIGNATURE_P( res ); +} + + +PG_FUNCTION_INFO_V1( sig_set ); + +Datum +sig_set( PG_FUNCTION_ARGS ) +{ + Signature *sig, + *res; + int32 sigbytes, + resbytes, + index, + bit, + byte_offset, + bit_offset; + uint8 c; + + sig = PG_GETARG_SIGNATURE_P(0); + sigbytes = VARSIZE(sig) - VARHDRSZ - SIGNATUREHDRSZ; + + index = PG_GETARG_INT32(1); + if (PG_NARGS() == 3) { + bit = PG_GETARG_INT32(2); + } else { + bit = 1; + } + + byte_offset = index / 8; + bit_offset = index % 8; + + if (byte_offset >= sigbytes) { + resbytes = byte_offset + (IN_AGGR ? AGGR_GROW_SIZE : 1); + } else { + resbytes = sigbytes; + } + + res = (Signature *) palloc0( resbytes + VARHDRSZ + SIGNATUREHDRSZ ); + SET_VARSIZE(res, resbytes + VARHDRSZ + SIGNATUREHDRSZ ); + memcpy(res->data, sig->data, MIN(sigbytes, resbytes)); + + res->len = MAX(sig->len, index+1); + + c = res->data[byte_offset]; + if (bit) { + c |= (0x80 >> bit_offset); + } else { + c &= ~(0x80 >> bit_offset); + } + res->data[byte_offset] = c; + + PG_FREE_IF_COPY(sig, 0); + + PG_RETURN_SIGNATURE_P( res ); +} + + +PG_FUNCTION_INFO_V1( sig_get ); + +Datum +sig_get( PG_FUNCTION_ARGS ) +{ + Signature *sig; + int32 index, + byte_offset, + bit_offset, + c, + bit; + + sig = PG_GETARG_SIGNATURE_P(0); + index = PG_GETARG_INT32(1); + + if (index > sig->len) { + bit = 0; + } else { + byte_offset = index / 8; + bit_offset = index % 8; + + c = sig->data[byte_offset]; + if (c & (0x80 >> bit_offset)) { + bit = 1; + } else { + bit = 0; + } + } + + PG_FREE_IF_COPY(sig, 0); + + PG_RETURN_INT32( bit ); +} + + +PG_FUNCTION_INFO_V1( contains ); + +Datum +contains( PG_FUNCTION_ARGS ) +{ + Signature *sig; + int32 sigbytes, + index, + byte_offset, + bit_offset, + c, + bit; + + sig = PG_GETARG_SIGNATURE_P(0); + sigbytes = VARSIZE(sig) - VARHDRSZ - SIGNATUREHDRSZ; + + index = PG_GETARG_INT32(1); + + if (index > sig->len) { + bit = 0; + } else { + byte_offset = index / 8; + bit_offset = index % 8; + + c = sig->data[byte_offset]; + if (c & (0x80 >> bit_offset)) { + bit = 1; + } else { + bit = 0; + } + } + + PG_FREE_IF_COPY(sig, 0); + + PG_RETURN_BOOL( bit == 1 ); +} + + +PG_FUNCTION_INFO_V1( members ); + +typedef struct +{ + Signature *sig; + int32 index; +} members_fctx; + +Datum +members( PG_FUNCTION_ARGS ) +{ + FuncCallContext *funcctx; + members_fctx *fctx; + MemoryContext oldcontext; + + int32 result; + + // based on set-returning examples in postgresql source's contrib/tablefunc directory + + // executed on first entry to function + if ( SRF_IS_FIRSTCALL() ) { + Signature *sig; + + funcctx = SRF_FIRSTCALL_INIT(); + oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx); + funcctx->max_calls = PG_GETARG_UINT32(0); + + // save signature, position state across calls + sig = PG_GETARG_SIGNATURE_P(0); + fctx = (members_fctx *) palloc(sizeof(members_fctx)); + fctx->sig = sig; + fctx->index = 0; + + funcctx->user_fctx = fctx; + MemoryContextSwitchTo(oldcontext); + } + + // executed every entry to function + funcctx = SRF_PERCALL_SETUP(); + fctx = funcctx->user_fctx; + + // return position of next non-zero bit + result = -1; + while( result < 0 && (fctx->index < fctx->sig->len) ) { + int32 byte_offset, + bit_offset, + c; + + byte_offset = fctx->index / 8; + bit_offset = fctx->index % 8; + + c = fctx->sig->data[byte_offset]; + if (c & (0x80 >> bit_offset)) { + result = fctx->index; + } + + fctx->index++; + } + + // return result + if( result >= 0 ) + SRF_RETURN_NEXT(funcctx, Int32GetDatum(result)); + else + SRF_RETURN_DONE(funcctx); // pgsql documentation claims no palloc mgmt necessary +} + + +PG_FUNCTION_INFO_V1( sig_length ); + +Datum +sig_length( PG_FUNCTION_ARGS ) +{ + Signature *sig; + int32 length; + + sig = PG_GETARG_SIGNATURE_P(0); + length = sig->len; + + PG_FREE_IF_COPY( sig, 0 ); + + PG_RETURN_INT32( length ); +} + + +PG_FUNCTION_INFO_V1( sig_min ); + +Datum +sig_min( PG_FUNCTION_ARGS ) +{ + Signature *sig; + int32 sigbytes, + min, + i; + uint8 ch, x; + + sig = PG_GETARG_SIGNATURE_P(0); + sigbytes = VARSIZE(sig) - VARHDRSZ - SIGNATUREHDRSZ; + + min = - 1; + i = 0; + while (i < sigbytes && min < 0) { + ch = sig->data[i]; + if (ch > 0) { + min = i * 8; + x = 0x80; + while ((ch & x) == 0) { + x >>= 1; + min++; + } + } + i++; + } + + PG_FREE_IF_COPY( sig, 0 ); + + if (min < 0) { + PG_RETURN_NULL(); + } else { + PG_RETURN_INT32( min ); + } +} + + +PG_FUNCTION_INFO_V1( sig_and ); + +Datum +sig_and( PG_FUNCTION_ARGS ) +{ + Signature *sig1, + *sig2, + *res; + int32 sig1bytes, + sig2bytes, + resbytes, + i; + uint8 byte1, + byte2; + + sig1 = PG_GETARG_SIGNATURE_P(0); + sig1bytes = VARSIZE(sig1) - VARHDRSZ - SIGNATUREHDRSZ; + + sig2 = PG_GETARG_SIGNATURE_P(1); + sig2bytes = VARSIZE(sig2) - VARHDRSZ - SIGNATUREHDRSZ; + + resbytes = MAX(sig1bytes, sig2bytes); + + res = (Signature *) palloc0( resbytes + VARHDRSZ + SIGNATUREHDRSZ ); + SET_VARSIZE(res, resbytes + VARHDRSZ + SIGNATUREHDRSZ ); + + res->len = MAX(sig1->len, sig2->len); + + for(i=0; idata[i]; + if (i < sig2bytes) + byte2 = sig2->data[i]; + + res->data[i] = byte1 & byte2; + } + + PG_FREE_IF_COPY(sig1, 0); + PG_FREE_IF_COPY(sig2, 1); + + PG_RETURN_SIGNATURE_P( res ); +} + + +PG_FUNCTION_INFO_V1( sig_or ); + +Datum +sig_or( PG_FUNCTION_ARGS ) +{ + Signature *sig1, + *sig2, + *res; + int32 sig1bytes, + sig2bytes, + resbytes, + i; + uint8 byte1, + byte2; + + sig1 = PG_GETARG_SIGNATURE_P(0); + sig1bytes = VARSIZE(sig1) - VARHDRSZ - SIGNATUREHDRSZ; + + sig2 = PG_GETARG_SIGNATURE_P(1); + sig2bytes = VARSIZE(sig2) - VARHDRSZ - SIGNATUREHDRSZ; + + resbytes = MAX(sig1bytes, sig2bytes); + + res = (Signature *) palloc0( resbytes + VARHDRSZ + SIGNATUREHDRSZ ); + SET_VARSIZE(res, resbytes + VARHDRSZ + SIGNATUREHDRSZ ); + + res->len = MAX(sig1->len, sig2->len); + + for(i=0; idata[i]; + if (i < sig2bytes) + byte2 = sig2->data[i]; + + res->data[i] = byte1 | byte2; + } + + PG_FREE_IF_COPY(sig1, 0); + PG_FREE_IF_COPY(sig2, 1); + + PG_RETURN_SIGNATURE_P( res ); +} + + +PG_FUNCTION_INFO_V1( sig_xor ); + +Datum +sig_xor( PG_FUNCTION_ARGS ) +{ + Signature *sig, + *res; + int32 bytes, + bits, + i; + uint8 x; + + sig = PG_GETARG_SIGNATURE_P(0); + bytes = sig->len / 8; + bits = sig->len % 8; + + res = (Signature *) palloc0( bytes + 1 + VARHDRSZ + SIGNATUREHDRSZ ); + SET_VARSIZE(res, bytes + 1 + VARHDRSZ + SIGNATUREHDRSZ ); + res->len = sig->len; + + for(i=0; i<=bytes; i++) { + res->data[i] = ~(sig->data[i]); + } + + if (bits > 0) { + x = 0xFF >> bits; + res->data[bytes] &= ~x; + } + + PG_FREE_IF_COPY(sig, 0); + + PG_RETURN_SIGNATURE_P( res ); +} + + +PG_FUNCTION_INFO_V1( count ); + +Datum +count( PG_FUNCTION_ARGS ) +{ + Signature *sig; + int32 sigbytes, + count, + i; + uint8 ch; + + sig = PG_GETARG_SIGNATURE_P(0); + sigbytes = VARSIZE(sig) - VARHDRSZ - SIGNATUREHDRSZ; + + count = 0; + for(i=0; i < sigbytes; i++) { + ch = sig->data[i]; + count += COUNT_TABLE[ch]; + } + + PG_FREE_IF_COPY( sig, 0 ); + + PG_RETURN_INT32( count ); +} + + +static int32 +signature_cmp_internal(Signature *sig1, Signature *sig2) +{ + int32 sig1bytes, + sig2bytes, + sig1bits, + sig2bits, + result, + i, + maxbytes; + uint8 ch1, ch2, + x; + + sig1bytes = sig1->len / 8; + sig1bits = sig1->len % 8; + + sig2bytes = sig2->len / 8; + sig2bits = sig2->len % 8; + + maxbytes = MAX(sig1bytes, sig2bytes); + + result = i = 0; + while(result == 0 && i <= maxbytes) { + ch1 = ch2 = 0; + if(i <= sig1bytes) { + ch1 = sig1->data[i]; + if(i == sig1bytes && sig1bits > 0) { + x = 0xFF >> sig1bits; + ch1 &= ~x; + } + } + if(i <= sig2bytes) { + ch2 = sig2->data[i]; + if(i == sig2bytes && sig2bits > 0) { + x = 0xFF >> sig2bits; + ch2 &= ~x; + } + } + if (ch1 > ch2) { + result = 1; + } else if (ch1 < ch2) { + result = -1; + } + i++; + } + + return result; +} + + +PG_FUNCTION_INFO_V1(sig_cmp); + +Datum +sig_cmp(PG_FUNCTION_ARGS) +{ + Signature *sig1 = (Signature *) PG_GETARG_POINTER(0); + Signature *sig2 = (Signature *) PG_GETARG_POINTER(1); + int32 result; + + result = signature_cmp_internal(sig1, sig2); + + PG_FREE_IF_COPY(sig1, 0); + PG_FREE_IF_COPY(sig2, 1); + + PG_RETURN_INT32(result); +} + + +PG_FUNCTION_INFO_V1(sig_lt); + +Datum +sig_lt(PG_FUNCTION_ARGS) +{ + Signature *sig1 = (Signature *) PG_GETARG_POINTER(0); + Signature *sig2 = (Signature *) PG_GETARG_POINTER(1); + bool result; + + result = signature_cmp_internal(sig1, sig2) < 0; + + PG_FREE_IF_COPY(sig1, 0); + PG_FREE_IF_COPY(sig2, 1); + + PG_RETURN_BOOL(result); +} + + +PG_FUNCTION_INFO_V1(sig_lte); + +Datum +sig_lte(PG_FUNCTION_ARGS) +{ + Signature *sig1 = (Signature *) PG_GETARG_POINTER(0); + Signature *sig2 = (Signature *) PG_GETARG_POINTER(1); + bool result; + + result = signature_cmp_internal(sig1, sig2) <= 0; + + PG_FREE_IF_COPY(sig1, 0); + PG_FREE_IF_COPY(sig2, 1); + + PG_RETURN_BOOL(result); +} + + +PG_FUNCTION_INFO_V1(sig_eq); + +Datum +sig_eq(PG_FUNCTION_ARGS) +{ + Signature *sig1 = (Signature *) PG_GETARG_POINTER(0); + Signature *sig2 = (Signature *) PG_GETARG_POINTER(1); + bool result; + + result = signature_cmp_internal(sig1, sig2) == 0; + + PG_FREE_IF_COPY(sig1, 0); + PG_FREE_IF_COPY(sig2, 1); + + PG_RETURN_BOOL(result); +} + + +PG_FUNCTION_INFO_V1(sig_gte); + +Datum +sig_gte(PG_FUNCTION_ARGS) +{ + Signature *sig1 = (Signature *) PG_GETARG_POINTER(0); + Signature *sig2 = (Signature *) PG_GETARG_POINTER(1); + bool result; + + result = signature_cmp_internal(sig1, sig2) >= 0; + + PG_FREE_IF_COPY(sig1, 0); + PG_FREE_IF_COPY(sig2, 1); + + PG_RETURN_BOOL(result); +} + + +PG_FUNCTION_INFO_V1(sig_gt); + +Datum +sig_gt(PG_FUNCTION_ARGS) +{ + Signature *sig1 = (Signature *) PG_GETARG_POINTER(0); + Signature *sig2 = (Signature *) PG_GETARG_POINTER(1); + bool result; + + result = signature_cmp_internal(sig1, sig2) > 0; + + PG_FREE_IF_COPY(sig1, 0); + PG_FREE_IF_COPY(sig2, 1); + + PG_RETURN_BOOL(result); +} \ No newline at end of file diff --git a/vendor/gems/repertoire-faceting-0.6.0/ext/signature/signature.sql b/vendor/gems/repertoire-faceting-0.6.0/ext/signature/signature.sql new file mode 100644 index 0000000..0abd41e --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/ext/signature/signature.sql @@ -0,0 +1,217 @@ +-- ============================================================================ +-- Faceting API implementing bitmap indices using a custom C datatype and +-- associated functions. +-- +-- This API is to be preferred in all situations where it is possible to +-- build and install the datatype (this requires superuser access to PostgreSQL) +-- +-- Christopher York +-- MIT Hyperstudio +-- February 2014 +-- ============================================================================ + +-- complain if script is sourced in psql, rather than via CREATE EXTENSION +\echo Use "CREATE EXTENSION faceting" to load this the default faceting API.\quit + +-- functions for bitmap indices using datatype written in C + +CREATE TYPE @extschema@.signature; + +-- basic i/o functions for signatures + +CREATE FUNCTION @extschema@.sig_in(cstring) + RETURNS signature + AS 'signature.so', 'sig_in' + LANGUAGE C STRICT; + +CREATE FUNCTION @extschema@.sig_out(signature) + RETURNS cstring + AS 'signature.so', 'sig_out' + LANGUAGE C STRICT; + +-- signature postgresql type + +CREATE TYPE @extschema@.signature ( + INTERNALLENGTH = VARIABLE, + INPUT = sig_in, + OUTPUT = sig_out, + STORAGE = extended +); + +-- functions for signatures + +CREATE FUNCTION @extschema@.sig_resize( signature, INT ) + RETURNS signature + AS 'signature.so', 'sig_resize' + LANGUAGE C STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_set( signature, INT, INT ) + RETURNS signature + AS 'signature.so', 'sig_set' + LANGUAGE C STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_set( signature, INT ) + RETURNS signature + AS 'signature.so', 'sig_set' + LANGUAGE C STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_get( signature, INT ) + RETURNS INT + AS 'signature.so', 'sig_get' + LANGUAGE C STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_length( signature ) + RETURNS INT + AS 'signature.so', 'sig_length' + LANGUAGE C STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_min( signature ) + RETURNS INT + AS 'signature.so', 'sig_min' + LANGUAGE C STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_and( signature, signature ) + RETURNS signature + AS 'signature.so', 'sig_and' + LANGUAGE C STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_or( signature, signature ) + RETURNS signature + AS 'signature.so', 'sig_or' + LANGUAGE C STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_xor( signature ) + RETURNS signature + AS 'signature.so', 'sig_xor' + LANGUAGE C STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.count( signature ) + RETURNS INT + AS 'signature.so', 'count' + LANGUAGE C STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.contains( signature, INT ) + RETURNS BOOL + AS 'signature.so', 'contains' + LANGUAGE C STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.members( signature ) + RETURNS SETOF INT + AS 'signature.so', 'members' + LANGUAGE C STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_cmp( signature, signature ) + RETURNS INT + AS 'signature.so', 'sig_cmp' + LANGUAGE C STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_lt( signature, signature ) + RETURNS BOOL + AS 'signature.so', 'sig_lt' + LANGUAGE C STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_lte( signature, signature ) + RETURNS BOOL + AS 'signature.so', 'sig_lte' + LANGUAGE C STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_eq( signature, signature ) + RETURNS BOOL + AS 'signature.so', 'sig_eq' + LANGUAGE C STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_gt( signature, signature ) + RETURNS BOOL + AS 'signature.so', 'sig_gt' + LANGUAGE C STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_gte( signature, signature ) + RETURNS BOOL + AS 'signature.so', 'sig_gte' + LANGUAGE C STRICT IMMUTABLE; + + +-- operators for signatures + +CREATE OPERATOR @extschema@.& ( + leftarg = signature, + rightarg = signature, + procedure = @extschema@.sig_and, + commutator = & +); + +CREATE OPERATOR @extschema@.| ( + leftarg = signature, + rightarg = signature, + procedure = @extschema@.sig_or, + commutator = | +); + +CREATE OPERATOR @extschema@.+ ( + leftarg = signature, + rightarg = int, + procedure = @extschema@.sig_set +); + +CREATE OPERATOR @extschema@.< ( + leftarg = signature, rightarg = signature, procedure = sig_lt, + commutator = > , negator = >= , + restrict = scalarltsel, join = scalarltjoinsel +); + +CREATE OPERATOR @extschema@.<= ( + leftarg = signature, rightarg = signature, procedure = sig_lte, + commutator = >= , negator = > , + restrict = scalarltsel, join = scalarltjoinsel +); + +CREATE OPERATOR @extschema@.= ( + leftarg = signature, rightarg = signature, procedure = sig_eq, + commutator = = , negator = <> , + restrict = eqsel, join = eqjoinsel +); + +CREATE OPERATOR >= ( + leftarg = signature, rightarg = signature, procedure = sig_gte, + commutator = <= , negator = < , + restrict = scalargtsel, join = scalargtjoinsel +); + +CREATE OPERATOR @extschema@.> ( + leftarg = signature, rightarg = signature, procedure = sig_gt, + commutator = < , negator = <= , + restrict = scalargtsel, join = scalargtjoinsel +); + +-- index operator classes for signatures + +CREATE OPERATOR CLASS @extschema@.signature_ops + DEFAULT FOR TYPE signature USING btree AS + OPERATOR 1 < , + OPERATOR 2 <= , + OPERATOR 3 = , + OPERATOR 4 >= , + OPERATOR 5 > , + FUNCTION 1 sig_cmp(signature, signature); + + +-- aggregate functions for faceting + +CREATE AGGREGATE @extschema@.collect( signature ) +( + sfunc = @extschema@.sig_or, + stype = signature +); + +CREATE AGGREGATE @extschema@.filter( signature ) +( + sfunc = @extschema@.sig_and, + stype = signature +); + +CREATE AGGREGATE @extschema@.signature( INT ) +( + sfunc = @extschema@.sig_set, + stype = signature, + initcond = '0' +); \ No newline at end of file diff --git a/vendor/gems/repertoire-faceting-0.6.0/ext/varbit/faceting_varbit.control b/vendor/gems/repertoire-faceting-0.6.0/ext/varbit/faceting_varbit.control new file mode 100644 index 0000000..cf44a9b --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/ext/varbit/faceting_varbit.control @@ -0,0 +1,7 @@ +# Faceting PostgreSQL extension module + +comment = 'API for faceted indexing and queries (based on builtin VARBIT bit strings)' +requires = plpgsql +superuser = false +default_version = '0.6.0' +schema = 'facet' \ No newline at end of file diff --git a/vendor/gems/repertoire-faceting-0.6.0/ext/varbit/varbit.sql b/vendor/gems/repertoire-faceting-0.6.0/ext/varbit/varbit.sql new file mode 100644 index 0000000..3b7da11 --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/ext/varbit/varbit.sql @@ -0,0 +1,164 @@ +-- ============================================================================ +-- Faceting API implementing bitmap indices using PostgreSQL's built-in VARBIT +-- type, processed using the built-in language pl/pgsql. +-- +-- This API is suitable for deployment on any host, since it requires no +-- PostgreSQL extensions outside the default install. +-- +-- However, performance is limited to around 30,000 items in practice (in part +-- because of unnecessary duplication of varbit values when pl/pgsql evaluates +-- the count function.) +-- +-- The 'signature' C-based faceting API is preferable for any install where +-- you have superuser access to the database. +-- +-- Christopher York +-- MIT Hyperstudio +-- February 2014 +-- ============================================================================ + +CREATE FUNCTION @extschema@.sig_resize( sig VARBIT, bits INT ) RETURNS VARBIT AS $$ +DECLARE + len INT; +BEGIN + len := length(sig); + IF bits > len THEN + RETURN sig || repeat('0', bits - len)::VARBIT; + ELSIF bits < len THEN + RETURN substring(sig FROM 1 FOR bits); + END IF; + RETURN sig; +END $$ LANGUAGE plpgsql STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_set( sig VARBIT, pos INT, val INT) RETURNS VARBIT AS $$ +DECLARE + len INT; +BEGIN + len := length(sig); + IF pos >= len THEN + IF val > 0 THEN + RETURN set_bit(@extschema@.sig_resize(sig, pos+1), pos, 1); + ELSE + RETURN sig; + END IF; + ELSE + RETURN set_bit(sig, pos, val); + END IF; +END $$ LANGUAGE plpgsql STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_set( sig VARBIT, pos INT) RETURNS VARBIT AS $$ +BEGIN + RETURN @extschema@.sig_set(sig, pos, 1); +END $$ LANGUAGE plpgsql STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_get( sig VARBIT, pos INT ) RETURNS INT AS $$ +DECLARE + len INT; +BEGIN + len := length(sig); + IF pos >= len THEN + RETURN 0; + ELSE + RETURN get_bit(sig, pos); + END IF; +END $$ LANGUAGE plpgsql STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_length( sig VARBIT ) RETURNS INT AS $$ +BEGIN + RETURN length(sig); +END $$ LANGUAGE plpgsql STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_min( sig VARBIT ) RETURNS INT AS $$ +BEGIN + RETURN position('1' in sig) - 1; +END $$ LANGUAGE plpgsql STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_and( sig1 VARBIT, sig2 VARBIT ) RETURNS VARBIT AS $$ +DECLARE + len INT; +BEGIN + len := GREATEST(length(sig1), length(sig2)); + RETURN bitand(@extschema@.sig_resize(sig1, len), @extschema@.sig_resize(sig2, len)) ; +END $$ LANGUAGE plpgsql STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_or( sig1 VARBIT, sig2 VARBIT ) RETURNS VARBIT AS $$ +DECLARE + len INT; +BEGIN + len := GREATEST(length(sig1), length(sig2)); + RETURN bitor(@extschema@.sig_resize(sig1, len), @extschema@.sig_resize(sig2, len)) ; +END $$ LANGUAGE plpgsql STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.sig_xor( sig1 VARBIT, sig2 VARBIT ) RETURNS VARBIT AS $$ +DECLARE + len INT; +BEGIN + len := GREATEST(length(sig1), length(sig2)); + RETURN bitxor(@extschema@.sig_resize(sig1, len), @extschema@.sig_resize(sig2, len)) ; +END $$ LANGUAGE plpgsql STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.count( sig VARBIT ) RETURNS INT AS $$ +BEGIN + -- This is, by any measure, horrific. However, it appears to be the only + -- way to use PostgreSQL built in functions to count bits in a bit string. + RETURN length(replace(sig::TEXT, '0', '')); +END $$ LANGUAGE plpgsql STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.contains( sig VARBIT, pos INT ) RETURNS BOOL AS $$ +BEGIN + RETURN @extschema@.sig_get(sig, pos) = 1; +END $$ LANGUAGE plpgsql STRICT IMMUTABLE; + +CREATE FUNCTION @extschema@.members( sig VARBIT ) RETURNS SETOF INT AS $$ +BEGIN + FOR i IN 0 .. length(sig) - 1 LOOP + IF @extschema@.contains(sig, i) THEN + RETURN NEXT i; + END IF; + END LOOP; +END $$ LANGUAGE plpgsql STRICT IMMUTABLE; + + +-- operators for faceting + +CREATE OPERATOR @extschema@.& ( + leftarg = VARBIT, + rightarg = VARBIT, + procedure = @extschema@.sig_and, + commutator = & +); + +CREATE OPERATOR @extschema@.| ( + leftarg = VARBIT, + rightarg = VARBIT, + procedure = @extschema@.sig_or, + commutator = | +); + +CREATE OPERATOR @extschema@.+ ( + leftarg = VARBIT, + rightarg = int, + procedure = @extschema@.sig_set +); + + +-- aggregate functions for faceting + +CREATE AGGREGATE @extschema@.collect( VARBIT ) +( + sfunc = @extschema@.sig_or, + stype = VARBIT +); + +CREATE AGGREGATE @extschema@.filter( VARBIT ) +( + sfunc = @extschema@.sig_and, + stype = VARBIT +); + +CREATE AGGREGATE @extschema@.signature( INT ) +( + sfunc = @extschema@.sig_set, + stype = VARBIT, + initcond = '0' +); \ No newline at end of file diff --git a/vendor/gems/repertoire-faceting-0.6.0/lib/assets/images/repertoire-faceting/proportional_symbol.png b/vendor/gems/repertoire-faceting-0.6.0/lib/assets/images/repertoire-faceting/proportional_symbol.png new file mode 100644 index 0000000..d99bb95 Binary files /dev/null and b/vendor/gems/repertoire-faceting-0.6.0/lib/assets/images/repertoire-faceting/proportional_symbol.png differ diff --git a/vendor/gems/repertoire-faceting-0.6.0/lib/assets/images/repertoire-faceting/spinner_sm.gif b/vendor/gems/repertoire-faceting-0.6.0/lib/assets/images/repertoire-faceting/spinner_sm.gif new file mode 100644 index 0000000..d0bce15 Binary files /dev/null and b/vendor/gems/repertoire-faceting-0.6.0/lib/assets/images/repertoire-faceting/spinner_sm.gif differ diff --git a/vendor/gems/repertoire-faceting-0.6.0/lib/assets/javascripts/rep.faceting.js b/vendor/gems/repertoire-faceting-0.6.0/lib/assets/javascripts/rep.faceting.js new file mode 100644 index 0000000..b903152 --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/lib/assets/javascripts/rep.faceting.js @@ -0,0 +1,49 @@ +/* +* Repertoire faceting ajax widgets +* +* Copyright (c) 2009 MIT Hyperstudio +* Christopher York, 09/2009 +* +* Requires jquery 1.3.2+ +* Support: Firefox 3+ & Safari 4+. IE emphatically not supported. +* +* The package provides 4 jquery plugins: +* $elem.facet_context() : used to group facets that share state together and extract page state to send to the web service +* $elem.results() : display results from the current context facet query +* $elem.facet() : display an extensible facet value and count refining widget +* $elem.nesting_facet(): display and refine on nested facet values +* +* You can configure an existing facet widget using its supported configuration options; or by adding a small handler that inserts +* your own control and event hook into the widget; by subclassing an existing widget to alter its functionality; or by writing +* an entirely new widget based on the core functionality. It is relatively easy to create widgets that use visualization toolkits +* like Protovis or Processing.js for rendering. +* +* Facet widgets are always collected together in groups that share the same query context (range of potential items and current +* facet refinements). In practice, the context is simply an enclosing div and facet widgets are contained elements. +* +* Basic example, using faceting widgets out of the box (urls are calculated by default using element ids, or can be set explicitly). +* +* [ faceting over all plays, with two facets defined (genre and era) ] +* +* $().ready(function() { +* $('#plays').facet_context(); +* $('.facet').facet(); +* $('#results').results(); +* }); +* ... +*
+*
+*
+*
+*
+* +* See the README and FAQ for more information. +* +* TODO. can the css for this module be namespaced? +*/ + +//= require ./rep.faceting/context +//= require ./rep.faceting/facet +//= require ./rep.faceting/facet_widget +//= require ./rep.faceting/nested_facet +//= require ./rep.faceting/results \ No newline at end of file diff --git a/vendor/gems/repertoire-faceting-0.6.0/lib/assets/javascripts/rep.faceting/context.js b/vendor/gems/repertoire-faceting-0.6.0/lib/assets/javascripts/rep.faceting/context.js new file mode 100644 index 0000000..df1d47e --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/lib/assets/javascripts/rep.faceting/context.js @@ -0,0 +1,150 @@ +/* +* Repertoire faceting ajax widgets +* +* Copyright (c) 2009 MIT Hyperstudio +* Christopher York, 09/2009 +* +* Requires jquery 1.3.2+ +* Support: Firefox 3+ & Safari 4+. IE emphatically not supported. +* +* +* Register an element as the faceting context, +* and provide user data extraction function +* +* Handles: +* - manipulation of faceting refinements +* - url/query-string construction +* - data assembly for sending to webservice +* - change publication and observing +* - grouping faceting widgets into shared context +* - facet count/results ajax api +* - hooks for managing custom data +*/ + +//= require jquery + +//= require rep.widgets + +repertoire.facet_context = function(context_name, state_fn, options) { + var self = repertoire.model(options); + + // current query state for all facets in context + var filter = {}; + + // + // Return the current refinements for one facet, or all if no facet given + // + // Changes to the returned object are persistent, but you must call self.state_changed() + // to trigger an update event. + // + self.refinements = function(name) { + if (!name) { + // if no facet provided, return all + return filter; + } else { + // set up refinements for this facet + if (!filter[name]) + filter[name] = []; + + // return current refinements + return filter[name]; + } + }; + + // + // Calculate facet value counts from webservice + // + // By default, the url is '//counts/' + // + self.counts = function(facet_name, callback, $elem) { + var url = self.facet_url('counts', facet_name); + // package up the faceting state and send back to results rendering service + self.fetch(self.params(), url, 'json', callback, $elem); + }; + + // + // Update query results from webservice + // + // By default, the url is '//counts/' + // + self.results = function(type, callback, $elem) { + var url = self.facet_url('results'); + // package up the faceting state and send back to results rendering service + self.fetch(self.params(), url, type, callback, $elem); + }; + + // + // Convenience function for constructing faceting urls + // + self.facet_url = function(action, facet, ext, params) { + var paths = [context_name, action] + if (facet) + paths.push(facet) + var url = self.default_url(paths, ext); + if (params) + return url + '?' + self.to_query_string(params); + else + return url; + }; + + // + // Return the state for the entire faceting context (group of widgets), + // with any context-specific additions + // + self.params = function() { + var state = state_fn ? state_fn() : {}; + return $.extend({}, { filter: self.refinements() }, state); + }; + + + // + // Added to allow facet widgets to pass arguments to context. + // + // ( Have to revisit with Christopher, it seems like he had an + // pre_update() function in mind for this kind of behavior, + // But I'm not sure how he envisioned it functioning. ) + // + self.update_state = function(new_state) { + var current_state = state_fn ? state_fn() : {}; + state_fn = function () { + return $.extend({}, current_state, new_state); + } + }; + + + // + // Return the identifying name for this context (usually the model class, pluralized) + // + self.name = function() { + return context_name; + } + + // + // Toggle whether facet value is selected + // + self.toggle = function(name, item) { + var values = self.refinements(name); + var index = $.inArray(item, values); + + if (index == -1) + values.push(item); + else + values.splice(index, 1); + + return values; + }; + + // end of context factory method + return self; +} + +$.fn.facet_context = function(state_fn) { + return this.each(function() { + // add locator css class to element, and store faceting context data model in it + var $elem = $(this); + var name = $elem.attr('id'); + var model = repertoire.facet_context(name, state_fn, repertoire.defaults); + $elem.addClass('facet_refinement_context'); + $elem.data('context', model); + }); +}; \ No newline at end of file diff --git a/vendor/gems/repertoire-faceting-0.6.0/lib/assets/javascripts/rep.faceting/ext/earth_facet.js b/vendor/gems/repertoire-faceting-0.6.0/lib/assets/javascripts/rep.faceting/ext/earth_facet.js new file mode 100644 index 0000000..702debc --- /dev/null +++ b/vendor/gems/repertoire-faceting-0.6.0/lib/assets/javascripts/rep.faceting/ext/earth_facet.js @@ -0,0 +1,437 @@ +/* +* Repertoire faceting ajax widgets +* +* Copyright (c) 2009 MIT Hyperstudio +* Christopher York, 12/2009 +* +* Requires jquery 1.3.2+, OpenLayers 2.8 +* Support: Firefox 3+ & Safari 4+. IE emphatically not supported. +* +* Earth browser widget for GIS-based facets +* +* Usage: +* +* $('#birthplace_geom').earth_facet() +* +* Note! +* You must activate Google's API by including a