Merge pull request #2067 from ualbertalib/circuit_breaker_memory

rake ingest[sfx] after adding the circuit breaker caused the ingest script to run out of memory:

rake aborted!
Errno::ENOMEM: Cannot allocate memory - java

    Kilmarnock in Prod has 4Gb memory / 2Gb swap and performs this feat monthly without complaint.
    Forest has the same virtual h/w, but was throwing off Netdata alarms about memory & swap usage while ingesting SFX specifically

Basically we read the file twice with two different GC'd languages, thus the file's in memory twice. oops

.travis.yml

@@ -1,7 +1,9 @@
 language: ruby
-sudo: required
 addons:
   chrome: stable
+  apt:
+    packages:
+    - libxml2-utils
 rvm:
   - 2.5
   - 2.6

CHANGELOG.md

@@ -6,6 +6,11 @@ and releases in Discovery project adheres to [Semantic Versioning](http://semver
 
 ## [Unreleased]
 
+## [3.5.3]
+
+### Fixed
+- memory use by circuit breaker [PR#2067](https://github.com/ualbertalib/discovery/pull/2067)
+
 ## [3.5.2]
 
 ### Added

README.md

@@ -6,6 +6,7 @@ discovery platform. Based on [Project Blacklight](projectblacklight.org).
 
 *   Depends on [Ruby](https://www.ruby-lang.org/en/) 2.5.x
 *   Depends on Java (for SolrMarc and Ingestion scripts)
+*   Depends on [xmllint](http://xmlsoft.org/xmllint.html) (for SFX Ingestion scripts) which is available in `sudo apt install libxml2-utils` on Ubuntu.
 *   Depends on an instance of [Solr](https://lucene.apache.org/solr/) with [this configuration](https://github.com/ualbertalib/blacklight_solr_conf)
 *   If you wish to use docker for the datastores install [docker](https://docs.docker.com/install/) and [docker-compose](https://docs.docker.com/compose/install/) first.
 

config/application.rb

@@ -8,7 +8,7 @@
 Bundler.require(*Rails.groups)
 
 module Discovery
-  VERSION = '3.5.2'.freeze # used in application layout meta generator tag
+  VERSION = '3.5.3'.freeze # used in application layout meta generator tag
 
   class Application < Rails::Application
     # Settings in config/environments/* take precedence over those specified here.

lib/tasks/ingest.rake

@@ -8,6 +8,7 @@ require "#{Rails.root}/lib/ingest/promoted_services_om.rb"
 require_relative './ingest_configuration.rb'
 
 require 'yaml'
+require 'open3'
 
 import 'lib/tasks/delete.rake'
 
@@ -51,13 +52,15 @@ task :ingest, [:collection] => [:update_solr_marc_maps] do |_t, args|
 
   # circuit breaker to prevent unparsable data from changing our index
   if File.extname(Rails.root.join(@c.path)) == '.xml'
-    doc = File.open(Rails.root.join(@c.path)) { |f| Nokogiri::XML(f) }
-    if doc.errors.count.positive?
-      unparsable = "#{@c.path} is unparsable #{doc.errors}"
-      Rollbar.error(unparsable)
-      @ingest_log.fatal(unparsable)
-      at_exit { puts unparsable }
-      exit
+    Open3.popen3("xmllint --encode utf-8 --noout #{Rails.root.join(@c.path)}") do |_stdout, _stderr, status, _thread|
+      response = status.read
+      unless response.empty?
+        unparsable = "#{@c.path} is unparsable #{response}"
+        Rollbar.error(unparsable)
+        @ingest_log.fatal(unparsable)
+        at_exit { puts unparsable }
+        exit
+      end
     end
   end