Pubid nist integration #106

andrew2net · 2023-09-16T20:11:37Z

No description provided.

hound · 2023-09-16T20:11:42Z

lib/relaton_nist/hit_collection.rb

      d = Date.strptime year, "%Y" if year
-      statuses = %w[draft-public draft-prelim]
+      ref = "#{@pubid_ref.serie.serie.sub(/^NIST\s|\sPUB$/, '')} #{@pubid_ref.code}" # TODO: Pubid shouldn't have PUB


Line is too long. [117/80]

@andrew2net can't we move all of these functions into pubid-nist?

hound · 2023-09-16T20:11:42Z

lib/relaton_nist/hit_collection.rb

+
+    def json_pubid(json)
+      if json["doi"]
+        id = json["doi"].split("/").last.sub(/-draft\d*/, "").sub(/\.\wpd/, "").sub(/-(Add\d*)/, "")


Line is too long. [100/80]

@andrew2net can't we move all of these functions into pubid-nist?

hound · 2023-09-16T20:11:42Z

lib/relaton_nist/hit_collection.rb

+      end
+    end
+
+    def json_pubid(json)


Assignment Branch Condition size for json_pubid is too high. [<17, 33, 15> 40.04/15]
Cyclomatic complexity for json_pubid is too high. [15/6]
Method has too many lines. [19/10]
Perceived complexity for json_pubid is too high. [16/7]

hound · 2023-09-16T20:11:42Z

lib/relaton_nist/hit_collection.rb

-                  url: h["uri"], status: h["status"],
-                  release_date: release_date, json: h }, self)
+        Hit.new({ code: json_pubid(h), series: series.upcase, title: title, url: h["uri"],
+                  status: h["status"], release_date: release_date, json: h }, self)


Line is too long. [83/80]

hound · 2023-09-16T20:11:42Z

lib/relaton_nist/hit_collection.rb

-        Hit.new({ code: docidentifier(h), series: series.upcase, title: title,
-                  url: h["uri"], status: h["status"],
-                  release_date: release_date, json: h }, self)
+        Hit.new({ code: json_pubid(h), series: series.upcase, title: title, url: h["uri"],


Line is too long. [90/80]

hound · 2023-09-16T20:11:43Z

lib/relaton_nist/scrapper.rb

-        iter = doc["iteration"] == "initial" ? 1 : doc["iteration"]
-        RelatonNist::DocumentStatus.new stage: stage, substage: subst, iteration: iter.to_s
+        iter = doc["iteration"] == "ipd" ? "1" : doc["iteration"].match(/\d+/)&.to_s
+        RelatonNist::DocumentStatus.new stage: stage, substage: subst, iteration: iter


Line is too long. [86/80]

hound · 2023-09-16T20:11:43Z

lib/relaton_nist/scrapper.rb

@@ -80,8 +80,8 @@ def fetch_docid(hit)
      def fetch_status(doc)
        stage = doc["status"]
        subst = doc["substage"]
-        iter = doc["iteration"] == "initial" ? 1 : doc["iteration"]
-        RelatonNist::DocumentStatus.new stage: stage, substage: subst, iteration: iter.to_s
+        iter = doc["iteration"] == "ipd" ? "1" : doc["iteration"].match(/\d+/)&.to_s


Line is too long. [84/80]

hound · 2023-09-16T20:11:43Z

lib/relaton_nist/scrapper.rb

@@ -68,7 +68,7 @@ def fetch_docid(hit)
        # json["docidentifier"]
        # item_ref ||= "?"
        # item_ref.sub!(/\sAddendum$/, "-Add")
-        ids = [RelatonBib::DocumentIdentifier.new(id: hit[:code], type: "NIST", primary: true)]
+        ids = [RelatonBib::DocumentIdentifier.new(id: hit[:code].to_s, type: "NIST", primary: true)]


Line is too long. [100/80]

hound · 2023-09-16T20:11:44Z

spec/relaton_nist/pubs_export_spec.rb

@@ -19,7 +19,7 @@
      expect(File).to receive(:exist?).with(RelatonNist::PubsExport::DATAFILE).and_return true
      ctime = Time.now
      expect(File).to receive(:ctime).with(RelatonNist::PubsExport::DATAFILE).and_return ctime
-      expect(File).to receive(:size).with(RelatonNist::PubsExport::DATAFILE).and_return 0
+      expect(File).to receive(:empty?).with(RelatonNist::PubsExport::DATAFILE).and_return true


Line is too long. [94/80]

hound · 2023-09-16T20:11:44Z

spec/relaton_nist_spec.rb

-        expect(result.id).to eq "NISTSP800-162"
-      end
+      # it "final without updated-date", vcr: { cassette_name: "nist_sp_800_162_2014_01" } do
+      #   result = RelatonNist::NistBibliography.get "NIST SP 800-162 (January 2014)"


Line is too long. [85/80]

ronaldtse · 2023-09-17T09:04:38Z

lib/relaton_nist/hit_collection.rb

+
+    def json_pubid(json)
+      if json["doi"]
+        id = json["doi"].split("/").last.sub(/-draft\d*/, "").sub(/\.\wpd/, "").sub(/-(Add\d*)/, "")


@andrew2net can't we move all of these functions into pubid-nist?

ronaldtse · 2023-09-17T09:05:14Z

lib/relaton_nist/hit_collection.rb

+      if json["doi"]
+        id = json["doi"].split("/").last.sub(/-draft\d*/, "").sub(/\.\wpd/, "").sub(/-(Add\d*)/, "")
+        add = Regexp.last_match(1)
+        id.sub!(/-upd(\d)/, "")


@andrew2net can't we move all of these functions into pubid-nist?

ronaldtse · 2023-09-17T09:05:42Z

lib/relaton_nist/hit_collection.rb

+            when "ipd" then "i"
+            when "final" then "f"
+            else iteration.match(/^\d+/)&.to_s
+            end


@andrew2net can't we move all of these functions into pubid-nist?

@ronaldtse I've created issues but they have not been solved yet. These are just temp solutions.

ronaldtse · 2023-09-17T09:05:50Z

lib/relaton_nist/hit_collection.rb

      d = Date.strptime year, "%Y" if year
-      statuses = %w[draft-public draft-prelim]
+      ref = "#{@pubid_ref.serie.serie.sub(/^NIST\s|\sPUB$/, '')} #{@pubid_ref.code}" # TODO: Pubid shouldn't have PUB


@andrew2net can't we move all of these functions into pubid-nist?

ronaldtse · 2023-09-17T09:06:41Z

lib/relaton_nist/hit_collection.rb

-        if @opts[:stage]&.include?("PD")
-          next unless statuses.include? doc["status"]
+        if @pubid_ref.stage&.type == "pd"
+          next unless doc["status"] == "draft-public"
        else


Can't we move the detection of draft-public to an earlier stage to store in pubid?

This is a loop through all the items in pubs-export dataset. The Parslet, used in Pubid gems, is quite slow. So, parsing every ID in the dataset will slow down serachig. In this loop a few elements seelcted by code and stage. Next, IDs of the selected items are parsed and compared to reference pubid. Does it make sense?

…elaton-nist into pubid-nist_integration

hound · 2023-09-29T14:28:38Z

lib/relaton_nist/nist_bibliography.rb

-        warn "[relaton-nist] WARNING: no match found online for #{id}. " \
-             "The code must be exactly like it is on the standards website."
+        Util.warn "WARNING: no match found online for `#{id}`. " \
+                  "The code must be exactly like it is on the standards website."


Line is too long. [81/80]

hound · 2023-09-29T14:28:38Z

spec/relaton_nist_spec.rb

-        result = RelatonNist::NistBibliography.get "NIST SP 800-162 (January 2014)"
-        expect(result.id).to eq "NISTSP800-162"
-      end
+      # it "final without updated-date", vcr: { cassette_name: "nist_sp_800_162_2014_01" } do


Line is too long. [93/80]

hound · 2023-09-29T14:28:38Z

spec/relaton_nist_spec.rb

      ).to_stderr
    end
+
+    # it "contains EP at the end" do
+    #   expect { RelatonNist::NistBibliography.get "NIST FIPS 201 EP" }.to output(


Line is too long. [82/80]

hound · 2023-09-29T14:28:38Z

spec/relaton_nist_spec.rb

@@ -410,41 +407,37 @@
        expect do
          RelatonNist::NistBibliography.get("NISTIR 8200", "2017", {})
        end.to output(
-          /\[relaton-nist\] WARNING: no match found online for NISTIR 8200:2017/,
+          /\[relaton-nist\] WARNING: no match found online for `NISTIR 8200:2017`/,


Line is too long. [83/80]

hound · 2023-09-29T14:28:38Z

spec/relaton_nist_spec.rb

-      it "with volume and version", vcr: { cassette_name: "nist_sp_800_60ver2v1" } do
-        bib = RelatonNist::NistBibliography.get "NIST.SP.800-60.v-1.ver2.eng"
-        expect(bib.docidentifier[0].id).to eq "NIST SP 800-60 Vol. 1 Ver. 2 Rev. 2"
+      it "with volume and revision", vcr: { cassette_name: "nist_sp_800_60v1r1" } do


Line is too long. [84/80]

hound · 2023-09-29T14:28:38Z

spec/relaton_nist_spec.rb

+      result = RelatonNist::NistBibliography.get("NIST SP 800-189 2PD", nil, {})
+        .to_xml bibdata: true
+      file_path = "spec/examples/draft.xml"
+      File.write file_path, result, encoding: "UTF-8" unless File.exist? file_path


Line is too long. [82/80]

hound · 2023-09-29T14:28:38Z

spec/relaton_nist_spec.rb

+    it "a code with an year from json", vcr: { cassette_name: "json_data" } do
+      result = RelatonNist::NistBibliography.get "FIPS 140-2", "2001"
+      expect(result.docidentifier.first.id).to eq "NIST FIPS 140-2r2 fpd"
+      expect(result.date.find { |d| d.type == "published" }.on).to eq "2001-05-25"


Line is too long. [82/80]

ronaldtse · 2023-10-26T04:18:48Z

@andrew2net can we move forward with this? Thanks!

andrew2net · 2023-10-26T17:06:58Z

@ronaldtse we need to release the latest updates in pubid-nist gem. @mico and I discussed the pubid-nist functionality 2 weeks ago. I'll ask Artur how is the update going on.

andrew2net added 3 commits August 19, 2023 20:33

fix issue #81

fcf67db

Merge branch 'main' into pubid-nist_integration

fdc8fbb

WIP

2511c66

hound bot reviewed Sep 16, 2023

View reviewed changes

andrew2net requested a review from ronaldtse September 16, 2023 20:12

ronaldtse requested changes Sep 17, 2023

View reviewed changes

andrew2net added 6 commits September 19, 2023 17:55

use gem version as trigger to flush cache relaton/relaton#115

bf60ede

add config & logger

b8777f1

update README

19a6b16

fix issue #81

5c12918

WIP

df43062

Merge branch 'pubid-nist_integration' of https://github.com/relaton/r…

fd85d4f

…elaton-nist into pubid-nist_integration

hound bot reviewed Sep 29, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pubid nist integration #106

Pubid nist integration #106

andrew2net commented Sep 16, 2023

hound bot Sep 16, 2023

ronaldtse Sep 17, 2023

hound bot Sep 16, 2023

ronaldtse Sep 17, 2023

hound bot Sep 16, 2023

hound bot Sep 16, 2023

hound bot Sep 16, 2023

hound bot Sep 16, 2023

hound bot Sep 16, 2023

hound bot Sep 16, 2023

hound bot Sep 16, 2023

hound bot Sep 16, 2023

ronaldtse Sep 17, 2023

ronaldtse Sep 17, 2023

ronaldtse Sep 17, 2023

andrew2net Sep 17, 2023

ronaldtse Sep 17, 2023

ronaldtse Sep 17, 2023

andrew2net Sep 17, 2023

hound bot Sep 29, 2023

hound bot Sep 29, 2023

hound bot Sep 29, 2023

hound bot Sep 29, 2023

hound bot Sep 29, 2023

hound bot Sep 29, 2023

hound bot Sep 29, 2023

ronaldtse commented Oct 26, 2023

andrew2net commented Oct 26, 2023

Pubid nist integration #106

Are you sure you want to change the base?

Pubid nist integration #106

Conversation

andrew2net commented Sep 16, 2023

hound bot Sep 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hound bot Sep 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hound bot Sep 16, 2023

Choose a reason for hiding this comment

hound bot Sep 16, 2023

Choose a reason for hiding this comment

hound bot Sep 16, 2023

Choose a reason for hiding this comment

hound bot Sep 16, 2023

Choose a reason for hiding this comment

hound bot Sep 16, 2023

Choose a reason for hiding this comment

hound bot Sep 16, 2023

Choose a reason for hiding this comment

hound bot Sep 16, 2023

Choose a reason for hiding this comment

hound bot Sep 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hound bot Sep 29, 2023

Choose a reason for hiding this comment

hound bot Sep 29, 2023

Choose a reason for hiding this comment

hound bot Sep 29, 2023

Choose a reason for hiding this comment

hound bot Sep 29, 2023

Choose a reason for hiding this comment

hound bot Sep 29, 2023

Choose a reason for hiding this comment

hound bot Sep 29, 2023

Choose a reason for hiding this comment

hound bot Sep 29, 2023

Choose a reason for hiding this comment

ronaldtse commented Oct 26, 2023

andrew2net commented Oct 26, 2023