Merge remote-tracking branch 'upstream/8.0.x' into refactor/latex-doc…

…ker-downsize-api

CHANGELOG.md

@@ -2,63 +2,57 @@
 
 All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
 
-### [8.0.25](https://github.com/macite/doubtfire-deploy/compare/v8.0.24...v8.0.25) (2024-08-09)
+### [8.0.32](https://github.com/macite/doubtfire-deploy/compare/v8.0.31...v8.0.32) (2024-09-05)
+
+
+### Features
+
+* add support for upload of vue components ([7c85aaf](https://github.com/macite/doubtfire-deploy/commit/7c85aaf1e1080554f4132bd7da18c1e67e2d2aea))
+
+### [8.0.31](https://github.com/doubtfire-lms/doubtfire-deploy/compare/v8.0.30...v8.0.31) (2024-08-29)
 
 
 ### Bug Fixes
 
-* ensure schema has index for auth token type ([7d3e4d3](https://github.com/macite/doubtfire-deploy/commit/7d3e4d369e66815b422faf46f8924397600266f1))
-* ensure test attempt review exception is handled ([bb3590c](https://github.com/macite/doubtfire-deploy/commit/bb3590c14c5c66191833fa98ee6c6eeebc2a3d78))
-* logging of fail to send message in accept submission ([38abe9e](https://github.com/macite/doubtfire-deploy/commit/38abe9eeb7dedf8f7d26b7b1c659be94d9c42d4a))
-* remove default from cmi_datamodel in test attempt ([ccb20dc](https://github.com/macite/doubtfire-deploy/commit/ccb20dc5c1efea2e5d0331026bc17d39dda3db11))
+* ensure sidekiq logs latex errors to stdout ([78151b3](https://github.com/doubtfire-lms/doubtfire-deploy/commit/78151b3c00f768ee83dd6838628eee2163bd6cde))
+* limit sidekiq concurrency to 1 ([0046562](https://github.com/doubtfire-lms/doubtfire-deploy/commit/004656216508f5469b234f3024c0d95a19d3b014))
+* revert delay in sidekiq pdf generation ([904ca34](https://github.com/doubtfire-lms/doubtfire-deploy/commit/904ca3432cf777f88121e1cd1cf59284c628e1cf))
 
-### [8.0.24](https://github.com/doubtfire-lms/doubtfire-deploy/compare/v8.0.23...v8.0.24) (2024-08-09)
+### [8.0.30](https://github.com/doubtfire-lms/doubtfire-deploy/compare/v8.0.29...v8.0.30) (2024-08-29)
 
 
-### Features
+### Bug Fixes
+
+* add short delay for accept submission job ([b3861ff](https://github.com/doubtfire-lms/doubtfire-deploy/commit/b3861ff2f44467e135a92427141844f9d33d6164))
+
+### [8.0.29](https://github.com/macite/doubtfire-deploy/compare/v8.0.28...v8.0.29) (2024-08-28)
+
+
+### Bug Fixes
+
+* correct email reporting of pdf errors in sidekiq ([ff2686a](https://github.com/macite/doubtfire-deploy/commit/ff2686ab0074c5f9442debdaddc9fce02dcdae54))
+
+### [8.0.28](https://github.com/macite/doubtfire-deploy/compare/v8.0.27...v8.0.28) (2024-08-28)
 
-* add attribute to allow file upload before scorm is passed ([fce7e75](https://github.com/doubtfire-lms/doubtfire-deploy/commit/fce7e7519bb9171726a030b409aee23de65f44fd))
-* add Numbas config options to task def backend ([d53610a](https://github.com/doubtfire-lms/doubtfire-deploy/commit/d53610a3f4b0c8077aea34cbfa2924e301914e1f))
-* add numbas task comment on test completion ([3f5aa2b](https://github.com/doubtfire-lms/doubtfire-deploy/commit/3f5aa2be6bd69441730375b689751fe881d7617a))
-* add test attempt auth ([7d31f7c](https://github.com/doubtfire-lms/doubtfire-deploy/commit/7d31f7caaae6dc1efa24f78842873e9f55796279))
-* change Numbas time delay config to enable incremental delays ([54c27ce](https://github.com/doubtfire-lms/doubtfire-deploy/commit/54c27cef2b8ff57fd8ac972728ec3d249e2862b8))
-* create unique token for scorm asset retrieval ([fc8134a](https://github.com/doubtfire-lms/doubtfire-deploy/commit/fc8134ab6b734b7daf064a67ad15f3cefba1d7d6))
-* enable reviewing, passing, and deleting test attempts ([8c9a68b](https://github.com/doubtfire-lms/doubtfire-deploy/commit/8c9a68ba6b3914da24ba33ee62f6a5a00e101c76))
-* enable students to request extra scorm attempt ([c5055b8](https://github.com/doubtfire-lms/doubtfire-deploy/commit/c5055b858c30ba693c535590e1ccff0e8e0b42da))
-* restrict test attempts by limit and comments to when test is completed ([26d75f5](https://github.com/doubtfire-lms/doubtfire-deploy/commit/26d75f51b7fcf11dac0834ddc5a46f40c07407de))
-
-
-### Bug Fixes
-
-* add allow review property to task def related files ([3539d95](https://github.com/doubtfire-lms/doubtfire-deploy/commit/3539d957022f0c6310a2939dd6eccad946cb6610))
-* add missing numbas config fields to fix unit tests ([89a6615](https://github.com/doubtfire-lms/doubtfire-deploy/commit/89a66157b4fde887a19912ca40261243b4961e2f))
-* add scorm bypass to excel file ([4139690](https://github.com/doubtfire-lms/doubtfire-deploy/commit/413969069969316f6ea9c515e4ec9da6b332be0a))
-* calculate attempt number and limit instead of using stored int ([28f3279](https://github.com/doubtfire-lms/doubtfire-deploy/commit/28f327964edb0c9326b487a674d19b7da7da8c89))
-* change scorm comment text ([69053ee](https://github.com/doubtfire-lms/doubtfire-deploy/commit/69053ee147503e7916e929aac5c834903c0087ba))
-* check for attempts before accessing properties ([4255347](https://github.com/doubtfire-lms/doubtfire-deploy/commit/42553479eb2a018a9273931e033084e26b3d18d5))
-* check if no old scorm tokens exist ([6108b52](https://github.com/doubtfire-lms/doubtfire-deploy/commit/6108b52bc04d7866548c8738b39d37c30d24f602))
-* consolidate numbas api endpoints ([27253bd](https://github.com/doubtfire-lms/doubtfire-deploy/commit/27253bd1b1d5640d00098f692160dd4b50675640))
-* enforce attempt limit ([d71ea14](https://github.com/doubtfire-lms/doubtfire-deploy/commit/d71ea14d319a59ba1e96bbd5bf34c85a21f0c0f6))
-* expose enable Numbas test config to all users ([20d5265](https://github.com/doubtfire-lms/doubtfire-deploy/commit/20d526533a2ecab592d7d22f3330d37cee7e0f45))
-* expose scorm configs to student ([910eecd](https://github.com/doubtfire-lms/doubtfire-deploy/commit/910eecdc218f52e572d39059e64a0b28acb44dce))
-* grant same number of extra attempts as scorm limit ([3d44ef2](https://github.com/doubtfire-lms/doubtfire-deploy/commit/3d44ef2ea57829131cc3c70d1655ccd996154ee2))
-* post scorm comment after test attempt termination ([0812e20](https://github.com/doubtfire-lms/doubtfire-deploy/commit/0812e206a9dadcfe7d575feec04e49b15b412556))
-* preload unit in test attempt and ensure limit flexibility in validation ([8059213](https://github.com/doubtfire-lms/doubtfire-deploy/commit/80592130bfb33bbb74322c5950e62e2663223af1))
-* prevent new attempt if last is incomplete or passed ([1240b3f](https://github.com/doubtfire-lms/doubtfire-deploy/commit/1240b3fa853d3a3f3fd1ad061f9cc6f6635c2c37))
-* prevent scorm extensions if no attempt limit ([1ae0347](https://github.com/doubtfire-lms/doubtfire-deploy/commit/1ae03478bb2c55b5e281a876bae37f730206ac3e))
-* refactor numbas config reset logic ([ff5ff62](https://github.com/doubtfire-lms/doubtfire-deploy/commit/ff5ff62061c05e509f15af3048fe047b0d69dc68))
-* rename entity file and add update fields in task spreadsheet ([b498924](https://github.com/doubtfire-lms/doubtfire-deploy/commit/b4989242e37ccd046651bfc8db32934ee94e190a))
-* reorder columns for csv export ([5db5f35](https://github.com/doubtfire-lms/doubtfire-deploy/commit/5db5f35dc6cc1874c50f5891ca7bbd752ea32b55))
-* reset Numbas configs if no zip file has been uploaded ([3f19ffa](https://github.com/doubtfire-lms/doubtfire-deploy/commit/3f19ffa6f4f465ed0691582b5012cf997ec62852))
-* temporarily disable auth and fix test attempt lookup ([b4d3f9d](https://github.com/doubtfire-lms/doubtfire-deploy/commit/b4d3f9dc1661b733eaf704c551ceb5836789db22))
-* update auth token to work with scorm and general ([e7a6eed](https://github.com/doubtfire-lms/doubtfire-deploy/commit/e7a6eed53d8e7049b6144e2b07b8018725be01fb))
-* use correct endpoint url and include exam result for numbas test attempts ([ee992f4](https://github.com/doubtfire-lms/doubtfire-deploy/commit/ee992f4218b8ca07c9259d6569c9c946af7701ef))
-* use correct Numbas data path in Numbas api ([5d80830](https://github.com/doubtfire-lms/doubtfire-deploy/commit/5d80830d3564bb7137db3c4adb3b1d906342e851))
-* use custom endpoint for Numbas ([0cc4915](https://github.com/doubtfire-lms/doubtfire-deploy/commit/0cc4915c85d7d55b48ca6832f6779e49362a7870))
-* use project and task def to fix issue where task is undefined on launching scorm test ([2a04a06](https://github.com/doubtfire-lms/doubtfire-deploy/commit/2a04a068282f69b11a6243a590bb25edcdd5c2c1))
-* use test attempt entity in file instead ([a7c4006](https://github.com/doubtfire-lms/doubtfire-deploy/commit/a7c400669bf199f30b627b54c4ed49157ff88222))
-* use unique perms for scorm test retrieval ([08a0090](https://github.com/doubtfire-lms/doubtfire-deploy/commit/08a00906019ce0c2706c34cf053a511b6e5ddca2))
-* validate attempt id ([c5240d8](https://github.com/doubtfire-lms/doubtfire-deploy/commit/c5240d8da378b84deb3ac64e1584808b07d5e671))
+
+### Bug Fixes
+
+* ensure that TII can log multiple similarity issues for each task ([55aa194](https://github.com/macite/doubtfire-deploy/commit/55aa1940b418d5bcb7d43663d5453e7cc6f8610a))
+
+### [8.0.27](https://github.com/macite/doubtfire-deploy/compare/v8.0.26...v8.0.27) (2024-08-28)
+
+
+### Bug Fixes
+
+* correct link to error log mailer and add test ([312f22e](https://github.com/macite/doubtfire-deploy/commit/312f22eacead8b8d666116df52a7c11e49ce1794))
+
+### [8.0.26](https://github.com/macite/doubtfire-deploy/compare/v8.0.23...v8.0.26) (2024-08-26)
+
+
+### Bug Fixes
+
+* logging of fail to send message in accept submission ([38abe9e](https://github.com/macite/doubtfire-deploy/commit/38abe9eeb7dedf8f7d26b7b1c659be94d9c42d4a))
+* use system timeout command with timeout helper ([b77147c](https://github.com/macite/doubtfire-deploy/commit/b77147c791396e202bbf2e01eb60385a1ae6cd7b))
 
 ### [8.0.23](https://github.com/macite/doubtfire-deploy/compare/v8.0.22...v8.0.23) (2024-08-05)
 

app/helpers/file_helper.rb

@@ -11,7 +11,7 @@ module FileHelper
   extend MimeCheckHelpers
 
   def known_extension?(extn)
-    allow_extensions = %w(pdf ps csv xls xlsx pas cpp c cs csv h hpp java py js html coffee scss yaml yml xml json ts r rb rmd rnw rhtml rpres tex vb sql txt md jack hack asm hdl tst out cmp vm sh bat dat ipynb css png bmp tiff tif jpeg jpg gif zip gz tar wav ogg mp3 mp4 webm aac pcm aiff flac wma alac pml)
+    allow_extensions = %w(pdf ps csv xls xlsx pas cpp c cs csv h hpp java py js html coffee scss yaml yml xml json ts r rb rmd rnw rhtml rpres tex vb sql txt md jack hack asm hdl tst out cmp vm sh bat dat ipynb css png bmp tiff tif jpeg jpg gif zip gz tar wav ogg mp3 mp4 webm aac pcm aiff flac wma alac pml vue)
 
     # Allow empty or nil extensions for blobs otherwise check that it matches the allowed list
     extn.blank? || allow_extensions.include?(extn)

app/helpers/timeout_helper.rb

@@ -25,12 +25,10 @@ def try_within(sec, timeout_message = 'operation')
   #
   def system_try_within(sec, timeout_message, command)
     # shell script to kill command after timeout
-    timeout_exec = Rails.root.join('lib/shell/timeout.sh')
-    result = false
-    try_within sec, timeout_message do
-      result = system "#{timeout_exec} -t #{sec} nice -n 10 #{command}"
-    end
-    result
+    system "timeout -k 2 #{sec} nice -n 10 #{command}"
+  rescue
+    logger.error "Timeout when #{timeout_message} after #{sec}s"
+    false
   end
 
   # Export functions as module functions

app/models/task.rb

@@ -1101,7 +1101,7 @@ def self.pygments_lang(extn)
     elsif ['cpp', 'hpp', 'c++', 'h++', 'cc', 'cxx', 'cp'].include?(extn) then 'cpp'
     elsif ['java'].include?(extn) then 'java'
     elsif %w(js json ts).include?(extn) then 'js'
-    elsif ['html', 'rhtml'].include?(extn) then 'html'
+    elsif ['html', 'rhtml', 'vue'].include?(extn) then 'html'
     elsif %w(css scss).include?(extn) then 'css'
     elsif ['rb'].include?(extn) then 'ruby'
     elsif ['coffee'].include?(extn) then 'coffeescript'

app/models/turn_it_in/tii_action_upload_submission.rb

@@ -17,8 +17,8 @@ def update_from_pdf_report_status(response)
     case response
     when 'FAILED' # The report failed to be generated
       error_message = 'similarity PDF failed to be created'
-    when 'SUCCESS' # Similarity report is complete
-      entity.status = :similarity_pdf_requested
+    when 'SUCCESS' # Similarity report is complete - pdf is available
+      entity.status = :similarity_pdf_available
       entity.save
       save_progress
       download_similarity_report_pdf(skip_check: true)
@@ -64,17 +64,15 @@ def update_from_similarity_status(response)
     # when 'PROCESSING' # Similarity report is being generated
     #   return
     when 'COMPLETE' # Similarity report is complete
-      entity.overall_match_percentage = response.overall_match_percentage
-
-      flag = response.overall_match_percentage.present? && response.overall_match_percentage.to_i > task.tii_match_pct(idx)
-
       # Update the status of the entity
-      entity.update(status: flag ? :similarity_report_complete : :complete_low_similarity)
+      entity.overall_match_percentage = response.overall_match_percentage.present? ? response.overall_match_percentage.to_i : -1
+      flag = entity.should_flag?
+      entity.status = flag ? :similarity_report_complete : :complete_low_similarity
+      entity.save
 
-      # Create the similarity record
-      TiiTaskSimilarity.find_or_initialize_by task: entity.task do |similarity|
-        similarity.pct = response.overall_match_percentage
-        similarity.tii_submission = entity
+      # Create the similarity record - for task and this turn it in submission
+      TiiTaskSimilarity.find_or_initialize_by task: entity.task, tii_submission: entity do |similarity|
+        similarity.pct = entity.overall_match_percentage # record percentage
         similarity.flagged = flag
         similarity.save
       end
@@ -108,7 +106,7 @@ def next_step
       "awaiting similarity report"
     when :similarity_pdf_available
       "downloading similarity report"
-    when "similarity_pdf_downloaded"
+    when :similarity_pdf_downloaded
       "complete - report available"
     when :to_delete
       "awaiting deletion"

app/models/turn_it_in/tii_submission.rb

@@ -68,6 +68,13 @@ def create_viewer_url(user)
     ).perform
   end
 
+  # Should we flag this task for high similarity?
+  #
+  # @return [Boolean] true if the task should be flagged, false otherwise
+  def should_flag?
+    overall_match_percentage > task.tii_match_pct(idx)
+  end
+
   private
 
   # Delete the turn it in submission for a task

app/sidekiq/accept_submission_job.rb

@@ -16,12 +16,12 @@ def perform(task_id, user_id, accepted_tii_eula)
     begin
       logger.info "Accepting submission for task #{task.id} by user #{user.id}"
       # Convert submission to PDF
-      task.convert_submission_to_pdf(log_to_stdout: false)
+      task.convert_submission_to_pdf(log_to_stdout: true)
     rescue StandardError => e
       # Send email to student if task pdf failed
       if task.project.student.receive_task_notifications
         begin
-          PortfolioEvidenceMailer.task_pdf_failed(project, [task]).deliver
+          PortfolioEvidenceMailer.task_pdf_failed(task.project, [task]).deliver
         rescue StandardError => e
           logger.error "Failed to send task pdf failed email for project #{task.project.id}!\n#{e.message}"
         end

config/sidekiq.yml

@@ -0,0 +1 @@
+:concurrency: 1

test/mailers/error_log_mailer_test.rb

@@ -0,0 +1,19 @@
+require 'test_helper'
+require 'grade_helper'
+
+class ErrorLogMailerTest < ActionMailer::TestCase
+
+  def test_can_send_error_log_mail
+    Doubtfire::Application.config.email_errors_to = 'test <[email protected]>'
+    begin
+      raise 'test'
+    rescue StandardError => e
+      mail = ErrorLogMailer.error_message('test', 'test message', e)
+    end
+
+    assert mail.present?
+    assert mail.to.include? '[email protected]'
+    assert mail.body.include? e.message
+    assert mail.body.include? e.backtrace.join("\n")
+  end
+end

test_files/submissions/test.vue

@@ -0,0 +1,23 @@
+<template>
+  <div class="example">{{ msg }}</div>
+</template>
+
+<script>
+export default {
+  data() {
+    return {
+      msg: 'Hello world!'
+    }
+  }
+}
+</script>
+
+<style>
+.example {
+  color: red;
+}
+</style>
+
+<custom1>
+  This could be e.g. documentation for the component.
+</custom1>