Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GO-128 Download messages from FS #483

Merged
merged 38 commits into from
Nov 13, 2024
Merged
Changes from 12 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
bb0a0a0
Draft downloading sent FS message
luciajanikova Oct 15, 2024
0ac9aa8
Move creating message with thread to Fs::Message
luciajanikova Oct 15, 2024
ce6ab5b
Save correlation_id to Fs::MessageDraft
luciajanikova Oct 16, 2024
95e9b4e
Update downloading FS sent messages & Add downloading received messages
luciajanikova Oct 16, 2024
154ee47
Remove format_html_visualization method
luciajanikova Oct 16, 2024
6e6ac84
Update Fs::Api
luciajanikova Oct 16, 2024
823fe35
Small syntax updates
luciajanikova Oct 16, 2024
0dbe7f0
Draft custom FS visualizations
luciajanikova Oct 17, 2024
41e10c9
Update visualizations
luciajanikova Oct 22, 2024
47899d5
Create PDF from message HTML visualization
luciajanikova Oct 22, 2024
fdbe251
Update FS HTML visualizations templates
luciajanikova Oct 22, 2024
a32b402
Update datetime operations & FS API response updates
luciajanikova Oct 22, 2024
8ac7511
Add submission_type_identifier attribute to Fs::Form
luciajanikova Oct 22, 2024
2f935b6
Use paging when requesting FS messages
luciajanikova Oct 22, 2024
351dce0
Draft Fs boxes sync
luciajanikova Oct 23, 2024
fd14363
Update brakeman ignore
luciajanikova Oct 23, 2024
be51628
Update checks in download messages jobs
luciajanikova Oct 23, 2024
e19f6e4
Small syntax updates
luciajanikova Oct 23, 2024
1b81b73
Update sync FS box jobs
luciajanikova Oct 23, 2024
7d2fd3e
Formatting updates
luciajanikova Oct 23, 2024
0747cc4
Update Fs::MessageDraft visualizations
luciajanikova Oct 23, 2024
6840c4f
Add display_url to generated PDF
luciajanikova Oct 23, 2024
50ca8c5
Small fixes after PROD testing
luciajanikova Oct 23, 2024
6dbbe2c
Update Fs::DownloadSentMessageJob
luciajanikova Oct 24, 2024
58bfcc2
Merge branch 'main' into GO-128/download_messages_from_fs
luciajanikova Oct 24, 2024
63ce7b3
Schedule Fs::FetchFormsJob in latest migration
luciajanikova Oct 25, 2024
a5167d9
Add style to Fs::Messages HTML visualizations (to use same font as in…
luciajanikova Oct 25, 2024
1c38460
Merge branch 'main' into GO-128/download_messages_from_fs
luciajanikova Oct 29, 2024
266ffa0
Make download jobs fail if any error
luciajanikova Nov 6, 2024
84f1650
Send uuids to FS API when submitting
luciajanikova Nov 7, 2024
dcfd011
Copy tags to downloaded FS outbox message from draft
luciajanikova Nov 11, 2024
5d9e89d
Update Automation::Condition in order to be able to create Automation…
luciajanikova Nov 11, 2024
5d135b3
Add missing fixtures
luciajanikova Nov 12, 2024
7307fae
Add missing fixtures 2
luciajanikova Nov 12, 2024
9de9135
Update automation to be able to add tag to thread after form object i…
luciajanikova Nov 12, 2024
16dc11c
make grover work in docker container
celuchmarek Nov 12, 2024
6a97f21
Update download message object event
luciajanikova Nov 12, 2024
8474ba5
Try to update Dockerfile
luciajanikova Nov 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Gemfile
Original file line number Diff line number Diff line change
@@ -39,6 +39,7 @@ gem 'jwt'
gem 'stimulus-rails'
gem 'jsbundling-rails'
gem 'pdf-reader'
gem 'grover'

# Monitoring
gem 'rollbar'
7 changes: 7 additions & 0 deletions Gemfile.lock
Original file line number Diff line number Diff line change
@@ -117,6 +117,9 @@ GEM
activesupport
tzinfo
coderay (1.1.3)
combine_pdf (1.0.26)
matrix
ruby-rc4 (>= 0.1.5)
concurrent-ruby (1.2.3)
connection_pool (2.4.1)
crack (1.0.0)
@@ -164,6 +167,9 @@ GEM
fugit (>= 1.1)
railties (>= 6.0.0)
thor (>= 0.14.1)
grover (1.1.11)
combine_pdf (~> 1.0)
nokogiri (~> 1.0)
hashdiff (1.1.0)
hashery (2.1.2)
hashie (5.0.0)
@@ -512,6 +518,7 @@ DEPENDENCIES
erb_lint
foreman
good_job
grover
htmlbeautifier
importmap-rails
jbuilder
2 changes: 1 addition & 1 deletion app/components/message_draft_body_component.html.erb
Original file line number Diff line number Diff line change
@@ -93,7 +93,7 @@

<% if @message.html_visualization.present? %>
<div class="w-full">
<%= tag.iframe class: "relative border-none overflow-hidden h-full w-full", srcdoc: @message.format_html_visualization, onload: "(
<%= tag.iframe class: "relative border-none overflow-hidden h-full w-full", srcdoc: @message.html_visualization, onload: "(
function(iframe) {
iframe.contentWindow.document.body.style['height'] = 'unset';
iframe.contentWindow.document.body.style['min-height'] = 'unset';
4 changes: 2 additions & 2 deletions app/helpers/application_helper.rb
Original file line number Diff line number Diff line change
@@ -9,10 +9,10 @@ def nice_datetime(datetime)
end
end

def nice_datetime_with_time(datetime)
def nice_datetime_with_time(datetime, keep_year: false)
if datetime.today?
l(datetime, format: '%H:%M')
elsif datetime.year == Date.current.year
elsif datetime.year == Date.current.year && !keep_year
l(datetime, format: '%e. %b %H:%M')
else
l(datetime, format: '%e. %b %Y %H:%M')
22 changes: 22 additions & 0 deletions app/helpers/fs/message_helper.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
module Fs::MessageHelper
def self.build_html_visualization(message)
return [ActionController::Base.new.render_to_string('fs/messages/_submission', layout: false, locals: { message: message }), build_html_visualization_from_form(message)].compact.join('<hr>') if message.outbox?

# TODO: Vieme aj lepsie identifikovat? Nejake dalsie typy v tejto kategorii neexistuju?
Copy link
Member Author

@luciajanikova luciajanikova Oct 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Toto tu je otvorena vec. FS API v response vracia aj message_type, da sa identifikovat aj podla toho (ale tiez to je trochu len take hadanie, ked nemame nejaku dokumentaciu k tomu). Nevieme ako casto sa mozu zmenit bud tieto nazvy alebo message_type.
Samotna dorucenka si myslim, ze problem nie je (tam by som sa spolahla aj na message_type), ale tieto negativne verzie su narocnejsie. Nevieme ani kolko typov existuje.

template = if message.title.in?(['Informácia o podaní', 'Informácia o odmietnutí podania'])
'fs/messages/_delivery_report'
else
'fs/messages/_generic_message'
end

ActionController::Base.new.render_to_string(template, layout: false, locals: { message: message })
end

def self.build_html_visualization_from_form(message)
return unless message.form&.xslt_txt
return unless message.form_object&.unsigned_content

template = Nokogiri::XSLT(message.form.xslt_txt)
ActionController::Base.helpers.simple_format(template.transform(message.form_object.xml_unsigned_content).to_s)
end
end
6 changes: 0 additions & 6 deletions app/helpers/message_helper.rb
Original file line number Diff line number Diff line change
@@ -2,10 +2,4 @@ module MessageHelper
def self.export_filename(message)
"#{message.delivered_at.to_date}-sprava-#{message.id}.zip"
end

def format_html_visualization
return ActionController::Base.helpers.simple_format(html_visualization) if is_a?(Fs::MessageDraft)

html_visualization
end
end
11 changes: 11 additions & 0 deletions app/jobs/fs/download_received_message_job.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
class Fs::DownloadReceivedMessageJob < ApplicationJob
def perform(fs_message_id, box:, fs_client: FsEnvironment.fs_client)
return if Message.where("metadata ->> 'fs_message_id' = ?", fs_message_id).any?
luciajanikova marked this conversation as resolved.
Show resolved Hide resolved

fs_api = fs_client.api(api_connection: box.api_connection, box: box)

raw_message = fs_api.fetch_received_message(fs_message_id)

Fs::Message.create_inbox_message_with_thread!(raw_message, box: box)
end
end
9 changes: 9 additions & 0 deletions app/jobs/fs/download_sent_message_job.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
class Fs::DownloadSentMessageJob < ApplicationJob
def perform(fs_message_id, box:, fs_client: FsEnvironment.fs_client)
fs_api = fs_client.api(api_connection: box.api_connection, box: box)

raw_message = fs_api.fetch_sent_message(fs_message_id)

Fs::Message.create_outbox_message_with_thread!(raw_message, box: box)
end
end
11 changes: 11 additions & 0 deletions app/jobs/fs/download_sent_message_related_messages_job.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
class Fs::DownloadSentMessageRelatedMessagesJob < ApplicationJob
def perform(outbox_message, fs_client: FsEnvironment.fs_client)
fs_api = fs_client.api(api_connection: outbox_message.box.api_connection, box: outbox_message.box)

received_messages = fs_api.fetch_received_messages(sent_message_id: outbox_message.metadata['fs_message_id'])

received_messages['messages'].each do |received_message|
::Fs::DownloadReceivedMessageJob.perform_later(received_message['message_id'], box: outbox_message.box)
end
end
end
3 changes: 3 additions & 0 deletions app/jobs/fs/submit_message_draft_result_job.rb
Original file line number Diff line number Diff line change
@@ -4,7 +4,10 @@ def perform(message_draft, location_header, fs_client: FsEnvironment.fs_client)

if 200 == response[:status]
message_draft.metadata[:status] = 'submitted'
message_draft.metadata[:fs_message_id] = response[:body]['sent_message_id']
message_draft.save

::Fs::DownloadSentMessageJob.perform_later(response[:body]['sent_message_id'], box: message_draft.box)
elsif [400, 422].include?(response[:status])
message_draft.metadata[:status] = 'submit_fail'
message_draft.add_cascading_tag(message_draft.tenant.submission_error_tag)
9 changes: 5 additions & 4 deletions app/lib/fs/api.rb
Original file line number Diff line number Diff line change
@@ -33,15 +33,16 @@ def fetch_sent_messages(page: 1, count: 100, obo: @obo)
end

def fetch_sent_message(message_id, obo: @obo)
request(:get, "sent-messages/#{message_id}", {}, jwt_header(obo).merge(fs_credentials_header))[:body]
request(:get, "sent-messages/#{CGI.escape(message_id)}", {}, jwt_header(obo).merge(fs_credentials_header))[:body]
end

def fetch_received_messages(page: 1, count: 100, obo: @obo)
request(:get, "received-messages", {}, jwt_header(obo).merge(fs_credentials_header))[:body]
def fetch_received_messages(page: 1, count: 100, sent_message_id: nil, obo: @obo)
query = "?sent_message_id=#{CGI.escape(sent_message_id)}" if sent_message_id
request(:get, "received-messages#{query}", {}, jwt_header(obo).merge(fs_credentials_header))[:body]
end

def fetch_received_message(message_id, obo: @obo)
request(:get, "received-messages/#{message_id}", {}, jwt_header(obo).merge(fs_credentials_header))[:body]
request(:get, "received-messages/#{CGI.escape(message_id)}", {}, jwt_header(obo).merge(fs_credentials_header))[:body]
end

def post_validation(form_identifier, content)
1 change: 0 additions & 1 deletion app/lib/utils.rb
Original file line number Diff line number Diff line change
@@ -100,7 +100,6 @@ def file_extension_by_mimetype(mimetype)

# TODO use UPVS API to detect if document is signed
def is_signed?(entry_name:, content:)

case File.extname(entry_name).downcase
when '.asice', '.asics', '.xzep'
true
26 changes: 25 additions & 1 deletion app/models/concerns/pdf_visualization_operations.rb
Original file line number Diff line number Diff line change
@@ -7,6 +7,10 @@ module PdfVisualizationOperations

included do
def prepare_pdf_visualization
prepare_pdf_visualization_from_template || prepare_pdf_visualization_from_html
end

def prepare_pdf_visualization_from_template
return unless form&.xsl_fo
return unless unsigned_content
return unless xml?
@@ -39,6 +43,13 @@ def prepare_pdf_visualization
end
end

def prepare_pdf_visualization_from_html
return unless form?
return unless message.html_visualization.present?

Grover.new(full_html_document_from_body_content(message.html_visualization), format: 'A4', margin: {top: '15px', bottom: '15px', left: '15px', right: '15px'}).to_pdf
end

def xml_unsigned_content
document = Nokogiri::XML(unsigned_content) do |config|
config.noblanks
@@ -75,7 +86,7 @@ def find_or_create_form
end

def downloadable_as_pdf?
xml? && form&.xsl_fo&.present?
(xml? && form&.xsl_fo&.present?) || (form? && message.html_visualization.present?)
luciajanikova marked this conversation as resolved.
Show resolved Hide resolved
end

def xml?
@@ -85,5 +96,18 @@ def xml?
Utils::XML_MIMETYPES.any? { |xml_mimetype| xml_mimetype == Utils.mimetype_without_optional_params(mimetype) }
end
end

def full_html_document_from_body_content(body_content)
<<-HTML
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
</head>
<body>
#{body_content}
</body>
</html>
HTML
end
end
end
133 changes: 133 additions & 0 deletions app/models/fs/message.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
class Fs::Message
luciajanikova marked this conversation as resolved.
Show resolved Hide resolved
FS_SUBJECT_NAME = 'Finančná správa'

def self.create_inbox_message_with_thread!(raw_message, box:)
message = nil
associated_outbox_message = box.messages.where("metadata ->> 'fs_message_id' = ?", raw_message['sent_message_id']).take

MessageThread.with_advisory_lock!(associated_outbox_message.metadata['correlation_id'], transaction: true, timeout_seconds: 10) do
message = create_inbox_message(raw_message)

message.thread = associated_outbox_message.thread

message.save!
end

create_message_objects(message, raw_message)
update_html_visualization(message)

EventBus.publish(:message_created, message)
end

def self.create_outbox_message_with_thread!(raw_message, box:)
message = nil
associated_message_draft = box.messages.where(type: 'Fs::MessageDraft').where("metadata ->> 'fs_message_id' = ?", raw_message['message_id']).take

merge_identifier = (associated_message_draft.metadata['correlation_id'] if associated_message_draft) || SecureRandom.uuid

MessageThread.with_advisory_lock!(merge_identifier, transaction: true, timeout_seconds: 10) do
message = create_outbox_message(raw_message)

message.thread = box.message_threads.find_or_create_by_merge_uuid!(
box: box,
merge_uuid: merge_identifier,
title: message.title,
delivered_at: message.delivered_at
)

associated_message_draft.destroy if associated_message_draft
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nebude tu rovnaky problem so stracajucimi sa tagmi?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bude, myslim na to, ale logiku toho by som vyladila v tom otvorenom PR najprv a potom to lahko preklopim aj sem. Nech to neriesime teraz na dvoch miestach.


message.save!
end

create_message_objects(message, raw_message)
update_html_visualization(message)

EventBus.publish(:message_created, message)
end

private

def self.create_inbox_message(raw_message)
Message.create(
uuid: SecureRandom.uuid,
title: raw_message['message_type_name'],
sender_name: FS_SUBJECT_NAME,
recipient_name: raw_message['subject'],
delivered_at: Time.parse(raw_message['created_at']).getlocal,
replyable: false,
collapsed: collapsed?,
outbox: false,
metadata: {
"fs_message_id": raw_message['message_id'],
"fs_sent_message_id": raw_message['sent_message_id'],
"fs_status": raw_message['status'],
"fs_submitting_subject": raw_message['submitting_subject'],
"fs_submission_status": raw_message['submission_status'],
"fs_submission_created_at": Time.parse(raw_message['submission_created_at']).getlocal,
"dic": raw_message['dic']
},
)
end

def self.create_outbox_message(raw_message, associated_message_draft: nil)
Message.create(
uuid: (associated_message_draft.uuid if associated_message_draft) || SecureRandom.uuid,
title: raw_message['submission_type_name'],
sender_name: raw_message['subject'],
recipient_name: FS_SUBJECT_NAME,
delivered_at: Time.parse(raw_message['created_at']).getlocal,
replyable: false,
collapsed: collapsed?,
outbox: true,
metadata: {
"fs_form_id": (associated_message_draft.metadata['fs_form_id'] if associated_message_draft) || Fs::Form.where("identifier LIKE '#{raw_message['submission_type_id']}_%'")&.take&.id,
"fs_message_id": raw_message['message_id'],
"fs_status": raw_message['status'],
"submitting_subject": raw_message['submitting_subject'],
"dic": raw_message['dic']
},
)
end

def self.create_message_objects(message, raw_message)
raw_message.dig('message_container', 'objects').each do |raw_object|
tags = raw_object["signed"] ? [message.thread.box.tenant.signed_externally_tag!] : []

message_object = message.objects.create!(
# uuid: raw_object["id"], TODO uncomment when GO-130 is closed
is_signed: raw_object["signed"],
mimetype: raw_object["mime_type"],
name: raw_object["name"],
object_type: raw_object["class"],
tags: tags
)

if raw_object["encoding"] == "Base64"
message_object_content = Base64.decode64(raw_object["content"])
else
message_object_content = raw_object["content"]
end

MessageObjectDatum.create!(
blob: message_object_content,
message_object: message_object
)
end
end

def self.update_html_visualization(message)
message.update(
html_visualization: Fs::MessageHelper.build_html_visualization(message)
)

message.form_object&.update(
visualizable: message.html_visualization.present?
)
end

def self.collapsed?
# TODO urcit podmienky: odoslana sprava s potvrdenkou by mohla byt collapsed
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kedze ta dorucenka je extremne dolezita pre nich, tak skor prave naopak.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Je to tu zle naformulovane, mala som na mysli, ze podanie by mohlo byt collapsed v pripade, ze uz prisla dorucenka. Toto su zatial rychle komentare v ramci draftu, doladim este.

false
end
end
11 changes: 3 additions & 8 deletions app/models/fs/message_draft.rb
Original file line number Diff line number Diff line change
@@ -53,7 +53,8 @@ def self.create_and_validate_with_fs_form(form_files: [], author:, fs_client: Fs
delivered_at: Time.now,
metadata: {
'status': 'being_loaded',
'fs_form_id': fs_form.id
'fs_form_id': fs_form.id,
'correlation_id': SecureRandom.uuid
},
author: author
)
@@ -103,13 +104,7 @@ def attachments_allowed?
end

def build_html_visualization
return self.html_visualization if self.html_visualization.present?

return unless form&.xslt_txt
return unless form_object&.unsigned_content

template = Nokogiri::XSLT(form.xslt_txt)
template.transform(form_object.xml_unsigned_content)
Fs::MessageHelper.build_html_visualization(self)
end

def form
2 changes: 1 addition & 1 deletion app/models/message.rb
Original file line number Diff line number Diff line change
@@ -101,7 +101,7 @@ def form
::Upvs::Form.find_by(
identifier: all_metadata['posp_id'],
version: all_metadata['posp_version']
)
) || (::Fs::Form.find(metadata['fs_form_id']) if metadata['fs_form_id'].present?)
luciajanikova marked this conversation as resolved.
Show resolved Hide resolved
end

def update_html_visualization
2 changes: 1 addition & 1 deletion app/models/nested_message_object.rb
Original file line number Diff line number Diff line change
@@ -16,7 +16,7 @@ class NestedMessageObject < ApplicationRecord

belongs_to :message_object, inverse_of: :nested_message_objects

delegate :message, to: :message_object
delegate :message, :form?, to: :message_object

validates :name, presence: true, on: :validate_data

Loading