Add api endpoint to duplicate document #570

sampaccoud · 2025-01-21T23:07:27Z

Purpose

We want users to be able to duplicate a document.

resolves #335 and #567

Proposal

Refactor the way access control is done on attachment files to play well with document duplication
Add a "duplicate" action to the document API endpoint.
Extract attachment keys from content when a document is saved, and add them to the list of readable attachments on the current document but only if they were already readable on another document by the editing user...

While doing this work I was brought to refactor the way the media-auth and collaboration-auth actions were factorizing code and add missing tests for these actions.

src/backend/core/tests/test_utils_base64_yjs_to_text.py

Dockerfile

src/backend/core/api/viewsets.py

src/backend/core/tests/documents/test_api_documents_media_auth.py

src/backend/core/api/viewsets.py

src/backend/core/tests/test_utils_base64_yjs_to_text.py

sampaccoud · 2025-01-28T07:44:50Z

@lunika done implementing your feedbacks 🙏

AntoLC

If you have existing data the last migration does not succeed.

I got a bit confused about the with_accesses as a query string instead of from the payload.
Except that works well! I used it frontend side, I challenged it a bit, works like excepted.

AntoLC · 2025-01-27T16:35:17Z

src/backend/core/migrations/0013_remove_is_public_add_field_attachments_and_duplicated_from.py

+def populate_attachments_on_all_documents(apps, schema_editor):
+    """Populate "attachments" field on all existing documents in the database."""
+    Document = apps.get_model("core", "Document")
+
+    for document in  Document.objects.all():
+        document.attachments = extract_attachments(document.content)
+        document.save(update_fields=['attachments'])
+
+
+class Migration(migrations.Migration):
+


When I try to apply the migration I get AttributeError: 'Document' object has no attribute 'content'.

It could be interesting to test the migration, there is a nice lib to do it django-test-migrations.
If you are interested, here some examples:
7e05800
5711bf5#diff-b94dbfcff3e13c1f70f5c42f5c071dde9fb1a4f884f80fe18a6ce522d5019476

Exact, I have the same error ! The first time I applied the migration I didn't have documents saved !

@lunika @AntoLC fixed. Can you check again?

src/backend/core/api/viewsets.py

src/backend/core/models.py

AntoLC · 2025-01-28T08:44:01Z

src/backend/core/utils.py

+from core import enums
+
+
+def base64_yjs_to_xml(base64_string):


Nice to see it used ^^

Documents content is stored in the Ydoc format. We need a util to extract it as xml/text.

These 2 actions had factorized code but a few iterations lead to spaghetti code where factorized code includes "if" clauses. Refactor abstractions so that code factorization really works.

…xibility

Tests were forgotten. While writing the tests, I fixed a few edge cases like the possibility to connect to the collaboration server for an anonymous user.

These methods were involved in a bug that was fixed without first evidencing the error in a test: #556 Fixes #567

We took this opportunity to refactor the way access is controlled on media attachments. We now add the media key to a list on the document instance each time a media is uploaded to a document. This list is passed along when a document is duplicated, allowing us to grant access to readers on the new document, even if they don't have or lost access to the original document. We also propose an option to reproduce the same access rights on the duplicate document as what was in place on the original document. This can be requested by passing the "with_accesses=true" option in the query string. The tricky point is that we need to extract attachment keys from the existing documents and set them on the new "attachments" field that is now used to track access rights on media files.

We can't prevent document editors from copy/pasting content to from one document to another. The problem is that copying content, will copy the urls pointing to attachments but if we don't do anything, the reader of the document to which the content is being pasted, may not be allowed to access the attachment files from the original document. Using the work from the previous commit, we can grant access to the readers of the target document by extracting the attachment keys from the content and adding themto the target document's "attachments" field. Before doing this, we check that the current user can indeed access the attachment files extracted from the content and that they are allowed to edit the current document.

…cess

The idea behind wrapping choices in `lazy` function was to allow overriding the list of languages in tests with `override_settings`. This was causin makemigrations to keep on including the field in migrations when it is not needed. Since we finally don't override the LANGUAGES setting in tests, we can remove it to fix the problem.

sampaccoud · 2025-02-02T14:20:56Z

@lunika @AntoLC ready for a new review 😉

lunika · 2025-02-06T13:52:04Z

src/backend/core/enums.py

+ATTACHMENTS_FOLDER = "attachments"
+UUID_REGEX = (
+    r"[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}"
+)
+FILE_EXT_REGEX = r"\.[a-zA-Z]{3,4}"
+MEDIA_STORAGE_URL_PATTERN = re.compile(
+    f"{settings.MEDIA_URL:s}(?P<pk>{UUID_REGEX:s})/"
+    f"(?P<attachment>{ATTACHMENTS_FOLDER:s}/{UUID_REGEX:s}{FILE_EXT_REGEX:s})$"
+)
+MEDIA_STORAGE_URL_EXTRACT = re.compile(
+    f"{settings.MEDIA_URL:s}({UUID_REGEX}/{ATTACHMENTS_FOLDER}/{UUID_REGEX}{FILE_EXT_REGEX})"
+)
+COLLABORATION_WS_URL_PATTERN = re.compile(rf"(?:^|&)room=(?P<pk>{UUID_REGEX})(?:&|$)")


All except MEDIA_STORAGE_URL_EXTRACT are duplicated in the same file (from line 11 to 20)

lunika · 2025-02-06T13:53:23Z

src/backend/core/factories.py

@@ -75,7 +91,7 @@ class Meta:

    title = factory.Sequence(lambda n: f"document{n}")
    excerpt = factory.Sequence(lambda n: f"excerpt{n}")
-    content = factory.Sequence(lambda n: f"content{n}")
+    content = YDOC_HELLO_WORLD_BASE64


To be consistent maybe excerpt can be computed from this content ?

AntoLC · 2025-02-10T10:01:08Z

src/backend/core/migrations/0018_remove_is_public_add_field_attachments_and_duplicated_from.py

+    Document = apps.get_model("core", "Document")
+
+    for document in  Document.objects.all():
+        document.attachments = extract_attachments(document.content)


I still have the error here during the migration.

Applying core.0018_remove_is_public_add_field_attachments_and_duplicated_from...Traceback (most recent call last): File "/app/manage.py", line 15, in <module> execute_from_command_line(sys.argv) File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line utility.execute() File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 436, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/local/lib/python3.12/site-packages/django/core/management/base.py", line 413, in run_from_argv self.execute(*args, **cmd_options) File "/usr/local/lib/python3.12/site-packages/django/core/management/base.py", line 459, in execute output = self.handle(*args, **options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/django/core/management/base.py", line 107, in wrapper res = handle_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/django/core/management/commands/migrate.py", line 357, in handle post_migrate_state = executor.migrate( ^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/django/db/migrations/executor.py", line 135, in migrate state = self._migrate_all_forwards( ^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/django/db/migrations/executor.py", line 255, in apply_migration state = migration.apply(state, schema_editor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/django/db/migrations/migration.py", line 132, in apply operation.database_forwards( File "/usr/local/lib/python3.12/site-packages/django/db/migrations/operations/special.py", line 196, in database_forwards self.code(from_state.apps, schema_editor) File "/app/core/migrations/0018_remove_is_public_add_field_attachments_and_duplicated_from.py", line 17, in populate_attachments_on_all_documents document.attachments = extract_attachments(document.content)

sampaccoud self-assigned this Jan 21, 2025

sampaccoud requested review from qbey, AntoLC, lunika and virgile-dev January 21, 2025 23:07

lunika reviewed Jan 22, 2025

View reviewed changes

sampaccoud force-pushed the add-api-endpoint-to-duplicate-document branch from a138321 to 95241cd Compare January 28, 2025 07:44

sampaccoud requested a review from lunika January 28, 2025 07:44

AntoLC reviewed Jan 28, 2025

View reviewed changes

virgile-dev mentioned this pull request Jan 29, 2025

Add an onboarding template doc for new users #393

Open

sampaccoud added 4 commits January 29, 2025 14:56

✨(backend) add util to extract text from Ydoc content

75ed4f5

Documents content is stored in the Ydoc format. We need a util to extract it as xml/text.

♻️(backend) refactor media_auth and collaboration_auth for flexibility

7d496cd

These 2 actions had factorized code but a few iterations lead to spaghetti code where factorized code includes "if" clauses. Refactor abstractions so that code factorization really works.

fixup! ♻️(backend) refactor media_auth and collaboration_auth for fle…

852f386

…xibility

✅(backend) add missing tests for collaboration auth

778f9e2

Tests were forgotten. While writing the tests, I fixed a few edge cases like the possibility to connect to the collaboration server for an anonymous user.

sampaccoud mentioned this pull request Jan 29, 2025

⚗️ Extract text from base64 yjs document #270

Closed

sampaccoud added 6 commits January 30, 2025 00:32

✅(backend) add missing test on media-auth and collaboration-auth

397242c

These methods were involved in a bug that was fixed without first evidencing the error in a test: #556 Fixes #567

fixup! ✨(backend) add duplicate action to the document API endpoint

c749287

fixup! ✨(backend) extract attachment keys from updated content for ac…

5411210

…cess

sampaccoud force-pushed the add-api-endpoint-to-duplicate-document branch from 95241cd to bddfd20 Compare January 30, 2025 18:42

sampaccoud requested a review from AntoLC February 2, 2025 13:59

lunika reviewed Feb 6, 2025

View reviewed changes

AntoLC reviewed Feb 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add api endpoint to duplicate document #570

Add api endpoint to duplicate document #570

sampaccoud commented Jan 21, 2025 •

edited

Loading

sampaccoud commented Jan 28, 2025

AntoLC left a comment

AntoLC Jan 27, 2025

lunika Jan 28, 2025

sampaccoud Feb 2, 2025

AntoLC Jan 28, 2025

sampaccoud commented Feb 2, 2025

lunika Feb 6, 2025

lunika Feb 6, 2025

AntoLC Feb 10, 2025 •

edited

Loading

Add api endpoint to duplicate document #570

Are you sure you want to change the base?

Add api endpoint to duplicate document #570

Conversation

sampaccoud commented Jan 21, 2025 • edited Loading

Purpose

Proposal

sampaccoud commented Jan 28, 2025

AntoLC left a comment

Choose a reason for hiding this comment

AntoLC Jan 27, 2025

Choose a reason for hiding this comment

lunika Jan 28, 2025

Choose a reason for hiding this comment

sampaccoud Feb 2, 2025

Choose a reason for hiding this comment

AntoLC Jan 28, 2025

Choose a reason for hiding this comment

sampaccoud commented Feb 2, 2025

lunika Feb 6, 2025

Choose a reason for hiding this comment

lunika Feb 6, 2025

Choose a reason for hiding this comment

AntoLC Feb 10, 2025 • edited Loading

Choose a reason for hiding this comment

sampaccoud commented Jan 21, 2025 •

edited

Loading

AntoLC Feb 10, 2025 •

edited

Loading