Linked attachments on the issues/notes aren't transferred

[georgearmenta]

Hello @matm,

When I run gitlab-copy to transfer between different instances/hosts of Gitlab, linked attachments on the issues or notes aren't transferred.

Also, this comment points out the same problem: #15 (comment)

[matm]

Interesting, thanks for reporting.

[matm]

Related: https://gitlab.com/gitlab-org/gitlab/-/issues/17303

[matm]

Hum ... https://gitlab.com/gitlab-org/gitlab/-/issues/24155. Still open for many years and is not fixed yet apparently.

[matm]

@georgearmenta After some investigation I am not sure how to fix this: it used to work a few years ago because every issue's note came with an Attachment field which is now deprecated. From the 2 gitlab issues I added in the comments yesterday, GitLab's API appears to have an upload endpoint (to upload files attached to an issue) but nothing to download them.

This basic resource download feature is not available, which is very surprising. Even large customers of their enterprise solution are asking for it (you can read it in the thread of https://gitlab.com/gitlab-org/gitlab/-/issues/24155).

So for now I have no way to access the attachments, hence no way to migrate them from one GitLab instance to another.

If you have any idea or find anything of interest that would allow me to make some progress here, please let me know.

[georgearmenta]

@matm I have checked those open issues on Gitlab and realized that that is an open issue for years. From time-to-time different enterprise customers requested that feature Gitlab company hasn't prioritized and assigned issue someone to work on it.

Currently, the only way to get attachments on the issues and notes are on a hacky way like using cookies 🥲

curl --cookie "_gitlab_session=xxxxxxxxx" or through automated login to get the file contents (https://stackoverflow.com/a/54706046) (Someone also managed solve this problem on his side with those methos https://gitlab.com/gitlab-org/gitlab/-/issues/25838#note_844882300 )

[matm]

Yeah I saw the "cookie-login" ugly hack. Super unsafe, I don't plan to use it.

[matm]

This really needs to be fixed upstream IMO, not with a dirty hack.

[georgearmenta]

@matm so that means we are stuck at this point ☹️

I know, the cookie-login is the worst solution. But it looks like Gitlab won't have this update for years, even enterprise customers requested.

[georgearmenta]

Oh, I maybe found a solution while checking the API for wiki pages.

Wiki pages API supports uploading attachments to be used for Wiki pages. But upload is upload 😄

Do you think it would work for our case?

https://docs.gitlab.com/ee/api/wikis.html#upload-an-attachment-to-the-wiki-repository

[georgearmenta]

@matm Do you think wiki upload API would fix the issue?

[georgearmenta]

@matm did you have a chance to look on it?

[georgearmenta]

Hello @matm

Did you have a chance to look on it?

[matm]

Hi @georgearmenta, I'm sorry I am super busy at work atm. Had no chance to contribute to any of my open source projects. My analysis on this issue is not complete yet, I need more free time to address it.

Sorry for the delay.

[matm]

Issue copy attachments

Hi @georgearmenta, let's find a clean solution together.

AFAIK GitLab is not going to fix this anytime soon so we need to find our way.

This issue on permissions on project wiki pages per person is 5 years old and still open.

From what I have read in many places, files added in GitLab 11.3 and later are stored in the wiki’s Git repository.

As stated here, while one can upload files to a project (in the wiki), files are not tied to the issue in any way.

There was a proposal to implement an attachment manager into GitLab. Unfortunately this issue is 6 years old, still open, with recent but low activity.

The only endpoint I have found so far is the ability to upload an attachment to the wiki.

Unfortunately, the file would be uploaded to an internal uploads/ folder that is not part of the wiki's git repository, meaning that there would be no easy way to delete any attachment sent there (GitLab's API don't provide any CRUD operation on wiki attachments).

So, as of today, I think 2 solutions are worth investigating:

Using the Project's Wiki

If your destination/target project has a wiki, gitlab-copy could

• Clone the project's wiki locally
• Create a new wiki page named after the (source) issue's title
• Copy all issue's attachments to this wiki page
• Git commit and push the changed to the projet's wiki
• In the target issue, add all attachments references into the issue's description (as a list)

Note that I am not sure adding a link to a wiki resource in an issue is possible, but some have managed to do it in the past ...

Also using the wiki could raise some concerns regarding privacy since some issues can have restricted access to some people inside an organization, while GitLab's wikis don't allow any user access management: any attachment linked to a restricted/private issue could be accessed by all project members in the wiki.

Using an External File Hosting Service

If we don't go the wiki way, we could upload all issue attachments to a separate file hosting service instead, like DropBox or Amazon S3 (or really any other one providing an HTTP REST API for uploading files).

The tool would

• Upload all issue's attachments to the file hosting service (leaving the CRUD responsibility to the end user)
• In the target issue, add all attachments references into the issue's description (as a list)

What do you think about it?

[georgearmenta]

Hello @matm,

I totally agree with you. Gitlab won't provide a solution in near 10+ years, so we are on our own.

There is a no way to delete the attached files inside the issues and also there is 6 years old open issue related to attachment management is also on hold, as you mentioned.

I think, it would be better to stick with Gitlab only solution, so there won't be any third-party dependency.

Am I missing something or Gitlab is providing a way to delete attachments on issues? If you delete any issue using the web interface, the attachments still waiting there on the server.

So, as Gitlab doesn't offer an official solution for this problem (attachment deleting), I think we shouldn't focus on that, as basically gitlab-copy is mirroring the repos.

So, my vote is going with solution 1, using the projects wiki functionality.

For the workflow, reason for creating a wiki page is not clear for me, other than mirroring wiki pages. As the wiki upload api does not expects the wiki page to upload.

For instance, we should change this issue (which is from source repo issue)

This is my test issue
[issueattachment.zip](/uploads/77c38313a2e033b8653f7120be22468e/issueattachment.zip)

to

This is my test issue
[issueattachment.zip](/uploads/new_upload_generated_folder_id/issueattachment.zip)

What do you think about it? Or are we talking about the same with different words 😄

[georgearmenta]

Hello @matm Did you have a chance to consider the available options? What do you think about uploading to the wiki?

[georgearmenta]

Hello @matm,

Did you have a chance to look the options?