Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New API for downloading LDML files from projects that allow sharing WS data #1309

Draft
wants to merge 1 commit into
base: develop
Choose a base branch
from

Conversation

rmunn
Copy link
Contributor

@rmunn rmunn commented Dec 9, 2024

Currently this adds a "ldmlzip" command to our hg command runner.

Will return 403 Forbidden if project does not allow sharing ws data with SLDR. Will also return same 403 Forbidden error code if project does not exist, to avoid possibly leaking project codes.

If project exists and allows data sharing, command will return a zipfile containing CachedSettings/WritingSystems/*.ldml from the tip revision.

Future plans: C# code will unzip the *.ldml files into a directory by project ID, then zip up the whole thing for the API result. Then we'll create a job to delete the LDML directory four hours later, to clean up.

Ideally we'd be able to start streaming the output zip file as we receive data from each project, rather than waiting until all projects have been accessed and only then starting to stream the output zip. But we'll see.

Will return 403 Forbidden if project does not allow sharing ws data with
SLDR. Will also return same 403 Forbidden error code if project does not
exist, to avoid possibly leaking project codes.

If project exists and allows data sharing, command will return a zipfile
containing CachedSettings/WritingSystems/*.ldml from the tip revision.
@rmunn rmunn self-assigned this Dec 9, 2024
@rmunn
Copy link
Contributor Author

rmunn commented Dec 9, 2024

Unfortunately it looks as though we won't be able to use ZipArchive to add files to the output stream asynchronously as they come in: we'd probably run into this issue where ZipArchive tries to write the end-of-zipfile directory synchronously when it's disposed, but Kestrel doesn't allow synchronous writes to its HTTP output stream. So we might have to extract everything, then send the zip file separately. Which means that the HTTP connection might time out, so it's possible we'll have to make two API endpoints, one to prepare the zip file and one to download it. Testing required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

New API to export ldml files for projects which have "Share ws data" set
1 participant