Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement support for National cloud deployments (Issue #937) #938

Merged
merged 35 commits into from
Jun 16, 2020

Conversation

abraunegg
Copy link
Owner

abraunegg added 6 commits May 29, 2020 08:51
* Add documentation for option
* Update readme
Update documentation
Update readme
* Remove debug logging
@abraunegg abraunegg added this to the v2.4.3 milestone May 29, 2020
@abraunegg
Copy link
Owner Author

@norbusan
Whilst the code changes do work - that is ability to set the Auth / Graph URL's to specific objects, unable to test this myself to make sure that there are no other issues.

Documentation for this feature 'looks' right based on existing application configuration, but again, no way to test (except for using new application id which we know works)

@abraunegg abraunegg requested a review from norbusan June 3, 2020 20:59
@abraunegg abraunegg changed the title WIP: Implement support for National cloud deployments Implement support for National cloud deployments (Issue #937) Jun 3, 2020
norbusan
norbusan previously approved these changes Jun 4, 2020
Copy link
Collaborator

@norbusan norbusan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like you, I cannot actually test it, but the changes look reasonable

@abraunegg abraunegg linked an issue Jun 6, 2020 that may be closed by this pull request
* Generage a /delta compatible response when using National Azure AD deployments that do not support /delta queries
* USL4 needs to be checked
abraunegg added 7 commits June 9, 2020 17:42
* Add error handling for OneDrive Exceptions (HTTP errors etc)
* Add logging so the user 'knows' that stuff is being queried ... otherwise the OneDrive walk is occuring .. and there is zero visibility what is going on
* Update error handling that makes more sense
* Handle nextLink when /children responses have greater than 200 elements
* Add folder being scanned to better show progress
* Handle remote deletes when /delta is not possible by flagging all files as in-sync, but if using National Cloud Deployment, before calling /children, flag all files as out-of-sync locally. As each JSON response from OneDrive is processed, it resets the flag for that item as being 'in-sync', thus we can query those that were not set, thus, those are files that were remove from OneDrive, and should be removed locally.
* resetting this to false (was true) to test locally
@abraunegg
Copy link
Owner Author

@norbusan
This should be good for another review now. It is quite a large feature implementation now, including a DB schema update, but those changes ONLY ever get used / called if a National Cloud Deployment is configured for use.

When in use, the output is like this:

./onedrive --confdir '~/.config/onedrive-business/' --synchronize --verbose
Using 'user' Config Dir: /home/alex/.config/onedrive-business/
Using 'system' Config Dir: 
Configuration file successfully loaded
Initializing the OneDrive API ...
Configuring Global Azure AD Endpoints
Opening the item database ...
All operations will be performed in: /home/alex/OneDriveBusiness
Application version: v2.4.2-29-g58423d8
Account Type: business
Default Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA
Default Root ID: 01WIXGO5V6Y2GOVW7725BZO354PWSELRRZ
Remaining Free Space: 1098476449374
Fetching details for OneDrive Root
OneDrive Root exists in the database
Initializing the Synchronization Engine ...
Syncing changes from OneDrive ...
Applying changes of Path ID: 01WIXGO5V6Y2GOVW7725BZO354PWSELRRZ
Adding OneDrive root details for processing
Adding 5 OneDrive items for processing from OneDrive root
Adding 2 OneDrive items for processing from /SMPP
Processing 8 OneDrive items to ensure consistent local state
Uploading differences of .
Processing root
The directory has not changed
Processing random_files
The directory has not changed
Processing random_images
The directory has not changed
Processing random_videos
The directory has not changed
Processing newfile.txt
The file has not changed
Processing SMPP
The directory has not changed
Processing SMPP-IF-SPEC_v3_3-24858.pdf
The file has not changed
Processing SMPP_v3_4_Issue1_2-24857.pdf
The file has not changed
Uploading new items of .
Applying changes of Path ID: 01WIXGO5V6Y2GOVW7725BZO354PWSELRRZ
Adding OneDrive root details for processing
Adding 5 OneDrive items for processing from OneDrive root
Adding 2 OneDrive items for processing from /SMPP
Processing 8 OneDrive items to ensure consistent local state

Deleting data on OneDrive:

./onedrive --confdir '~/.config/onedrive-business/' --synchronize --verbose
Using 'user' Config Dir: /home/alex/.config/onedrive-business/
Using 'system' Config Dir: 
Configuration file successfully loaded
Initializing the OneDrive API ...
Configuring Global Azure AD Endpoints
Opening the item database ...
All operations will be performed in: /home/alex/OneDriveBusiness
Application version: v2.4.2-29-g58423d8
Account Type: business
Default Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA
Default Root ID: 01WIXGO5V6Y2GOVW7725BZO354PWSELRRZ
Remaining Free Space: 1098476449374
Fetching details for OneDrive Root
OneDrive Root exists in the database
Initializing the Synchronization Engine ...
Syncing changes from OneDrive ...
Applying changes of Path ID: 01WIXGO5V6Y2GOVW7725BZO354PWSELRRZ
Adding OneDrive root details for processing
Adding 4 OneDrive items for processing from OneDrive root
Processing 5 OneDrive items to ensure consistent local state
Trying to delete item SMPP
Deleting item SMPP
Uploading differences of .
Processing root
The directory has not changed
Processing random_files
The directory has not changed
Processing random_images
The directory has not changed
Processing random_videos
The directory has not changed
Processing newfile.txt
The file has not changed
Uploading new items of .
Applying changes of Path ID: 01WIXGO5V6Y2GOVW7725BZO354PWSELRRZ
Adding OneDrive root details for processing
Adding 4 OneDrive items for processing from OneDrive root
Processing 5 OneDrive items to ensure consistent local state

@abraunegg abraunegg requested a review from norbusan June 11, 2020 00:48
norbusan
norbusan previously approved these changes Jun 11, 2020
Copy link
Collaborator

@norbusan norbusan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Alex,
since I cannot test, I only read through the massive changes ... not that I can say from reading that this is semantically correct.

The is one comment about the storage size/type of the syncStatus, but that is not that important.

@abraunegg
Copy link
Owner Author

Hi Alex,
since I cannot test, I only read through the massive changes ... not that I can say from reading that this is semantically correct.

The is one comment about the storage size/type of the syncStatus, but that is not that important.

You should be able to test this - if you 'modify' manually sync.d:

private bool nationalCloudDeployment = false;

to

private bool nationalCloudDeployment = true;

This will 'bypass' the checks where the flag gets set, and uses all the new code, but without using the National Azure Deployment ... it is how I have been testing.

@abraunegg abraunegg merged commit 1c10eff into master Jun 16, 2020
@abraunegg abraunegg deleted the Implement-National-Azure-AD-Clouds branch June 20, 2020 22:32
@github-actions
Copy link

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked and limited conversation to collaborators May 27, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support Office 365 Gov Cloud
2 participants