Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Office 365 Gov Cloud #937

Closed
spinkernel opened this issue May 28, 2020 · 41 comments · Fixed by #938
Closed

Support Office 365 Gov Cloud #937

spinkernel opened this issue May 28, 2020 · 41 comments · Fixed by #938
Labels
Feature Request Feature Request | Enhancement Request Implemented

Comments

@spinkernel
Copy link

Is your feature request related to a problem? Please describe.
Greetings, attempting to use this tool with Gov Cloud (office365.us / microsoft.us)
When attempting to authenticate I get:

$ onedrive -v+   
Initializing the OneDrive API ...
Authorize this app visiting:
https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id=22c49a0d-d21c-4792-aed1-8f163c982546&scope=Files.ReadWrite%20Files.ReadWrite.all%20Sites.ReadWrite.All%20offline_access&response_type=code&redirect_uri=https://login.microsoftonline.cNo config file found, using application defaults
Initializing the OneDrive API ...
Authorize this app visiting:
https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id=22c49a0d-d21c-4792-aed1-8f163c982546&scope=Files.ReadWrite%20Files.ReadWrite.all%20Sites.ReadWrite.All%20offline_access&response_type=code&redirect_uri=https://login.microsoftonline.com/common/oauth2/nativeclient

Enter the response uri: https://login.microsoftonline.com/common/oauth2/nativeclient?code=<omitted>
OneDrive returned a 'HTTP 400 - Bad Request' - gracefully handling error
onedrive.OneDriveException@src/onedrive.d(874): HTTP request returned status code 400 (Bad Request)
{
    "correlation_id": "7d410b32-c79b-42d6-8bb0-6ada768d612d",
    "error": "invalid_grant",
    "error_codes": [
        90051
    ],
    "error_description": "AADSTS90051: Invalid Delegation Token. Invalid national Cloud ID (2) is specified.\r\nTrace ID: 5b4a68ef-4ba7-46fa-bd97-0032c20f4a00\r\nCorrelation ID: 7d410b32-c79b-42d6-8bb0-6ada768d612d\r\nTimestamp: 2020-05-28 20:10:06Z",
    "timestamp": "2020-05-28 20:10:06Z",
    "trace_id": "5b4a68ef-4ba7-46fa-bd97-0032c20f4a00"
}
----------------
??:? [0x55d2feff79d9]
??:? [0x55d2feff6cb5]
??:? [0x55d2feff7b65]
??:? [0x55d2feff5f58]
??:? [0x55d2feff58c5]
??:? [0x55d2ff0037a8]
??:? void rt.dmain2._d_run_main2(char[][], ulong, extern (C) int function(char[][])*).runAll() [0x7f555e5c69db]
??:? _d_run_main2 [0x7f555e5c67ee]
??:? _d_run_main [0x7f555e5c665d]
??:? __libc_start_main [0x7f555e1b20b2]
??:? [0x55d2fefce5ed]

Describe the solution you'd like
Authentication to complete.

Describe alternatives you've considered
None

Additional context
None.

@abraunegg
Copy link
Owner

@spinkernel
The client output is not correct.

What client version are you running ?

@spinkernel
Copy link
Author

spinkernel commented May 28, 2020 via email

@spinkernel
Copy link
Author

spinkernel commented May 28, 2020

onedrive v2.3.13-1build2
Ubuntu 20.04 LTS

I just built the latest (2.4.2), i did get slightly different results:

ERROR: OneDrive returned an error with the following message:                                                                   
  Error Message: HTTP request returned status code 400 (Bad Request)                                                            
  Error Reason:  AADSTS900384: JWT token failed signature validation [Reason - The provided signature value did not match the expected signature value., Thumbprint of key used by client: '<omitted>', Found key 'Start=10/14/2018 00:00:00, End=10/14/2020 00:00:00']                                                                                          
  Trace ID: <omitted>                                                                                  
  Correlation ID: <omitted>                                                                            
  Timestamp: 2020-05-28 21:30:03Z                                                                                                 
  
  Application has been successfully authorised, however no additional command switches were provided.                             
  
  Please use --help for further assistance in regards to running this application. 

@abraunegg
Copy link
Owner

@spinkernel
Can you please clean up the output .. all the '....' should not be there, and use triple back ticks as the correct code block please

@abraunegg
Copy link
Owner

abraunegg commented May 28, 2020

@spinkernel
As for the 2 Azure AD errors:

  • AADSTS90051
  • AADSTS900384

This is because the application itself does not know, that for US Gov, it should use different authentication / query URL's - thus it currently does not support National Azure AD endpoints as per https://docs.microsoft.com/en-us/graph/deployments - and it only supports the 'Global Service'

Currently writing a PR for you to test that will provide this functionality. I have however zero capability to test this, other than ensuring that the client passes the right URL's ... so this will require you do perform debug testing.

@spinkernel
Copy link
Author

Happy to test whatever, I figured this is what would happen, didn't expect such quick responses! I'm keeping network engineer in the loop too as he's interested. I'm the only developer and linux user in the company.

@abraunegg
Copy link
Owner

abraunegg commented May 28, 2020

@spinkernel
Please can you test the following PR:

git clone https://github.com/abraunegg/onedrive.git
cd onedrive
git fetch origin pull/938/head:pr938
git checkout pr938
./configure; make clean; make;

You will need to add the following into your config file as well:

azure_ad_endpoint =

Where the following are valid options:

  • USL4
  • USL5
  • DE
  • CN

Example:

azure_ad_endpoint = "USL4"

Example output:

./onedrive --confdir '~/.config/onedrive-personal/' --synchronize --verbose
  ing Config Dir: /home/alex/.config/onedrive-personal/
Using config option for Azure AD for US Government Endpoints
Configuration file successfully loaded
Initializing the OneDrive API ...
Configuring Azure AD for US Government Endpoints
Opening the item database ...
All operations will be performed in: /home/alex/OneDrivePersonal

ERROR: OneDrive returned an error with the following message:
  Error Message: HTTP request returned status code 400 (Bad Request)
  Error Reason:  AADSTS90023: The provided 'msproxy' param is not valid for the instance indicated by the proxy assertion.
Trace ID: 2ea14c4a-559e-4d1a-85a4-8394119a1000
Correlation ID: f1a112de-fbff-4b8b-9426-6e1d0d04b788
Timestamp: 2020-05-28 22:49:25Z

ERROR: OneDrive returned an error with the following message:
  Error Message: HTTP request returned status code 401 (Unauthorized)
  Error Reason:  Access token is empty.

ERROR: Check your configuration as your refresh_token may be empty or invalid. You may need to issue a --logout and re-authorise this client.

Note: This is what I expect for me ... as I do not have a valid login / credentials. I am only going to test that once as per above.

@spinkernel
Copy link
Author

spinkernel commented May 29, 2020

Thanks!
I tried this and received:

 Request Id: 7bec40c6-5aee-454b-9a30-<omitted>
Correlation Id: 881510da-b39d-4b84-9d70-<omitted>
Timestamp: 2020-05-29T17:42:07Z
Message: AADSTS50011: The reply URL specified in the request does not match the reply URLs configured for the application: 'd50ca740-c83f-4d1b-b616-12c519384f0c'.

But i got this in the web sign in page not in onedrive output.
Did a full logout before trying, still gave the same error. Tried USL4 and 5, my IT guys says pretty sure were not DE or CN (which i assume is Europe and china?)

@abraunegg
Copy link
Owner

abraunegg commented May 29, 2020

@spinkernel
This sounds to me like the app registration which is done for the global instance of Azure is not able to be used with national instances Authentication Redirect URL - so that makes sense here.

Before I go changing the application registration, can you, as a test, follow the directions listed here:

This will register you a new application against your Azure AD instance. When selecting the redirect URI, select Public client / native as per below:

Capture

Once you have your new client id, add this to your config file as per:

application_id = "<new app id>"

If then that is working correctly, I can then add the expected response URI to the main / default application ID, negating the need to register a new application ID all the time.

@abraunegg
Copy link
Owner

@spinkernel
Any update on this?

@spinkernel
Copy link
Author

I'm waiting for approval from my IT Dept.
He wants to ensure no data gets exchanged between Microsoft any a 3rd party other than local to my machine.
By registering this application, is the cert that's exchanged local to my machine? Can you explain the exchange of data when registering?

Thanks!

@abraunegg
Copy link
Owner

@spinkernel
By registering an application, all you are doing is, on Azure AD, configuring that a 'new application' will communicate with Azure AD, and will use the Microsoft internally generated application ID.

AFAIK there is no cert / data exchange in this process.

If this will be too difficult, I can just add the URL's to the current application ID - however unsure if that will 'break' for all other users ....

@abraunegg
Copy link
Owner

@spinkernel
As another thought you may also need to configure the application 'scopes' as well. Refer to this URL for what the current application has configured: #769 (comment)

@spinkernel
Copy link
Author

No difficulty its just satisfying paranoia.

You're team does not have any servers which this application ID references or transmits data to correct? That's all that needs to be known. After that we can press forward.

@abraunegg
Copy link
Owner

@spinkernel
100% zero reference or use. You are using a Microsoft Azure AD instance, where you are logging in with your credentials, to create a 'new' application ID, which you can then use in the application configuration.

Feel free to reach out to me offline if you need me to have a chat with anyone.

@abraunegg
Copy link
Owner

@spinkernel
Also - as an FYI - I have tried to add the following response URI's to the current default application:

Azure AD fails to add these, as these are prohibited domains:

authentication_response_uri

So the only way that this will work is:

  1. Register a new application on Azure
  2. Configure the application with the appropriate scopes
  3. Ensure that the application authentication / redirect URI is correctly listed
  4. Configure the onedrive client to use the correct Azure AD instance
  5. Configure the onedrive client to use the new application 'id' as provided in step 1 above

Will have to get this correctly documented, however have no real way to test / validate the steps / process if not using a 'global' Azure AD instance.

@abraunegg
Copy link
Owner

@spinkernel
Any update from your side?

@abraunegg
Copy link
Owner

@abraunegg
Copy link
Owner

@spinkernel
Any update from your side?

1 similar comment
@abraunegg
Copy link
Owner

@spinkernel
Any update from your side?

@spinkernel
Copy link
Author

@abraunegg
Sorry my IT guy has been super busy and hasn't given me the green light yet. He suspects by tomorrow or monday he should have time to help me dig into this. Thanks for being diligent, I can't wait to make this work!

@abraunegg
Copy link
Owner

@spinkernel
No problem - if you could help review the documentation / have your IT guy review the documentation to validate that would be greatly appreciated.

@spinkernel
Copy link
Author

spinkernel commented Jun 8, 2020

@abraunegg
Reviewing your documentation, It's not clear how you get to the deaktop and devices URIs, might want to put one more step in there to show. I had to "add" a service in order for it to show up.

As for the test, things went will until the end:

~/repos/onedrive   pr938  ./onedrive --confdir '~/.config/onedrive/' --synchronize --verbose
Using Config Dir: /home/ <omitted>/.config/onedrive/
Using config option for Azure AD for US Government Endpoints
Configuration file successfully loaded
config file has been updated, checking if --resync needed
Initializing the OneDrive API ...
Configuring Azure AD for US Government Endpoints
Authorize this app visiting:

https://login.microsoftonline.us/common/oauth2/v2.0/authorize?client_id= <omitted>&scope=Files.ReadWrite%20Files.ReadWrite.all%20Sites.Read.All%20Sites.ReadWrite.All%20offline_access&response_type=code&redirect_uri=https://login.microsoftonline.us/common/oauth2/nativeclient

Enter the response uri: https://login.microsoftonline.us/common/oauth2/nativeclient?code= <omitted>
Opening the item database ...
All operations will be performed in: /home/<omitted>
Application version: v2.4.2-9-g7a0915d
Account Type: business
Default Drive ID: <omitted>
Default Root ID:  <omitted>
Remaining Free Space:  <omitted>
Fetching details for OneDrive Root
OneDrive Root does not exist in the database. We need to add it.
Added OneDrive Root to the local database
Initializing the Synchronization Engine ...
Syncing changes from OneDrive ...
Applying changes of Path ID: <omitted>

ERROR: OneDrive returned an error with the following message:
  Error Message: HTTP request returned status code 400 (Bad Request)
  Error Reason:  Unsupported request: Change tracking is not supported against 'microsoft.graph.driveItem'.
Uploading differences of .
Processing root
The directory has not changed
Uploading new items of .
Applying changes of Path ID: <omitted>

ERROR: OneDrive returned an error with the following message:
  Error Message: HTTP request returned status code 400 (Bad Request)
  Error Reason:  Unsupported request: Change tracking is not supported against 'microsoft.graph.driveItem'.

Also updated the branch to the latest, same issue.

config/onedrive/' --synchronize --verbose
Using 'user' Config Dir: /home/ <omitted>
Using 'system' Config Dir: 
Using config option for Azure AD for US Government Endpoints
Configuration file successfully loaded
Initializing the OneDrive API ...
Configuring Azure AD for US Government Endpoints
Opening the item database ...
All operations will be performed in: /home/ <omitted>
Application version: v2.4.2-29-g58423d8
Account Type: business
Default Drive ID: b! <omitted>
Default Root ID:  <omitted>
Remaining Free Space: 5488506826683
Fetching details for OneDrive Root
OneDrive Root exists in the database
Initializing the Synchronization Engine ...
Syncing changes from OneDrive ...
Applying changes of Path ID:  <omitted>

ERROR: OneDrive returned an error with the following message:
  Error Message: HTTP request returned status code 400 (Bad Request)
  Error Reason:  Unsupported request: Change tracking is not supported against 'microsoft.graph.driveItem'.
Uploading differences of .
Processing root
The directory has not changed
Uploading new items of .
Applying changes of Path ID:  <omitted>

ERROR: OneDrive returned an error with the following message:
  Error Message: HTTP request returned status code 400 (Bad Request)
  Error Reason:  Unsupported request: Change tracking is not supported against 'microsoft.graph.driveItem'.

@abraunegg
Copy link
Owner

abraunegg commented Jun 8, 2020

@spinkernel

Reviewing your documentation, It's not clear how you get to the deaktop and devices URIs, might want to put one more step in there to show. I had to "add" a service in order for it to show up.

If you can submit the required changes / any new images to update the documentation that would be greatly appreciated - as I have no way to validate this at all.

However .. this issue is going to be a bigger challenge:

Error Reason: Unsupported request: Change tracking is not supported against

Based on https://docs.microsoft.com/en-us/graph/deployments#supported-features despite the OneDrive graph feature being available (which I was hoping on being fully supported), the 'Delta query' looks like this has been 'disabled', thus the delta queries (basically tracking 'what changed' on OneDrive) is not possible - which is how the client works today.

What this means is, that when National Cloud Deployments is used, the query that tells the client what files are on OneDrive needs to be updated.

Question: How quickly are you able to respond to code changes / development updates? I am cognitive of TZ differences here, and because of clearance requirements - so no way to send data beyond what you have above. Basically - what I need to do is create a new function that looks at the children of the drive rather than the delta - then that 'may' work, but then run into another issue to solve. What are your thoughts here in assisting?

@spinkernel
Copy link
Author

spinkernel commented Jun 8, 2020

@abraunegg
Ha, my IT guy was like, "oh that explains some issues I was having with other SharePoint apps"

I'm happy to test new builds and so is he. I suspect these alternative methods could be tested with the global since Gov cloud is just an older version of global.

@abraunegg
Copy link
Owner

abraunegg commented Jun 9, 2020

@spinkernel
Please can you rebuild PR #938 to pick up the latest code changes.

I have tested this locally, futzing the queries to force to use the alternative method to generate compatible file list JSON .. so far so good & no issues.

The only concern however is - because there is no way to track changes with /delta - the only method is to walk all files and folders via query - which could either be time & processing sensitive.

In the current code, still to do:

  • API error handling (4xx, 5xx errors) Done
  • Retry Done
  • Most likely, in this scenario, drop the double full scan totally - as every sync it will do an entire walk so no need to do this twice every time

It would be great to get feedback on how this is performing for you as well - curious on how long it takes to generate Processing 63 OneDrive items to ensure consistent local state this for you ..

@abraunegg
Copy link
Owner

@spinkernel
Any update on your testing?

The latest version of the PR also now correctly handles seeking all files:

Adding 200 OneDrive items for processing
Adding 200 OneDrive items for processing
Adding 200 OneDrive items for processing
Adding 2 OneDrive items for processing
Adding 1 OneDrive items for processing
Adding 1 OneDrive items for processing
Adding 1 OneDrive items for processing
Processing 47122 OneDrive items to ensure consistent local state
Downloading file 50k_files/random_files/random_files_dir_set_1/8bIEx5iJTassf9QH9JShSjXuM5CwrGLz/file217.data ... done.
Downloading file 50k_files/random_files/random_files_dir_set_1/8bIEx5iJTassf9QH9JShSjXuM5CwrGLz/file218.data ... done.
Downloading file 50k_files/random_files/random_files_dir_set_1/8bIEx5iJTassf9QH9JShSjXuM5CwrGLz/file219.data ... done.
Downloading file 50k_files/random_files/random_files_dir_set_1/8bIEx5iJTassf9QH9JShSjXuM5CwrGLz/file22.data ... done.

So it would be great to get some reference from your experience.

Your version should be onedrive v2.4.2-35-g064c42d or greater (depending on when you read this)

@spinkernel
Copy link
Author

spinkernel commented Jun 9, 2020

Seems to be working....

 pr938  ./onedrive --confdir '~/.config/onedrive/' --synchronize --verbose
Using 'user' Config Dir: /home/<omitted>/.config/onedrive/
Using 'system' Config Dir: 
Using config option for Azure AD for US Government Endpoints
Configuration file successfully loaded
Initializing the OneDrive API ...
Configuring Azure AD for US Government Endpoints
Opening the item database ...
All operations will be performed in: /home/<omitted>
Application version: v2.4.2-34-g317287b
Account Type: business
Default Drive ID: <omitted>
Default Root ID: <omitted>
Remaining Free Space: 5488506439581
Fetching details for OneDrive Root
OneDrive Root exists in the database
Initializing the Synchronization Engine ...
Syncing changes from OneDrive ...
Applying changes of Path ID: <omitted>
Adding 71 OneDrive items for processing
Adding 13 OneDrive items for processing
Adding 1 OneDrive items for processing
Adding 1 OneDrive items for processing

@abraunegg
Copy link
Owner

@spinkernel
It would be worth rebuilding your client as v2.4.2-34-g317287b is < onedrive v2.4.2-35-g064c42d where all files are correctly scanned - thus you may 'miss' files.

I have also updated the Adding 71 OneDrive items for processing line to be more helpful - if you rebuild now the client version should be onedrive v2.4.2-36-g3a82c97

@spinkernel
Copy link
Author

Great, I just updated it after sending that. I'll update again. No method of downloading on demand?

How would sharepoint sites work? I don't see anything int he config to access them. Normally there's some url that gets opened with window's onedrive app in order to do the sync.

@abraunegg
Copy link
Owner

abraunegg commented Jun 9, 2020

@spinkernel

No method of downloading on demand

Not at the moment - see #757

How would sharepoint sites work?

Refer to https://github.com/abraunegg/onedrive/blob/master/docs/Office365.md

(That .md file really should be renamed)

The following I have confirmed as working:

  • Downloading / Uploading works without issue
  • Local deletes of files / folders are processed correctly
  • Local file updates / remove file updates are processed correctly

However this is not working right now:

  • The delete a file or folder on OneDrive (which, when using /delta provides what has been deleted) does not work - so the files & folders still reside locally, and are attempted to get uploaded again Done

@abraunegg
Copy link
Owner

@spinkernel
Please can you update your client to onedrive v2.4.2-38-g0e70a9a

This should now handle remote deletes from OneDrive - thus should now completely support National Cloud Deployments.

Please can you test this extensively with any usage scenarios you see yourself using to assist in shaking out any remaining issues.

@abraunegg
Copy link
Owner

@spinkernel
Any update in your testing?

@abraunegg abraunegg removed the In Progress Currently being worked on label Jun 12, 2020
@spinkernel
Copy link
Author

I've updated to the latest but I haven't done much testing, I had something come up that has been taking all my time. I'll play with it more this weekend.

@spinkernel
Copy link
Author

spinkernel commented Jun 14, 2020

@abraunegg
Playing around this weekend. I'm getting:

uploading new items of .
Uploading new file ./<omitted>.xlsx ... 
skipped.

ERROR: OneDrive returned an error with the following message:
  Error Message: HTTP request returned status code 400 (Bad Request)
  Error Reason:  A valid path must be provided.

Applying changes of Path ID: <omitted>

sometimes when updating.

Also seems like the way O365 vs Sharepoint Sites could be different? I don't have the "shared libraries" you talk about in your 365 instructions. On the windows client I would click a sync button on sharepoint site, and would get opened by the onedrive app:
image
image

would there be a way to create a separate config for each "site" to sync using the url they generate? Not sure how I can help troubleshoot that.

@spinkernel
Copy link
Author

spinkernel commented Jun 14, 2020

Also getting:

ERROR: OneDrive returned an error with the following message:
  Error Message: HTTP request returned status code 409 (Conflict)
  Error Reason:  Cannot create an upload session on a folder
[M] File changed: <omitted>.graphml
Uploading differences of <omitted>s.graphml
Uploading new items of <omitted>.graphml
Uploading new file <omitted>.graphml ... 
skipped.

When uploading new files to a folder.

@abraunegg
Copy link
Owner

@spinkernel
RE: HTTP request returned status code 400 (Bad Request) & HTTP request returned status code 409 (Conflict)

The 'only' way to debug this further would be to look at a verbose debug log which show out what is going on.

Have a look at this process: https://github.com/abraunegg/onedrive/wiki/Generate-debug-log-for-support

See if you can generate the required debug log, look at the information it contains - then have a chat with your security team as to if these details can be sent / shared. 90% of the data is internal 'what is the application doing', but there are details like drive ID and the like as well. You could 'redact' those, but then all the correlation mapping in the debug log is lost. You could 'mask / substitute' the sensitive details which would be better than pure 'redaction'

I am also OK with you contacting me off GitHub via email, and signing any sort of NDA or whatever with your organisation as well.

@abraunegg
Copy link
Owner

@spinkernel
Any update here on looking at the debug log?

Given the original issue of supporting the National Cloud instances is working, the PR as developed should be merged into master and any issues, like your seeing, tracked as separate items.

@spinkernel
Copy link
Author

I agree, it does work, we will need to track that sure. I have not had time to pull the log, i also have to get the OK from my IT guy to go forward. I can ping you directly. PM your direct contact so we can go from there.

@abraunegg
Copy link
Owner

@spinkernel
No way to PM directly off here. I can be reached at firstname dot lastname at google mail service or if you search LinkedIn via by day job in the financial sector.

@github-actions
Copy link

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 28, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Feature Request Feature Request | Enhancement Request Implemented
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants