Implement support for National cloud deployments (Issue #937) (#938)

* Implement support for National cloud deployments as per https://docs.microsoft.com/en-us/graph/deployments

README.md

@@ -18,10 +18,11 @@ This client is a 'fork' of the [skilion](https://github.com/skilion/onedrive) cl
 *   Resumable uploads
 *   Support OneDrive for Business (part of Office 365)
 *   Shared folders (OneDrive Personal)
-*   SharePoint / Office 365 Shared Libraries (refer to [docs/Office365.md](https://github.com/abraunegg/onedrive/blob/master/docs/Office365.md) to configure)
+*   SharePoint / Office 365 Shared Libraries
 *   Desktop notifications via libnotify
 *   Dry-run capability to test configuration changes
 *   Prevent major OneDrive accidental data deletion after configuration change
+*   Support for National cloud deployments (Microsoft Cloud for US Government, Microsoft Cloud Germany, Azure and Office 365 operated by 21Vianet in China)
 
 ## What's missing
 *   While local changes are uploaded right away, remote changes are delayed until next sync when using --monitor
@@ -36,9 +37,12 @@ See [docs/USAGE.md](https://github.com/abraunegg/onedrive/blob/master/docs/USAGE
 ## Docker support
 See [docs/Docker.md](https://github.com/abraunegg/onedrive/blob/master/docs/Docker.md)
 
-## Sharepoint group drive in Office 365 business or education
+## SharePoint / Office 365 Shared Libraries (Business or Education)
 See [docs/Office365.md](https://github.com/abraunegg/onedrive/blob/master/docs/Office365.md)
 
+## National Cloud support
+See [docs/national-cloud-deployments.md](https://github.com/abraunegg/onedrive/blob/master/docs/national-cloud-deployments.md)
+
 ## Reporting issues
 If you encounter any bugs you can report them here on Github. Before filing an issue be sure to:
 

docs/Office365.md

@@ -15,7 +15,7 @@ URL:      <your site URL>
 ```
 
 ## Configuring the onedrive client
-Once you have obtained the 'drive_id' above, add to your 'onedrive' configuration file (`~/.config/onedrive/config`)the following:
+Once you have obtained the 'drive_id' above, add to your 'onedrive' configuration file (`~/.config/onedrive/config`) the following:
 ```text
 drive_id = "insert the drive id from above here"
 ```

docs/national-cloud-deployments.md

@@ -0,0 +1,113 @@
+# How to configure access to specific Microsoft Azure deployments
+In some cases it is a requirement to utilise specific Microsoft Azure cloud deployments to conform with data and security reuqirements that requires data to reside within the geographic borders of that country.
+Current national clouds that are supported are:
+*   Microsoft Cloud for US Government
+*   Microsoft Cloud Germany
+*   Azure and Office 365 operated by 21Vianet in China
+
+In order to sucessfully use these specific Microsoft Azure deployments, the following steps are required:
+1. Register an application with the Microsoft identity platform using the Azure portal
+2. Configure the new application with the appropriate authentication scopes
+3. Validate that the authentication / redirect URI is correct for your application registration
+4. Configure the onedrive client to use the new application id as provided during application registration
+5. Configure the onedrive client to use the right Microsoft Azure deployment region that your application was registered with
+6. Authenticate the client
+
+## Step 1: Register a new application with Microsoft Azure
+1. Log into [Microsoft Azure](https://portal.azure.com/) with your applicable identity
+2. Select 'Azure Active Directory' as the service you wish to configure
+3. Under 'Manage', select 'App registrations' to register a new application
+4. Click 'New registration'
+5. Type in the appropriate details required as per below:
+
+![application_registration](./images/application_registration.jpg)
+
+6. To save the application registration, click 'Register' and something similar to the following will be displayed:
+
+![application_registration_done](./images/application_registration_done.jpg)
+
+**Note:** The Application (client) ID UUID as displayed after client registration, is what is required as the 'application_id' for Step 4 below.
+
+## Step 2: Configure application authentication scopes
+Configure the API permissions as per the following:
+
+| API / Permissions name | Type | Description | Admin consent required |
+|---|---|---|---|
+| Files.ReadWrite | Delegated | Have full access to user files | No |
+| Files.ReadWrite.All  | Delegated | Have full access to all files user can access | No |
+| offline_access   | Delegated | Maintain access to data you have given it access to | No |
+| Sites.Read.All   | Delegated | Read items in all site collections | No |
+| Sites.ReadWrite.All   | Delegated | Edit or delete items in all site collections | No |
+
+![authentication_scopes](./images/authentication_scopes.jpg)
+
+## Step 3: Validate that the authentication / redirect URI is correct
+Add the appropriate redirect URI for your Azure deployment:
+
+![authentication_response_uri](./images/authentication_response_uri.jpg)
+
+A valid entry for the response URI should be one of:
+*   https://login.microsoftonline.us/common/oauth2/nativeclient (Microsoft Cloud for US Government)
+*   https://login.microsoftonline.de/common/oauth2/nativeclient (Microsoft Cloud Germany)
+*   https://login.chinacloudapi.cn/common/oauth2/nativeclient (Azure and Office 365 operated by 21Vianet in China)
+
+## Step 4: Configure the onedrive client to use new application registration
+Update to your 'onedrive' configuration file (`~/.config/onedrive/config`) the following:
+```text
+application_id = "insert valid entry here"
+```
+
+This will reconfigure the client to use the new application registration you have created.
+
+**Example:**
+```text
+application_id = "22c49a0d-d21c-4792-aed1-8f163c982546"
+```
+
+## Step 5: Confgure the onedrive client to use the specific Microsoft Azure deployment
+Update to your 'onedrive' configuration file (`~/.config/onedrive/config`) the following:
+```text
+azure_ad_endpoint = "insert valid entry here"
+```
+
+Valid entries are:
+*   USL4 (Microsoft Cloud for US Government)
+*   USL5 (Microsoft Cloud for US Government - DOD)
+*   DE (Microsoft Cloud Germany)
+*   CN (Azure and Office 365 operated by 21Vianet in China)
+
+This will configure your client to use the correct Azure AD and Graph endpoints as per [https://docs.microsoft.com/en-us/graph/deployments](https://docs.microsoft.com/en-us/graph/deployments)
+
+**Example:**
+```text
+azure_ad_endpoint = "USL4"
+```
+
+## Step 6: Authenticate the client
+Run the application without any additional command switches.
+
+You will be asked to open a specific URL by using your web browser where you will have to login into your Microsoft Account and give the application the permission to access your files. After giving permission to the application, you will be redirected to a blank page. Copy the URI of the blank page into the application.
+```text
+[user@hostname ~]$ onedrive 
+
+Authorize this app visiting:
+
+https://.....
+
+Enter the response uri: 
+
+```
+
+**Example:**
+```
+[user@hostname ~]$ onedrive 
+Authorize this app visiting:
+
+https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id=22c49a0d-d21c-4792-aed1-8f163c982546&scope=Files.ReadWrite%20Files.ReadWrite.all%20Sites.ReadWrite.All%20offline_access&response_type=code&redirect_uri=https://login.microsoftonline.com/common/oauth2/nativeclient
+
+Enter the response uri: https://login.microsoftonline.com/common/oauth2/nativeclient?code=<redacted>
+
+Application has been successfully authorised, however no additional command switches were provided.
+
+Please use --help for further assistance in regards to running this application.
+```

src/config.d

@@ -84,6 +84,22 @@ final class Config
 		// Ignore data safety checks and overwrite local data rather than preserve & rename
 		// This is a config file option ONLY
 		boolValues["bypass_data_preservation"] = false;
+		// Support National Azure AD endpoints as per https://docs.microsoft.com/en-us/graph/deployments
+		// By default, if empty, use standard Azure AD URL's
+		// Will support the following options:
+		// - USL4
+		//     AD Endpoint:    https://login.microsoftonline.us
+		//     Graph Endpoint: https://graph.microsoft.us
+		// - USL5
+		//     AD Endpoint:    https://login.microsoftonline.us
+		//     Graph Endpoint: https://dod-graph.microsoft.us
+		// - DE
+		//     AD Endpoint:    https://portal.microsoftazure.de
+		//     Graph Endpoint: 	https://graph.microsoft.de
+		// - CN
+		//     AD Endpoint:    https://login.chinacloudapi.cn
+		//     Graph Endpoint: 	https://microsoftgraph.chinacloudapi.cn
+		stringValues["azure_ad_endpoint"] = "";
 
 		// DEVELOPER OPTIONS 
 		// display_memory = true | false
@@ -512,6 +528,30 @@ final class Config
 						if (key == "sync_dir") configFileSyncDir = c.front.dup;
 						if (key == "skip_file") configFileSkipFile = c.front.dup;
 						if (key == "skip_dir") configFileSkipDir = c.front.dup;
+						// Azure AD Configuration
+						if (key == "azure_ad_endpoint") {
+							string azureConfigValue = c.front.dup;
+							switch(azureConfigValue) {
+								case "":
+									log.log("Using config option for Global Azure AD Endpoints");
+									break;
+								case "USL4":
+									log.log("Using config option for Azure AD for US Government Endpoints");
+									break;
+								case "USL5":
+									log.log("Using config option for Azure AD for US Government Endpoints (DOD)");
+									break;
+								case "DE":
+									log.log("Using config option for Azure AD Germany");
+									break;
+								case "CN":
+									log.log("Using config option for Azure AD China operated by 21Vianet");
+									break;
+								// Default - all other entries 
+								default:
+									log.log("Unknown Azure AD Endpoint - using Global Azure AD Endpoints");
+							}
+						}
 					} else {
 						auto ppp = key in longValues;
 						if (ppp) {

src/itemdb.d

@@ -26,12 +26,13 @@ struct Item {
 	string   quickXorHash;
 	string   remoteDriveId;
 	string   remoteId;
+	string   syncStatus;
 }
 
 final class ItemDatabase
 {
 	// increment this for every change in the db schema
-	immutable int itemDatabaseVersion = 9;
+	immutable int itemDatabaseVersion = 10;
 
 	Database db;
 	string insertItemStmt;
@@ -83,12 +84,12 @@ final class ItemDatabase
 		db.exec("PRAGMA auto_vacuum = FULL");
 
 		insertItemStmt = "
-			INSERT OR REPLACE INTO item (driveId, id, name, type, eTag, cTag, mtime, parentId, crc32Hash, sha1Hash, quickXorHash, remoteDriveId, remoteId)
-			VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13)
+			INSERT OR REPLACE INTO item (driveId, id, name, type, eTag, cTag, mtime, parentId, crc32Hash, sha1Hash, quickXorHash, remoteDriveId, remoteId, syncStatus)
+			VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14)
 		";
 		updateItemStmt = "
 			UPDATE item
-			SET name = ?3, type = ?4, eTag = ?5, cTag = ?6, mtime = ?7, parentId = ?8, crc32Hash = ?9, sha1Hash = ?10, quickXorHash = ?11, remoteDriveId = ?12, remoteId = ?13
+			SET name = ?3, type = ?4, eTag = ?5, cTag = ?6, mtime = ?7, parentId = ?8, crc32Hash = ?9, sha1Hash = ?10, quickXorHash = ?11, remoteDriveId = ?12, remoteId = ?13, syncStatus = ?14
 			WHERE driveId = ?1 AND id = ?2
 		";
 		selectItemByIdStmt = "
@@ -117,6 +118,7 @@ final class ItemDatabase
 				remoteDriveId    TEXT,
 				remoteId         TEXT,
 				deltaLink        TEXT,
+				syncStatus       TEXT,
 				PRIMARY KEY (driveId, id),
 				FOREIGN KEY (driveId, parentId)
 				REFERENCES item (driveId, id)
@@ -300,13 +302,14 @@ final class ItemDatabase
 			bind(11, quickXorHash);
 			bind(12, remoteDriveId);
 			bind(13, remoteId);
+			bind(14, syncStatus);
 		}
 	}
 
 	private Item buildItem(Statement.Result result)
 	{
 		assert(!result.empty, "The result must not be empty");
-		assert(result.front.length == 14, "The result must have 14 columns");
+		assert(result.front.length == 15, "The result must have 15 columns");
 		Item item = {
 			driveId: result.front[0].dup,
 			id: result.front[1].dup,
@@ -319,7 +322,8 @@ final class ItemDatabase
 			sha1Hash: result.front[9].dup,
 			quickXorHash: result.front[10].dup,
 			remoteDriveId: result.front[11].dup,
-			remoteId: result.front[12].dup
+			remoteId: result.front[12].dup,
+			syncStatus: result.front[14].dup
 		};
 		switch (result.front[3]) {
 			case "file":    item.type = ItemType.file;    break;
@@ -417,4 +421,28 @@ final class ItemDatabase
 		stmt.bind(3, deltaLink);
 		stmt.exec();
 	}
+
+	// National Cloud Deployments (US and DE) do not support /delta as a query
+	// We need to track in the database that this item is in sync
+	// As we query /children to get all children from OneDrive, update anything in the database 
+	// to be flagged as not-in-sync, thus, we can use that flag to determing what was previously
+	// in-sync, but now deleted on OneDrive
+	void downgradeSyncStatusFlag()
+	{
+		db.exec("UPDATE item SET syncStatus = 'N'");
+	}
+
+	// National Cloud Deployments (US and DE) do not support /delta as a query
+	// Select items that have a out-of-sync flag set
+	Item[] selectOutOfSyncItems()
+	{
+		Item[] items;
+		auto stmt = db.prepare("SELECT * FROM item WHERE syncStatus = 'N'");
+		auto res = stmt.exec();
+		while (!res.empty) {
+			items ~= buildItem(res);
+			res.step();
+		}
+		return items;
+	}
 }

src/main.d

@@ -681,6 +681,15 @@ int main(string[] args)
 		sync.setBypassDataPreservation();
 	}
 
+	// Are we configured to use a National Cloud Deployment
+	if (cfg.getValueString("azure_ad_endpoint") != "") {
+		// value is configured, is it a valid value?
+		if ((cfg.getValueString("azure_ad_endpoint") == "USL4") || (cfg.getValueString("azure_ad_endpoint") == "USL5") || (cfg.getValueString("azure_ad_endpoint") == "DE") || (cfg.getValueString("azure_ad_endpoint") == "CN")) {
+			// valid entries to flag we are using a National Cloud Deployment
+			sync.setNationalCloudDeployment();
+		}
+	}
+
 	// Do we need to validate the syncDir to check for the presence of a '.nosync' file
 	if (cfg.getValueBool("check_nomount")) {
 		// we were asked to check the mounts