Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge #2

Merged
merged 15 commits into from
Aug 7, 2020
Merged

Merge #2

merged 15 commits into from
Aug 7, 2020

Conversation

a9s2w5
Copy link
Owner

@a9s2w5 a9s2w5 commented Aug 7, 2020

What is the purpose of this change?

Was the change discussed in an issue or in the forum before?

Checklist

  • I have read the contribution guidelines.
  • I have added tests for all changes in this PR if appropriate.
  • I have added documentation for the changes if appropriate.
  • All commit messages are in house style.
  • I'm done, this Pull Request is ready for review :-)

tyhuber1 and others added 15 commits July 30, 2020 16:43
If this option is enabled, rclone will not set modtime of uploaded files and
the backend will return ModTimeNotSupported as its Precision.

Normally rclone updates modification time of files after they are done
uploading. This can cause permissions issues on Linux platforms when
rclone is copying to a CIFS mount where the user rclone is
running as does not own the file uploaded. If this option is enabled,
rclone will no longer update the modtime after copying a file.

See: https://forum.rclone.org/t/chtimes-error-on-local-mounted-copy/17784
- add a directory to the optional Purge interface
- fix up all the backends
- add an additional integration test to test for the feature
- use the new feature in operations.Purge

Many of the backends had been prepared in advance for this so the
change was trivial for them.
`rclone obscure` currently only accepts a command line argument of `password` to generate
an obfuscated password. This is an issue since generating obfuscated passwords programatically
requires sending the plain text password as a shell argument, which can cause problems if the
password contains shell characters, or if the password is from an untrusted source.

This patch opens up STDIN which will allow developers to open the STDIN source and print a password
directly to `rclone obscure`, which can increase safety and convenince.
Previous to this change rclone cached the looked up root_folder_id in
the root_folder_id config variable.

This has caused a lot of confusion and a few attempts at workarounds
and ultimately was a mistake.

This reverts rclone attempting to cache anything in root_folder_id and
returns that variable to be entirely user modified.

It gives a little hint in the debug that rclone could be sped up
slightly by setting it, but it is up to the user to think about
whether that would be OK or not.

    Google drive root '': root_folder_id = "XXX" - save this in the config to speed up startup

It does not change root_folder_id itself, leaving this to the user.

See: https://forum.rclone.org/t/rclone-gdrive-no-longer-returning-anything/17215
After uploading a multipart object, rclone deletes any unused parts.

Probably as part of the listing unification, the detection of the
parts beloning to the current upload was failing and calling Update
was deleting the parts for the current object.

This change fixes the detection and deletes all the old parts but none
of the new ones now.

Fixes #4075
The deadlock was caused in transfermap.go by calling mu.RLock() in one
function then calling it again in a sub function. Normally this is
fine, however this leaves a window where mu.Lock() can be called. When
mu.Lock() is called it doesn't allow the second mu.RLock() and
deadlocks.

    Thead 1                    Thread 2
    String():mu.RLock()
                               del():mu.Lock()
    sortedSlice():mu.RLock()                     - DEADLOCK

Lesson learnt: don't try using locks recursively ever!

This patch fixes the problem by removing the second mu.RLock(). This
was done by factoring the code that was calling it into the
transfermap.go file so all the locking can be seen at once which was
ultimately the cause of the problem - the code which used the locks
was too far away from the rest of the code using the lock.

This problem was introduced in:

bfa5715 fs/accounting: sort transfers by start time

Which hasn't been released in a stable version yet
This parameter causes extra read-ahead over --buffer-size which is not
buffered in memory but on disk.
Before this fix, download threads would fill up the buffer and then
timeout even though data was still being read from them. If the client
was streaming slower than network speed this caused the downloader to
stop and be restarted continuously. This caused more potential for
skips in the download and unecessary network transactions.

This patch fixes that behaviour - as long as a downloader is being
read from more often than once every 5 seconds, it won't timeout.

This was done by:

- kicking the downloader whenever ensureDownloader is called
- making the downloader loop if it has already downloaded past the maxOffset
- making setRange() always kick the downloader
1. adds SharedOptions data structure to oauthutil
2. adds config.ConfigToken option to oauthutil.SharedOptions
3. updates the backends that have oauth functionality

Fixes #2849
@a9s2w5 a9s2w5 merged commit 1af309a into a9s2w5:feat/sa-cycle Aug 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants