Bulk upload issues #6990
Comments
|
Yeah, when uploading bulk of files, it wastes cpu and after a tiny upload goes rescan etc in a loop. With fiber network uploading like 1 GB of photos can take even an hour to complete. So solution still needed |
|
thanks for the help raising the importance of this |
Ok. Giving information. Latest NextCloud stable release. Latest Linux desktop client. Now situation is that to client side dropped 700 mb of files (approx 1-1.5 Mb per file, photos). So key scenario to test reasoanble size like 700 MB or above, while they should be small and a lot of them, like 100 files or above. Sync will be like dropping fast maybe 30-40 mb, then huuuge wait like few minutes, then again short time upload of 30-40 mb and so on in the loop till all uploaded. Expected behavious is if dropped 700mb of many files to upload, then upload them all non stop and only then resync. What I suspect as a reason of this performance bug, is that it starts upload, uploads tiny files like 40 mb in total and then server reports changes (those same files), upload gets held, resync and then proceed. It should throw all files in a bunch and only then resync something. Also a tray icon and clicking on it while such sync doesn't often show floating window of nextcloud desktop client, by which I assume that there is process violation as well, that sync or something being done in main process and blocking UI, while it should be a separate thread. I tried the best I could to explain. If any more info needed, do tell, I will try to assist. |
|
My Windows10 Nextcloud client is 3.13.2 (Stable). I have "'bulkupload.enabled' => false," in my server nextcloud.cfg (though I am not convinced this is effective). My Windows10 Nextcloud client settings shows 20TB in use, and this includes Sync Connections for AppData/Roaming. The important point being that files are changing constantly at the whim of Windows. Bulk transfer seems a good option IMHO, but there is a high likelyhood of file changes before the transfer is complete. Hence I am expecting all files uploaded to the server to retain the file modified date, as captured in the bulk, and a subsequent transfer to handle files changed since the previous bulk capture. I am getting "Error no app in context Computed md5 hash is incorrect" every 10 minutes or so, therefore I think I have a suitable environment for testing possible fixes/patches. Please advise if there is any patch you wish me to apply to verify possible solutions. |
|
another frustrated user here. One more catch for bulk uploading files. it is not even browser or OS issue. OneDrive used to be notoriously slow, but it got way faster now. Yet, many people still stick to Dropbox's paid option because of the performance. |
This comment has been minimized.
This comment has been minimized.
|
Having the same issue - added about 3000 files, but all totaling only 32MB. It uploads 100 files, then pauses for 30-60 seconds (no network activity). After 30-60 seconds, it instantly copies another 100 files and pauses again. I have tried changing the minChunkSize/maxChunkSize values in the desktop config.ini (even went so far as to set the minChunkSize value to 1). I disabled the redis container to rule that out (this is mostly just for me, so file locking isn't a big concern). I added some timeout settings to the fastcgi nginx proxy. I also tried a few other things, but I can't remember what they were at the moment. They didn't help either. The only thing that seems to help is adding |
|
Same issue on nextcloud-client 3.14.2, with a 7GB folder of photos. |
|
Same here. A few hundred new files (~10 GB). I killed nextcloud and restarted. After a few seconds everything froze, consuming ~75-80% CPU. The last loglines before the freeze (I anonymized the information and removed duplicates): And I just noticed (at least once) this error appearing in the console completely at nextcloud start (before complaining about config-folder migration):
Best Regards, |
|
How frustrating that this still exists years later. I'm moving from NC to NC AIO on a new server (latest client and AIO versions) and only remembered just now (after 2 days of frustration) that this was an issue in the past. I disabled bulk upload on the server and now my client is speeding through the files. |
|
I encountered the same issue, I had to disable bulk upload, this bug should get a high priority! Bug like this can make nextcloud unusable. |
|
I'll add +1 - same issue, same work around. It sounds like a tricky issue to resolve, its just in a production environment it is undermining confidence that all of your files are indeed synced on the server, because for me the desktop app shows green ticks across the board. So naturally you think everything is fine. When working with others sharing/syncing files, confidence in those green ticks erodes quickly with this issue. To help any who find this, save you some time researching and (as I am) are new to docker, here's a quick how to for this work around: The only thing that seems to help at this time is adding How to:
<?php
$CONFIG = array (
'one-click-instance' => true,
'one-click-instance.user-limit' => 100,
'memcache.local' => '\\OC\\Memcache\\APCu',
// lots of other existing configurations
);
<?php
$CONFIG = array (
'one-click-instance' => true,
'one-click-instance.user-limit' => 100,
'memcache.local' => '\\OC\\Memcache\\APCu',
'bulkupload.enabled' => false,
'apps_paths' =>
array (
// lots of other configs
);
Hope that helps make it easier to get this work around for those new to this. If I can help test any possible future fixes to this issue, let me know |
|
Hi, this is a major issue for me as well, especially when migrating all my files to Nextcloud. And sadly, running Nextcloud on TrueNAS prevents me from manually editing the |
thank you, it helped me on my windows client (3.14.3 and 3.15.3 versions) /config/www/nextcloud/config# nano config.php |
(WIP: will be updated as needed)
Summary: Indications are that there are one or more scenarios that trigger problems with bulk uploads, creating serious performance problems (e.g. 100% CPU usage, non-responsive). Many find success through workarounds like disabling bulk uploads or adjusting upload limits. Some reporters have noted that #5680 resolved their situation, while others it has not. There may either be additional triggers or simply multiple unrelated root causes.
Note: Many of these are likely duplicates (which is part of what needs to get sorted out)
Issues:
PRs:
The text was updated successfully, but these errors were encountered: