knife download sometimes fails with EEXIST

[jkeiser]

I suspect parallelism. Just let this method continue silently or something.

Created data_bags
Created data_bags/aws
Created data_bags/aws/rs-preprod.json
ERROR: Errno::EEXIST: File exists - /home/jkeiser/testit/data_bags

[yoniLavi]

I get this as well.
I'm using 11.6.0, and when running knife download '*' it keeps failing similarly and leaving empty directories. When I then delete the directory mentioned in the error and retry, it gets further. So it takes me several such iterations to eventually finish the download.

[jkeiser]

I believe this was just fixed. If you see this continue, please reopen!

[davekonopka]

I'm still running into this with knife-essentials (1.5.3) installed running knife download . and knife download \* in an otherwise empty repo folder. It's seemingly random as to which folder triggers it.

[jkeiser]

OK, I'll take a look today.

[davekonopka]

Here's one thing I just tested. Against a Chef 10.20.0 server knife download . works without issue in an empty directory.

Against an 11.6.0 server it dies. Same setup: empty repo directory with just a .chef/ folder containing knife.rb. The exception is actually consistent:

ERROR: Errno::EEXIST: File exists - /Users/davek/projects/backup-test/clientsCreated cookbooks

I think the variation I was seeing in what folder the exception would hit came from some earlier tries against a partially populated repo folder.

[jkeiser]

The variation is actually almost certainly more to do with parallelism
problems, so they will occur randomly.

On Tue, Aug 13, 2013 at 8:29 AM, Dave Konopka [email protected]:

Here's one thing I just tested. Against a Chef 10.20.0 server knife
download . works without issue in an empty directory.

Against an 11.6.0 server it dies. Same setup: empty repo directory with
just a .chef/ folder containing knife.rb. The exception is actually
consistent:

ERROR: Errno::EEXIST: File exists -
/Users/davek/projects/backup-test/clientsCreated cookbooks

I think the variation I was seeing in what folder the exception would hit
came from some earlier tries against a partially populated repo folder.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/98#issuecomment-22573290
.

[luisbosque]

This also happens to me.
Chef - 10.16.4
knife-essentials - 1.5.5

I'm using "versioned_cookbooks true"
The more cookbooks I have already downloaded the more frequently the error is raised. I'm at a point when I'm not able to download more cookbooks since the error is always raised in different cookbooks/versions.

[luisbosque]

Not sure if it's related to concurrency. I tried with --concurrency 1 and it also fails

[luisbosque]

Actually increasing (dramatically) the value of concurrency reduces the chance to get errors.

[jkeiser]

If it fails with concurrency 1, it's not concurrency at all. Would you
mind posting a log with -VV on the command line of it failing with
--concurrency 1? I can't be 100% sure I will know the error, but it could
well help.

On Mon, Dec 16, 2013 at 8:12 AM, Luis Bosque [email protected]:

Actually increasing (dramatically) the value of concurrency reduces the
chance to get errors.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/98#issuecomment-30673272
.

[lamont]

Here's my repeatable crash. It occurs with anything from concurrency 1 to 50, and if I just re-run it seems to get stuck at the exact same directory each time. if I rmdir the (sometimes empty) directory and re-run it goes for a few minutes before getting stuck on the next one.

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/file_system/file_system_entry.rb:53:in `mkdir': File exists - /home/users/me/transfer/cookbooks/cookbookWithLotsOfVersions-0.2.43/templates (Errno::EEXIST)
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/file_system/file_system_entry.rb:53:in `create_child'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/file_system.rb:313:in `copy_entries'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/file_system.rb:356:in `block in copy_entries'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/parallelizer.rb:94:in `call'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/parallelizer.rb:94:in `process_input'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/parallelizer.rb:69:in `each'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/file_system.rb:421:in `to_a'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/file_system.rb:421:in `parallel_do'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/file_system.rb:355:in `copy_entries'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/file_system.rb:356:in `block in copy_entries'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/parallelizer.rb:94:in `call'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/parallelizer.rb:94:in `process_input'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/parallelizer.rb:111:in `worker_loop'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/parallelizer.rb:23:in `block (2 levels) in initialize'