Problem with git-lfs-migrate and BitBucket 4.11 with LFS support

[mbrennan]

Hello,

I'm trying to use git-lfs-migrate to migrate an internal Subversion repository to Git for use with Atlassian BitBucket 4.11. This repository has a lot of binary files and I'd like to remove them from the git history using git-lfs-migrate. I'm using Windows 7, Java 1.8.0_91, Git LFS 1.5.5, git 2.10.0.windows.1.

In the commands and output below, I've changed the repository name to repo.git, the BitBucket project to 'project', the BitBucket server name to somebitbucketserver.net and the BitBucket IP address to 111.111.111.111 for illustration purposes.

When I issue any of the following commands:
java -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git --lfs https://somebitbucketserver.net/myproject/repo.git/info/lfs --no-check-certificate --glob-file binaries.txt
java -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git --lfs https://somebitbucketserver.net --no-check-certificate --glob-file binaries.txt
java -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git --lfs https://user:[email protected]/myproject/repo.git/info/lfs --no-check-certificate --glob-file binaries.txt
java -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git --lfs https://user:[email protected] --no-check-certificate --glob-file binaries.txt

I always get this error:

[main] INFO git.lfs.migrate.Main - LFS server: Batch API request exception
org.apache.http.conn.HttpHostConnectException: Connect to somebitbucketserver.net:443 [somebitbucketserver.net/111.111.111.111] failed: Connection refused: connect
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at ru.bozaro.gitlfs.client.internal.HttpClientExecutor.executeMethod(HttpClientExecutor.java:26)
at ru.bozaro.gitlfs.client.Client.doRequest(Client.java:275)
at ru.bozaro.gitlfs.client.Client.lambda$postBatch$7(Client.java:118)
at ru.bozaro.gitlfs.client.Client.doWork(Client.java:252)
at ru.bozaro.gitlfs.client.Client.postBatch(Client.java:118)
at git.lfs.migrate.Main.checkLfsAuthenticate(Main.java:113)
at git.lfs.migrate.Main.main(Main.java:66)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:337)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
... 17 more
[main] ERROR git.lfs.migrate.Main - LFS server: Invalid base URL

I suspect it has something to do with authentication because in order to view the repository via HTTPS in a browser, I have to authenticated using our third-party SSO service. However, repos are fully accessible via SSH. Which led me to try the following in a git bash session after I added my SSH key to ssh-agent:

java -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git -g ssh://[email protected]:7999/project/repo.git --glob-file binaries.txt --no-check-certificate
java -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git -g [email protected]:project/repo.git --glob-file binaries.txt --no-check-certificate
java -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git -g ssh://[email protected]:7999/project/repo.git --glob-file binaries.txt
java -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git -g [email protected]:project/repo.git --glob-file binaries.txt

These commands always gives this error:

[main] INFO git.lfs.migrate.Main - LFS server: Batch API request exception
java.io.IOException: Command returned with non-zero exit code 255: [ssh, [email protected]:7999, -oBatchMode=yes, -C, git-lfs-authenticate, project/repo.git, upload]
at ru.bozaro.gitlfs.client.auth.ExternalAuthProvider.getAuthUncached(ExternalAuthProvider.java:82)
at ru.bozaro.gitlfs.client.auth.CachedAuthProvider.getAuth(CachedAuthProvider.java:38)
at ru.bozaro.gitlfs.client.Client.doWork(Client.java:248)
at ru.bozaro.gitlfs.client.Client.postBatch(Client.java:118)
at git.lfs.migrate.Main.checkLfsAuthenticate(Main.java:113)
at git.lfs.migrate.Main.main(Main.java:66)
[main] ERROR git.lfs.migrate.Main - LFS server: Invalid base URL

I also tried:
java -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git --lfs ssh://[email protected]:7999/project/repo.git --glob-file binaries.txt --no-check-certificate

Which errors with:

Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (org.apache.http.conn.UnsupportedSchemeException) caught when processing request to {}->ssh://somebitbucketserver.net:7999: ssh protocol is not supported
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->ssh://somebitbucketserver.net:7999
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (org.apache.http.conn.UnsupportedSchemeException) caught when processing request to {}->ssh://somebitbucketserver.net:7999: ssh protocol is not supported
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->ssh://somebitbucketserver.net:7999
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (org.apache.http.conn.UnsupportedSchemeException) caught when processing request to {}->ssh://somebitbucketserver.net:7999: ssh protocol is not supported
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->ssh://somebitbucketserver.net:7999
[main] INFO git.lfs.migrate.Main - LFS server: Batch API request exception
org.apache.http.conn.UnsupportedSchemeException: ssh protocol is not supported
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:108)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at ru.bozaro.gitlfs.client.internal.HttpClientExecutor.executeMethod(HttpClientExecutor.java:26)
at ru.bozaro.gitlfs.client.Client.doRequest(Client.java:275)
at ru.bozaro.gitlfs.client.Client.lambda$postBatch$7(Client.java:118)
at ru.bozaro.gitlfs.client.Client.doWork(Client.java:252)
at ru.bozaro.gitlfs.client.Client.postBatch(Client.java:118)
at git.lfs.migrate.Main.checkLfsAuthenticate(Main.java:113)
at git.lfs.migrate.Main.main(Main.java:66)
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (org.apache.http.conn.UnsupportedSchemeException) caught when processing request to {}->ssh://somebitbucketserver.net:7999: ssh protocol is not supported
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->ssh://somebitbucketserver.net:7999
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (org.apache.http.conn.UnsupportedSchemeException) caught when processing request to {}->ssh://somebitbucketserver.net:7999: ssh protocol is not supported
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->ssh://somebitbucketserver.net:7999
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (org.apache.http.conn.UnsupportedSchemeException) caught when processing request to {}->ssh://somebitbucketserver.net:7999: ssh protocol is not supported
Feb 02, 2017 5:22:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->ssh://somebitbucketserver.net:7999
[main] ERROR git.lfs.migrate.Main - LFS server: Invalid base URL

Is there some way to get this working other than resorting to filter-branch or cluttering my repo with .gitattributes files from bfg --convert-to-git-lfs?

Thanks,
Mike

[leth]

As a workaround, you can run the "lfs-test-server" locally for the conversion; then use git config --local --add lfs.url URL, git lfs fetch --all and git lfs push -all to migrate the LFS objects to a new server.

[mbrennan]

Thanks @leth,

I didn't realize the source repository had to be bare, so I did:
git clone --mirror repo.git repo.bare.git

Then I managed to get git-lfs-migrate to run properly with lfs-test-server using the default options. This starts the LFS server on http://localhost:8080. This is the command I invoked:
java -jar git-lfs-migrate.jar -s repo.bare.git -d repo.lfs.git --lfs http://localhost:8080 --glob-file binaries.txt

Everything seemed to work fine. Then to test out the new LFS-based repo I did:
git clone repo.lfs.git repo.lfs.not_bare

When that command tries to checkout the HEAD of master, it fails with this output:

Cloning into 'repo.lfs.not_bare'...
done.
Downloading subdir/Resources/112_LeftArrowLong_Blue.ico (29.22 KB)
Error downloading object: subdir/Resources/112_LeftArrowLong_Blue.ico (d8e5f5b20eff4ab7e1023e33ca69cdbf9023288ef436769f37606e4d88ca9cee)

Errors logged to E:\repo.lfs.not_bare\.git\lfs\objects\logs\20170203T105759.3777981.log
Use git lfs logs last to view the log.
error: external filter git-lfs smudge -- %f failed 2
error: external filter git-lfs smudge -- %f failed
fatal: subdir/Resources/112_LeftArrowLong_Blue.ico: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'

That log file referenced shows this:

git-lfs/1.5.5 (GitHub; windows amd64; go 1.7.4; git c2dcd6f5)
git version 2.10.0.windows.1

$ git-lfs.exe smudge -- subdir/Resources/112_LeftArrowLong_Blue.ico
Error downloading object: subdir/Resources/112_LeftArrowLong_Blue.ico (d8e5f5b20eff4ab7e1023e33ca69cdbf9023288ef436769f37606e4d88ca9cee)

Smudge error: Error downloading d8e5f5b20eff4ab7e1023e33ca69cdbf9023288ef436769f37606e4d88ca9cee: http: Post e:///.%5Crepo.lfs.not_bare/info/lfs/objects/batch: unsupported protocol scheme "e": http: Post e:///.%5Crepo.lfs.not_bare/info/lfs/objects/batch: unsupported protocol scheme "e"
github.com/git-lfs/git-lfs/errors.newWrappedError
C:/Go/src/github.com/git-lfs/git-lfs/errors/types.go:166
github.com/git-lfs/git-lfs/errors.NewSmudgeError
C:/Go/src/github.com/git-lfs/git-lfs/errors/types.go:252
github.com/git-lfs/git-lfs/lfs.PointerSmudge
C:/Go/src/github.com/git-lfs/git-lfs/lfs/pointer_smudge.go:70
github.com/git-lfs/git-lfs/lfs.(*Pointer).Smudge
C:/Go/src/github.com/git-lfs/git-lfs/lfs/pointer.go:65
github.com/git-lfs/git-lfs/commands.smudge
C:/Go/src/github.com/git-lfs/git-lfs/commands/command_smudge.go:84
github.com/git-lfs/git-lfs/commands.smudgeCommand
C:/Go/src/github.com/git-lfs/git-lfs/commands/command_smudge.go:112
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).execute
C:/Go/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:477
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).Execute
C:/Go/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:551
github.com/git-lfs/git-lfs/commands.Run
C:/Go/src/github.com/git-lfs/git-lfs/commands/run.go:66
main.main
C:/Go/src/github.com/git-lfs/git-lfs/git-lfs.go:33
runtime.main
C:/Go/src/runtime/proc.go:183
runtime.goexit
C:/Go/src/runtime/asm_amd64.s:2086

It seems lfs-test-server isn't compatible with local directory-based repositories. Is this benign? I assumed it was and tried to do this:
cd repo.lfs.git
git lfs fetch --all
git config --local --add lfs.url [email protected]:project/repo.git
git remote add origin ssh://[email protected]:7999/project/repo.git
git lfs push origin --all

This errors with:

open subdir\Resources\112_LeftArrowLong_Blue.ico: The system cannot find the path specified.

Any other approaches anyone can think of?

[leth]

Git LFS urls are always HTTP; by default it will autodiscover the correct url. I'm only suggesting using the url override for the one-off transform, then pushing them across to their ideal server.

Here's roughly how I'm doing my migration, I'm doing all this on linux, so YMMV:

• start lfs-test-server in another directory

LFS_ADMINUSER=admin LFS_ADMINPASS=password ./lfs-test-server-linux-amd64/lfs-test-server > lfs-server.log &

• log in and create a user

wget -O - --post-data "name=user&password=password" http://admin:password@localhost:8080/mgmt/add

• migrate with --lfs http://user:password@localhost:8080/
• in newly-created lfs-ed repo create your remote git add remote my_remote ...
• set the url: git config --local --add lfs.url http://user:password@localhost:8080/
• run git lfs fetch --all my_remote
• remove the config option git config --local --unset lfs.url
• run git lfs push --all my_remote

Hope that helps!

[bozaro]

If you don't set --lfs and --git option, git-lfs-migrate should create lfs files locally in lfs/object subdirectory inside destination repository. Theoretically you can upload it later by git lfs push --all command (see https://help.github.com/articles/duplicating-a-repository/).
Also you can provide user/password in --lfs key for working with custom authorization.

[flipside]

finally got it working using this to convert and push a local repository! more or less what i did was:

git clone --mirror ./path/to/repo.git
java -jar git-lfs-migrate.jar \ -s repo.git \ -d repo-converted.git \ "*.psd"
cd repo-converted.git
git fsck
git add remote origin https://github.com/user/repo-converted.git
git push origin master or git push --all origin
git lfs push --all origin

[pwagland]

Should this issue be closed? Or is there still some work that needs to happen?

[svendhhh]

Not sure... If it IS all working, I think the instructions/tutorial on how to use git-lfs-migrate should be updated. I'm getting the same issue, and am having to read through this conversation to try to work out how to actually do the conversion. Would be cool if the instructions were a bit more detailed, to help getting the migration working :)

[svendhhh]

I tried to follow the steps given by flipside as what was working:

time java -jar git-lfs-migrate/git-lfs-migrate.jar -s repo.git -d repo-converted.git "*.png" "*.jpg"

This seems to run OK. As far as I can tell it's put the binaries into folder called "lfs" inside "lfs-converted-repo.git". Then I try:

cd repo-converted.git
git fsck
git add remote origin https://github.com/user/repo-converted.git

This last command fails, because repo-converted.git is a bare repository, and you have to run git add remote in a work tree. So I try to first clone it:

git clone repo-converted.git repo-converted-cloned

But this now fails with errors trying to download the tracked files:

$ git clone repo-converted.git repo-converted-cloned
Cloning into 'repo-converted-cloned'...
done.
Downloading android/DeviceUtilityBridgeAndroid/res/drawable-hdpi/ic_launcher.png (9.18 KB)
Error downloading object: android/DeviceUtilityBridgeAndroid/res/drawable-hdpi/ic_launcher.png (28c8991): Smudge error: Error downloading android/DeviceUtilityBridgeAndroid/res/drawable-hdpi/ic_launcher.png (28c89918a8d235fcadf2e83bb14a6d9da9eab9e665857ad43b9f00e038f1f615): batch response: Post p:///repo-converted.git/info/lfs/objects/batch: unsupported protocol scheme "p"

Apparently (git-lfs/git-lfs#1207) this is because Git LFS only works when downloading from a server (not from file)... so I guess I'd need to be able to do the push, but I can't do the add of the remote... catch 22? :P

I'm not running the lfs test server, but it didn't seem from that latest post that that was necessary?

[svendhhh]

I now see that the error I was getting was due to the remote command being wrong. It should be:

git remote add origin https://github.com/user/repo-converted.git

Otherwise it was trying to add some (non-existent) files "origin" and "http...", which isn't allowed when not on a work tree.

I then did git push --all origin which worked, but then the git lfs push --all origin complains that:

$ git lfs push --all origin
Error scanning for Git LFS files in the "origin/master" ref
Error: ref 'origin/master' is ambiguous

So, I've gotten a bit further, but not quite there yet :)

Maybe this is the problem:

$ git branch -a
* master
  origin/3dBranch
  origin/master
  remotes/origin/master
  remotes/origin/origin/3dBranch
  remotes/origin/origin/master

Not sure where the "origin/origin/.." branches came from, but might be the issue...

[svendhhh]

I think I've worked out a solution to the new problem. When I do the conversion I end up with branches master, origin/master and origin/3dBranch. I presume the master one has been added as a default when during the conversion, as it doesn't exist in the bare repo before. So I've renamed the branches and deleted the new one:

git branch -D master
git branch -m origin/master master 
git branch -m origin/3dBranch 3dBranch

After this the rest of flipside's solution works for me (with the "add" and "remote" swapped).