Composer plugin is not working on NXRM 3.71 #157
Comments
|
I have the same problem |
|
Obviously. 3.71 dropped OrientDB support and this plugin is several versions behind, so quite a few transitions to new internal structures to be done.
I tried to get into it and start migration (see #158), but this will take some time and does not have any priority on my side. Current status: plugin starts, tables are created, hosted repository works mostly (cleanup and deletion to be done) |
|
Update from my side: Think I'm mostly done with the migration. As far as my tests tell me, the project is now on par with the 0.29 state.
I'd appreciate testing, if anyone is willing to do it. Can attach a snapshot JAR or KAR, if requested. |
|
Thank you |
|
This one is built from 527a512 using OpenJDK 17.0.12 Use at own risk 😉 Already mentioned, but like overread... It does not support migration from OrientDB to H2 (or at least I did not test it) |
composer hosted works fine. but composer proxy respond with 500 ERROR |
|
Thanks for testing! Good catch. Missed one leading slash, so fetching Fixed that one: [removed old version link, see below] |
Could you please provide the scripts for downloading and uploading the files? |
Thank you for fixing previous error. now it works but as you mentioned there is another problem with composer proxy. all requests respond with 404. |
|
also in previous stable version there is no leading slash for path
|
|
Yes, paths do require leading slashes for most internal APIs now. Will have a look into it and see at which point I have to remove it again. The proxy should also be a solvable problem, fetch the upstream link and store the cached asset. But likely not before the weekend. |
|
Got it. In both cases I just had to remove a leading [removed old version, see below] |
|
One think I thought about when looking into the plugin: Is the component structure really "good"? We have Maybe this is a good point in time to switch to NPM-style layout skipping the version path: |
|
Internal server errors produced when uploading different packages concurrent: Sometimes the error is triggered by I have no idea if this is an old existing bug or a new one. I have never uploaded so many packages as now. Thank you so much for working on this issues |
|
Interesting, did not experience that one, tried with ~300 artifacts several times both on a local container and a remote system. My scripts. Not very elegant, but straight forward and likely good enough for one-time use... backup-composer.sh#!/usr/bin/env bash
set -euo pipefail
REPO_URL="http://localhost:8081/repository/composer-hosted"
TARGET_DIR="composer"
# get package list
PACKAGE_NAMES=$(curl --fail -s "${REPO_URL}/packages/list.json" | jq -r '.packageNames[]')
# iterate over packages
for P in ${PACKAGE_NAMES}; do
VENDOR="${P%/*}"
PROJECT="${P#*/}"
# fetch package details
INFOS="$(curl -s "${REPO_URL}/p2/$VENDOR/$PROJECT.json" | jq -r ".packages.\"$VENDOR/$PROJECT\"[] | .version + \";\" + .dist.url")"
# create target directory structure and download blob
mkdir -p "${TARGET_DIR}/$VENDOR/$PROJECT"
echo "$VENDOR/$PROJECT"
# iterate over versions
for INFO in $INFOS; do
VERSION="${INFO%;*}"
DIST_URL="${INFO#*;}"
# download blob
curl --fail -s -o "${TARGET_DIR}/$VENDOR/$PROJECT/$VENDOR-$PROJECT-$VERSION.zip" "$DIST_URL"
echo " $VERSION"
done
donerestore-composer.sh#!/usr/bin/env bash
set -euo pipefail
# repo URL (add credentials like "https://user:[email protected]/..."
REPO_URL="http://localhost:8081/repository/composer-hosted"
# local directory with vendor/project/artifacts structure
SOURCE_DIR="composer"
# iterate over packages
for VD in $(find "${SOURCE_DIR}" -mindepth 1 -maxdepth 1 -type d); do
VENDOR="$(basename "$VD")"
for PD in $(find "$VD" -mindepth 1 -maxdepth 1 -type d); do
PROJECT="$(basename "$PD")"
echo "$VENDOR/$PROJECT"
for FILE in $(find "$PD" -mindepth 1 -maxdepth 1 -type f -name "*.zip"); do
VERSION="$(basename "$FILE")"
VERSION="${VERSION#"$VENDOR-$PROJECT-"}"
VERSION="${VERSION%.zip}"
# upload
curl --fail --show-error --upload-file "$FILE" "${REPO_URL}/packages/upload/$VENDOR/$PROJECT/$VERSION"
echo " $VERSION"
done
done
done |
|
Your scripts does not work concurrent. When I disable concurrency on my application it works like a charm. I have updated the restore script to work concurrent. The error only occurred when the packages not exists yet. I have tested it with only 54 packages per project. The errors should look like. restore.sh#!/bin/bash
set -euo pipefail
upload() {
curl -k --fail --show-error --upload-file "$1" "$2" &
}
# repo URL (add credentials like "https://user:[email protected]/..."
REPO_URL="http://localhost:8081/repository/composer-hosted"
# local directory with vendor/project/artifacts structure
SOURCE_DIR="composer"
for FILE in $(find "$SOURCE_DIR" -type f -name "*.zip"); do
RF=${FILE//$SOURCE_DIR\/}
VENDOR=${RF%%/*}
PROJECT="${RF//$VENDOR\/}"
PROJECT="${PROJECT%%/*}"
VERSION="$(basename "$FILE")"
VERSION="${VERSION#"$VENDOR-$PROJECT-"}"
VERSION="${VERSION%.zip}"
# upload
upload $FILE $REPO_URL/packages/upload/$VENDOR/$PROJECT/$VERSION &
done
wait |
Thank you, Composer proxy now works fine. I think npm style directory structure is better. |
|
Dear @stklcode Do you have a plan to change directory structure? or any other update? |
|
I'm currently looking into layout migration. I do have some other points in mind that would be nice to adjust, bug I don't want to get this too big at this point, so we actually have a chance to get the update released. Looking at NPM the node above the tarballs is actually a PACKAGE_ROOT node which is somewhat equivalent to our PACKAGE asset. With provider and package JSON it might also be a possibility to move these out of p/p2 and place them next to the ZIPBALLs or maybe (actual package/provider URLs have to remain at p/p2 of course) Layout could also be migrated in a following change with a corresponding migration step, it's just some URL replacement in the BrowseNode and/or Component paths. |
|
I also tried the following zip(kar) and got an error when restoring. However, the curl upload returns a 500 error, but the artifact exists in the repository. The following is logged in the nexus log, does this have anything to do with it? |
|
Hi @stklcode, Could you please send latest KAR bundle? |
|
This one's built from stklcode@23ddc10 Not much difference from the previous version, some minor fix and working integration tests. [removed old link, see new version below] |
Hi, this snapshot is terribly slow and has error |
|
Sorry, I messed up with the modified URL schema which should have been extracted to a different branch. Missing |
|
Apparently the code is okay and I just uploaded the wrong bundle. But anyway, I've added a unit test for this and ran a clean build. |
For which plugin/nexus version are these scripts? I still have < 3.71 with 0.0.29 for exporting, but the repository/composer-hosted/packages/list.json always returns a 404 for different (test) instances. I came to it by this answer #163 (comment), the problem of migration is still not really solved, right? |
Should work with any version as the public API has not changed. Tested backup with 3.70/0.0.29 and restore with 3.71/0.1.x (that’s obviously the use case)
Is the repository you want to backup actually named |
|
same i have, composer fully lost |
|
I used https://github.com/Col-E/Recaf to edit the nexus-db-migrator.jar a bit and just added composer in the com.sonatype.nexus.db.migrator.config. |
Could you clarify what exactly needs to be done in com.sonatype.nexus.db.migrator.config. ? |
|
@EvgenPavlyuchek public enum Format {
RAW(true),
MAVEN2(true),
NUGET(true),
HELM(true),
DOCKER(true),
PYPI(true),
YUM(true),
NPM(true),
APT(true),
CONDA(true),
CONAN(true),
R(true),
GITLFS(true),
COCOAPODS(true),
RUBYGEMS(true),
GO(true),
COMPOSER(true), // add this line to migrate Composer
P2(true);I then migrated the database as described in the official documentation. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I tried to use composer bundle/plugin 0.0.29 with nexus repository version 3.71 but i got the following error during nexus start.
so the plugin is not working
The text was updated successfully, but these errors were encountered: