Renovate internal artifactory config

[ghost]

Hi,

I am configuring Renovate for my projects and trying to point so renovate could also look for dependencies in internal artifactory, but I see that there are some issues which I might suspect is due to my config for Renovate:

module.exports = {
    platform: 'gitlab',
    endpoint: 'https://gitlab.com/api/v4/',
    token: process.env.RENOVATE_TOKEN,
    gitAuthor: 'Renovate Bot',
    labels: ['renovate', 'dependencies', 'automated'],
    assignees: ['username'],
    logLevel: 'DEBUG',
    onboarding: true,
    onboardingConfig: {
        extends: ['config:base'],
    },
    repositories: [
        'username/repo'
    ],
    maven: {
        "enabled": true
    },
    hostRules: [{
        "platform": "maven",
        "endpoint": "https://artifactory.internal.net/java-release/",
        "username": process.env.ARTIFACTORY_USERNAME,
        "password": process.env.ARTIFACTORY_PASSWORD
    }],
    "packageRules": [{
        "managers": ["maven"],
        "registryUrls": ["https://artifactory.internal.net/java-release/"]
    }],
};

In logs I see that artifactory credentials are resolved correctly but it fails to download any dependencies:

DEBUG: Content is not found for Maven url: https://artifactory.internal.net/java-release/org/mockito/mockito-core/maven-metadata.xml (repository=username/repo)
DEBUG: Failed to look up dependency org.mockito:mockito-core (repository=username/repo, packageFile=pom.xml, dependency=org.mockito:mockito-core)

And I can open maven-metadata.xml and see that there are release versions and it's correct.
At the end of the Renovate log there is this message:

               {
                 "datasource": "maven",
                 "depName": "org.mockito:mockito-core",
                 "currentValue": "3.6.28",
                 "fileReplacePosition": 2938,
                 "registryUrls": ["https://repo.maven.apache.org/maven2"],
                 "depType": "test",
                 "groupName": "mockito-core.version",
                 "depIndex": 32,
                 "updates": [],
                 "warnings": [
                   {
                     "depName": "org.mockito:mockito-core",
                     "message": "Failed to look up dependency org.mockito:mockito-core"
                   }
                 ],
                 "fixedVersion": "3.6.28"
               },

Strangely registryUrl is not of my internal artifactory.
Is it a config issue or it is communication between Renovate and internal artifactory?

[viceice]

Your hoste rule is wrong, checkout the docs:

• https://docs.renovatebot.com/configuration-options/#hostrules
• https://docs.renovatebot.com/java/#maven

[ghost]

I have changed my config to the following:
config.js

module.exports = {
    platform: 'gitlab',
    endpoint: 'https://gitlab.com/api/v4/',
    token: process.env.RENOVATE_TOKEN,
    gitAuthor: 'Renovate Bot',
    labels: ['renovate', 'dependencies', 'automated'],
    assignees: ['username'],
    logLevel: 'DEBUG',
    onboarding: true,
    onboardingConfig: {
        extends: ['config:base'],
    },
    repositories: [
        'username/repo'
    ],
    hostRules: [{
        "baseUrl": "https://artifactory.internal.net/java-release/",
        "username": process.env.ARTIFACTORY_USERNAME,
        "password": process.env.ARTIFACTORY_PASSWORD
    }],
};

But still, my internal dependencies aren't downloaded.
Pipeline logs:

DEBUG: Looking up net.internal:service-application in repository https://artifactory.internal.net/java-release/ (repository=username/repo)
DEBUG: Content is not found for Maven url: https://artifactory.internal.net/java-release/net/internal/service-application/maven-metadata.xml (repository=username/repo)

maven-metadata.xml

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>net.internal</groupId>
  <artifactId>service-application</artifactId>
  <versioning>
    <latest>4.31</latest>
    <release>4.31</release>
    <versions>
      ...
      <version>4.30</version>
      <version>4.31</version>
    </versions>
    <lastUpdated>20200908074251</lastUpdated>
  </versioning>
</metadata>

"datasource": "maven",
depName": "net.internal:service-application",
currentValue": "4.31",
fileReplacePosition": 1212,
registryUrls": [
  "https://repo.maven.apache.org/maven2",
  "https://artifactory.internal.net/java-release",
  "https://maven.google.com/"
],
"groupName": "lib-service.version",
"depIndex": 28,
"updates": [],
"warnings": [
  {
    "depName": "net.internal:service-application",
    "message": "Failed to look up dependency net.internal:service-application"
   }
],
"fixedVersion": "4.31"
},

Lastly my .gitlab-ci.yml

image: renovate/renovate

services:
  - docker:20.10.1-dind

variables:
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""
  DOCKER_HOST: tcp://docker:2375

renovate:
  stage: deploy
  script:
    - renovate $RENOVATE_EXTRA_FLAGS

The interesting thing which I find in pipeline logs, maybe the culprit is that there is a timeout happening and artifactory is not reached:

       "hostStats": [
         "auth.docker.io, 1 request, 212ms request average, 0ms queue average",
         "gitlab.com, 7 requests, 600ms request average, 1ms queue average",
         "index.docker.io, 4 requests, 643ms request average, 0ms queue average",
         "repo.maven.apache.org, 2654 requests, 45ms request average, 0ms queue average"
       ],

@rarkins @viceice So even after fixing hostRules internal dependencies aren't picked up, I have double-checked artifactory credentials and they are correct.

[viceice]

for running on gitlab pipelin please checkout our official docu: https://gitlab.com/renovate-bot/renovate-runner/-/blob/master/README.md

[rarkins]

This is the function that should be getting called:

renovate/lib/datasource/maven/util.ts

Lines 57 to 97 in 74b7038

	export async function downloadHttpProtocol(
	pkgUrl: url.URL | string,
	hostType = id
	): Promise<string | null> {
	let raw: { body: string };
	try {
	const httpClient = httpByHostType(hostType);
	raw = await httpClient.get(pkgUrl.toString());
	return raw.body;
	} catch (err) {
	const failedUrl = pkgUrl.toString();
	if (err.message === HOST_DISABLED) {
	// istanbul ignore next
	logger.trace({ failedUrl }, 'Host disabled');
	} else if (isNotFoundError(err)) {
	logger.trace({ failedUrl }, `Url not found`);
	} else if (isHostError(err)) {
	// istanbul ignore next
	logger.debug({ failedUrl }, `Cannot connect to ${hostType} host`);
	} else if (isPermissionsIssue(err)) {
	logger.debug(
	{ failedUrl },
	'Dependency lookup unauthorized. Please add authentication with a hostRule'
	);
	} else if (isTemporalError(err)) {
	logger.debug({ failedUrl, err }, 'Temporary error');
	if (isMavenCentral(pkgUrl)) {
	throw new ExternalHostError(err);
	}
	} else if (isConnectionError(err)) {
	// istanbul ignore next
	logger.debug({ failedUrl }, 'Connection refused to maven registry');
	} else if (isUnsupportedHostError(err)) {
	// istanbul ignore next
	logger.debug({ failedUrl }, 'Unsupported host');
	} else {
	logger.info({ failedUrl, err }, 'Unknown error');
	}
	return null;
	}
	}

If you don't see any instances of failedUrl in the debug logs then try logging that repo with LOG_LEVEL=trace. I recommend you may want to have a test repo with only one dependency though so that your logs aren't too long.

[ghost]

@rarkins I have created a repo with one dependency and checked the pipeline log for those listed errors in debug log level, but there are none of them. Sadly when changing log level to trace, it starts flooding, and Gitlab pipelines have a log file size limit of 4MBs. I am thinking to run renovate from my local machine. Maybe that way I could see if there is an error within trace log level.

[ghost]

So I have managed to find root cause and fix it, the issue was that our Artifactory is not publicly accessible only within internal network. It can't be reached from public networks. You can close this thread it works as expected.