High Level

I am trying to allow Renovate to manage Python dependency updates, including updates dependent upon our own internal registry, which lives in Google's Artifact Registry. We use GitHub to manage our repositories, and Github's provision of Renovate.

Details

To do this, I need to update the renovate.json5 file to allow access to the repo. There is scant information in the documentation, and I have looked in the following places:

and as far as I can tell, I am supposed to add something like this to get access to my private repo:

{
  "hostRules": [
    {
      "matchHost": "https://registry.company.com/pypi-simple/",
      "username": "engineering",
      "password": "abc123"
    }
  ]
}

It seems that Renovate can only use username + password for Python packages. However, I am using a service account, for which Google uses tokens (and other passwordless authentication means). For example, to access the repo, I add the following to pip.conf (or to [[tool.pdm.source]] in my pyproject.toml since I am using PDM to build):

[global]
extra-index-url = https://_json_key_base64:<long_string_for_private_info>@us-west1-python.pkg.dev/my-project/my-repo/simple/

In the case of Google / GCP, that <long_string_for_private_info> is actually a base 64 encoded JSON blob with a bunch of information:

> echo <long_string_for_private_info> | base64 --decode
{
  "type": "service_account",
  "project_id": "my-project",
  "private_key_id": "<filtered out for security>",
  "private_key": "-----BEGIN PRIVATE KEY-----\n<Filtered out for security>\n-----END PRIVATE KEY-----\n",
  "client_email": "my-service-account@my-project.iam.gserviceaccount.com",
  "client_id": "<filtered out for security>",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/my-service-account%40my-project.iam.gserviceaccount.com",
  "universe_domain": "googleapis.com"
}

The filtered out private key is also base64 encoded (so that there is a base64 encoded object that after decoding, it has another base64 encoded object in it).

Attempts so far

I have tried the following:

Using a token:

{
  "matchHost": "us-west1-python.pkg.dev",
  "hostType": "pypi",
  "encrypted": {
    "token": <hidden for security>
  }
}

Using a username and password:

{
  "matchHost": "us-west1-python.pkg.dev",
  "hostType": "pypi",
  "username": "_json_key_base64"
  "encrypted": {
    "token": <hidden for security>
  }
}

In each of the cases above (token or username/password), I have:

Always encoded using Renovate's encryption page
Tried the base64 encoded version of just the "private_key" mentioned above
Tried the decoded version of just the "private_key" mentioned above
Tried the base64 encoded entire "long_string_for_private_info" object mentioned in the extra-index-url in pip.conf.
- Remember that I am able to download (and upload) Python packages to the registry, so I know that pip.conf file is correct.
Tried the decoded version of the entire "long_string_for_private_info".

At this point, I have run out of ideas, and the documentation does not really explain how to use tokens with Python packages at all, let alone with Google's cloud infrastructure.

Error Reported

No matter how the secret is seen, the error is always the same, a variant of:

{
  "err": {
    "validationError": "Encrypted secret is scoped to a different repository: \"US-WEST1-PYTHON.PKG.DEV/MY-PROJ/MY-REPO\".",
    "message": "config-validation",
    "stack": "Error: config-validation\n    at tryDecrypt (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/config/decrypt.ts:124:31)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at decryptConfig (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/config/decrypt.ts:186:30)\n    at decryptConfig (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/config/decrypt.ts:238:13)\n    at mergeRenovateConfig (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/repository/init/merge.ts:252:27)\n    at getRepoConfig (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/repository/init/config.ts:12:12)\n    at initRepo (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/repository/init/index.ts:44:12)\n    at Object.renovateRepository (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/repository/index.ts:55:14)\n    at attributes.repository (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/global/index.ts:184:11)\n    at start (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/global/index.ts:169:7)\n    at /opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/renovate.ts:18:22"

.
.
.

{
  "error": {
    "validationError": "Failed to decrypt field token. Please re-encrypt and try again.",
    "message": "config-validation",
    "stack": "Error: config-validation\n    at decryptConfig (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/config/decrypt.ts:192:27)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at decryptConfig (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/config/decrypt.ts:238:13)\n    at mergeRenovateConfig (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/repository/init/merge.ts:252:27)\n    at getRepoConfig (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/repository/init/config.ts:12:12)\n    at initRepo (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/repository/init/index.ts:44:12)\n    at Object.renovateRepository (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/repository/index.ts:55:14)\n    at attributes.repository (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/global/index.ts:184:11)\n    at start (/opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/workers/global/index.ts:169:7)\n    at /opt/containerbase/tools/renovate/36.83.0/node_modules/renovate/lib/renovate.ts:18:22"
  }
}

I have even inspected those two functions on Renovate's github repo, but they are very "introspective" and javascript is not my best language, so I could not get too far.

from Renovate with Google Cloud, Github, Python project, and pyproject.toml

Hemant Vishwakarma

Monday, 11 September 2023

Renovate with Google Cloud, Github, Python project, and pyproject.toml