Coder.com•5mo ago

Has anyone included coder modules into renovate?

Hi there, I'm trying to include coder modules into my renovate bot, that should update the terraform and dockerfiles when a update is available. This is my current config:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:recommended"],
  "regexManagers": [
    {
      "fileMatch": ["^.*\\.tf$"],
      "matchStrings": [
        "install_version\\s*=\\s*\"(?<currentValue>[^\"]+)\""
      ],
      "depNameTemplate": "code-server",
      "datasourceTemplate": "npm",
      "versioningTemplate": "semver"
    }
  ],
  "packageRules": [
    {
      "matchDatasources": ["terraform-module"],
      "registryUrls": ["https://registry.coder.com"]
    }
  ]
}

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:recommended"],
  "regexManagers": [
    {
      "fileMatch": ["^.*\\.tf$"],
      "matchStrings": [
        "install_version\\s*=\\s*\"(?<currentValue>[^\"]+)\""
      ],
      "depNameTemplate": "code-server",
      "datasourceTemplate": "npm",
      "versioningTemplate": "semver"
    }
  ],
  "packageRules": [
    {
      "matchDatasources": ["terraform-module"],
      "registryUrls": ["https://registry.coder.com"]
    }
  ]
}

This covers the terraform providers, the code-server version in the code-server module and images in the dockerfile. But I can't seem to figure out, how to include the coder modules. There is a terraform-modules provider within renovate, but it seems, that renovate can't handle the coder registry entities. Would be amazing, if someone could show me how to fix this. Thanks!

26 Replies

ZokerOP•5mo ago

Is there any API returning the current version of the modules? Until now I could use the github coder/modules repo for releases, but with the new coder/registry this is not possible anymore (since there is no release in that repo) @Atif @Phorcys Any idea?

Phorcys•5mo ago

yeah you should be able to use the terraform registry api e.g https://registry.coder.com/terraform_protocol/modules/jetbrains-gateway/coder/versions the syntax is GET /terraform_protocol/{namespace}/{module}/coder/versions where namespace is modules/templates and module is the module name without the author in he URL e.g https://registry.coder.com/modules/nataindata/apache-airflow would become https://registry.coder.com/terraform_protocol/modules/apache-airflow/coder/versions the registry is closed source atm, but fyi we support all these routes

GET /.well-known/terraform.json
GET /terraform_protocol/{namespace}/{module}/coder/versions
GET /terraform_protocol/{namespace}/{module}/coder/{version}/download
GET /download/{namespace}/{versionModulePath}
GET /{namespace}/{name}

GET /.well-known/terraform.json
GET /terraform_protocol/{namespace}/{module}/coder/versions
GET /terraform_protocol/{namespace}/{module}/coder/{version}/download
GET /download/{namespace}/{versionModulePath}
GET /{namespace}/{name}

I think you might have to use the last item of the array or perform a semver sort though, it doesn't seem that there's a proper "get latest" endpoint which kinda sucks tbh

ZokerOP•5mo ago

Thank you very much, that helped. Just curios why the default terraform path does not work: https://developer.hashicorp.com/terraform/registry/api-docs#list-available-versions-for-a-specific-module According to this, the URL should be: <base_url>/:namespace/:name/:provider/versions Or is https://registry.coder.com/terraform_protocol/ the base url? That's maybe why the default renovate terraform config is not working

Phorcys•5mo ago

Or is https://registry.coder.com/terraform_protocol/ the base url

yes, see https://registry.coder.com/.well-known/terraform.json

ZokerOP•5mo ago

Hmm ok, that I would the default config to work

Phorcys•5mo ago

not sure whether it'll work because it seems we implement a very small subset of the registry api spec

ZokerOP•5mo ago

No idea, why it does not work currently

Phorcys•5mo ago

so maybe it calls for an endpoint we don't have

ZokerOP•5mo ago

Ah I see

Phorcys•5mo ago

could you get full logs somehow? or will it just tell you "failed to look up" and that's it?

ZokerOP•5mo ago

In the logs I only see:

 WARN: Package lookup failures (repository=images/custom-coder-images)
       "warnings": [
         "Failed to look up terraform-module package registry.coder.com/modules/code-server/coder",
         "Failed to look up terraform-module package registry.coder.com/modules/coder-login/coder",
         "Failed to look up terraform-module package registry.coder.com/modules/git-commit-signing/coder",
         "Failed to look up terraform-module package registry.coder.com/modules/git-config/coder",
         "Failed to look up terraform-module package registry.coder.com/modules/nodejs/coder",
         "Failed to look up terraform-module package registry.coder.com/modules/personalize/coder"
       ],
       "files": ["images/base/Base.tf", "images/salesforce/Salesforce.tf", "images/vnc/VNC.tf"]

 WARN: Package lookup failures (repository=images/custom-coder-images)
       "warnings": [
         "Failed to look up terraform-module package registry.coder.com/modules/code-server/coder",
         "Failed to look up terraform-module package registry.coder.com/modules/coder-login/coder",
         "Failed to look up terraform-module package registry.coder.com/modules/git-commit-signing/coder",
         "Failed to look up terraform-module package registry.coder.com/modules/git-config/coder",
         "Failed to look up terraform-module package registry.coder.com/modules/nodejs/coder",
         "Failed to look up terraform-module package registry.coder.com/modules/personalize/coder"
       ],
       "files": ["images/base/Base.tf", "images/salesforce/Salesforce.tf", "images/vnc/VNC.tf"]

Phorcys•5mo ago

is there any way to make it more verbose?

ZokerOP•5mo ago

I enabled the debug logs and it seems, that this is the issue:

TypeError: Cannot read properties of undefined (reading 'length') 
  at TerraformModuleDatasource.queryRegistryVersions (/usr/local/renovate/lib/modules/datasource/terraform-module/index.ts:158:23)
  at processTicksAndRejections (node:internal/process/task_queues:105:5)
  at TerraformModuleDatasource.getReleases (/usr/local/renovate/lib/modules/datasource/terraform-module/index.ts:78:12)
  at /usr/local/renovate/lib/util/cache/package/decorator.ts:131:20
  at getRegistryReleases (/usr/local/renovate/lib/modules/datasource/index.ts:86:15)
  at fetchReleases (/usr/local/renovate/lib/modules/datasource/index.ts:361:15)"

TypeError: Cannot read properties of undefined (reading 'length') 
  at TerraformModuleDatasource.queryRegistryVersions (/usr/local/renovate/lib/modules/datasource/terraform-module/index.ts:158:23)
  at processTicksAndRejections (node:internal/process/task_queues:105:5)
  at TerraformModuleDatasource.getReleases (/usr/local/renovate/lib/modules/datasource/terraform-module/index.ts:78:12)
  at /usr/local/renovate/lib/util/cache/package/decorator.ts:131:20
  at getRegistryReleases (/usr/local/renovate/lib/modules/datasource/index.ts:86:15)
  at fetchReleases (/usr/local/renovate/lib/modules/datasource/index.ts:361:15)"

https://github.com/renovatebot/renovate/blob/main/lib/modules/datasource/terraform-module/index.ts#L158

Phorcys•5mo ago

ah yeah so we're most definitely missing some kind of property i'll try to take a look this week or next week, though it is a more time consuming debugging process so i likely won't be very quick on it!

ZokerOP•5mo ago

Thank you very much! I will also try to do some debugging with the linked code Ok so after some investigation, I think I found two issues (at least with the renovate case). Here is a sample of that endpoint with a default terraform registry module: https://registry.terraform.io/v1/modules/terraform-aws-modules/iam/aws/versions First I noticed, that with the official registry the name is not Modules, but modules (lowercase). That's why when the renovate bot tries to get res.modules.length it's called on a undefined property. Additional the renovate code uses the source parameter (but it seems to be optional). Source contains again the path to the module, like this: terraform-aws-modules/iam/aws This could also be included into the registry entry. @Phorcys Is there any repo that contains your implementation of the registry? If yes I can provide a PR fixing these issues. If not: Would you be able to fix that? Thanks!

Phorcys•5mo ago

@Zoker thanks for the info!

Is there any repo that contains your implementation of the registry?

There is one, but it's closed source, i'm not going to ask you to contribute to proprietary code, i'll ask around internally to see why it isn't open source. I'll raise an issue about this internally and see if I can find the time to get it fixed but it'll likely be a task for one of our engineers

ZokerOP•5mo ago

Thank you very much!

matifali•5mo ago

Let's enable dependabot on the coder/coder repo to see if dependabot also fails cc: @Phorcys

Phorcys•5mo ago

@Zoker fyi Dependabot also fails, we have an internal issue about this, will keep you updated

ZokerOP•5mo ago

I think they both expect the Terraform default format Any news on resolve this issue? Thanks!

Kor3un•5mo ago

Phorcys•5mo ago

nothing yet, i'm subscribed to the issue, i've asked internally for an ETA quoting internal chats:

I think this is something we can look into before the next (Coder) release, [...] in July

ZokerOP•4mo ago

I just received a notification from the renovate bot, that there is an update for a module. It seems to work now 🎉 Thank you very much!

matifali•4mo ago

Wow Dependabot is still failing though. We did make some chnages but i think we need to investigate more for making dependabot happy.

Phorcys•4mo ago

nice!

Codercord•4mo ago

@Phorcys closed the thread.

Gaming

Programming

Has anyone included coder modules into renovate?

Did you find this page helpful?