Coder.com•5mo ago

Envbuilder with Kubernetes: image cache not working

Layer caching appears to work fine. And I see both the image cache and layer caches in my DockerHub private registry (see screenshoot) I'm just using the example template in Coder called kubernetes-devcontainer https://github.com/coder/coder/blob/main/examples/templates/kubernetes-devcontainer/main.tf However, when starting a workspace, I get this:

Warning: Cached image not found.
on main.tf line 203, in resource "envbuilder_cached_image" "cached":
  203: resource "envbuilder_cached_image" "cached" {
Failed to find cached image in repository "<removed>/kaniko-cache". It will be rebuilt in the next apply. Error: get cached image: error probing build cache: uncached COPY command is not supported in cache probe mode

Warning: Cached image not found.
on main.tf line 203, in resource "envbuilder_cached_image" "cached":
  203: resource "envbuilder_cached_image" "cached" {
Failed to find cached image in repository "<removed>/kaniko-cache". It will be rebuilt in the next apply. Error: get cached image: error probing build cache: uncached COPY command is not supported in cache probe mode

And when I stop a workspace I get this:

Warning: Re-running cache probe due to previous miss.
on main.tf line 203, in resource "envbuilder_cached_image" "cached":
  203: resource "envbuilder_cached_image" "cached" {
The previous state specifies image == builder_image "ghcr.io/coder/envbuilder:latest", which indicates a previous cache miss.

Warning: Re-running cache probe due to previous miss.
on main.tf line 203, in resource "envbuilder_cached_image" "cached":
  203: resource "envbuilder_cached_image" "cached" {
The previous state specifies image == builder_image "ghcr.io/coder/envbuilder:latest", which indicates a previous cache miss.

I did ENVBUILDER_GET_CACHED_IMAGE=1 and the workspace logs says it sees the matched image in the registry, but the build goes into an endless loop which I get since the probe should be occurring in the build, and not with a manual set of the env var. Thanks team

GitHub

coder/examples/templates/kubernetes-devcontainer/main.tf at main ·...

Provision remote development environments via Terraform - coder/coder

7 Replies

Cian•5mo ago

Hey Mark, what repo are you using? I've found there can be issues with caching when using features.

Phorcys•5mo ago

(@sharkymark) EDIT: sorry for the failed ping to the other Mark

sharkymarkOP•5mo ago

@Cian @Phorcys Hi guys. I have a simple fork of the example repo https://github.com/sharkymark/envbuilder-starter-devcontainer with no features. I have a flask one as well. https://github.com/sharkymark/envb-flask-hw I've tried various envbuilder images, including the preview in dogfood's template with the same result. My K8s is on colima on a mac, but oddly I see the image and layers in my dockerhub cache repo as I said earlier and the GET env var saw the image. thanks

GitHub

GitHub - sharkymark/envbuilder-starter-devcontainer: A sample proje...

A sample project for getting started with devcontainer.json in envbuilder - sharkymark/envbuilder-starter-devcontainer

GitHub

GitHub - sharkymark/envb-flask-hw

Contribute to sharkymark/envb-flask-hw development by creating an account on GitHub.

0xg4b•5mo ago

I'm experiencing the same issue with a template based on the kubernetes-devcontainer sample. Layers are pushed as expected to cache_repo as well. When the workspace is restarted, the cached image is never found and a rebuild is always triggered:

Warning: Cached image not found.
on main.tf line 231, in resource "envbuilder_cached_image" "cached":
  231: resource "envbuilder_cached_image" "cached" {
Failed to find cached image in repository "ghcr.io/xxx/yyy". It will be rebuilt in the next apply. Error: get cached image: error probing build cache: uncached RUN command is not supported in cache probe mode

Warning: Cached image not found.
on main.tf line 231, in resource "envbuilder_cached_image" "cached":
  231: resource "envbuilder_cached_image" "cached" {
Failed to find cached image in repository "ghcr.io/xxx/yyy". It will be rebuilt in the next apply. Error: get cached image: error probing build cache: uncached RUN command is not supported in cache probe mode

I've removed devcontainers' features as it was reported that they're not cacheable:

TL;DR features are not cacheable at the moment.

Ref: https://github.com/coder/terraform-provider-envbuilder/issues/68#issuecomment-2557247792 The same issue still happens, though.

GitHub

RUN command is not supported in cache probe mode · Issue #68 · co...

I am using the envbuilder terraform provider to cache my workspace image in coder. Could somebody help me understand why I always see the following error which results in a cache miss? Failed to fi...

Phorcys•5mo ago

@0xg4b for the features side of things, yeah it's just not going to cache them, but not fail completely we are looking into this

0xg4b•5mo ago

message.txt

Cian•4mo ago

Hi, the behaviour you're seeing is expected if the envbuilder terraform provider is unable to recreate the build. This can happen for a number of reasons. It would be simplest to troubleshoot if you provided a minimal example and opened a bug in github.com/coder/terraform-provider-envbuilder But it's likely a similar issue as Mark's that was posted above -- I have seen instances where a build cannot be recreated even without features.

Gaming

Programming

Envbuilder with Kubernetes: image cache not working

Did you find this page helpful?