Deploys broken after updating to 3.12.0

Rack (Version 3)
1

Hello,

I’m running into an authorization error after updating to 3.12.0 with the new buildkit for image building. I had to re-add my Docker Hub registry, and attempted to re-add the AWS ECR registry, but I’m getting a 403 forbidden error.

$ convox build --id --app "myapp" --rack "production-v3"
Packaging source... OK
Uploading source... OK
Starting build... OK
time="2023-05-15T01:28:31Z" level=warning msg="local cache import at /var/lib/buildkit not found due to err: could not read /var/lib/buildkit/index.json: open /var/lib/buildkit/index.json: no such file or directory" spanID=575d8d3a48bdc8a7 traceID=048bec6d79cef5c5f60dd34c9b0fbf0a
#1 [internal] load build definition from Dockerfile.0
#1 transferring dockerfile: 111B done
#1 DONE 0.1s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.1s

#3 [auth] myapp/app:pull token for registry-1.docker.io
#3 DONE 0.0s

#4 [internal] load metadata for docker.io/myapp/app:c5b5cab13c8d421aec2460366bd7b48896a3da98
#4 DONE 0.3s

#5 [1/1] FROM docker.io/myapp/app:c5b5cab13c8d421aec2460366bd7b48896a3da98@sha256:cf02acf6136cb60b2e85531b4031d7d841124c924422840bf8866e80b75d9cf7
#5 resolve docker.io/myapp/app:c5b5cab13c8d421aec2460366bd7b48896a3da98@sha256:cf02acf6136cb60b2e85531b4031d7d841124c924422840bf8866e80b75d9cf7 done
#5 DONE 0.6s

#6 [auth] sharing credentials for 123412341234.dkr.ecr.us-east-1.amazonaws.com
#6 DONE 0.0s

#7 exporting to image
#7 exporting layers done
#7 exporting manifest sha256:ff755a695772b8ad54bb50bb0c51f944f2d191c4ce5e82ffbdcd97db0a731c8a 0.0s done
#7 exporting config sha256:3ca616149c892178e8228898e8b50bf2fdb379c05abc2c2776e942a63d122487 done
#7 pushing layers 0.0s done
#7 ERROR: unexpected status: 403 Forbidden
------
 > exporting to image:
------
error: failed to solve: unexpected status: 403 Forbidden
ERROR: exit status 1
ERROR: exit status 1
ERROR: build failed

I attempted to update my IAM user policy to allow more ECR actions:

        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": [
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage",
                "ecr:BatchCheckLayerAvailability",
                "ecr:InitiateLayerUpload",
                "ecr:CompleteLayerUpload",
                "ecr:GetDownloadUrlForLayer",
                "ecr:PutImage",
                "ecr:UploadLayerPart"
            ],
            "Resource": "arn:aws:ecr:us-east-1:123412341234:repository/myapp/convox"
        },

Is there something else I need to add? Or any way to get more detailed error logs?

Any help would be greatly appreciated!

2

@nathan.f77 can you share your convox.yaml file

3

@nathan.f77 also check the iam role in the aws account. it will be <rack-name>-api