Potential break in Rack v2 due to ECS API request validation improvements

Rack (Version 2)
21

Upgraded to 20211019100155 on both the rack and the CLI and still get 502

convox build
Packaging source... OK
Uploading source... OK
Starting build... ERROR: response status 502

Still seeing the same error:

2021-10-19T22:14:55Z service/web/b0d586c9-305c-4c0d-b26b-22ec7105540c id=cefe3391bff7 ns=api at=ReleaseGet method="GET" path="/apps/app-name/releases/RXSXTSIKTXY" response=200 elapsed=340.306
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/go/src/github.com/convox/rack/pkg/api/controllers.go:192 +0x23f
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 goroutine 1058 [running]:
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 net/http.(*conn).serve.func1(0xc0008325a0)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/usr/local/go/src/net/http/server.go:1769 +0x139
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 panic(0x1da1300, 0x3aa7290)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/usr/local/go/src/runtime/panic.go:522 +0x1b5
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 github.com/convox/rack/provider/aws.(*Provider).waitForTask(0xc000322240, 0xc001974190, 0x4c, 0x0, 0xc00043e748, 0xc00051dce0, 0x0)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/go/src/github.com/convox/rack/provider/aws/processes.go:1510 +0x12b
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 github.com/convox/rack/provider/aws.(*Provider).runBuild(0xc000322240, 0xc00058afc0, 0xc000985f80, 0x3a, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/go/src/github.com/convox/rack/provider/aws/builds.go:927 +0x1781
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 github.com/convox/rack/provider/aws.(*Provider).BuildCreate(0xc000322240, 0xc00004b92b, 0xa, 0xc000985f80, 0x3a, 0x0, 0x0, 0x0, 0x0, 0x6f523a, ...)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/go/src/github.com/convox/rack/provider/aws/builds.go:69 +0x5c7
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 github.com/convox/rack/pkg/api.(*Server).BuildCreate(0xc00079e1b0, 0xc0002320e0, 0x24, 0xc001719901)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 2021/10/19 22:14:58 http: panic serving 10.0.3.83:57530: runtime error: invalid memory address or nil pointer dereference
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 github.com/convox/rack/pkg/api.(*Server).authenticate.func1(0xc0002320e0, 0xc00047fcc0, 0xc000011c28)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/go/src/github.com/convox/rack/pkg/api/api.go:60 +0x17a
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 github.com/convox/rack/vendor/github.com/convox/stdapi.(*Router).handle(0xc00079e420, 0xc00047fcc0, 0xc0002320e0, 0x0, 0x0)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/go/src/github.com/convox/rack/vendor/github.com/convox/stdapi/router.go:158 +0x434
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 github.com/convox/rack/vendor/github.com/convox/stdapi.(*Router).http.func1(0x25198a0, 0xc0006a5a40, 0xc001357300)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/go/src/github.com/convox/rack/vendor/github.com/convox/stdapi/router.go:184 +0xf4
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 net/http.HandlerFunc.ServeHTTP(0xc000799380, 0x25198a0, 0xc0006a5a40, 0xc001357300)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/usr/local/go/src/net/http/server.go:1995 +0x44
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 github.com/convox/rack/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc0007fb320, 0x25198a0, 0xc0006a5a40, 0xc001357300)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/go/src/github.com/convox/rack/vendor/github.com/gorilla/mux/mux.go:150 +0x105
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 github.com/convox/rack/vendor/github.com/convox/stdapi.(*Server).ServeHTTP(0xc0002defc0, 0x25198a0, 0xc0006a5a40, 0xc001356e00)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/go/src/github.com/convox/rack/vendor/github.com/convox/stdapi/server.go:77 +0x50
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 net/http.serverHandler.ServeHTTP(0xc0002df008, 0x25198a0, 0xc0006a5a40, 0xc001356e00)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/usr/local/go/src/net/http/server.go:2774 +0xa8
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 net/http.(*conn).serve(0xc0008325a0, 0x252c4e0, 0xc00096fd40)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/usr/local/go/src/net/http/server.go:1878 +0x851
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 created by net/http.(*Server).Serve
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/usr/local/go/src/net/http/server.go:2884 +0x2f4
22

@scott Im not able to reproduce this. Is this happening on all your racks?

23

2 out of 4. The 2 that is happening for are both in the US region, and the 2 that are fine are in Canada and Australia (Syndey). A little bit before you asked me this question, I tried rolling my US test rack back to 20210822164712 and am running through a build right now. It appears to be working, which makes no sense since that release pre-dates the API change from AWS. The entire reason I found this thread and updated is because my Canadia/Australian racks were breaking from this error. Does anyone know if the US region just doesn’t have the ECS API update live and/or required yet?

24

Typically AWS would roll this out at the account level so it’s possible some accounts do and some don’t have the change within the same region. However our release is backwards compatible.

Based on the behaviors you are describing I think it might be unrelated to the ECS changes.

I would check the drift status of your cloud formation stacks as a first step.

25

If anyone else runs into this issue, it is because we had the settings under " Amazon ECS ARN and resource ID settings" in ECS Account Settings section set to disabled. My understanding was that these were supposed to be for an older migration/ECS change, but it looks like this also turns off the new ECS API changes and somehow broke rack builds with the new version. After changing these toggles to enabled (for the root account, changing when logging in as an IAM user is broken), builds started failing on the old version again and passing on the new version. So we are back to good again.