Convox Community

Potential break in Rack v2 due to ECS API request validation improvements

We also received the same notification and are on 20210319134118. Upgrading to latest v2 didn’t show any issue for us in the past.
By the way, how to know what is the latest v2 available?
And yes, @ddollar we really need to know if this impacts the latest v2 version.

Also received the same notice from AWS.
Will this be supported?

@ddollar any update?
@Brian-G ?

It appears these emails were sent to anyone making the ECS API calls mentioned regardless of wether the requests will pass the future validation checks. We are still in the process of reviewing our code to confirm but I don’t think this will have any impact on v2 racks.

1 Like

@eravelo Regarding what is the latest v2 available? This is what we do for before upgrading…

I may be conflating two different changes, but it seems like we did need to worry?

Since today we’re also experiencing the following issue:

convox ps --app <ourAPP> --rack <dev-rack>
ERROR: InvalidParameterException: Invalid identifier: Identifier is for cluster dev-rack-Cluster-123123. Your cluster is dev-rack-BuildCluster-456456
	status code: 400, request id: e2a2d3f3-fb70-4aa6-84c9-b60b392eab30

Same, we are experiencing similar issues on our V2 racks.

This change did break some of the ECS API calls. We did a release this week to mitigate the problem.

We have a test release out to address some further issues and should have it released publicly after further testing.

Any idea when [RELEASE] 20211015 by beastawakens · Pull Request #3476 · convox/rack · GitHub will be out? We aren’t able to do deploys on most of our racks at the moment.

1 Like

We are waiting for feedback from some testers before releasing. If you have a test rack you can try this on please give it a try and let us know how it goes.

convox rack update 20211015153539-20211015 -r dev-rack

Sweet thanks for letting me know. I’m upgrading our test rack now to see if it works.

After updating still getting 502s during build:

Status    running
Version   20211015153539-20211015
convox_release_id=$(convox build --app app-name --id) && echo "export CONVOX_RELEASE_ID=\"$convox_release_id\"" >> $BASH_ENV

Packaging source... OK
Uploading source... OK
Starting build... ERROR: response status 502

Exited with code exit status 1
CircleCI received exit code 1

Edit: My co-worker pointed out that the releases actually were created even though I got an immediate 502. Looks like the builds are sort of working now, just reporting back to command line that it’s not:
Builds:

BXGKRSSGGET  complete  RBRRZDDBTSX  28 minutes ago  2m57s               
BKGXFGFZYDB  complete  RCFHEXZLUDQ  56 minutes ago  10s                 
BIVJHNOLWGA  complete  RVWSLEFFZTY  1 hour ago      3m3s                
BJHIRVGDWBP  complete  RHFABUVRQAC  1 hour ago      2m56s               
BIZRCZETQSF  complete  RRMHGLHVZPO  3 hours ago     13m51s

Releases:

ID           STATUS  BUILD        CREATED         DESCRIPTION
RBRRZDDBTSX          BXGKRSSGGET  25 minutes ago            
RCFHEXZLUDQ          BKGXFGFZYDB  56 minutes ago             
RVWSLEFFZTY          BIVJHNOLWGA  57 minutes ago             
RHFABUVRQAC          BJHIRVGDWBP  1 hour ago                 
RRMHGLHVZPO          BIZRCZETQSF  3 hours ago

Thanks for the update. We will see if we can replicate this. Can you post the output of convox rack also?

You mean convox rack logs correct?

2021-10-18T21:26:22Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 ns=provider.aws at=SystemGet state=success elapsed=471.188
2021-10-18T21:26:22Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 id=b27cd803fe28 ns=api at=SystemGet method="GET" path="/system" response=200 elapsed=471.567
2021-10-18T21:26:24Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 ns=provider.aws at=stackResource stack=plic-apps resource=ApiBuildTasks state=success physical=arn:aws:ecs:us-east-1:awsid:task-definition/plic-apps-build:57 elapsed=0.011
2021-10-18T21:26:24Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 ns=provider.aws at=ObjectList prefix="system/registries/" state=success elapsed=49.396
2021-10-18T21:26:24Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 ns=provider.aws at=RegistryList state=success elapsed=49.428
2021-10-18T21:26:24Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 ns=provider.aws at=stackResource stack=plic-apps-app-name resource=Registry state=success physical=plic-regis-1hi9kzljylsnt elapsed=205.622
2021-10-18T21:26:26Z service/web/d392f3db-4b64-4827-9068-2df235bf43a6 id=43d8077a1a82 ns=api at=BuildGet method="GET" path="/apps/app-name/builds/BXHXZUZQELN" response=200 elapsed=35.861
2021-10-18T21:26:26Z service/web/d392f3db-4b64-4827-9068-2df235bf43a6 ns=provider.aws at=stackResource stack=plic-apps-app-name resource=Settings state=success physical=plic-apps-app-name-settings-718iz2hdtzee elapsed=0.010
2021-10-18T21:26:27Z service/web/d392f3db-4b64-4827-9068-2df235bf43a6 id=51e5c4ce27ef ns=api at=ObjectFetch method="GET" path="/apps/app-name/objects//tmp/06a12f61a693ea7eeff6da5c7de7ff.tgz" response=200 elapsed=542.406
2021-10-18T21:26:28Z service/web/d392f3db-4b64-4827-9068-2df235bf43a6 id=f1a457d28b60 ns=api at=BuildUpdate method="PUT" path="/apps/app-name/builds/BXHXZUZQELN" response=200 elapsed=62.169
2021-10-18T21:26:28Z service/web/d392f3db-4b64-4827-9068-2df235bf43a6 id=c79fddd869ab ns=api at=ReleaseList method="GET" path="/apps/app-name/releases" response=200 elapsed=8.113
2021-10-18T21:26:28Z service/web/d392f3db-4b64-4827-9068-2df235bf43a6 ns=provider.aws at=stackResource stack=plic-apps-app-name resource=Settings state=success physical=plic-apps-app-name-settings-718iz2hdtzee elapsed=198.540
2021-10-18T21:26:29Z service/web/d392f3db-4b64-4827-9068-2df235bf43a6 ns=provider.aws at=stackResource stack=plic-apps resource=EncryptionKey state=success physical=arn:aws:kms:us-east-1:awsid:key/be8c9b87-165d-43d1-8c86-5dc9bd60eb68 elapsed=235.900
2021-10-18T21:26:29Z service/web/d392f3db-4b64-4827-9068-2df235bf43a6 id=cf4e4c729428 ns=api at=ReleaseGet method="GET" path="/apps/app-name/releases/RPRDTFIDYYW" response=200 elapsed=498.982
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 2021/10/18 21:26:32 http: panic serving 10.0.2.204:17980: runtime error: invalid memory address or nil pointer dereference
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 goroutine 9316 [running]:
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 net/http.(*conn).serve.func1(0xc001000fa0)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/usr/local/go/src/net/http/server.go:1769 +0x139
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 panic(0x1da1300, 0x3aa7290)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/usr/local/go/src/runtime/panic.go:522 +0x1b5
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 github.com/convox/rack/provider/aws.(*Provider).waitForTask(0xc0003d9b00, 0xc00102b680, 0x4c, 0x0, 0xc0003781a8, 0xc000440620, 0x0)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/go/src/github.com/convox/rack/provider/aws/processes.go:1510 +0x12b
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 github.com/convox/rack/provider/aws.(*Provider).runBuild(0xc0003d9b00, 0xc003647880, 0xc003a82740, 0x3a, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/go/src/github.com/convox/rack/provider/aws/builds.go:927 +0x1781
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 github.com/convox/rack/provider/aws.(*Provider).BuildCreate(0xc0003d9b00, 0xc003a881bb, 0xa, 0xc003a82740, 0x3a, 0x0, 0x0, 0x0, 0x0, 0x6f523a, ...)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/go/src/github.com/convox/rack/provider/aws/builds.go:69 +0x5c7
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 github.com/convox/rack/pkg/api.(*Server).BuildCreate(0xc0004cee40, 0xc00025cbd0, 0x24, 0xc00075d901)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/go/src/github.com/convox/rack/pkg/api/controllers.go:192 +0x23f
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 github.com/convox/rack/pkg/api.(*Server).authenticate.func1(0xc00025cbd0, 0xc00078b180, 0xc0037e2690)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/go/src/github.com/convox/rack/pkg/api/api.go:60 +0x17a
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 github.com/convox/rack/vendor/github.com/convox/stdapi.(*Router).handle(0xc0004cf0b0, 0xc00078b180, 0xc00025cbd0, 0x0, 0x0)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/go/src/github.com/convox/rack/vendor/github.com/convox/stdapi/router.go:158 +0x434
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 github.com/convox/rack/vendor/github.com/convox/stdapi.(*Router).http.func1(0x25198a0, 0xc0039e8620, 0xc00312cc00)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/go/src/github.com/convox/rack/vendor/github.com/convox/stdapi/router.go:184 +0xf4
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 net/http.HandlerFunc.ServeHTTP(0xc00078f0e0, 0x25198a0, 0xc0039e8620, 0xc00312cc00)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/usr/local/go/src/net/http/server.go:1995 +0x44
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 github.com/convox/rack/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc00001cd20, 0x25198a0, 0xc0039e8620, 0xc00312cc00)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/go/src/github.com/convox/rack/vendor/github.com/gorilla/mux/mux.go:150 +0x105
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 github.com/convox/rack/vendor/github.com/convox/stdapi.(*Server).ServeHTTP(0xc0002c3b00, 0x25198a0, 0xc0039e8620, 0xc00312ca00)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/go/src/github.com/convox/rack/vendor/github.com/convox/stdapi/server.go:77 +0x50
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 net/http.serverHandler.ServeHTTP(0xc0002c3b48, 0x25198a0, 0xc0039e8620, 0xc00312ca00)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/usr/local/go/src/net/http/server.go:2774 +0xa8
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 net/http.(*conn).serve(0xc001000fa0, 0x252c4e0, 0xc0032f9a00)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/usr/local/go/src/net/http/server.go:1878 +0x851
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 created by net/http.(*Server).Serve
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/usr/local/go/src/net/http/server.go:2884 +0x2f4

PS I also tried updating the convox CLI to that version and that didn’t work either

We just did a public release on this. Do you mind updating and letting us know if this is still happening?

Upgraded to 20211019100155 on both the rack and the CLI and still get 502

convox build
Packaging source... OK
Uploading source... OK
Starting build... ERROR: response status 502

Still seeing the same error:

2021-10-19T22:14:55Z service/web/b0d586c9-305c-4c0d-b26b-22ec7105540c id=cefe3391bff7 ns=api at=ReleaseGet method="GET" path="/apps/app-name/releases/RXSXTSIKTXY" response=200 elapsed=340.306
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/go/src/github.com/convox/rack/pkg/api/controllers.go:192 +0x23f
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 goroutine 1058 [running]:
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 net/http.(*conn).serve.func1(0xc0008325a0)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/usr/local/go/src/net/http/server.go:1769 +0x139
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 panic(0x1da1300, 0x3aa7290)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/usr/local/go/src/runtime/panic.go:522 +0x1b5
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 github.com/convox/rack/provider/aws.(*Provider).waitForTask(0xc000322240, 0xc001974190, 0x4c, 0x0, 0xc00043e748, 0xc00051dce0, 0x0)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/go/src/github.com/convox/rack/provider/aws/processes.go:1510 +0x12b
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 github.com/convox/rack/provider/aws.(*Provider).runBuild(0xc000322240, 0xc00058afc0, 0xc000985f80, 0x3a, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/go/src/github.com/convox/rack/provider/aws/builds.go:927 +0x1781
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 github.com/convox/rack/provider/aws.(*Provider).BuildCreate(0xc000322240, 0xc00004b92b, 0xa, 0xc000985f80, 0x3a, 0x0, 0x0, 0x0, 0x0, 0x6f523a, ...)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/go/src/github.com/convox/rack/provider/aws/builds.go:69 +0x5c7
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 github.com/convox/rack/pkg/api.(*Server).BuildCreate(0xc00079e1b0, 0xc0002320e0, 0x24, 0xc001719901)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 2021/10/19 22:14:58 http: panic serving 10.0.3.83:57530: runtime error: invalid memory address or nil pointer dereference
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 github.com/convox/rack/pkg/api.(*Server).authenticate.func1(0xc0002320e0, 0xc00047fcc0, 0xc000011c28)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/go/src/github.com/convox/rack/pkg/api/api.go:60 +0x17a
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 github.com/convox/rack/vendor/github.com/convox/stdapi.(*Router).handle(0xc00079e420, 0xc00047fcc0, 0xc0002320e0, 0x0, 0x0)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/go/src/github.com/convox/rack/vendor/github.com/convox/stdapi/router.go:158 +0x434
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 github.com/convox/rack/vendor/github.com/convox/stdapi.(*Router).http.func1(0x25198a0, 0xc0006a5a40, 0xc001357300)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/go/src/github.com/convox/rack/vendor/github.com/convox/stdapi/router.go:184 +0xf4
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 net/http.HandlerFunc.ServeHTTP(0xc000799380, 0x25198a0, 0xc0006a5a40, 0xc001357300)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/usr/local/go/src/net/http/server.go:1995 +0x44
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 github.com/convox/rack/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc0007fb320, 0x25198a0, 0xc0006a5a40, 0xc001357300)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/go/src/github.com/convox/rack/vendor/github.com/gorilla/mux/mux.go:150 +0x105
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 github.com/convox/rack/vendor/github.com/convox/stdapi.(*Server).ServeHTTP(0xc0002defc0, 0x25198a0, 0xc0006a5a40, 0xc001356e00)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/go/src/github.com/convox/rack/vendor/github.com/convox/stdapi/server.go:77 +0x50
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 net/http.serverHandler.ServeHTTP(0xc0002df008, 0x25198a0, 0xc0006a5a40, 0xc001356e00)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/usr/local/go/src/net/http/server.go:2774 +0xa8
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 net/http.(*conn).serve(0xc0008325a0, 0x252c4e0, 0xc00096fd40)
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/usr/local/go/src/net/http/server.go:1878 +0x851
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 created by net/http.(*Server).Serve
2021-10-19T22:14:58Z service/web/311c5868-bad7-4e2a-97cb-ca2921893f42 	/usr/local/go/src/net/http/server.go:2884 +0x2f4

@scott Im not able to reproduce this. Is this happening on all your racks?

2 out of 4. The 2 that is happening for are both in the US region, and the 2 that are fine are in Canada and Australia (Syndey). A little bit before you asked me this question, I tried rolling my US test rack back to 20210822164712 and am running through a build right now. It appears to be working, which makes no sense since that release pre-dates the API change from AWS. The entire reason I found this thread and updated is because my Canadia/Australian racks were breaking from this error. Does anyone know if the US region just doesn’t have the ECS API update live and/or required yet?

Typically AWS would roll this out at the account level so it’s possible some accounts do and some don’t have the change within the same region. However our release is backwards compatible.

Based on the behaviors you are describing I think it might be unrelated to the ECS changes.

I would check the drift status of your cloud formation stacks as a first step.