Potential break in Rack v2 due to ECS API request validation improvements

Rack (Version 2)
1

Hi there,

Got this email:

Your action is required to avoid potential service interruption once Amazon ECS API request validation improvements take effect on September 24, 2021. We have identified the following API requests to Amazon ECS from your account that could be impacted by these changes:
DescribeTasks

With these improvements, Amazon ECS APIs will validate that the Service and Cluster name parameters in the API match the Cluster and Service name in the ARN.

For Example:

Incorrect usage, will return an error:
aws ecs describe-tasks --cluster one --tasks arn:aws:ecs:us-east-1:123456789012:task/two/48d8e8a0ccf54d969a05099a269b7528

Correct usage, clusters match:
aws ecs describe-tasks --cluster one --tasks arn:aws:ecs:us-east-1:123456789012:task/one/48d8e8a0ccf54d969a05099a269b7528

The following APIs will include request validation improvements for the following parameters:

Cluster Name Consistency Validation Only:
DeregisterContainerInstance, RegisterContainerInstance, UpdateContainerAgent, DeleteAttributes, DescribeContainerInstances, DescribeTasks, ExecuteCommand, ListTasks, PutAttributes, StartTask, StopTask, UpdateContainerInstancesState, CreateTaskSet, DeleteService, DescribeServices, UpdateService

Cluster and Service Name Consistency Validation:
DeleteTaskSet, DescribeTaskSets, UpdateServicePrimaryTaskSet, UpdateTaskSet

To avoid any service interruption from upcoming Amazon ECS API request validation improvements, please update the API requests identified above to ensure that cluster and service parameters in the API request match those in the ARN. Please refer to this page for details on these APIs. [1]

If you have any questions, please contact AWS Premium Support[2].

[1] Actions - Amazon Elastic Container Service
[2] https://aws.amazon.com/support

Will this affect Rack v2? We’re still using 20200529011310 and are a bit wary about upgrading to latest Rack v2 version, as even with the latest version we still might run into issues.
Is it time to migrate to Rack v3?

3 Likes
2

We received the same notification and are on 20210701111905, so it seems to still impact later rack releases.

I’m also curious if there’s any action needed on our part.

3

@ddollar I have the same question. Does this impact current rack versions?

@alon - As a side-note, in May, I did get assurances that v2 rack support will continue for some time and that there’s no urgency/need to upgrade to v3. This spring we upgraded our v2 racks from 20191216210003 to 20210521142829 without any issues.

1 Like
4

@nolson Thanks, I appreciate you letting me know about this. Good to know. I guess we won’t go v3 just yet, then, and I’ll be more optimistic about upgrading to the latest v2. :slight_smile:

5

We also received the same notification and are on 20210319134118. Upgrading to latest v2 didn’t show any issue for us in the past.
By the way, how to know what is the latest v2 available?
And yes, @ddollar we really need to know if this impacts the latest v2 version.

6

Also received the same notice from AWS.
Will this be supported?

7

@ddollar any update?
@Brian-G ?

8

It appears these emails were sent to anyone making the ECS API calls mentioned regardless of wether the requests will pass the future validation checks. We are still in the process of reviewing our code to confirm but I don’t think this will have any impact on v2 racks.

1 Like
9

@eravelo Regarding what is the latest v2 available? This is what we do for before upgrading…

10

I may be conflating two different changes, but it seems like we did need to worry?

11

Since today we’re also experiencing the following issue:

convox ps --app <ourAPP> --rack <dev-rack>
ERROR: InvalidParameterException: Invalid identifier: Identifier is for cluster dev-rack-Cluster-123123. Your cluster is dev-rack-BuildCluster-456456
	status code: 400, request id: e2a2d3f3-fb70-4aa6-84c9-b60b392eab30
12

Same, we are experiencing similar issues on our V2 racks.

13

This change did break some of the ECS API calls. We did a release this week to mitigate the problem.

We have a test release out to address some further issues and should have it released publicly after further testing.

14

Any idea when [RELEASE] 20211015 by beastawakens · Pull Request #3476 · convox/rack · GitHub will be out? We aren’t able to do deploys on most of our racks at the moment.

1 Like
15

We are waiting for feedback from some testers before releasing. If you have a test rack you can try this on please give it a try and let us know how it goes.

convox rack update 20211015153539-20211015 -r dev-rack

16

Sweet thanks for letting me know. I’m upgrading our test rack now to see if it works.

17

After updating still getting 502s during build:

Status    running
Version   20211015153539-20211015

convox_release_id=$(convox build --app app-name --id) && echo "export CONVOX_RELEASE_ID=\"$convox_release_id\"" >> $BASH_ENV

Packaging source... OK
Uploading source... OK
Starting build... ERROR: response status 502

Exited with code exit status 1
CircleCI received exit code 1

Edit: My co-worker pointed out that the releases actually were created even though I got an immediate 502. Looks like the builds are sort of working now, just reporting back to command line that it’s not:
Builds:

BXGKRSSGGET  complete  RBRRZDDBTSX  28 minutes ago  2m57s               
BKGXFGFZYDB  complete  RCFHEXZLUDQ  56 minutes ago  10s                 
BIVJHNOLWGA  complete  RVWSLEFFZTY  1 hour ago      3m3s                
BJHIRVGDWBP  complete  RHFABUVRQAC  1 hour ago      2m56s               
BIZRCZETQSF  complete  RRMHGLHVZPO  3 hours ago     13m51s

Releases:

ID           STATUS  BUILD        CREATED         DESCRIPTION
RBRRZDDBTSX          BXGKRSSGGET  25 minutes ago            
RCFHEXZLUDQ          BKGXFGFZYDB  56 minutes ago             
RVWSLEFFZTY          BIVJHNOLWGA  57 minutes ago             
RHFABUVRQAC          BJHIRVGDWBP  1 hour ago                 
RRMHGLHVZPO          BIZRCZETQSF  3 hours ago
18

Thanks for the update. We will see if we can replicate this. Can you post the output of convox rack also?

19

You mean convox rack logs correct?

2021-10-18T21:26:22Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 ns=provider.aws at=SystemGet state=success elapsed=471.188
2021-10-18T21:26:22Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 id=b27cd803fe28 ns=api at=SystemGet method="GET" path="/system" response=200 elapsed=471.567
2021-10-18T21:26:24Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 ns=provider.aws at=stackResource stack=plic-apps resource=ApiBuildTasks state=success physical=arn:aws:ecs:us-east-1:awsid:task-definition/plic-apps-build:57 elapsed=0.011
2021-10-18T21:26:24Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 ns=provider.aws at=ObjectList prefix="system/registries/" state=success elapsed=49.396
2021-10-18T21:26:24Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 ns=provider.aws at=RegistryList state=success elapsed=49.428
2021-10-18T21:26:24Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 ns=provider.aws at=stackResource stack=plic-apps-app-name resource=Registry state=success physical=plic-regis-1hi9kzljylsnt elapsed=205.622
2021-10-18T21:26:26Z service/web/d392f3db-4b64-4827-9068-2df235bf43a6 id=43d8077a1a82 ns=api at=BuildGet method="GET" path="/apps/app-name/builds/BXHXZUZQELN" response=200 elapsed=35.861
2021-10-18T21:26:26Z service/web/d392f3db-4b64-4827-9068-2df235bf43a6 ns=provider.aws at=stackResource stack=plic-apps-app-name resource=Settings state=success physical=plic-apps-app-name-settings-718iz2hdtzee elapsed=0.010
2021-10-18T21:26:27Z service/web/d392f3db-4b64-4827-9068-2df235bf43a6 id=51e5c4ce27ef ns=api at=ObjectFetch method="GET" path="/apps/app-name/objects//tmp/06a12f61a693ea7eeff6da5c7de7ff.tgz" response=200 elapsed=542.406
2021-10-18T21:26:28Z service/web/d392f3db-4b64-4827-9068-2df235bf43a6 id=f1a457d28b60 ns=api at=BuildUpdate method="PUT" path="/apps/app-name/builds/BXHXZUZQELN" response=200 elapsed=62.169
2021-10-18T21:26:28Z service/web/d392f3db-4b64-4827-9068-2df235bf43a6 id=c79fddd869ab ns=api at=ReleaseList method="GET" path="/apps/app-name/releases" response=200 elapsed=8.113
2021-10-18T21:26:28Z service/web/d392f3db-4b64-4827-9068-2df235bf43a6 ns=provider.aws at=stackResource stack=plic-apps-app-name resource=Settings state=success physical=plic-apps-app-name-settings-718iz2hdtzee elapsed=198.540
2021-10-18T21:26:29Z service/web/d392f3db-4b64-4827-9068-2df235bf43a6 ns=provider.aws at=stackResource stack=plic-apps resource=EncryptionKey state=success physical=arn:aws:kms:us-east-1:awsid:key/be8c9b87-165d-43d1-8c86-5dc9bd60eb68 elapsed=235.900
2021-10-18T21:26:29Z service/web/d392f3db-4b64-4827-9068-2df235bf43a6 id=cf4e4c729428 ns=api at=ReleaseGet method="GET" path="/apps/app-name/releases/RPRDTFIDYYW" response=200 elapsed=498.982
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 2021/10/18 21:26:32 http: panic serving 10.0.2.204:17980: runtime error: invalid memory address or nil pointer dereference
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 goroutine 9316 [running]:
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 net/http.(*conn).serve.func1(0xc001000fa0)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/usr/local/go/src/net/http/server.go:1769 +0x139
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 panic(0x1da1300, 0x3aa7290)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/usr/local/go/src/runtime/panic.go:522 +0x1b5
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 github.com/convox/rack/provider/aws.(*Provider).waitForTask(0xc0003d9b00, 0xc00102b680, 0x4c, 0x0, 0xc0003781a8, 0xc000440620, 0x0)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/go/src/github.com/convox/rack/provider/aws/processes.go:1510 +0x12b
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 github.com/convox/rack/provider/aws.(*Provider).runBuild(0xc0003d9b00, 0xc003647880, 0xc003a82740, 0x3a, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/go/src/github.com/convox/rack/provider/aws/builds.go:927 +0x1781
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 github.com/convox/rack/provider/aws.(*Provider).BuildCreate(0xc0003d9b00, 0xc003a881bb, 0xa, 0xc003a82740, 0x3a, 0x0, 0x0, 0x0, 0x0, 0x6f523a, ...)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/go/src/github.com/convox/rack/provider/aws/builds.go:69 +0x5c7
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 github.com/convox/rack/pkg/api.(*Server).BuildCreate(0xc0004cee40, 0xc00025cbd0, 0x24, 0xc00075d901)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/go/src/github.com/convox/rack/pkg/api/controllers.go:192 +0x23f
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 github.com/convox/rack/pkg/api.(*Server).authenticate.func1(0xc00025cbd0, 0xc00078b180, 0xc0037e2690)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/go/src/github.com/convox/rack/pkg/api/api.go:60 +0x17a
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 github.com/convox/rack/vendor/github.com/convox/stdapi.(*Router).handle(0xc0004cf0b0, 0xc00078b180, 0xc00025cbd0, 0x0, 0x0)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/go/src/github.com/convox/rack/vendor/github.com/convox/stdapi/router.go:158 +0x434
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 github.com/convox/rack/vendor/github.com/convox/stdapi.(*Router).http.func1(0x25198a0, 0xc0039e8620, 0xc00312cc00)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/go/src/github.com/convox/rack/vendor/github.com/convox/stdapi/router.go:184 +0xf4
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 net/http.HandlerFunc.ServeHTTP(0xc00078f0e0, 0x25198a0, 0xc0039e8620, 0xc00312cc00)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/usr/local/go/src/net/http/server.go:1995 +0x44
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 github.com/convox/rack/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc00001cd20, 0x25198a0, 0xc0039e8620, 0xc00312cc00)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/go/src/github.com/convox/rack/vendor/github.com/gorilla/mux/mux.go:150 +0x105
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 github.com/convox/rack/vendor/github.com/convox/stdapi.(*Server).ServeHTTP(0xc0002c3b00, 0x25198a0, 0xc0039e8620, 0xc00312ca00)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/go/src/github.com/convox/rack/vendor/github.com/convox/stdapi/server.go:77 +0x50
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 net/http.serverHandler.ServeHTTP(0xc0002c3b48, 0x25198a0, 0xc0039e8620, 0xc00312ca00)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/usr/local/go/src/net/http/server.go:2774 +0xa8
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 net/http.(*conn).serve(0xc001000fa0, 0x252c4e0, 0xc0032f9a00)
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/usr/local/go/src/net/http/server.go:1878 +0x851
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 created by net/http.(*Server).Serve
2021-10-18T21:26:32Z service/web/67403fb2-286c-4cdc-8cb3-77d628ce6c68 	/usr/local/go/src/net/http/server.go:2884 +0x2f4

PS I also tried updating the convox CLI to that version and that didn’t work either

20

We just did a public release on this. Do you mind updating and letting us know if this is still happening?