Having an issue with deployment of a new app.
During the deploy 3 out of 4 services are getting stuck at UPDATE_COMPLETE_CLEANUP_IN_PROGRESS in the CloudFormation console. From the deploy output (below) they do hit UPDATE_COMPLETE, but only 1 of them (happens to be the frontend API) shows UPDATE_COMPLETE in the console. The app itself remains as UPDATE_IN_PROGRESS (20mins in).
The services are all responding to health checks fine. I can’t find anywhere in AWS that shows what its doing right now. CloudTrail is giving me a 500, but can’t recall if that is meant to have anything in.
Any clues?
Promoting RENGWRJZWTX...
2019-04-25T18:19:02Z system/aws/cfm gazs-gazette UPDATE_IN_PROGRESS gazs-gazette User Initiated
2019-04-25T18:19:12Z system/aws/cfm gazs-gazette UPDATE_IN_PROGRESS ServiceLedger
2019-04-25T18:19:12Z system/aws/cfm gazs-gazette CREATE_IN_PROGRESS BalancerApiCertificate
2019-04-25T18:19:13Z system/aws/cfm gazs-gazette CREATE_IN_PROGRESS BalancerApiCertificate Resource creation Initiated
2019-04-25T18:19:13Z system/aws/cfm gazs-gazette-ServiceLedger-UX66QQQL9DUE UPDATE_IN_PROGRESS gazs-gazette-ServiceLedger-UX66QQQL9DUE User Initiated
2019-04-25T18:19:13Z system/aws/cfm gazs-gazette UPDATE_IN_PROGRESS ServiceAccounts
2019-04-25T18:19:14Z system/aws/cfm gazs-gazette UPDATE_IN_PROGRESS ServiceContent
2019-04-25T18:19:14Z system/aws/cfm gazs-gazette-ServiceAccounts-17B13FV7N7NDH UPDATE_IN_PROGRESS gazs-gazette-ServiceAccounts-17B13FV7N7NDH User Initiated
2019-04-25T18:19:14Z system/aws/cfm gazs-gazette-ServiceContent-FB1XN0SV01MG UPDATE_IN_PROGRESS gazs-gazette-ServiceContent-FB1XN0SV01MG User Initiated
2019-04-25T18:19:23Z system/aws/cfm gazs-gazette-ServiceLedger-UX66QQQL9DUE UPDATE_IN_PROGRESS Tasks Requested update requires the creation of a new physical resource; hence creating one.
2019-04-25T18:19:24Z system/aws/cfm gazs-gazette-ServiceLedger-UX66QQQL9DUE UPDATE_IN_PROGRESS Tasks Resource creation Initiated
2019-04-25T18:19:24Z system/aws/cfm gazs-gazette-ServiceLedger-UX66QQQL9DUE UPDATE_COMPLETE Tasks
2019-04-25T18:19:25Z system/aws/cfm gazs-gazette-ServiceAccounts-17B13FV7N7NDH UPDATE_IN_PROGRESS Tasks Requested update requires the creation of a new physical resource; hence creating one.
2019-04-25T18:19:26Z system/aws/cfm gazs-gazette-ServiceAccounts-17B13FV7N7NDH UPDATE_IN_PROGRESS Tasks Resource creation Initiated
2019-04-25T18:19:26Z system/aws/cfm gazs-gazette-ServiceContent-FB1XN0SV01MG UPDATE_IN_PROGRESS Tasks Requested update requires the creation of a new physical resource; hence creating one.
2019-04-25T18:19:26Z system/aws/cfm gazs-gazette-ServiceAccounts-17B13FV7N7NDH UPDATE_COMPLETE Tasks
2019-04-25T18:19:26Z system/aws/cfm gazs-gazette-ServiceContent-FB1XN0SV01MG UPDATE_IN_PROGRESS Tasks Resource creation Initiated
2019-04-25T18:19:26Z system/aws/cfm gazs-gazette-ServiceContent-FB1XN0SV01MG UPDATE_COMPLETE Tasks
2019-04-25T18:19:27Z system/aws/cfm gazs-gazette-ServiceLedger-UX66QQQL9DUE UPDATE_IN_PROGRESS Service
2019-04-25T18:19:30Z system/aws/cfm gazs-gazette-ServiceContent-FB1XN0SV01MG UPDATE_IN_PROGRESS Service
2019-04-25T18:19:31Z system/aws/cfm gazs-gazette-ServiceAccounts-17B13FV7N7NDH UPDATE_IN_PROGRESS Service
2019-04-25T18:19:37Z system/aws/ecs (service gazs-gazette-ServiceContent-FB1XN0SV01MG-Service-1FPS72CJ455V4) has started 1 tasks: (task e2a7d471-9912-45b9-85df-2d32bc13df66).
2019-04-25T18:19:40Z system/aws/ecs (service gazs-gazette-ServiceAccounts-17B13FV7N7NDH-Service-16XQX9I3K6VK4) has started 1 tasks: (task 95e59a47-7e4c-4cc8-837d-f8c5ba3c9a40).
2019-04-25T18:19:46Z system/aws/ecs (service gazs-gazette-ServiceContent-FB1XN0SV01MG-Service-1FPS72CJ455V4) registered 1 targets in (target-group arn:aws:elasticloadbalancing:us-east-1:805781165890:targetgroup/gazs-Balan-J6EQ0Q5EG51S/23b805f120eae7e2)
2019-04-25T18:19:51Z system/aws/ecs (service gazs-gazette-ServiceLedger-UX66QQQL9DUE-Service-EIS6AW5F44IT) has started 1 tasks: (task b491a5e5-dbf3-4264-9471-3fb9d8e499f3).
2019-04-25T18:20:02Z system/aws/ecs (service gazs-gazette-ServiceAccounts-17B13FV7N7NDH-Service-16XQX9I3K6VK4) registered 1 targets in (target-group arn:aws:elasticloadbalancing:us-east-1:805781165890:targetgroup/gazs-Balan-1T37DYG08ELPZ/4fb9c62545387416)
2019-04-25T18:20:02Z system/aws/ecs (service gazs-gazette-ServiceLedger-UX66QQQL9DUE-Service-EIS6AW5F44IT) registered 1 targets in (target-group arn:aws:elasticloadbalancing:us-east-1:805781165890:targetgroup/gazs-Balan-1XFP08ZJCXXNK/47a14414d9af5c98)
2019-04-25T18:20:19Z system/aws/ecs (service gazs-gazette-ServiceContent-FB1XN0SV01MG-Service-1FPS72CJ455V4) has begun draining connections on 1 tasks.
2019-04-25T18:20:19Z system/aws/ecs (service gazs-gazette-ServiceContent-FB1XN0SV01MG-Service-1FPS72CJ455V4) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-east-1:805781165890:targetgroup/gazs-Balan-J6EQ0Q5EG51S/23b805f120eae7e2)
2019-04-25T18:20:35Z system/aws/ecs (service gazs-gazette-ServiceAccounts-17B13FV7N7NDH-Service-16XQX9I3K6VK4) has begun draining connections on 1 tasks.
2019-04-25T18:20:35Z system/aws/ecs (service gazs-gazette-ServiceAccounts-17B13FV7N7NDH-Service-16XQX9I3K6VK4) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-east-1:805781165890:targetgroup/gazs-Balan-1T37DYG08ELPZ/4fb9c62545387416)
2019-04-25T18:20:45Z system/aws/ecs (service gazs-gazette-ServiceLedger-UX66QQQL9DUE-Service-EIS6AW5F44IT) has begun draining connections on 1 tasks.
2019-04-25T18:20:45Z system/aws/ecs (service gazs-gazette-ServiceLedger-UX66QQQL9DUE-Service-EIS6AW5F44IT) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-east-1:805781165890:targetgroup/gazs-Balan-1XFP08ZJCXXNK/47a14414d9af5c98)
2019-04-25T18:20:51Z system/aws/ecs (service gazs-gazette-ServiceContent-FB1XN0SV01MG-Service-1FPS72CJ455V4) has stopped 1 running tasks: (task bc15929d-02e6-4419-9242-98c462cd6941).
2019-04-25T18:21:06Z system/aws/ecs (service gazs-gazette-ServiceAccounts-17B13FV7N7NDH-Service-16XQX9I3K6VK4) has stopped 1 running tasks: (task cdf9bb34-b767-47b6-a9e8-28c1c9b2a4fa).
2019-04-25T18:21:12Z system/aws/ecs (service gazs-gazette-ServiceContent-FB1XN0SV01MG-Service-1FPS72CJ455V4) has reached a steady state.
2019-04-25T18:21:16Z system/aws/ecs (service gazs-gazette-ServiceLedger-UX66QQQL9DUE-Service-EIS6AW5F44IT) has stopped 1 running tasks: (task b30df11e-eff4-4740-b207-a96571acfbb6).
2019-04-25T18:21:27Z system/aws/ecs (service gazs-gazette-ServiceAccounts-17B13FV7N7NDH-Service-16XQX9I3K6VK4) has reached a steady state.
2019-04-25T18:21:29Z system/aws/cfm gazs-gazette-ServiceLedger-UX66QQQL9DUE UPDATE_COMPLETE Service
2019-04-25T18:21:32Z system/aws/cfm gazs-gazette-ServiceLedger-UX66QQQL9DUE UPDATE_COMPLETE_CLEANUP_IN_PROGRESS gazs-gazette-ServiceLedger-UX66QQQL9DUE
2019-04-25T18:21:32Z system/aws/cfm gazs-gazette-ServiceAccounts-17B13FV7N7NDH UPDATE_COMPLETE Service
2019-04-25T18:21:34Z system/aws/cfm gazs-gazette-ServiceContent-FB1XN0SV01MG UPDATE_COMPLETE Service
2019-04-25T18:21:35Z system/aws/cfm gazs-gazette-ServiceAccounts-17B13FV7N7NDH UPDATE_COMPLETE_CLEANUP_IN_PROGRESS gazs-gazette-ServiceAccounts-17B13FV7N7NDH
2019-04-25T18:21:37Z system/aws/cfm gazs-gazette-ServiceContent-FB1XN0SV01MG UPDATE_COMPLETE_CLEANUP_IN_PROGRESS gazs-gazette-ServiceContent-FB1XN0SV01MG
2019-04-25T18:21:37Z system/aws/ecs (service gazs-gazette-ServiceLedger-UX66QQQL9DUE-Service-EIS6AW5F44IT) has reached a steady state.
2019-04-25T18:21:53Z system/aws/cfm gazs-gazette UPDATE_COMPLETE ServiceLedger
2019-04-25T18:21:55Z system/aws/cfm gazs-gazette UPDATE_COMPLETE ServiceContent
2019-04-25T18:21:56Z system/aws/cfm gazs-gazette UPDATE_COMPLETE ServiceAccounts
2019-04-25T18:23:23Z system/aws/ecs (service gazs-gazette-ServiceApi-1H84BFBZ0T9QT-Service-UPKOS4Y9574L) has started 1 tasks: (task 820bb314-969b-407b-989a-fd943bfaf09f).
2019-04-25T18:23:33Z system/aws/ecs (service gazs-gazette-ServiceApi-1H84BFBZ0T9QT-Service-UPKOS4Y9574L) registered 1 targets in (target-group arn:aws:elasticloadbalancing:us-east-1:805781165890:targetgroup/gazs-Balan-FSQP7GSDNN0Q/c0e27197920b6930)