Error creating EKS Cluster with unsupported availability zone

Rack (Version 3)
1

I got this EKS crash while running convox rack install aws convox-test:

$ convox version
3.0.41

$ convox rack install aws
…
module.system.module.cluster.aws_eks_cluster.cluster: Creating…
Error: error creating EKS Cluster (convox-test): UnsupportedAvailabilityZoneException: Cannot create cluster ‘convox-test’ because us-east-1c, the targeted availability zone, does not currently have sufficient capacity to support the cluster. Retry and choose from these availability zones: us-east-1a, us-east-1b, us-east-1d, us-east-1e, us-east-1f

Now I’m not too sure what to do. Rerunning the command:

 $  convox rack install aws convox-test
ERROR: rack name in use: convox-test

Would it be possible to retry automatically with a different availability zone?

Also, how can I tidy up the resources that it already created and start over?




Here’s the full logs:

$ convox rack install aws convox-test
Upgrading modules...
Downloading github.com/convox/convox?ref=3.0.41 for system...
- system in .terraform/modules/system/terraform/system/aws
- system.cluster in .terraform/modules/system/terraform/cluster/aws
- system.fluentd in .terraform/modules/system/terraform/fluentd/aws
- system.fluentd.k8s in .terraform/modules/system/terraform/fluentd/k8s
- system.rack in .terraform/modules/system/terraform/rack/aws
- system.rack.api in .terraform/modules/system/terraform/api/aws
- system.rack.api.k8s in .terraform/modules/system/terraform/api/k8s
- system.rack.k8s in .terraform/modules/system/terraform/rack/k8s
- system.rack.metrics in .terraform/modules/system/terraform/metrics/k8s
- system.rack.resolver in .terraform/modules/system/terraform/resolver/aws
- system.rack.resolver.k8s in .terraform/modules/system/terraform/resolver/k8s
- system.rack.router in .terraform/modules/system/terraform/router/aws
- system.rack.router.nginx in .terraform/modules/system/terraform/router/nginx

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/random versions matching "~> 2.2"...
- Finding hashicorp/external versions matching "~> 1.2"...
- Finding hashicorp/aws versions matching "~> 2.49, ~> 2.49, ~> 2.49, ~> 2.49, ~> 2.49, ~> 2.49, ~> 2.49"...
- Finding hashicorp/http versions matching "~> 1.1, ~> 1.1"...
- Finding hashicorp/kubernetes versions matching "~> 1.11, ~> 1.11, ~> 1.11, ~> 1.11, ~> 1.11, ~> 1.11, ~> 1.11, ~> 1.11, ~> 1.11, ~> 1.11, ~> 1.11, ~> 1.11, ~> 1.11"...
- Finding hashicorp/null versions matching "~> 2.1"...
- Finding hashicorp/local versions matching "~> 1.3"...
- Installing hashicorp/local v1.4.0...
- Installed hashicorp/local v1.4.0 (signed by HashiCorp)
- Installing hashicorp/random v2.3.0...
- Installed hashicorp/random v2.3.0 (signed by HashiCorp)
- Installing hashicorp/external v1.2.0...
- Installed hashicorp/external v1.2.0 (signed by HashiCorp)
- Installing hashicorp/aws v2.70.0...
- Installed hashicorp/aws v2.70.0 (signed by HashiCorp)
- Installing hashicorp/http v1.2.0...
- Installed hashicorp/http v1.2.0 (signed by HashiCorp)
- Installing hashicorp/kubernetes v1.13.2...
- Installed hashicorp/kubernetes v1.13.2 (signed by HashiCorp)
- Installing hashicorp/null v2.1.2...
- Installed hashicorp/null v2.1.2 (signed by HashiCorp)

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
module.system.data.http.releases: Refreshing state...
module.system.module.rack.module.api.data.aws_partition.current: Refreshing state...
module.system.module.cluster.data.aws_iam_policy_document.autoscale: Refreshing state...
module.system.module.rack.module.api.data.aws_caller_identity.current: Refreshing state...
module.system.module.cluster.data.aws_availability_zones.available: Refreshing state...
module.system.module.cluster.data.aws_iam_policy_document.assume_eks: Refreshing state...
module.system.module.fluentd.data.aws_partition.current: Refreshing state...
module.system.module.cluster.data.aws_partition.current: Refreshing state...
module.system.module.fluentd.data.aws_caller_identity.current: Refreshing state...
module.system.module.rack.module.api.data.aws_region.current: Refreshing state...
module.system.module.cluster.data.aws_region.current: Refreshing state...
module.system.module.fluentd.data.aws_region.current: Refreshing state...
module.system.module.rack.module.router.data.aws_region.current: Refreshing state...
module.system.module.cluster.data.aws_iam_policy_document.assume_ec2: Refreshing state...
module.system.module.rack.module.api.data.aws_iam_policy_document.logs: Refreshing state...
module.system.module.fluentd.data.aws_iam_policy_document.fluentd: Refreshing state...
module.system.module.rack.module.api.module.k8s.random_string.password: Creating...
module.system.module.rack.module.api.module.k8s.random_string.password: Creation complete after 0s [id=fJojA8WBEZPpqxOXSvJ6jo6oqee5l4zCznSPaHFYUd8nYyxEAaMmbSyMMPs5EE73]
module.system.module.cluster.data.aws_availability_zones.available: Reading... [id=2020-09-20 16:02:13.24668 +0000 UTC]
module.system.module.cluster.data.aws_availability_zones.available: Read complete after 0s [id=2020-09-20 16:02:19.805155 +0000 UTC]
module.system.module.cluster.aws_eip.nat[2]: Creating...
module.system.module.cluster.aws_iam_role.cluster: Creating...
module.system.module.cluster.aws_iam_role.nodes: Creating...
module.system.module.cluster.aws_eip.nat[0]: Creating...
module.system.module.cluster.aws_eip.nat[1]: Creating...
module.system.module.rack.module.api.aws_s3_bucket.storage: Creating...
module.system.module.cluster.aws_eip.nat[1]: Creation complete after 3s [id=eipalloc-06513c7177444313b]
module.system.module.cluster.aws_eip.nat[2]: Creation complete after 4s [id=eipalloc-092af7dc23983b01f]
module.system.module.cluster.aws_eip.nat[0]: Creation complete after 4s [id=eipalloc-0303b673dd690c47d]
module.system.module.cluster.aws_iam_role.nodes: Creation complete after 4s [id=convox-test-nodes]
module.system.module.cluster.aws_iam_role_policy_attachment.nodes_ecr: Creating...
module.system.module.cluster.aws_iam_role_policy_attachment.nodes_eks_worker: Creating...
module.system.module.cluster.aws_iam_role_policy_attachment.nodes_eks_cni: Creating...
module.system.module.cluster.random_id.node_group: Creating...
module.system.module.cluster.random_id.node_group: Creation complete after 0s [id=tX-loWiUhL4]
module.system.module.cluster.aws_iam_role.cluster: Creation complete after 4s [id=convox-test-cluster]
module.system.module.cluster.aws_iam_role_policy_attachment.cluster_ec2_readonly: Creating...
module.system.module.cluster.aws_iam_role_policy_attachment.cluster_eks_cluster: Creating...
module.system.module.cluster.aws_iam_role_policy_attachment.cluster_eks_service: Creating...
module.system.module.cluster.aws_iam_role_policy_attachment.nodes_eks_worker: Creation complete after 3s [id=convox-test-nodes-20200920160233290900000002]
module.system.module.cluster.aws_iam_role_policy_attachment.nodes_ecr: Creation complete after 3s [id=convox-test-nodes-20200920160233331500000003]
module.system.module.cluster.aws_iam_role_policy_attachment.nodes_eks_cni: Creation complete after 3s [id=convox-test-nodes-20200920160233365300000004]
module.system.module.cluster.aws_iam_role_policy_attachment.cluster_eks_cluster: Creation complete after 3s [id=convox-test-cluster-20200920160233431600000007]
module.system.module.cluster.aws_iam_role_policy_attachment.cluster_ec2_readonly: Creation complete after 3s [id=convox-test-cluster-20200920160233428100000006]
module.system.module.cluster.aws_iam_role_policy_attachment.cluster_eks_service: Creation complete after 3s [id=convox-test-cluster-20200920160233427400000005]
module.system.module.cluster.null_resource.iam: Creating...
module.system.module.cluster.null_resource.delay_cluster: Creating...
module.system.module.cluster.null_resource.iam: Creation complete after 0s [id=4496035886750418158]
module.system.module.cluster.aws_vpc.nodes: Creating...
module.system.module.cluster.null_resource.delay_cluster: Provisioning with 'local-exec'...
module.system.module.cluster.null_resource.delay_cluster (local-exec): Executing: ["/bin/sh" "-c" "sleep 15"]
module.system.module.rack.module.api.aws_s3_bucket.storage: Still creating... [10s elapsed]
module.system.module.cluster.null_resource.delay_cluster: Still creating... [10s elapsed]
module.system.module.cluster.aws_vpc.nodes: Still creating... [10s elapsed]
module.system.module.rack.module.api.aws_s3_bucket.storage: Creation complete after 17s [id=convox-test-storage-20200920160228120100000001]
module.system.module.rack.module.api.data.aws_iam_policy_document.storage: Reading...
module.system.module.rack.module.api.data.aws_iam_policy_document.storage: Read complete after 0s [id=2169334174]
module.system.module.cluster.aws_vpc.nodes: Creation complete after 14s [id=vpc-0e4a929903dedd488]
module.system.module.cluster.aws_internet_gateway.nodes: Creating...
module.system.module.cluster.aws_route_table.private[1]: Creating...
module.system.module.cluster.aws_route_table.private[0]: Creating...
module.system.module.cluster.aws_route_table.public: Creating...
module.system.module.cluster.aws_subnet.public[2]: Creating...
module.system.module.cluster.aws_subnet.private[2]: Creating...
module.system.module.cluster.aws_subnet.private[1]: Creating...
module.system.module.cluster.aws_subnet.public[0]: Creating...
module.system.module.cluster.aws_subnet.private[0]: Creating...
module.system.module.cluster.null_resource.delay_cluster: Creation complete after 15s [id=9212518504656927493]
module.system.module.cluster.aws_subnet.public[1]: Creating...
module.system.module.cluster.aws_route_table.public: Creation complete after 5s [id=rtb-000a29393943d160f]
module.system.module.cluster.aws_route_table.private[2]: Creating...
module.system.module.cluster.aws_route_table.private[1]: Creation complete after 5s [id=rtb-04a5a99c168d8b698]
module.system.module.cluster.aws_security_group.cluster: Creating...
module.system.module.cluster.aws_subnet.private[2]: Creation complete after 5s [id=subnet-01bdf1da307492292]
module.system.module.cluster.aws_subnet.private[0]: Creation complete after 5s [id=subnet-0eaa1cb232c86f7e7]
module.system.module.cluster.aws_subnet.public[0]: Creation complete after 5s [id=subnet-0cc72aa47c1823f66]
module.system.module.cluster.aws_subnet.private[1]: Creation complete after 5s [id=subnet-0692f76ef7ffec014]
module.system.module.cluster.aws_route_table.private[0]: Creation complete after 5s [id=rtb-085a849d7008b3100]
module.system.module.cluster.aws_subnet.public[2]: Creation complete after 5s [id=subnet-0ecf7ae60d3ae9347]
module.system.module.cluster.aws_subnet.public[1]: Creation complete after 4s [id=subnet-0240c95b3fe3f9fb2]
module.system.module.cluster.aws_nat_gateway.private[1]: Creating...
module.system.module.cluster.aws_route_table_association.public[2]: Creating...
module.system.module.cluster.aws_route_table_association.public[1]: Creating...
module.system.module.cluster.aws_route_table_association.public[0]: Creating...
module.system.module.cluster.aws_nat_gateway.private[2]: Creating...
module.system.module.cluster.aws_nat_gateway.private[0]: Creating...
module.system.module.cluster.aws_internet_gateway.nodes: Creation complete after 7s [id=igw-01ae4b35f1b5e6b9e]
module.system.module.cluster.aws_route_table_association.public[2]: Creation complete after 2s [id=rtbassoc-020dd3e56db94f6ea]
module.system.module.cluster.aws_route_table_association.public[0]: Creation complete after 2s [id=rtbassoc-0d6dbe66c2edcb985]
module.system.module.cluster.aws_route_table_association.public[1]: Creation complete after 2s [id=rtbassoc-012dc6fc258234583]
module.system.module.cluster.aws_route.public-default: Creating...
module.system.module.cluster.aws_route_table.private[2]: Creation complete after 4s [id=rtb-0e225ad66528df1a7]
module.system.module.cluster.aws_route_table_association.private[0]: Creating...
module.system.module.cluster.aws_route_table_association.private[1]: Creating...
module.system.module.cluster.aws_route_table_association.private[2]: Creating...
module.system.module.cluster.aws_route.public-default: Creation complete after 3s [id=r-rtb-000a29393943d160f1080289494]
module.system.module.cluster.aws_route_table_association.private[1]: Creation complete after 1s [id=rtbassoc-0cd356348e875dbba]
module.system.module.cluster.aws_route_table_association.private[2]: Creation complete after 1s [id=rtbassoc-071dc7c6e4cf240b8]
module.system.module.cluster.aws_route_table_association.private[0]: Creation complete after 2s [id=rtbassoc-0a0734f2317c5ebad]
module.system.module.cluster.aws_security_group.cluster: Creation complete after 7s [id=sg-06a5a0cdee7f23638]
module.system.module.cluster.aws_nat_gateway.private[2]: Still creating... [10s elapsed]
module.system.module.cluster.aws_nat_gateway.private[0]: Still creating... [10s elapsed]
module.system.module.cluster.aws_nat_gateway.private[1]: Still creating... [10s elapsed]
module.system.module.cluster.aws_nat_gateway.private[0]: Still creating... [20s elapsed]
module.system.module.cluster.aws_nat_gateway.private[1]: Still creating... [20s elapsed]
module.system.module.cluster.aws_nat_gateway.private[2]: Still creating... [20s elapsed]
module.system.module.cluster.aws_nat_gateway.private[2]: Still creating... [30s elapsed]
module.system.module.cluster.aws_nat_gateway.private[0]: Still creating... [30s elapsed]
module.system.module.cluster.aws_nat_gateway.private[1]: Still creating... [30s elapsed]
module.system.module.cluster.aws_nat_gateway.private[0]: Still creating... [40s elapsed]
module.system.module.cluster.aws_nat_gateway.private[1]: Still creating... [40s elapsed]
module.system.module.cluster.aws_nat_gateway.private[2]: Still creating... [40s elapsed]
module.system.module.cluster.aws_nat_gateway.private[1]: Still creating... [50s elapsed]
module.system.module.cluster.aws_nat_gateway.private[2]: Still creating... [50s elapsed]
module.system.module.cluster.aws_nat_gateway.private[0]: Still creating... [50s elapsed]
module.system.module.cluster.aws_nat_gateway.private[0]: Still creating... [1m0s elapsed]
module.system.module.cluster.aws_nat_gateway.private[1]: Still creating... [1m0s elapsed]
module.system.module.cluster.aws_nat_gateway.private[2]: Still creating... [1m0s elapsed]
module.system.module.cluster.aws_nat_gateway.private[2]: Still creating... [1m10s elapsed]
module.system.module.cluster.aws_nat_gateway.private[0]: Still creating... [1m10s elapsed]
module.system.module.cluster.aws_nat_gateway.private[1]: Still creating... [1m10s elapsed]
module.system.module.cluster.aws_nat_gateway.private[1]: Still creating... [1m20s elapsed]
module.system.module.cluster.aws_nat_gateway.private[0]: Still creating... [1m20s elapsed]
module.system.module.cluster.aws_nat_gateway.private[2]: Still creating... [1m20s elapsed]
module.system.module.cluster.aws_nat_gateway.private[2]: Still creating... [1m30s elapsed]
module.system.module.cluster.aws_nat_gateway.private[0]: Still creating... [1m30s elapsed]
module.system.module.cluster.aws_nat_gateway.private[1]: Still creating... [1m30s elapsed]
module.system.module.cluster.aws_nat_gateway.private[1]: Creation complete after 1m31s [id=nat-0415054d71f37f3d4]
module.system.module.cluster.aws_nat_gateway.private[0]: Still creating... [1m40s elapsed]
module.system.module.cluster.aws_nat_gateway.private[2]: Still creating... [1m40s elapsed]
module.system.module.cluster.aws_nat_gateway.private[0]: Creation complete after 1m42s [id=nat-0693534d337bf9dbf]
module.system.module.cluster.aws_nat_gateway.private[2]: Still creating... [1m50s elapsed]
module.system.module.cluster.aws_nat_gateway.private[2]: Still creating... [2m0s elapsed]
module.system.module.cluster.aws_nat_gateway.private[2]: Creation complete after 2m4s [id=nat-07ea34d3ec15f0fc8]
module.system.module.cluster.aws_route.private-default[1]: Creating...
module.system.module.cluster.aws_route.private-default[0]: Creating...
module.system.module.cluster.aws_route.private-default[2]: Creating...
module.system.module.cluster.aws_route.private-default[1]: Creation complete after 4s [id=r-rtb-04a5a99c168d8b6981080289494]
module.system.module.cluster.aws_route.private-default[2]: Creation complete after 4s [id=r-rtb-0e225ad66528df1a71080289494]
module.system.module.cluster.aws_route.private-default[0]: Creation complete after 4s [id=r-rtb-085a849d7008b31001080289494]
module.system.module.cluster.null_resource.network: Creating...
module.system.module.cluster.null_resource.network: Creation complete after 0s [id=1806055981947219842]
module.system.module.cluster.aws_eks_cluster.cluster: Creating...

Error: error creating EKS Cluster (convox-test): UnsupportedAvailabilityZoneException: Cannot create cluster 'convox-test' because us-east-1c, the targeted availability zone, does not currently have sufficient capacity to support the cluster. Retry and choose from these availability zones: us-east-1a, us-east-1b, us-east-1d, us-east-1e, us-east-1f
{
  RespMetadata: {
    StatusCode: 400,
    RequestID: "0b2cebb1-7674-4e9f-b561-9bc70af951c7"
  },
  ClusterName: "convox-test",
  Message_: "Cannot create cluster 'convox-test' because us-east-1c, the targeted availability zone, does not currently have sufficient capacity to support the cluster. Retry and choose from these availability zones: us-east-1a, us-east-1b, us-east-1d, us-east-1e, us-east-1f",
  ValidZones: [
    "us-east-1a",
    "us-east-1b",
    "us-east-1d",
    "us-east-1e",
    "us-east-1f"
  ]
}

  on .terraform/modules/system/terraform/cluster/aws/main.tf line 54, in resource "aws_eks_cluster" "cluster":
  54: resource "aws_eks_cluster" "cluster" {


ERROR: exit status 1
2

Argh this same error just happened again when I retried with a different rack name.

module.system.module.cluster.aws_eks_cluster.cluster: Creating...

Error: error creating EKS Cluster (another-test): UnsupportedAvailabilityZoneException: Cannot create cluster 'another-test' because us-east-1c, the targeted availability zone, does not currently have sufficient capacity to support the cluster. Retry and choose from these availability zones: us-east-1a, us-east-1b, us-east-1d, us-east-1e, us-east-1f
{
  RespMetadata: {
    StatusCode: 400,
    RequestID: "cb81cd5b-5628-4259-b8cf-f6d6643f8037"
  },
  ClusterName: "another-test",
  Message_: "Cannot create cluster 'another-test' because us-east-1c, the targeted availability zone, does not currently have sufficient capacity to support the cluster. Retry and choose from these availability zones: us-east-1a, us-east-1b, us-east-1d, us-east-1e, us-east-1f",
  ValidZones: [
    "us-east-1a",
    "us-east-1b",
    "us-east-1d",
    "us-east-1e",
    "us-east-1f"
  ]
}

  on .terraform/modules/system/terraform/cluster/aws/main.tf line 54, in resource "aws_eks_cluster" "cluster":
  54: resource "aws_eks_cluster" "cluster" {


ERROR: exit status 1

Not sure how to proceed. I could try with a different region (maybe us-west-2). But it would be great if Convox automatically tried a different EKS availability zone, or if I could configure this with an option.

3

I think I’ve figured out how to delete all the partially created resources. (NAT gateways, Elastic IPs, VPCs, Security Groups)

I had to delete everything from the VPC service (not EC2): https://console.aws.amazon.com/vpc/home

4

Did you ever manage to get your EKS cluster up and running? I’m experiencing the same error.

5

Hi @erik, I’m setting up a v3 cluster again, and I’ve figured out how to avoid the us-east-1c AZ. I just got the same error again today.

When creating a new Rack from the convox UI, you can add a new parameter for “availability_zones”, and set this to the following string: us-east-1a,us-east-1b,us-east-1d,us-east-1e,us-east-1f. This will skip the 1c zone that causes problems, and only use a, b, d, e, and f. This worked for me and I was able to set up the rack.