Problem installing Gen 3 local rack

julian · March 12, 2020, 1:43am

When Gen 3 left beta I tried to set up a local rack and ran into some issues. Today, I wiped my entire docker disk image and reset Kubernetes, and I’m still having problems.

Terraform starts chugging away but then times out waiting for an IP for some load balancer.

$ convox3 --version
3.0.1
$ terraform --version
Terraform v0.12.20

Logs:

$ convox3 rack install local convox
Initializing modules...
Downloading github.com/convox/convox?ref=3.0.8 for system...
- system in .terraform/modules/system/terraform/system/local
- system.platform in .terraform/modules/system/terraform/platform
- system.rack in .terraform/modules/system/terraform/rack/local
- system.rack.api in .terraform/modules/system/terraform/api/local
- system.rack.api.k8s in .terraform/modules/system/terraform/api/k8s
- system.rack.k8s in .terraform/modules/system/terraform/rack/k8s
- system.rack.resolver in .terraform/modules/system/terraform/resolver/local
- system.rack.resolver.k8s in .terraform/modules/system/terraform/resolver/k8s
- system.rack.router in .terraform/modules/system/terraform/router/local
- system.rack.router.k8s in .terraform/modules/system/terraform/router/k8s

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "local" (hashicorp/local) 1.4.0...
- Downloading plugin for provider "null" (hashicorp/null) 2.1.2...
- Downloading plugin for provider "random" (hashicorp/random) 2.2.1...
- Downloading plugin for provider "tls" (hashicorp/tls) 2.1.1...
- Downloading plugin for provider "http" (hashicorp/http) 1.1.1...
- Downloading plugin for provider "kubernetes" (hashicorp/kubernetes) 1.10.0...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
module.system.data.http.releases: Refreshing state...
module.system.module.rack.random_string.secret: Creating...
module.system.module.platform.null_resource.platform: Creating...
module.system.module.rack.module.router.tls_private_key.ca-private: Creating...
module.system.module.platform.null_resource.platform: Provisioning with 'local-exec'...
module.system.module.rack.random_string.secret: Creation complete after 0s [id=NmMrl2Aub1225fvxWdWwAxZ5IwAs3X]
module.system.module.platform.null_resource.platform (local-exec): Executing: ["/bin/sh" "-c" "mkdir -p .terraform/modules/system/terraform/platform && uname -s > .terraform/modules/system/terraform/platform/convox.platform"]
module.system.module.platform.null_resource.platform: Creation complete after 0s [id=2362315601528036223]
module.system.module.platform.data.local_file.platform: Refreshing state...
module.system.module.rack.module.k8s.kubernetes_namespace.system: Creating...
module.system.module.rack.module.resolver.module.k8s.kubernetes_cluster_role.resolver: Creating...
module.system.module.rack.module.api.module.k8s.kubernetes_cluster_role.atom: Creating...
module.system.module.rack.module.api.module.k8s.kubernetes_cluster_role.api: Creating...
module.system.module.rack.module.router.module.k8s.kubernetes_cluster_role.router: Creating...
module.system.module.rack.module.api.module.k8s.random_string.password: Creating...
module.system.module.rack.module.api.module.k8s.random_string.password: Creation complete after 0s [id=GeL5DZ0qXrFnLL5qoRX2oAwpvknnDyT8nXba9P3pFy8xbIZwY6BbY5GYgg2bUas6]
module.system.module.rack.module.router.module.k8s.kubernetes_cluster_role.router: Creation complete after 0s [id=router]
module.system.module.rack.module.api.module.k8s.kubernetes_cluster_role.atom: Creation complete after 0s [id=atom]
module.system.module.rack.module.k8s.kubernetes_namespace.system: Creation complete after 0s [id=convox-system]
module.system.module.rack.module.resolver.module.k8s.kubernetes_cluster_role.resolver: Creation complete after 0s [id=resolver]
module.system.module.rack.module.api.module.k8s.kubernetes_cluster_role.api: Creation complete after 0s [id=convox-api]
module.system.module.rack.module.k8s.kubernetes_config_map.rack: Creating...
module.system.module.rack.module.api.module.k8s.kubernetes_cluster_role_binding.atom: Creating...
module.system.module.rack.module.api.module.k8s.kubernetes_service_account.atom: Creating...
module.system.module.rack.module.api.module.k8s.kubernetes_service_account.api: Creating...
module.system.module.rack.module.router.kubernetes_service.router: Creating...
module.system.module.rack.module.resolver.module.k8s.kubernetes_service.resolver: Creating...
module.system.module.rack.module.resolver.kubernetes_service.resolver-external: Creating...
module.system.module.rack.module.resolver.module.k8s.kubernetes_service_account.resolver: Creating...
module.system.module.rack.module.router.tls_private_key.ca-private: Creation complete after 0s [id=da307c69cc6f9da9418f3edd3f18932fad1428ea]
module.system.module.rack.module.api.module.k8s.kubernetes_service.api: Creating...
module.system.module.rack.module.api.module.k8s.kubernetes_cluster_role_binding.atom: Creation complete after 0s [id=atom]
module.system.module.rack.module.api.module.k8s.kubernetes_persistent_volume_claim.api_volume[0]: Creating...
module.system.module.rack.module.api.module.k8s.kubernetes_deployment.atom: Creating...
module.system.module.rack.module.k8s.kubernetes_config_map.rack: Creation complete after 1s [id=convox-system/rack]
module.system.module.rack.kubernetes_service.registry: Creating...
module.system.module.rack.module.resolver.module.k8s.kubernetes_service.resolver: Creation complete after 2s [id=convox-system/resolver]
module.system.module.rack.kubernetes_persistent_volume_claim.registry: Creating...
module.system.module.rack.module.api.module.k8s.kubernetes_service.api: Creation complete after 3s [id=convox-system/api]
module.system.module.rack.module.router.tls_self_signed_cert.ca: Creating...
module.system.module.rack.module.router.tls_self_signed_cert.ca: Creation complete after 0s [id=1355263026195828405863486357396425702]
module.system.module.rack.module.router.module.k8s.kubernetes_horizontal_pod_autoscaler.router: Creating...
module.system.module.rack.module.router.module.k8s.kubernetes_horizontal_pod_autoscaler.router: Creation complete after 0s [id=convox-system/router]
module.system.module.rack.module.router.module.k8s.kubernetes_service_account.router: Creating...
module.system.module.rack.module.api.module.k8s.kubernetes_service_account.atom: Creation complete after 3s [id=convox-system/atom]
module.system.module.rack.module.api.module.k8s.kubernetes_ingress.api: Creating...
module.system.module.rack.module.api.module.k8s.kubernetes_ingress.api: Creation complete after 0s [id=convox-system/api]
module.system.module.rack.module.router.kubernetes_secret.ca: Creating...
module.system.module.rack.module.resolver.kubernetes_service.resolver-external: Creation complete after 4s [id=convox-system/resolver-external]
module.system.module.rack.module.api.module.k8s.kubernetes_service_account.api: Creation complete after 4s [id=convox-system/api]
module.system.module.rack.module.api.module.k8s.kubernetes_cluster_role_binding.api: Creating...
module.system.module.rack.module.api.module.k8s.kubernetes_cluster_role_binding.api: Creation complete after 0s [id=convox-api]
module.system.module.rack.kubernetes_service.registry: Creation complete after 3s [id=convox-system/registry]
module.system.module.rack.kubernetes_ingress.registry: Creating...
module.system.module.rack.kubernetes_ingress.registry: Creation complete after 0s [id=convox-system/registry]
module.system.module.rack.module.api.module.k8s.kubernetes_deployment.api: Creating...
module.system.module.rack.module.api.module.k8s.kubernetes_deployment.atom: Creation complete after 4s [id=convox-system/atom]
module.system.module.rack.module.api.module.k8s.kubernetes_persistent_volume_claim.api_volume[0]: Creation complete after 5s [id=convox-system/api-storage]
module.system.module.rack.module.resolver.module.k8s.kubernetes_service_account.resolver: Creation complete after 5s [id=convox-system/resolver]
module.system.module.rack.module.resolver.module.k8s.kubernetes_cluster_role_binding.resolver: Creating...
module.system.module.rack.module.resolver.module.k8s.kubernetes_cluster_role_binding.resolver: Creation complete after 0s [id=resolver]
module.system.module.rack.module.resolver.module.k8s.kubernetes_deployment.resolver: Creating...
module.system.module.rack.kubernetes_persistent_volume_claim.registry: Creation complete after 4s [id=convox-system/registry]
module.system.module.rack.module.router.kubernetes_secret.ca: Creation complete after 3s [id=convox-system/ca]
module.system.module.rack.kubernetes_deployment.registry: Creating...
module.system.module.rack.module.router.module.k8s.kubernetes_service_account.router: Creation complete after 4s [id=convox-system/router]
module.system.module.rack.module.router.module.k8s.kubernetes_cluster_role_binding.router: Creating...
module.system.module.rack.module.router.module.k8s.kubernetes_cluster_role_binding.router: Creation complete after 0s [id=router]
module.system.module.rack.module.router.module.k8s.kubernetes_deployment.router: Creating...
module.system.module.rack.module.resolver.module.k8s.kubernetes_deployment.resolver: Creation complete after 4s [id=convox-system/resolver]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [10s elapsed]
module.system.module.rack.module.router.module.k8s.kubernetes_deployment.router: Creation complete after 4s [id=convox-system/router]
module.system.module.rack.module.api.module.k8s.kubernetes_deployment.api: Creation complete after 8s [id=convox-system/api]
module.system.module.rack.kubernetes_deployment.registry: Creation complete after 8s [id=convox-system/registry]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [20s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [30s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [40s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [50s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [1m0s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [1m10s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [1m20s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [1m30s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [1m40s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [1m50s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [2m0s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [2m10s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [2m20s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [2m30s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [2m40s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [2m50s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [3m0s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [3m10s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [3m20s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [3m30s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [3m40s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [3m50s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [4m0s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [4m10s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [4m20s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [4m30s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [4m40s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [4m50s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [5m0s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [5m10s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [5m20s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [5m30s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [5m40s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [5m50s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [6m0s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [6m10s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [6m20s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [6m30s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [6m40s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [6m50s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [7m0s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [7m10s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [7m20s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [7m30s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [7m40s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [7m50s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [8m0s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [8m10s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [8m20s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [8m30s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [8m40s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [8m50s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [9m0s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [9m10s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [9m20s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [9m30s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [9m40s elapsed]
module.system.module.rack.module.router.kubernetes_service.router: Still creating... [9m50s elapsed]

Error: Waiting for service "convox-system/router" to assign IP/hostname for a load balancer

  on .terraform/modules/system/terraform/router/local/main.tf line 40, in resource "kubernetes_service" "router":
  40: resource "kubernetes_service" "router" {


ERROR: exit status 1

mrluc · May 25, 2020, 2:03pm

Can confirm – I get the resolver thing, and also the registry's PersistentVolumeClaim failing for the local registry.

module.system.module.rack.module.resolver.kubernetes_service.resolver-external: Still creating... [9m50s elapsed]
module.system.module.rack.module.resolver.kubernetes_service.resolver-external: Still creating... [10m0s elapsed]

Error: timeout while waiting for state to become 'Bound' (last state: 'Pending', timeout: 5m0s)
   * registry (PersistentVolumeClaim): ProvisioningFailed: failed to provision volume with StorageClass "do-block-storage": rpc error: code = OutOfRange desc = invalid capacity range: required (20) can not be less than minimum supported volume size (1Gi)

  on .terraform/modules/system/terraform/rack/local/registry.tf line 86, in resource "kubernetes_persistent_volume_claim" "registry":
  86: resource "kubernetes_persistent_volume_claim" "registry" {

Error: Waiting for service "dev-system/resolver-external" to assign IP/hostname for a load balancer
   * resolver-external (Service): SyncLoadBalancerFailed: Error syncing load balancer: failed to ensure load balancer: failed to build load-balancer request: only TCP protocol is supported, got: "UDP"

  on .terraform/modules/system/terraform/resolver/local/main.tf line 19, in resource "kubernetes_service" "resolver-external":
  19: resource "kubernetes_service" "resolver-external" {

ERROR: exit status 1

NBD/not blocking anything for me at the moment.

julian · May 26, 2020, 11:19pm

I think it was actually a problem with /etc/resolver/convox. It seems sometimes you have to restart OSX to get it to acknowledge changes to those files.

My V3 local rack works now after:

fully wiping Docker + Kubernetes
Deleting ~/Library/Preferences/convox/racks
Updating terraform, convox cli, and Docker
Installing the local rack

My /etc/resolver/convox:

nameserver 127.0.0.1

You can debug resolver issues with dig, e.g. does dig api.dev.convox @localhost return 127.0.0.1 A record, does dig api.dev.convox without specifying a nameserver, etc. And try a restart if @localhost works but without does not.

Topic		Replies	Views
Installing local rack on ARM64 / M1 Mac Rack (Version 3)	1	396	November 15, 2021
Trouble with running rack locally Rack (Version 3)	1	359	March 17, 2022
Local rack startup problems on Ubuntu after update Rack (Version 2)	3	902	February 22, 2019
Version 3.x stopped working with Gen1 racks? CLI	8	1079	February 25, 2020
Error while trying to install local rack Rack (Version 2)	7	1979	April 25, 2019

Problem installing Gen 3 local rack

Related topics