Convox Community

[RESOLVED] Logging not working

I’m running into an issue on a new v3 rack where logging does not appear to be working (from my Rails app.) It looks like the logs are accessible via kubectl, but they are not getting forwarded to CloudWatch.

$ kubectl get pods
NAME                                 READY   STATUS    RESTARTS   AGE
resource-database-7b9cb5ddb5-vznrb   1/1     Running   0          98m
resource-redis-5fc7577b9d-h52tv      1/1     Running   0          98m
web-85c9d99477-4s2jl                 1/1     Running   0          12m
web-85c9d99477-6tq4c                 1/1     Running   0          12m
web-85c9d99477-hjxv7                 1/1     Running   0          12m
worker-5894f7db9-62hbn               1/1     Running   0          12m
worker-5894f7db9-996zt               1/1     Running   0          12m
worker-5894f7db9-bssld               1/1     Running   0          12m
worker-5894f7db9-vtvxz               1/1     Running   0          12m
worker-5894f7db9-w7vvh               1/1     Running   0          12m

Logs are showing up for web-85c9d99477-hjxv7 if I use kubectl logs:

$ kubectl logs web-85c9d99477-hjxv7
Skipping iptables configuration. (Requires --cap-add=NET_ADMIN or --privileged)
02:37:35 web.1     | started with pid 18
02:37:37 web.1     | [18] Puma starting in cluster mode...
02:37:37 web.1     | [18] * Puma version: 5.3.1 (ruby 2.7.1-p83) ("Sweetnighter")
02:37:37 web.1     | [18] *  Min threads: 5
02:37:37 web.1     | [18] *  Max threads: 5
02:37:37 web.1     | [18] *  Environment: production
02:37:37 web.1     | [18] *   Master PID: 18
02:37:37 web.1     | [18] *      Workers: 3
02:37:37 web.1     | [18] *     Restarts: (✔) hot (✖) phased
02:37:37 web.1     | [18] * Preloading application
...
02:45:01 web.1     | I, [2021-09-17T02:45:01.643308 #44]  INFO -- : [495e749b82326e28f358a47606941bb4] method=GET path=/users format=html controller=UsersController action=index status=200 duration=141.77 view=61.71 db=34.57 request_id=495e749b82326e28f358a47606941bb4 host=testapp.docspring.co source_ip=10.1.196.72 user_id=1 account_id=1 params={} basic_auth=false
02:46:18 web.1     | I, [2021-09-17T02:46:18.050010 #48]  INFO -- : [79e44d3c623e815f7ab806097cc59b13] method=GET path=/templates format=html controller=TemplatesController action=index status=200 duration=512.33 view=256.04 db=68.76 request_id=79e44d3c623e815f7ab806097cc59b13 host=testapp.docspring.co source_ip=10.1.196.72 user_id=1 account_id=1 params={} basic_auth=false

However, the logs are not showing up in the CloudWatch log stream:

How could I debug this further and fix the issue? Thanks!

I’ve tried deleting and re-creating my app, but it looks like that didn’t help. I think this is the last thing I need to fix before I can migrate to the gen 3 rack. @ddollar I was wondering if you might have any idea why my logging is broken? (Works in kubectl, but not appearing in CloudWatch or convox logs output.)

I’ve deleted my old rack and created a new rack from scratch. I’m following the “deploying an application” tutorial and deploying the demo node.js application: Convox Docs

I’m still seeing the same problem with logs. The page is loading fine:

Hello Convox!
I'm: nodejs
Build:BEPRRSMZKZV
Release: RKHBEMWRKSJ
running on: dstest20210918

But convox logs doesn’t show any log messages from the nodejs app.

$ convox logs
2021-09-17T21:02:46Z system/k8s/build-wrxrn Successfully assigned dstest20210918-nodejs/build-wrxrn to ip-10-1-124-208.ec2.internal
2021-09-17T21:02:47Z system/k8s/build-wrxrn Container image "convox/convox:3.0.52" already present on machine
2021-09-17T21:02:48Z system/k8s/build-wrxrn Created container nodejs
2021-09-17T21:02:48Z system/k8s/build-wrxrn Started container nodejs
2021-09-17T21:02:58Z system/k8s/atom/app Status: Running => Pending
2021-09-17T21:02:59Z system/k8s/web Scaled up replica set web-5787f85cff to 1
2021-09-17T21:02:59Z system/k8s/web-5787f85cff Created pod: web-5787f85cff-7cqrk
2021-09-17T21:02:59Z system/k8s/web-5787f85cff-7cqrk Successfully assigned dstest20210918-nodejs/web-5787f85cff-7cqrk to ip-10-1-255-105.ec2.internal
2021-09-17T21:02:59Z system/k8s/atom/app Status: Pending => Updating
2021-09-17T21:03:00Z system/k8s/web-5787f85cff-7cqrk Pulling image "635930426519.dkr.ecr.us-east-1.amazonaws.com/dstest20210918/nodejs:web.BEPRRSMZKZV"
2021-09-17T21:03:05Z system/k8s/web-5787f85cff-7cqrk Successfully pulled image "635930426519.dkr.ecr.us-east-1.amazonaws.com/dstest20210918/nodejs:web.BEPRRSMZKZV"
2021-09-17T21:03:05Z system/k8s/web-5787f85cff-7cqrk Created container nodejs
2021-09-17T21:03:05Z system/k8s/web-5787f85cff-7cqrk Started container nodejs
2021-09-17T21:03:12Z system/k8s/web Scaled down replica set web-6d8c7c554b to 0
2021-09-17T21:03:12Z system/k8s/web-6d8c7c554b Deleted pod: web-6d8c7c554b-nzkxp
2021-09-17T21:03:12Z system/k8s/atom/app Status: Updating => Running
$ convox ps
ID                    SERVICE  STATUS   RELEASE      STARTED        COMMAND
web-5787f85cff-7cqrk  web      running  RKHBEMWRKSJ  3 minutes ago
$ kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
web-5787f85cff-7cqrk   1/1     Running   0          3m27s

$ kubectl logs web-5787f85cff-7cqrk
[21:03:05.677] Server running at 3000/
[21:03:11.467] GET /   IP: ::ffff:10.1.123.123
[21:03:16.471] GET /   IP: ::ffff:10.1.123.123
...

(I added timestamps and IP to the console.log call in app.js.)

CloudWatch has two log groups for the new rack:

  • /convox/dstest20210918/nodejs
  • /convox/dstest20210918/system

Here’s all the CloudWatch logs for /convox/dstest20210918/nodejs:

(I selected all the log streams under /convox/dstest20210918/nodejs, and clicked “Search All”. Then “Actions” => “Copy search results (ASCII)”.)

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   timestamp   |                                                             message                                                             |            logStreamName             |
|---------------|---------------------------------------------------------------------------------------------------------------------------------|--------------------------------------|
| 1631912385000 | Status: Pending => Updating                                                                                                     | system/k8s/atom/app                  |
| 1631912385000 | Status: Updating => Running                                                                                                     | system/k8s/atom/app                  |
| 1631912430000 | Successfully assigned dstest20210918-nodejs/build-mnrgx to ip-10-1-179-88.ec2.internal                                          | system/k8s/build-mnrgx               |
| 1631912431000 | Container image "convox/convox:3.0.52" already present on machine                                                               | system/k8s/build-mnrgx               |
| 1631912431000 | Created container nodejs                                                                                                        | system/k8s/build-mnrgx               |
| 1631912431000 | Started container nodejs                                                                                                        | system/k8s/build-mnrgx               |
| 1631912445000 | Status: Running => Pending                                                                                                      | system/k8s/atom/app                  |
| 1631912446000 | Scaled up replica set web-6d8c7c554b to 1                                                                                       | system/k8s/web                       |
| 1631912446000 | Created pod: web-6d8c7c554b-nzkxp                                                                                               | system/k8s/web-6d8c7c554b            |
| 1631912446000 | Successfully assigned dstest20210918-nodejs/web-6d8c7c554b-nzkxp to ip-10-1-179-88.ec2.internal                                 | system/k8s/web-6d8c7c554b-nzkxp      |
| 1631912447000 | Container image "635930426519.dkr.ecr.us-east-1.amazonaws.com/dstest20210918/nodejs:web.BNGRTJGRNTN" already present on machine | system/k8s/web-6d8c7c554b-nzkxp      |
| 1631912448000 | Created container nodejs                                                                                                        | system/k8s/web-6d8c7c554b-nzkxp      |
| 1631912448000 | Started container nodejs                                                                                                        | system/k8s/web-6d8c7c554b-nzkxp      |
| 1631912448000 | Status: Pending => Updating                                                                                                     | system/k8s/atom/app                  |
| 1631912448000 | Successfully assigned dstest20210918-nodejs/cm-acme-http-solver-p6hwb to ip-10-1-124-208.ec2.internal                           | system/k8s/cm-acme-http-solver-p6hwb |
| 1631912449000 | Successfully assigned dstest20210918-nodejs/cm-acme-http-solver-r9h57 to ip-10-1-124-208.ec2.internal                           | system/k8s/cm-acme-http-solver-r9h57 |
| 1631912450000 | Container image "quay.io/jetstack/cert-manager-acmesolver:v0.12.0" already present on machine                                   | system/k8s/cm-acme-http-solver-p6hwb |
| 1631912450000 | Created container acmesolver                                                                                                    | system/k8s/cm-acme-http-solver-p6hwb |
| 1631912450000 | Started container acmesolver                                                                                                    | system/k8s/cm-acme-http-solver-p6hwb |
| 1631912450000 | Container image "quay.io/jetstack/cert-manager-acmesolver:v0.12.0" already present on machine                                   | system/k8s/cm-acme-http-solver-r9h57 |
| 1631912450000 | Created container acmesolver                                                                                                    | system/k8s/cm-acme-http-solver-r9h57 |
| 1631912450000 | Started container acmesolver                                                                                                    | system/k8s/cm-acme-http-solver-r9h57 |
| 1631912457000 | Status: Updating => Running                                                                                                     | system/k8s/atom/app                  |
| 1631912566000 | Successfully assigned dstest20210918-nodejs/build-wrxrn to ip-10-1-124-208.ec2.internal                                         | system/k8s/build-wrxrn               |
| 1631912567000 | Container image "convox/convox:3.0.52" already present on machine                                                               | system/k8s/build-wrxrn               |
| 1631912568000 | Created container nodejs                                                                                                        | system/k8s/build-wrxrn               |
| 1631912568000 | Started container nodejs                                                                                                        | system/k8s/build-wrxrn               |
| 1631912578000 | Status: Running => Pending                                                                                                      | system/k8s/atom/app                  |
| 1631912579000 | Scaled up replica set web-5787f85cff to 1                                                                                       | system/k8s/web                       |
| 1631912579000 | Created pod: web-5787f85cff-7cqrk                                                                                               | system/k8s/web-5787f85cff            |
| 1631912579000 | Successfully assigned dstest20210918-nodejs/web-5787f85cff-7cqrk to ip-10-1-255-105.ec2.internal                                | system/k8s/web-5787f85cff-7cqrk      |
| 1631912579000 | Status: Pending => Updating                                                                                                     | system/k8s/atom/app                  |
| 1631912580000 | Pulling image "635930426519.dkr.ecr.us-east-1.amazonaws.com/dstest20210918/nodejs:web.BEPRRSMZKZV"                              | system/k8s/web-5787f85cff-7cqrk      |
| 1631912585000 | Successfully pulled image "635930426519.dkr.ecr.us-east-1.amazonaws.com/dstest20210918/nodejs:web.BEPRRSMZKZV"                  | system/k8s/web-5787f85cff-7cqrk      |
| 1631912585000 | Created container nodejs                                                                                                        | system/k8s/web-5787f85cff-7cqrk      |
| 1631912585000 | Started container nodejs                                                                                                        | system/k8s/web-5787f85cff-7cqrk      |
| 1631912592000 | Scaled down replica set web-6d8c7c554b to 0                                                                                     | system/k8s/web                       |
| 1631912592000 | Deleted pod: web-6d8c7c554b-nzkxp                                                                                               | system/k8s/web-6d8c7c554b            |
| 1631912592000 | Status: Updating => Running                                                                                                     | system/k8s/atom/app                  |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Here’s all the CloudWatch logs for /convox/dstest20210918/system:

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   timestamp   |                                                                                  message                                                                                   |               logStreamName               |
|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------|
| 1631911706000 | Readiness probe failed: Get https://10.1.73.255:5443/check: dial tcp 10.1.73.255:5443: connect: connection refused                                                         | system/k8s/api-77f75f74b6-wp9gq           |
| 1631911707000 | Container image "convox/convox:3.0.52" already present on machine                                                                                                          | system/k8s/api-7445fbdb96-cgf9s           |
| 1631911707000 | Created container system                                                                                                                                                   | system/k8s/api-7445fbdb96-cgf9s           |
| 1631911707000 | Started container system                                                                                                                                                   | system/k8s/api-7445fbdb96-cgf9s           |
| 1631911708000 | Liveness probe failed: Get https://10.1.77.36:5443/check: dial tcp 10.1.77.36:5443: connect: connection refused                                                            | system/k8s/api-7445fbdb96-cgf9s           |
| 1631911710000 | Readiness probe failed: Get https://10.1.77.36:5443/check: dial tcp 10.1.77.36:5443: connect: connection refused                                                           | system/k8s/api-7445fbdb96-cgf9s           |
| 1631911710000 | Successfully pulled image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1"                                                                          | system/k8s/ingress-nginx-6c88d5b6db-vxlj5 |
| 1631911711000 | Successfully pulled image "convox/convox:3.0.52"                                                                                                                           | system/k8s/atom-64d8d87b6c-smv8m          |
| 1631911711000 | Created container system                                                                                                                                                   | system/k8s/ingress-nginx-6c88d5b6db-vxlj5 |
| 1631911711000 | Created container system                                                                                                                                                   | system/k8s/atom-64d8d87b6c-smv8m          |
| 1631911711000 | Started container system                                                                                                                                                   | system/k8s/ingress-nginx-6c88d5b6db-vxlj5 |
| 1631911711000 | Started container system                                                                                                                                                   | system/k8s/atom-64d8d87b6c-smv8m          |
| 1631911713000 | Successfully pulled image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1"                                                                          | system/k8s/ingress-nginx-6c88d5b6db-gqjv7 |
| 1631911713000 | Created container system                                                                                                                                                   | system/k8s/ingress-nginx-6c88d5b6db-gqjv7 |
| 1631911713000 | Started container system                                                                                                                                                   | system/k8s/ingress-nginx-6c88d5b6db-gqjv7 |
| 1631911713000 | Successfully pulled image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1"                                                                          | system/k8s/ingress-nginx-6c88d5b6db-gqjv7 |
| 1631911713000 | Created container system                                                                                                                                                   | system/k8s/ingress-nginx-6c88d5b6db-gqjv7 |
| 1631911713000 | Started container system                                                                                                                                                   | system/k8s/ingress-nginx-6c88d5b6db-gqjv7 |
| 1631911716000 | Scaled down replica set api-77f75f74b6 to 1                                                                                                                                | system/k8s/api                            |
| 1631911716000 | Deleted pod: api-77f75f74b6-wp9gq                                                                                                                                          | system/k8s/api-77f75f74b6                 |
| 1631911716000 | Scaled up replica set api-7445fbdb96 to 2                                                                                                                                  | system/k8s/api                            |
| 1631911716000 | Created pod: api-7445fbdb96-g696s                                                                                                                                          | system/k8s/api-7445fbdb96                 |
| 1631911717000 | Successfully assigned dstest20210918-system/api-7445fbdb96-g696s to ip-10-1-255-105.ec2.internal                                                                           | system/k8s/api-7445fbdb96-g696s           |
| 1631911717000 | Container image "convox/convox:3.0.52" already present on machine                                                                                                          | system/k8s/api-7445fbdb96-g696s           |
| 1631911717000 | Created container system                                                                                                                                                   | system/k8s/api-7445fbdb96-g696s           |
| 1631911717000 | Successfully assigned dstest20210918-system/api-7445fbdb96-g696s to ip-10-1-255-105.ec2.internal                                                                           | system/k8s/api-7445fbdb96-g696s           |
| 1631911717000 | Container image "convox/convox:3.0.52" already present on machine                                                                                                          | system/k8s/api-7445fbdb96-g696s           |
| 1631911717000 | Created container system                                                                                                                                                   | system/k8s/api-7445fbdb96-g696s           |
| 1631911718000 | Started container system                                                                                                                                                   | system/k8s/api-7445fbdb96-g696s           |
| 1631911718000 | Started container system                                                                                                                                                   | system/k8s/api-7445fbdb96-g696s           |
| 1631911719000 | Liveness probe failed: Get https://10.1.234.136:5443/check: dial tcp 10.1.234.136:5443: connect: connection refused                                                        | system/k8s/api-7445fbdb96-g696s           |
| 1631911719000 | Liveness probe failed: Get https://10.1.234.136:5443/check: dial tcp 10.1.234.136:5443: connect: connection refused                                                        | system/k8s/api-7445fbdb96-g696s           |
| 1631911724000 | Scaled down replica set api-77f75f74b6 to 0                                                                                                                                | system/k8s/api                            |
| 1631911724000 | Deleted pod: api-77f75f74b6-27cb7                                                                                                                                          | system/k8s/api-77f75f74b6                 |
| 1631911736000 | unable to get metrics for resource cpu: no metrics returned from resource metrics API                                                                                      | system/k8s/nginx                          |
| 1631911736000 | invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | system/k8s/nginx                          |
| 1631911767000 | did not receive metrics for any ready pods                                                                                                                                 | system/k8s/nginx                          |
| 1631911767000 | invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: did not receive metrics for any ready pods                                            | system/k8s/nginx                          |
| 1631911859000 | Successfully assigned dstest20210918-system/cm-acme-http-solver-fffjp to ip-10-1-124-208.ec2.internal                                                                      | system/k8s/cm-acme-http-solver-fffjp      |
| 1631911860000 | Pulling image "quay.io/jetstack/cert-manager-acmesolver:v0.12.0"                                                                                                           | system/k8s/cm-acme-http-solver-fffjp      |
| 1631911862000 | Successfully pulled image "quay.io/jetstack/cert-manager-acmesolver:v0.12.0"                                                                                               | system/k8s/cm-acme-http-solver-fffjp      |
| 1631911862000 | Created container acmesolver                                                                                                                                               | system/k8s/cm-acme-http-solver-fffjp      |
| 1631911862000 | Started container acmesolver                                                                                                                                               | system/k8s/cm-acme-http-solver-fffjp      |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

I’m getting somewhere! It’s been a good learning experience. I figured out how to show all pods across all namespaces with kubectl:

$ kubectl get pods -A
NAMESPACE               NAME                                       READY   STATUS             RESTARTS   AGE
cert-manager            cert-manager-5c47f46f57-86bjd              1/1     Running            0          52m
cert-manager            cert-manager-cainjector-6659d6844d-l26bw   1/1     Running            0          52m
cert-manager            cert-manager-webhook-547567b88f-jhhtl      1/1     Running            0          52m
dstest20210918-nodejs   web-5787f85cff-7cqrk                       1/1     Running            0          38m
dstest20210918-system   api-7445fbdb96-cgf9s                       1/1     Running            0          52m
dstest20210918-system   api-7445fbdb96-g696s                       1/1     Running            0          52m
dstest20210918-system   atom-64d8d87b6c-smv8m                      1/1     Running            0          53m
dstest20210918-system   ingress-nginx-6c88d5b6db-gqjv7             1/1     Running            0          52m
dstest20210918-system   ingress-nginx-6c88d5b6db-vxlj5             1/1     Running            0          53m
dstest20210918-system   resolver-68df959567-gxbjt                  1/1     Running            0          53m
dstest20210918-system   resolver-68df959567-wp2vh                  1/1     Running            0          53m
kube-system             aws-node-4xq89                             1/1     Running            0          54m
kube-system             aws-node-9c6qc                             1/1     Running            0          54m
kube-system             aws-node-z8mv5                             1/1     Running            0          54m
kube-system             cluster-autoscaler-5889899c8c-pm2xk        1/1     Running            0          53m
kube-system             coredns-75b44cb5b4-dtrzg                   1/1     Running            0          57m
kube-system             coredns-75b44cb5b4-fthds                   1/1     Running            0          57m
kube-system             fluentd-4vkj5                              0/1     CrashLoopBackOff   15         53m
kube-system             fluentd-pz96k                              0/1     CrashLoopBackOff   15         53m
kube-system             fluentd-tfszw                              0/1     CrashLoopBackOff   15         53m
kube-system             kube-proxy-2kcb9                           1/1     Running            0          54m
kube-system             kube-proxy-bq2g9                           1/1     Running            0          54m
kube-system             kube-proxy-mgqnk                           1/1     Running            0          54m
kube-system             metrics-server-579f9cff58-x7sgq            1/1     Running            0          53m

So I can see that the fluentd pods are stuck in the CrashLoopBackOff state, which is why fluentd isn’t forwarding any logs to CloudWatch.

$ kubectl logs fluentd-4vkj5 --namespace kube-system
2021-09-17 21:40:53 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2021-09-17 21:40:53 +0000 [warn]: 'log_level' is deprecated parameter name. use '@log_level' instead.
2021-09-17 21:40:53 +0000 [warn]: 'log_level' is deprecated parameter name. use '@log_level' instead.
2021-09-17 21:40:53 +0000 [info]: adding rewrite_tag_filter rule: $.kubernetes.labels.system [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00007f9f503f9958 @keys=["kubernetes", "labels", "system"], @last_key="system", @dig_keys=["kubernetes", "labels"]>, /^convox$/, "", "convox"]
2021-09-17 21:40:53 +0000 [info]: adding rewrite_tag_filter rule: $.kubernetes.labels.service [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00007f9f51562370 @keys=["kubernetes", "labels", "service"], @last_key="service", @dig_keys=["kubernetes", "labels"]>, /^(.+)$/, "", "service.$1"]
2021-09-17 21:40:53 +0000 [info]: adding rewrite_tag_filter rule: $.kubernetes.namespace_labels.app [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00007f9f4fd153f8 @keys=["kubernetes", "namespace_labels", "app"], @last_key="app", @dig_keys=["kubernetes", "namespace_labels"]>, /^(.+)$/, "", "app.$1.${tag}"]
2021-09-17 21:40:53 +0000 [info]: adding rewrite_tag_filter rule: $.kubernetes.namespace_labels.rack [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00007f9f4fd3cb88 @keys=["kubernetes", "namespace_labels", "rack"], @last_key="rack", @dig_keys=["kubernetes", "namespace_labels"]>, /^(.+)$/, "", "rack.$1.${tag}"]
2021-09-17 21:40:53 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Unknown parser plugin 'cri'. Run 'gem search -rd fluent-plugin' to find plugins"

So fluentd is crashing with a Fluent::ConfigError: Unknown parser plugin 'cri'

Thanks for the PR Nathan. We are working on this one also. It requires an update to the fluentd config as well but we haven’t gotten to the bottom of it yet.

Here’s what we know so far:
cri is the log parser for the containerd runtime(now shipped with new k8s versions instead of docker) and that runtime has a different log format.What we’ll need to do is

  • add the cri parser to the fluentd image
  • add multi parsers on the fluentd config to try cri format and json(docker format) format.

Which would look something like:

        <parse>
            @type multi_format
            <pattern>
                format cri
                time_format %Y-%m-%dT%H:%M:%S.%NZ
            </pattern>
            <pattern>
                format json
                time_format %Y-%m-%dT%H:%M:%S.%NZ
            </pattern>
        </parse>

As a immediate workaround you can try to manually change the fluentd config in kubernetes. It’s a one word change (to use the json formatter instead of cri)

1 Like

Awesome, thanks for your help! Changing @type cri to @type json fixed the issue, and now I’m seeing all my logs in CloudWatch and the convox logs output.

Here’s some tips for anyone else who comes across this (or a similar k8s issue). The first step is to get kubectl working: Convox Docs

$ convox switch myrack
$ convox rack kubeconfig > $HOME/.kube/myrack-config
$ export KUBECONFIG=$HOME/.kube/myrack-config

Then here’s how you can view all resources across all namespaces: kubectl get all -A. You’ll see all your pods, services, daemonsets, deployments, and replicasets. (A good crash course in how k8s works.)

Set your editor for kubectl:

  • VS Code: export KUBE_EDITOR='code --wait'
  • Vim: export KUBE_EDITOR='vim'

Edit the fluentd configmap:

$ kubectl edit configmap -n kube-system fluentd -o yaml

Under data => containers.conf, change @type cri to @type json. Then save and close the file, and it will updated in your cluster.

Finally, you need to restart the fluentd daemonset to use the new config:

kubectl rollout restart daemonset fluentd -n kube-system

Run kubectl -n kube-system get pods to see all the pods:

$ kubectl -n kube-system get pods
NAME                                  READY   STATUS    RESTARTS   AGE
aws-node-4xq89                        1/1     Running   0          147m
aws-node-9c6qc                        1/1     Running   0          147m
aws-node-z8mv5                        1/1     Running   0          148m
cluster-autoscaler-5889899c8c-pm2xk   1/1     Running   0          146m
coredns-75b44cb5b4-dtrzg              1/1     Running   0          151m
coredns-75b44cb5b4-fthds              1/1     Running   0          151m
fluentd-rs9rl                         1/1     Running   0          21m
fluentd-s7j2r                         1/1     Running   0          21m
fluentd-tjv28                         1/1     Running   0          21m
kube-proxy-2kcb9                      1/1     Running   0          147m
kube-proxy-bq2g9                      1/1     Running   0          148m
kube-proxy-mgqnk                      1/1     Running   0          147m
metrics-server-579f9cff58-x7sgq       1/1     Running   0          146m

You should see the fluentd pods are Running instead of crashed.

Check the logs:

$ kubectl -n kube-system logs fluentd-rs9rl

You should see some lines like this:

2021-09-17 22:56:54 +0000 [info]: #0 [Aws::CloudWatchLogs::Client 200 0.043858 0 retries] put_log_events(...

Also now you should see logs appearing in CloudWatch, and in the output of convox logs.

2 Likes

We have done a release to address this:

1 Like