This morning the Convox console started reporting a FAILED status for workflows while they are still in progress – well before the process actually terminates.
I first see this in the Build step while the build is still processing. I can pull up live logs from the CLI and see that the build is still in progress, with no errors reported yet:
$ convox builds logs BUILD_ID <build output streams>
When the workflow eventually finishes, the console updates the status from FAILED to COMPLETE as expected.
In my case, the FAILED status immediately triggers a Slack notification reporting the failure, which also immediately sets off some heart palpitations.
Did something recently change on the Console side?