Convox Community

Strange bug on upgrading from Rails 5.1

Hi all,

I’ve recently been attempting to upgrade our app from Rails 5.1 to 6.0.0. In the development environment everything (eventually) worked ok, and the tests passed on Circle CI, but then hit a pernicious bug when I came to deploy it to our staging environment:
Missing encryption key to decrypt file with. Ask your team for your master key and write it to /app/config/master.key or put it in the ENV['RAILS_MASTER_KEY'].

I had not originally touched the previous credentials, which were set in the RAILS_MASTER_KEY env var on both CircleCI and Convox, and which were working throughout before the upgrade from 5.1. I have quadruple-checked that the variable is present on both, and tried refreshing it multiple times. I feel like this has to be either a CircleCI or a Convox issue, but I can’t tell which - the tests run fine on Circle, which seems to prove that it’s picking up the env var there (and therefore ‘must’ be a Convox issue), but running the exact same command line command from my local terminal (convox deploy --app founderspledge-staging) deploys it ok, which seems to ‘prove’ it’s a Circle issue.

I really have no idea how to proceed here, so I’ll cross post this on the Convox and Circle forums, and see if anyone can identify what’s happening. Contents of various plausibly relevant files follow:

Dockerfile
FROM ruby:2.6.5-alpine

RUN apk add --update --no-cache \
    build-base \
    nodejs \
    tzdata \
    libxml2-dev \
    libxslt-dev \
    imagemagick \
    git \
    postgresql-client \
    postgresql-dev \
    yarn

EXPOSE 3000

WORKDIR /app

RUN yarn install

# Cache bundler
COPY Gemfile /app/Gemfile
COPY Gemfile.lock /app/Gemfile.lock
RUN bundle install --without development test --jobs 4 --retry 3

# Copy the rest of the app
COPY . /app

ENV RAILS_ENV production
ENV DEVISE_SECRET_KEY dummy_secret
ENV DATABASE_URL postgresql://localhost/dummy_url
ENV S3_BUCKET_NAME foo
ENV AWS_ACCESS_KEY_ID foo
ENV AWS_SECRET_ACCESS_KEY foo
ENV AWS_REGION eu-west-1
ENV APP_DOMAIN foo.com
ENV ASSET_HOST foo.com
RUN bundle exec rake assets:precompile --trace

.circleci/config.yml

This configuration was automatically generated from a CircleCI 1.0 config.

# It should include any build commands you had along with commands that CircleCI
# inferred from your project structure. We strongly recommend you read all the
# comments in this file to understand the structure of CircleCI 2.0, as the idiom
# for configuration has changed substantially in 2.0 to allow arbitrary jobs rather
# than the prescribed lifecycle of 1.0. In general, we recommend using this generated
# configuration as a reference rather than using it in production, though in most
# cases it should duplicate the execution of your original 1.0 config.
version: 2
jobs:
  build:
    working_directory: ~/FoundersPledge/founders-pledge
    parallelism: 1
    shell: /bin/bash --login
    # CircleCI 2.0 does not support environment variables that refer to each other the same way as 1.0 did.
    # If any of these refer to each other, rewrite them so that they don't or see https://circleci.com/docs/2.0/env-vars/#interpolating-environment-variables-to-set-other-environment-variables .
    environment:
      CIRCLE_ARTIFACTS: /tmp/circleci-artifacts
      CIRCLE_TEST_REPORTS: /tmp/circleci-test-results
    # In CircleCI 1.0 we used a pre-configured image with a large number of languages and other packages.
    # In CircleCI 2.0 you can now specify your own image, or use one of our pre-configured images.
    # The following configuration line tells CircleCI to use the specified docker image as the runtime environment for you job.
    # We have selected a pre-built image that mirrors the build environment we use on
    # the 1.0 platform, but we recommend you choose an image more tailored to the needs
    # of each job. For more information on choosing an image (or alternatively using a
    # VM instead of a container) see https://circleci.com/docs/2.0/executor-types/
    # To see the list of pre-built images that CircleCI provides for most common languages see
    # https://circleci.com/docs/2.0/circleci-images/
    docker:
    # SC- See https://circleci.com/docs/2.0/circleci-images/
    # - image: circleci/ruby:2.3-browsers
    - image: circleci/ruby:2.6.5-stretch-node-browsers
      environment:
        PGHOST: 127.0.0.1
        PGUSER: ubuntu
        RAILS_ENV: test
        POSTGRES_USER: ubuntu
    - image: circleci/postgres:9.6.9-alpine-ram
      environment:
        POSTGRES_USER: ubuntu
        POSTGRES_DB: circleci_test
        POSTGRES_PASSWORD: ""
      # environment:
      #   PGHOST: 127.0.0.1
    # - image: circleci/postgres:9.6-alpine-ram
    # - image: circleci/ruby:2.3

    # - image: circleci/build-image:ubuntu-14.04-XXL-upstart-1189-5614f37
      # command: /sbin/init
    steps:
    # Machine Setup
    #   If you break your build into multiple jobs with workflows, you will probably want to do the parts of this that are relevant in each
    # The following `checkout` command checks out your code to your working directory. In 1.0 we did this implicitly. In 2.0 you can choose where in the course of a job your code should be checked out.
    - checkout
    # Prepare for artifact and test results  collection equivalent to how it was done on 1.0.
    # In many cases you can simplify this from what is generated here.
    # 'See docs on artifact collection here https://circleci.com/docs/2.0/artifacts/'

    - run: mkdir -p $CIRCLE_ARTIFACTS $CIRCLE_TEST_REPORTS
    # Dependencies
    #   This would typically go in either a build or a build-and-test job when using workflows
    # Restore the dependency cache
    - restore_cache:
        keys:
        # This branch if available
        - v1-dep-{{ .Branch }}-
        # Default branch if not
        - v1-dep-master-
        # Any branch if there are none on the default branch - this should be unnecessary if you have your default branch configured correctly
        - v1-dep-
    # The following line was run implicitly in your 1.0 builds based on what CircleCI inferred about the structure of your project. In 2.0 you need to be explicit about which commands should be run. In some cases you can discard inferred commands if they are not relevant to your project.
    - run: echo -e "export RAILS_ENV=test\nexport RACK_ENV=test" >> $BASH_ENV
    - run: 'bundle check --path=vendor/bundle || bundle install --path=vendor/bundle
        --jobs=4 --retry=3 '
    # This is based on your 1.0 configuration file or project settings
    - run: curl -Ls https://install.convox.com/linux.zip > convox.zip
    - run: sudo unzip convox.zip -d /usr/local/bin
    # Save dependency cache
    - save_cache:
        key: v1-dep-{{ .Branch }}-{{ epoch }}
        paths:
        # This is a broad list of cache paths to include many possible development environments
        # You can probably delete some of these entries
        - vendor/bundle
        - ~/virtualenvs
        - ~/.m2
        - ~/.ivy2
        - ~/.bundle
        - ~/.go_workspace
        - ~/.gradle
        - ~/.cache/bower
    # The following line was run implicitly in your 1.0 builds based on what CircleCI inferred about the structure of your project. In 2.0 you need to be explicit about which commands should be run. In some cases you can discard inferred commands if they are not relevant to your project.
    - run: |-
        mkdir -p config && echo 'test:
          username: ubuntu
          database: circle_ruby_test
          adapter: postgresql
          encoding: unicode
          pool: 5
          host: localhost
        ' > config/database.yml
    - run:
        command: bundle exec rake db:create db:schema:load --trace
        environment:
          RAILS_ENV: test
          RACK_ENV: test
    # Test
    #   This would typically be a build job when using workflows, possibly combined with build
    # The following line was run implicitly in your 1.0 builds based on what CircleCI inferred about the structure of your project. In 2.0 you need to be explicit about which commands should be run. In some cases you can discard inferred commands if they are not relevant to your project.
    - run: mkdir -p $CIRCLE_TEST_REPORTS/rspec
    - run:
        name: Update Chromedriver
        command: |
          sudo apt-get update
          sudo apt-get install lsb-release libappindicator3-1
          curl -L -o google-chrome.deb https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
          sudo dpkg -i google-chrome.deb
          sudo sed -i 's|HERE/chrome"|HERE/chrome" --no-sandbox|g' /opt/google/chrome/google-chrome
          rm google-chrome.deb
    - run:
        command: bundle exec rspec --color --require spec_helper --format RspecJunitFormatter --out $CIRCLE_TEST_REPORTS/rspec/rspec.xml --format progress spec
        environment:
          RAILS_ENV: test
          RACK_ENV: test
    - run: mkdir -p $CIRCLE_TEST_REPORTS/cucumber
    - run:
        command: 'bundle exec cucumber --format json --out $CIRCLE_TEST_REPORTS/cucumber/cucumber.cucumber '
        environment:
          RAILS_ENV: test
          RACK_ENV: test
    # This is based on your 1.0 configuration file or project settings
    - run: bundle exec bin/circle
    # Deployment
    # Your existing circle.yml file contains deployment steps.
    # The config translation tool does not support translating deployment steps
    # since deployment in CircleCI 2.0 are better handled through workflows.
    # See the documentation for more information https://circleci.com/docs/2.0/workflows/
    # Teardown
    #   If you break your build into multiple jobs with workflows, you will probably want to do the parts of this that are relevant in each
    # Save test results
    - store_test_results:
        path: /tmp/circleci-test-results
    # Save artifacts
    - store_artifacts:
        path: /tmp/circleci-artifacts
    - store_artifacts:
        path: /tmp/circleci-test-results
  staging_deploy:
    docker:
      - image: circleci/ruby:2.6.5-stretch-node-browsers
    steps:
      - checkout
      - run: curl -L https://convox.com/cli/linux/convox -o /tmp/convox
      - run: sudo mv /tmp/convox /usr/local/bin/convox
      - run: sudo chmod 755 /usr/local/bin/convox
      - run: sudo convox update
      - run: convox login console.convox.com
      - run: convox switch founders-pledge/founderspledge
      - run: convox deploy --app founderspledge-staging
      - run: convox run worker rake db:migrate --app founderspledge-staging
  production_deploy:
    docker:
      - image: circleci/ruby:2.6.5-stretch-node-browsers
    steps:
      - checkout
      - run: curl -L https://convox.com/cli/linux/convox -o /tmp/convox
      - run: sudo mv /tmp/convox /usr/local/bin/convox
      - run: sudo chmod 755 /usr/local/bin/convox
      - run: sudo convox update
      - run: convox login console.convox.com
      - run: convox switch founders-pledge/founderspledge
      - run: convox deploy --app founderspledge-production
      - run: convox run worker rake db:migrate --app founderspledge-production


workflows:
  version: 2
  test_and_deploy:
    jobs:
      - build
      - staging_deploy:
          requires:
            - build
          filters:
            branches:
              only: staging
      - production_deploy:
          requires:
            - build
          filters:
            branches:
              only: production

.gitignore
# See https://help.github.com/articles/ignoring-files for more about ignoring files.
#
# If you find yourself ignoring temporary files generated by your text editor
# or operating system, you probably want to add a global ignore instead:
# git config --global core.excludesfile ‘~/.gitignore_global’

# Ignore bundler config.
/.bundle

# Ignore all logfiles and tempfiles.
/log/*
!/log/.keep
/tmp
# !/tmp/mail-merges

# Enviroment files
.env

# Coverage
/coverage

#Upload folder
/public/uploads

dump.rdb

.DS_Store


#
/helper-scripts/*.csv
/helper-scripts/*.xlsx
/helper-scripts/node_modules/

# Ignore master key for decrypting credentials and more.
/config/master.key
# /public/packs
# /public/packs-test
/node_modules
yarn-debug.log*
.yarn-integrity

/public/packs
# /public/packs-test
/node_modules
/yarn-error.log
yarn-debug.log*
.yarn-integrity

convox-start.yml
web: &web
build: .
command: bundle exec puma -C config/puma.rb
environment:
- CONTENTFUL_API_KEY
- CONTENTFUL_SPACE
- DEVISE_SECRET_KEY
- HELLOSIGN_API_KEY
- RACK_ENV=production
- SECRET_KEY_BASE
- SENTRY_DSN
ports:
- 80:3000
- 443:3000
links:
- database
- redis
worker:
<<: *web
ports: []
command: bundle exec sidekiq -C config/sidekiq.yml
redis:
image: convox/redis
ports:
- 6379
database:
image: convox/postgres
ports:
- 5432

docker-compose.yml
web: &web
build: .
command: bundle exec puma -C config/puma.rb
environment:
- CONTENTFUL_API_KEY
- CONTENTFUL_SPACE
- DEVISE_SECRET_KEY
- HELLOSIGN_API_KEY
- RACK_ENV=production
- SECRET_KEY_BASE
- SENTRY_DSN
- ASSET_HOST
- RAILS_MASTER_KEY
labels:
- convox.port.443.protocol=https
- convox.deployment.minimum=50
ports:
- 80:3000
- 443:3000
worker:
<<: *web
ports: []
command: bundle exec sidekiq -C config/sidekiq.yml -e production

Hello,

it seems that your convox.yml is missing the RAILS_MASTER_KEY declaration in the environment section? Convox will only propagate env variables that are declared there. Or you can also use a wildcard to get all env variables automatically propagated:

environment:
  - "*"

Hi Crohr, thanks for the reply (and sorry about the messy formatting on the OP, which I hadn’t noticed until now and no longer seem able to edit). I assume by convox.yml you mean convox-start.yml?

I tried editing that so it consists of just the wildcard (see contents at bottom), but I’m still getting an identical error.

I’m confused by why this would only have happened when I upgraded Rails, given that I hadn’t made any changes to the file in the process - if the ENV vars aren’t being picked up now, it seems like they shouldn’t in theory have been picked up before? We set several ENV vars in the Racks screen on Convox, and they’ve always worked, at least until now. TBH the config was created before I worked on the project, and I never fully understood its role - could it be getting overwritten by something else?

The new contents of convox-start.yml:

web: &web
  build: .
  command: bundle exec puma -C config/puma.rb
  environment:
    - "*"
  ports:
    - 80:3000
    - 443:3000
  links:
    - database
    - redis
worker:
  <<: *web
  ports: []
  command: bundle exec sidekiq -C config/sidekiq.yml
redis:
  image: convox/redis
  ports:
    - 6379
database:
  image: convox/postgres
  ports:
    - 5432