I have a kubernetes cluster and a PostgreSQL Database running on Google Cloud.

The pod that has the problem is a cronjob with the following configuration:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: taxiq-cronjob-reminder
  annotations:
    cloud.google/neg: '{"ingress": true}'
    cloud.google/backend-config: '{"default": "taxiq-healthconfig"}'
spec:
  schedule: "31 4 * * *"
  timeZone: "Europe/Berlin"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: taxiq-stage-gke
          initContainers:
            - image:  gcr.io/google/cloudsdktool/cloud-sdk:326.0.0-alpine
              name: workload-identity-initcontainer
              command:
              - '/bin/bash'
              - '-c'
              - "curl -s -H 'Metadata-Flavor: Google' 'http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token' --retry 30 --retry-connrefused --retry-max-time 30 > /dev/null || exit 1"
          containers:
          - name: cloud-sql-proxy
            image: gcr.io/cloudsql-docker/gce-proxy:1.33.5
            command:
              - "/cloud_sql_proxy"
              - "-instances=taxiq-stage-app:europe-west3:taxiq-stage=tcp:5432"
            securityContext:
              runAsNonRoot: true
          - name: taxiq-cronjob-reminder
            image: europe-west3-docker.pkg.dev/brantpoint-artifacts/taxiq/cronjob-reminder:beta09.7
            env:
              - name: PGURL
                valueFrom:
                  secretKeyRef:
                    name: secrets
                    key: PGURL
          restartPolicy: Never

I expect the proxy to start as well as the cronjob to run without errors.

But I get the following error:

failed to connect to `host=127.0.0.1 user=masterchief database=tasks_buildingblock`: dial error (dial tcp 127.0.0.1:5432: connect: connection refused)

I know this error is related to the cloud-sql-proxy not running (yet). I had removed it with restartPolicy: OnFailure.

But I can not use this restartPolicy value for the following reason:

The cronjob is supposed to send mails -> if I get a Failure for any other reason after some mails have already been sent, the cronjob will run again and send the mails multiple times, which might make customers/users unhappy

How can I ensure the cloud-sql-proxy is listening before the cronjob starts?

I have a kubernetes cluster and a PostgreSQL Database running on Google Cloud.

The pod that has the problem is a cronjob with the following configuration:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: taxiq-cronjob-reminder
  annotations:
    cloud.google/neg: '{"ingress": true}'
    cloud.google/backend-config: '{"default": "taxiq-healthconfig"}'
spec:
  schedule: "31 4 * * *"
  timeZone: "Europe/Berlin"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: taxiq-stage-gke
          initContainers:
            - image:  gcr.io/google/cloudsdktool/cloud-sdk:326.0.0-alpine
              name: workload-identity-initcontainer
              command:
              - '/bin/bash'
              - '-c'
              - "curl -s -H 'Metadata-Flavor: Google' 'http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token' --retry 30 --retry-connrefused --retry-max-time 30 > /dev/null || exit 1"
          containers:
          - name: cloud-sql-proxy
            image: gcr.io/cloudsql-docker/gce-proxy:1.33.5
            command:
              - "/cloud_sql_proxy"
              - "-instances=taxiq-stage-app:europe-west3:taxiq-stage=tcp:5432"
            securityContext:
              runAsNonRoot: true
          - name: taxiq-cronjob-reminder
            image: europe-west3-docker.pkg.dev/brantpoint-artifacts/taxiq/cronjob-reminder:beta09.7
            env:
              - name: PGURL
                valueFrom:
                  secretKeyRef:
                    name: secrets
                    key: PGURL
          restartPolicy: Never

I expect the proxy to start as well as the cronjob to run without errors.

But I get the following error:

failed to connect to `host=127.0.0.1 user=masterchief database=tasks_buildingblock`: dial error (dial tcp 127.0.0.1:5432: connect: connection refused)

I know this error is related to the cloud-sql-proxy not running (yet). I had removed it with restartPolicy: OnFailure.

But I can not use this restartPolicy value for the following reason:

The cronjob is supposed to send mails -> if I get a Failure for any other reason after some mails have already been sent, the cronjob will run again and send the mails multiple times, which might make customers/users unhappy

How can I ensure the cloud-sql-proxy is listening before the cronjob starts?

• kubernetes
• google-kubernetes-engine
• kubernetes-cronjob
• cloud-sql-proxy

Share Improve this question Follow asked Mar 23 at 4:17 code4Godcode4God 1355 bronze badges 2

• 2 You want a sidecar container. Move your sql-proxy container to initContainers and give it the "Always" restart policy so it starts before your main container and keeps running. – Botje Commented Mar 23 at 8:31
• 2 This question is similar to: "Sidecar" containers in Kubernetes pods. If you believe it’s different, please edit the question, make it clear how it’s different and/or how the answers on that question are not helpful for your problem. – Botje Commented Mar 23 at 8:44

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 1

A mentioned in one of the comments, deploying the Cloud SQL Proxy using initContainers is the solution to start the Proxy as a native sidecar.

Also worth pointing out that you are using the old v1 Cloud SQL Proxy. It is recommended to migrate to the new v2 Cloud SQL Proxy to leverage both performance and reliability benefits.

There are examples of using the v2 Cloud SQL Proxy as a sidecar here and example health check usage here.

Your sample updated would look like the following:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: taxiq-cronjob-reminder
  annotations:
    cloud.google/neg: '{"ingress": true}'
    cloud.google/backend-config: '{"default": "taxiq-healthconfig"}'
spec:
  schedule: "31 4 * * *"
  timeZone: "Europe/Berlin"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: taxiq-stage-gke
          initContainers:
            - image:  gcr.io/google/cloudsdktool/cloud-sdk:326.0.0-alpine
              name: workload-identity-initcontainer
              command:
              - '/bin/bash'
              - '-c'
              - "curl -s -H 'Metadata-Flavor: Google' 'http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token' --retry 30 --retry-connrefused --retry-max-time 30 > /dev/null || exit 1"
            - name: cloud-sql-proxy
              # v2 Cloud SQL Proxy image
              image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.15.2
              restartPolicy: Always
              env:
                - name: CSQL_PROXY_HEALTH_CHECK
                  value: "true"
                - name: CSQL_PROXY_HTTP_PORT
                  value: "9801"
                - name: CSQL_PROXY_HTTP_ADDRESS
                  value: 0.0.0.0
              startupProbe:
                failureThreshold: 60
                httpGet:
                  path: /startup
                  port: 9801
                  scheme: HTTP
                periodSeconds: 1
                successThreshold: 1
                timeoutSeconds: 10
              args:
                - "--port=5432"
                - "taxiq-stage-app:europe-west3:taxiq-stage"
              securityContext:
                runAsNonRoot: true
          containers:
          - name: taxiq-cronjob-reminder
            image: europe-west3-docker.pkg.dev/brantpoint-artifacts/taxiq/cronjob-reminder:beta09.7
            env:
              - name: PGURL
                valueFrom:
                  secretKeyRef:
                    name: secrets
                    key: PGURL
          restartPolicy: Never