Tripping over k8s

🍅 1 - Jumping right in

We’re using Rundeck to manage operations tasks, and we’ve chosen Kubernetes as a deployment platform. The reasons for this are not relevant to this post, so we have to take them for granted for now. Other pieces of tooling that we should take for granted are Jenkins for the continuous integration pipeline, Helm for the packaging model and Spinnaker for the continuous deployment pipeline.

I’m writing this post in order to reason about and hopefully resolve issues encountered in writing the helm chart. I will also be using the pomodoro method to keep pace and try to measure my progress, but mostly because I have this cool Sense HAT which counts down the time for me in a naff LED matrix.

I’m coming back to this deployment after a few days concentrating on other things. This morning, I made two simple changes to the chart metadata, which broke the deployment, even though the build has been passing for several months:

Strangely enough, even though there are builds going back to May, only the last deploy is showing. The manifest seems to have baked properly, it is correct YAML - but the deploy failed. The error message from Spinnaker is quite long, but this caught my eye:

"status": map[
  "availableReplicas":'\x01',
  "conditions": [
    map[
      "lastTransitionTime": "2020-05-26T08:33:20Z",
      "lastUpdateTime": "2020-05-26T08:33:20Z",
      "message": "Deployment has minimum availability.",
      "reason":"MinimumReplicasAvailable",
      "status":"True",
      "type":"Available"],
    map[
      "lastTransitionTime": "2020-05-25T17:28:37Z",
      "lastUpdateTime": "2020-05-26T08:33:20Z",
      "message": "ReplicaSet "infra-rundeck-7558cb87b8" has successfully progressed.",
      "reason": "NewReplicaSetAvailable" "status": "True" "type":"Progressing"
    ]
  ],
  "observedGeneration": '\x11',
  "readyReplicas": '\x01',
  "replicas": '\x01',
  "updatedReplicas": '\x01'
]
]
}
for: "STDIN": unrecognized type: string

… I have no idea what this means.

On the one hand, it seems like the replicas are ok, but there is some weird problem with the data set in the configMap. There seems to be some encoded data returned.

Let’s see what describe-pod says:

RUNDECK_FEATURE_REPOSITORY_ENABLED: <empty> 🤔 perhaps this is the issue? It’s set in values-nonprod.yaml as:

RUNDECK_FEATURE_REPOSITORY_ENABLED: "true"

Perhaps I need to let booleans be booleans?

🍅 2: Moving along

The next attempt to understand what is going on is to make small changes and see how the deployments go. First, we update the deployment to pass a list of strings to the entrypoint of the main container. Result? A big fat nope.

Next, we let booleans be booleans…

Success, of a sort!

Deploy failed: Error from server (BadRequest):

error when creating "STDIN":
  ConfigMap in version "v1" cannot be handled as a ConfigMap:
    v1.ConfigMap.Data: ReadString: expects " or n, but found t, error found in #10 byte of ...|ENABLED":true,"RUNDE|...,
    bigger context
     ...|":"rundeck","RUNDECK_FEATURE_REPOSITORY_ENABLED":true,"RUNDECK_FEATURE_REPOSITORY_SYNCONBOOTSTRAP":t|...

Ok, seems everything needs to be quoted. How does helm lint not catch these things though?

🍅 3: Let bools be strings

Searching around more for the error:

ConfigMap in version "v1" cannot be handled as a ConfigMap

I found this issue in helm, which points out:

I was able to make it work (fix) by using –set-string. For example: --set-string controller.config.ssl-redirect=false

While I didn’t use this workaround, it did give me the idea of setting all variables as strings instead of bare booleans. When I did this, the creation of the resources worked! Now, we have a continuous delivery pipeline through to the preproduction environment via GitHub, Jenkins and Spinnaker.

Next, why are logs not getting sent to S3?

🍅 4: Setting secrets properly

We want to send logs of the jobs to S3, using the rundeck S3 log plugin. Following the documentation, I added the JAR and config:

1. Ensure rundeck-s3-log-plugin jar is present
2. Enable the ExecutionFileStorage provider
3. We’re missing the framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.path property - but the default value (rundeck/project/$PROJECT/logs/$ID) should work.

Credentials are consumed from the environment, but they seem to be incorrect, since I get access denied errors:

ERROR S3LogFileStoragePlugin ---
  [ileStorageTask2] Access Denied
  (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; ... )

Perhaps there is something wrong with the bucket?

Nope, there’s nothing wrong with the bucket, I can write stuff to the bucket with the same credentials that should be passed to the container’s environment.

Stop here.