After a break we are back with another post about Kubernetes. This time we will focus on how to pass configuration to application running in K8s cluster. But first, to have some playground, we are going to build docker image with test application and deploy it to K8s cluster.
Image repository
To make images available to all cluster's nodes we create private docker repository running on master node:
docker run -d -p 5000:5000 --restart=always --name registry registry:2
On each of worker nodes we need to update /etc/docker/daemon.json with new repository definition:
"insecure-registries" : [ "k8admin:5000" ]
and then restart docker.
Docker image
We will use simple node.js script acting as cloud service. Let's create base.Dockerfile:
FROM node:15-alpine
WORKDIR /app
RUN npm install --production
RUN npm install express
RUN apk --no-cache add curl
This defines our base image: node.js with express framework on top of Alpine Linux, with extra curl for connectivity diagnostics. To build the image we execute command:
docker build --build-arg https_proxy=http://proxy.yoursite.local:8080 -f base.Dockerfile -t k8admin:5000/blog-base .
This runs image build and tags it with private repository address. That way the image will go to our repo instead of central Docker registry:
docker push k8admin:5000/blog-base
Let's define simple application image on top of base image:
FROM k8admin:5000/blog-base:latest
ARG SRCDIR=.
COPY $SRCDIR/index.js .
and the execute commands:
docker build --build-arg https_proxy=http://proxy.yoursite.local:8080 --build-arg SRCDIR=$1 -t k8admin:5000/blog-app.
docker push k8admin:5000/blog-app
blog-app image gets index.js from given directory. Simplest version of index.js is to listen on port 3000 and send process environment in return to http GET request:
const express = require('express')
const os = require('os')
function printObject(o) {
let out = '';
for (let p in o) {
out += p + ': ' + o[p] + '\n';
}
return out;
}
const app = express()
app.get('/', (req, res) => {
let r = 'Environment of ' + os.hostname + ':\n';
r += printObject(process.env)
res.send(r)
})
const port = process.env.BLOG_APP_SVC_SERVICE_PORT
app.listen(port, () => console.log(`listening on port ${port}`))
Deployment
Next - to make our app to the Kubernetes - we define deployment (dep.yml):
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
name: blog-app
name: blog-depl
namespace: blog
spec:
replicas: 1
selector:
matchLabels:
name: blog-app
template:
metadata:
labels:
name: blog-app
spec:
containers:
- name: blog-app
image: k8admin:5000/blog-app:latest
command: ["node", "index.js"]
ports:
- containerPort: 3000
env:
- name: SOME_ENV
value: some-value
- name: OTHER_ENV
value: OTHER-value
---
apiVersion: v1
kind: Service
metadata:
name: blog-app-svc
namespace: blog
spec:
type: NodePort
selector:
name: blog-app
ports:
- port: 3000
targetPort: 3000
nodePort: 30001
Some explanation for definition above:
kind: Deployment - the kind of object to be definedmetadata.name - the name of deploymentmetadata.namespace - the namespace where deployment and all of its subobjects are to be placed. To create blog namespace execute kubectl create namespace blog before yor create deployment. Using namespaces makes live easier. In case of this blog I remove blog namespace to delete all its objects before moving to next scenario.replicas - sets the number of our app instanceslabels and selectors are mechanisms to search objects in the clustercontainers - definition of application containers
name - distinguishing name of containerimage - tag of the imagecommand - command to run on the container in array mannerports.containerPort - port number to be exposed from the container; in our case it is the same port that was set in index.js.env - some environment variables to be passed to the container
--- - object separator in multiobject yaml definition filekind: Service - definition of service object; Service is K8s mechanism to enable communication to applicationtype: NodePort - the kind of service, that exposes static port on each cluster's node
port - internal service porttargetPort - the port of the container; usually the port and targetPort have the same valuenodePort - the port to be exposed on cluster node
Now we are ready deploy the app:
$ kubectl apply -f dep.yml
deployment.apps/blog-depl created
service/blog-app-svc created
Success! To 'taste' it let's invoke our app. First we need to determine to which cluster node it has been deployed (remember, we've chosen to run only 1 copy of the app):
$ kubectl get pods -n blog -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
blog-depl-6bd7865df7-t5pmh 1/1 Running 0 9m56s 10.40.0.2 k8work1 <none> <none>
The app is running on k8work1 node. Let's send it http GET request:
$ curl -X GET k8work1:30001
Environment of blog-depl-6bd7865df7-t5pmh:
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME: blog-depl-6bd7865df7-t5pmh
SOME_ENV: some-value
OTHER_ENV: OTHER-value
KUBERNETES_PORT_443_TCP_ADDR: 10.96.0.1
BLOG_APP_SVC_SERVICE_HOST: 10.106.234.15
KUBERNETES_SERVICE_HOST: 10.96.0.1
KUBERNETES_SERVICE_PORT_HTTPS: 443
KUBERNETES_PORT: tcp://10.96.0.1:443
BLOG_APP_SVC_PORT_3000_TCP_PORT: 3000
KUBERNETES_PORT_443_TCP: tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PROTO: tcp
BLOG_APP_SVC_PORT_3000_TCP_PROTO: tcp
BLOG_APP_SVC_PORT_3000_TCP_ADDR: 10.106.234.15
BLOG_APP_SVC_SERVICE_PORT: 3000
BLOG_APP_SVC_PORT: tcp://10.106.234.15:3000
BLOG_APP_SVC_PORT_3000_TCP: tcp://10.106.234.15:3000
KUBERNETES_SERVICE_PORT: 443
KUBERNETES_PORT_443_TCP_PORT: 443
NODE_VERSION: 15.4.0
YARN_VERSION: 1.22.5
HOME: /root
In return - as expected - we have got environment of the container. It is worth notice that - besides of env we have defined in deployment (SOME_ENV, OTHER_ENV) - there are several variables injected by Kubernetes. An application can use them for its own configuration. Because we defined the same port value for the service and the container, we can use BLOG_APP_SVC_SERVICE_PORT value in index.js. So instead of
const port = 3000
we can set
const port = process.env.BLOG_APP_SVC_SERVICE_PORT
and this way move port number from application code to K8s service definition.
To make calling cloud app easier we assembly the command:
curl -X GET "`kubectl get pods -n blog --selector=name=blog-app --field-selector status.phase=Running --template '{{range .items}}{{ if not .metadata.deletionTimestamp }}{{.spec.nodeName}}{{end}}{{end}}'`:30001"
As you can see it filters the name of pod running in blog namespace and uses it to build curl parameter (thanks to https://github.com/kubernetes/kubectl/issues/450#issuecomment-706677565). Let's make it bash function:
cg () {
curl -X GET "`kubectl get pods -n blog --selector=name=blog-app --field-selector status.phase=Running --template '{{range .items}}{{ if not .metadata.deletionTimestamp }}{{.spec.nodeName}}{{end}}{{end}}'`:30001"/"$1";
}
Now we can call our cloud app by simple cg command.
ConfigMaps
Now that we have our test cloud running (and verified passing environment variables) we can move to ConfigMaps. ConfigMaps are common, native way to pass non sensitive information to container.
To cleanup test environment delete and recreate blog namespace.
From text file
Let's have a text file, just like this file.cfg:
This is theoretical configuration file
being injected to container.
To convert it to ConfigMap we use command:
$ kubectl create cm -n blog file-cm --from-file=./file.cfg
configmap/file-cm created
Parameter --from-file indicates that all file content is to be copied to ConfigMap we named file-cm.
and created ConfigMaps looks like:
$ kubectl get cm -n blog file-cm -o yaml
apiVersion: v1
data:
file.cfg: |
This is theoretical configuration file
being injected to container.
kind: ConfigMap
metadata:
creationTimestamp: "2021-01-20T13:42:02Z"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:file.cfg: {}
manager: kubectl
operation: Update
time: "2021-01-20T13:42:02Z"
name: file-cm
namespace: blog
resourceVersion: "43679504"
selfLink: /api/v1/namespaces/blog/configmaps/file-cm
uid: 0aea8662-b765-4113-ba69-be821f9f83f7
Next prepare deployment to use ConfigMap:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
name: blog-app
name: blog-depl
namespace: blog
spec:
replicas: 1
selector:
matchLabels:
name: blog-app
template:
metadata:
labels:
name: blog-app
spec:
containers:
- name: blog-app
image: k8admin:5000/blog-app:latest
command: ["node", "index.js"]
ports:
- containerPort: 3000
volumeMounts:
- name: blog-vol
mountPath: /etc/blog.cfg
volumes:
- name: blog-vol
configMap:
name: file-cm
---
apiVersion: v1
kind: Service
metadata:
name: blog-app-svc
namespace: blog
spec:
type: NodePort
selector:
name: blog-app
ports:
- port: 3000
targetPort: 3000
nodePort: 30001
What changed:
- new element
volumes consists info of resources for the deployment
configMap.name - indicates the resource is ConfigMap with given name
volumeMounts describes how resources are to be mapped to blog-app container
name indicates volume - the source of datamountPath tells kubernetes where to mount the resource
We want our app to read ConfigMap and see what will be read, so we need to modify its source (index.js):
const os = require('os')
const express = require('express')
const fs = require('fs')
const app = express()
app.get('/', (req, res) => {
let r = 'Config map content from ' + os.hostname + ':\n'
r += fs.readFileSync('/etc/blog.cfg/file.cfg', 'utf8')
res.send(r)
})
const port = process.env.BLOG_APP_SVC_SERVICE_PORT
app.listen(port, () => console.log(`listening on port ${port}`))
Now we rebuild the app, deploy new version and execute
cg to see how it works:
$ cg
Config map content from blog-depl-659bfdd9c5-2pnbv:
This is theoretical configuration file
being injected to container.
Works fine. But what's all this for? Coudn't we just copy config to the image and forget about ConfigMaps? Sure we could, but the whole thing is that with ConfigMap we can update data without need to rebuild the image or even to restart the pod. After update of ConfigMap definition its corresponding container value is updated automatically. Let's update the ConfigMap:
$ echo "And now it's updated." >> file.cfg
$ kubectl create cm -n blog file-cm --from-file=./file.cfg --dry-run=client -o yaml | kubectl apply -f -
The latter command thanks to --dry-run=client parameter generates yaml with updated config map definition and then applies it. This way existing ConfigMap is actually updated.
After some time (needed to update cached values) query the app:
$ cg
Config map content from blog-depl-659bfdd9c5-2pnbv:
This is theoretical configuration file
being injected to container.
And now it's updated.
The app sees updated ConfigMap without need to be restarted.
From key-value
Let some.env have simple env like key-value content:
blog_env=some_value
blog_env2=another_value
We can convert it into ConfigMap with command:
$ kubectl create cm -n blog env-cm --from-env-file=./some.env -o yaml
apiVersion: v1
data:
blog_env: some_value
blog_env2: another_value
kind: ConfigMap
metadata:
creationTimestamp: "2021-01-27T12:31:15Z"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:blog_env: {}
f:blog_env2: {}
manager: kubectl
operation: Update
time: "2021-01-27T12:31:15Z"
name: env-cm
namespace: blog
resourceVersion: "45121359"
selfLink: /api/v1/namespaces/blog/configmaps/env-cm
uid: 3dbae9de-787d-4f5d-b564-8207840957c9
This type of ConfigMap can be mapped to container's environment with definition:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
name: blog-app
name: blog-depl
spec:
replicas: 1
selector:
matchLabels:
name: blog-app
template:
metadata:
labels:
name: blog-app
spec:
containers:
- name: blog-app
image: k8admin:5000/blog-app:latest
command: ["node", "index.js"]
ports:
- containerPort: 3000
env:
- name: blog_env
valueFrom:
configMapKeyRef:
name: env-cm
key: blog_env
- name: changed_env
valueFrom:
configMapKeyRef:
name: env-cm
key: blog_env2
Section env defines environment variable blog_env with value from ConfigMap env-cm key blog-env and variable changed_env analogically.
To see if it works we switch back to returning environment version of index.js, rebuild the app, deploy new version and execute cg:
$ cg | grep env
blog_env: some_value
changed_env: another_value
There is some weakness of this type of ConfigMap: corresponding container's values are not updated with map change automatically. To see modified ConfigMap value deployment has to be restarted:
$ kubectl rollout restart deployment -n blog blog-depl
Secrets
When it comes to sensitive data such as password ConfigMaps may not be secure enough.
Instead of them K8s offers Secrets. Just like ConfigMaps Secrets can be created from file or from literal. We have not tried creating ConfigMap from literal, so let's do this with Secrets:
$ kubectl create secret generic lit-sec --from-literal=pass=pass-value -o yaml
apiVersion: v1
data:
pass: cGFzcy12YWx1ZQ==
kind: Secret
metadata:
creationTimestamp: "2021-01-27T15:11:01Z"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:pass: {}
f:type: {}
manager: kubectl
operation: Update
time: "2021-01-27T15:11:01Z"
name: lit-sec
namespace: blog
resourceVersion: "45144690"
selfLink: /api/v1/namespaces/blog/secrets/lit-sec
uid: fd4c7427-2e47-43c6-8bf6-4d12366bd10f
type: Opaque
As you can see the value of created secret is not explicit but base64-encoded - so still not secure.
To access secret from container we need to mount it:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
name: blog-app
name: blog-depl
namespace: blog
spec:
replicas: 1
selector:
matchLabels:
name: blog-app
template:
metadata:
labels:
name: blog-app
spec:
containers:
- name: blog-app
image: k8admin:5000/blog-app:latest
command: ["node", "index.js"]
ports:
- containerPort: 3000
volumeMounts:
- name: sec-vol
mountPath: /etc/secrets
volumes:
- name: sec-vol
secret:
secretName: lit-sec
The value of the secret is going to be available in /etc/secrets/pass, so we need to modify get method in our index.js:
app.get('/', (req, res) => {
let r = 'Secret value from ' + os.hostname + ':\n'
r += fs.readFileSync('/etc/secrets/pass', 'utf8')
res.send(r)
})
rebuild, restart and try with cg:
$ cg
Secret value from blog-depl-84d59fd8b8-5sr7r:
pass-value
As the secret is mounted, its value is propagated to container automatically:
$ kubectl create secret generic lit-sec -n blog --dry-run=client \
> --from-literal=pass=new-pass-value -o yaml | kubectl apply -f -
secret/lit-sec configured
After short time:
$ cg
Secret value from blog-depl-84d59fd8b8-5sr7r:
new-pass-value
Limiting access to secret
One way to protected sensitive data is to limit access to secret. It can be done by setting access mode for certain item and configuring securityContext for the container. First let's define secret with two keys:
kubectl create secret generic -n blog lit-sec --from-literal=pass=pass-value --from-literal=user=user-value
Next set up deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
name: blog-app
name: blog-depl
namespace: blog
spec:
replicas: 1
selector:
matchLabels:
name: blog-app
template:
metadata:
labels:
name: blog-app
spec:
containers:
- name: blog-app
image: k8admin:5000/blog-app:latest
command: ["node", "index.js"]
ports:
- containerPort: 3000
volumeMounts:
- name: sec-vol
mountPath: /etc/secrets
securityContext:
runAsUser: 1000
allowPrivilegeEscalation: false
volumes:
- name: sec-vol
secret:
secretName: lit-sec
items:
- key: pass
mode: 0400
path: pass
- key: user
mode: 0444
path: user
Finally modify index.js so we could get user and pass separately:
app.get('/user', (req, res) => {
var r = 'User value from ' + os.hostname + ':\n'
r += fs.readFileSync('/etc/secrets/user', 'utf8')
res.send(r)
})
app.get('/pass', (req, res) => {
var r = 'User value from ' + os.hostname + ':\n'
r += fs.readFileSync('/etc/secrets/pass', 'utf8')
res.send(r)
})
When we issue cg user we get user-value, but when we want to get cg pass an error EACCES: permission denied, open '/etc/secrets/pass' occurs. This is because we defined mode: 0400 so only root can read /etc/secrets/pass and set containers' user id to 1000 indicating it should be run as regular user.
Disadvantages of Secrets
Despite its name Secrets are not secured by default. They are held non-ecrypted in Kubernetes etcd (which is storage mechanism for K8s cluster), so anyone who has access to etcd can know the Secrets. Fortunately encryption of Secrets at rest can be enabled using --encryption-provider-config of kube-apiserver and Key Management Service (KMS).
Still, anyone who can create any pod that uses a secret can know secret value. Disturbing, isn't it? This and other disadvantages of Secrets are described in theirs documentation.
HashiCorp Vault
One of solutions for safe storage and management of sensitive data is HashiCorp Vault. Vault can be setup standalone or can be deployed to K8s cluster. In this article we will use standalone Vault and access it from K8s container using REST API.
To install Vault on Centos we simply follow the documentation.
Then we start Vault Server in development configuration by issuing:
$ vault server -dev -dev-root-token-id root -dev-listen-address '10.92.29.12:8200'
Parameters:
- 'dev' - tells Vault to run in development mode - it is unsealed and uses volatile memory storage. Normally Vault Server starts in sealed state, where it knows its storage but does not know how to decrypt it. To unseal the Vault one must know master key.
- 'dev-root-token-id' - sets the token which allows to authenticate to Vault to value
root. Otherwise Vault would generate random root token that we would have to remember.
- 'dev-listen-address' - tells Vault to use certain interface and address. By default Vault in dev mode runs on '127.0.0.1' and may not be accessible from outside of the the host.
Export VAULT_ADDR='http://10.92.29.12:8200' and VAULT_TOKEN=root to configure your terminal session.
Let's add some secrets to our brand new Vault Server. First enable secret engine:
$ vault secrets enable -path=blog -version=2 kv
This turns on secrets key-value engine version 2 (with history) on blog path. Then add some secret:
$ vault kv put blog/entry some=thing
Key Value
--- -----
created_time 2021-01-28T15:44:10.978011886Z
deletion_time n/a
destroyed false
version 3
Secret revealed its secret: it's third time I've set its value 😉
Now we can retrieve secret from Vault using both vault command:
$ vault kv get blog/entry
====== Metadata ======
Key Value
--- -----
created_time 2021-01-28T15:44:10.978011886Z
deletion_time n/a
destroyed false
version 3
==== Data ====
Key Value
--- -----
some thing
and REST API:
$ curl -s -X GET --header "X-Vault-Token: $VAULT_TOKEN" $VAULT_ADDR/v1/blog/data/entry | jq -r '.data.data.some'
thing
Accessing Vault from K8s cluster
To make our Vault available from cluster we define service without a pod selector.
apiVersion: v1
kind: Service
metadata:
namespace: blog
name: external-vault
labels:
name: blog-app
spec:
ports:
- protocol: TCP
port: 80
To define how external-vault maps to network address we add an Endpoint:
apiVersion: v1
kind: Endpoints
metadata:
name: external-vault
namespace: blog
labels:
name: blog-app
subsets:
- addresses:
- ip: 10.92.29.12
ports:
- port: 8200
These two combined tell kubernetes that service external-vault port 80 is to be redirected to ip 10.92.29.12 port 8200.
Now let's assume that for some abstract reason we would like to forbid reading our secret
more than once. Vault offers Policies to limit access to its object (file ro.hcl):
path "blog/*" {
capabilities = ["read","list"]
}
We write policy to Vault with command:
$ vault policy write blog-ro ro.hcl
Success! Uploaded policy: blog-ro
and we create access token:
$ vault token create -use-limit 1 -policy blog-ro
Key Value
--- -----
token s.sa1RaGpfFPeCTYEkc5q6YsOY
token_accessor sxGyEDegi1DGNUzmp9Ev0jZT
token_duration 768h
token_renewable true
token_policies ["blog-ro" "default"]
identity_policies []
policies ["blog-ro" "default"]
Token defined this way allows only to read our secret and only to do it once.
Let's try it with our app. New deployment (service blog-app-svc remains unchanged):
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
name: blog-app
name: blog-depl
namespace: blog
spec:
replicas: 1
selector:
matchLabels:
name: blog-app
template:
metadata:
labels:
name: blog-app
spec:
containers:
- name: blog-app
image: k8admin:5000/blog-app:latest
command: ["node", "index.js"]
ports:
- containerPort: 3000
env:
- name: VAULT_PATH
value: "/v1/blog/data/entry"
- name: VAULT_TOKEN
value: "s.sa1RaGpfFPeCTYEkc5q6YsOY"
passes to container blog-app path to secret and one-time token to authorize the request. I know passing token via environment is not the best way, but this is just for example.
index.js has to be modified to access secret via external service:
const express = require('express')
const os = require('os')
function printObject(o) {
let out = '';
for (let p in o) {
out += p + ': ' + o[p] + '\n';
}
return out;
}
const http = require('http')
const options = {
hostname: 'external-vault',
path: process.env.VAULT_PATH,
headers: {
'X-Vault-Token': process.env.VAULT_TOKEN
}
}
function requestCall() {
return new Promise((resolve, reject) => {
http.get(options, (response) => {
let chunks = [];
response.on('data', (fragments) => {
chunks.push(fragments);
});
response.on('end', () => {
let body = Buffer.concat(chunks);
resolve(body.toString());
});
response.on('error', (error) => {
reject(error);
});
});
});
}
async function exGet(req, res) {
let r = 'Vault secret from ' + os.hostname + ':\n';
prom = requestCall();
try {
aw = await prom;
let j = JSON.parse(aw);
r += printObject(j.data.data);
res.send(r);
}
catch(e){
res.send(e);
}
}
const app = express();
app.get('/', exGet);
const port = process.env.BLOG_APP_SVC_SERVICE_PORT
app.listen(port, () => console.log(`listening on port ${port}`))
By the way, code above became more complicated because javascript is designed as asynchronous and we want to get response synchronously.
Again build the app, deploy it and execute
cg:
Vault secret from blog-depl-dfc994588-r8c4v:
some: thing
We've got out secret. But when we try to get it once more:
$ cg
{}
empty object is returned. The token was one-time only.
There is so much more you can achive with Vault and Kubernetes, starting with deploying in Vault to cluster, through injecting secrets into pods using sidecar, to advanced scenarios like those presented here.
That's it for now. This article has already gone too long 😉