How to use a private AWS Elastic Container Registry (ECR) with an external Kubernetes Cluster or any on-prem infrastructure

AWS Elastic Container Registry (ECR)

AWS Elastic Container Registry (ECR) is a fully managed container registry. AWS ECR natively integrates with AWS EKS, AWS ECS, AWS Lambda, and the Docker CLI, allowing you to simplify your development and production workflows. But sometimes you might need to use a private AWS ECR registry with an external Kubernetes Cluster or any on-prem infrastructure (or even within a distributed hybrid cloud infrastructure).

This was one of the technical requirements I experienced recently. The idea was to use AWS CodeBuild (using webhooks with the source repositories in GitHub) as a fully managed continuous integration service that compiles source code, runs tests, produces images and push the images to a private AWS ECR registry. But then, the produced images had to be pulled from an on-prem Kubernetes Cluster. Below I am including some technical challenges/details from this integration.

AWS ECR

AWS ECR is a fully managed container registry offering high-performance hosting, so you can reliably deploy application images and artifacts anywhere. As a fully managed registry, it eliminates the need to operate and scale the infrastructure required to power your container registry. Additionally, AWS ECR transfers your container images over HTTPS and automatically encrypts your images at rest.

I just wanted to clarify here that usually we are associating the AWS ECR with the application container images but AWS ECR supports also Open Container Initiative (OCI) artifacts, such as Helm charts. Open Container Initiative (OCI) artifacts are not a new specification, format, or API. They are a set of somewhat-contradictory conventions on how to store things other than images inside an OCI registry. You can find more details here https://aws.amazon.com/blogs/containers/oci-artifact-support-in-amazon-ecr/ about the OCI artifacts. But just keep in mind that you could also create an AWS ECR repository and use it as a Helm registry pushing there your Helm charts for proper version control.

Authenticate to a private AWS ECR registry

You can use a private AWS ECR registry to manage private image repositories consisting of Docker images and Open Container Initiative (OCI) artifacts. You can authenticate to a private AWS ECR registry with an authorization token by using AWS CLI. An authorization token represents your IAM authentication credentials and it can be used to access any AWS ECR registry that your IAM principal has access to. Please find below an example of an ECR policy with all the permissions that it can be attached to an IAM user. The resource property should be replaced with your repository in case you apply this.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "ecr:GetRegistryPolicy",
        "ecr:DescribeRegistry",
        "ecr:GetAuthorizationToken",
        "ecr:DeleteRegistryPolicy",
        "ecr:PutRegistryPolicy",
        "ecr:PutReplicationConfiguration"
      ],
      "Resource": "*"
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": "ecr:*",
      "Resource": "arn:aws:ecr:eu-central-1:782135427521:repository/my-app"
    }
  ]
}

The authorization token is valid for 12 hours. To obtain an authorization token, you can use the GetAuthorizationToken API. The AWS CLI get-login-password command simplifies this by retrieving and decoding the authorization token which you can then pipe into a docker login command to authenticate.

To authenticate Docker to a private Amazon ECR registry with get-login-password, you should run the aws ecr get-login-password https://docs.aws.amazon.com/cli/latest/reference/ecr/get-login-password.html command. When passing the authorization token to the docker login command, use the value AWS for the username and specify the Amazon ECR registry uri you want to authenticate to.

aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <aws_account_id>.dkr.ecr.<region>.amazonaws.com

After a successful login, you can start using the Docker CLI and pull images from any environment.

docker pull [OPTIONS] NAME[:TAG|@DIGEST]

Using a private AWS ECR registry in Kubernetes

Regarding the container images, we typically create a container image of our application and all its software dependencies and push it to a registry before referring to it in a Kubernetes Pod. When using a private registry in Kubernetes, the credentials can be provided in several ways:

1. Configuring the nodes to authenticate to a private registry
- All pods can read any configured private registries
- Requires node configuration by cluster administrator

2. Pre-pulled images
- All pods can use any images cached on a node
- Requires root access to all nodes to setup

3. Specifying ImagePullSecrets on a pod
- Only pods which provide own keys can access the private registry

4. Vendor-specific or local extensions
- If you’re using a custom node configuration, you (or your cloud provider) can implement your mechanism for authenticating the node to the container registry.

At this article, I will focus only on the third option. This is the option I used in this specific integration and this is a very common pattern especially in case the Kubernetes Cluster nodes are fully managed and they can be replaced at any time or you have not root access to the nodes or you are missing a node configuration management tool for re-applying the same settings/configurations to the nodes. Actually, this is even the Kubernetes recommended approach to run containers based on images in private registries.

Regarding the third option and the imagePullSecrets specification, Kubernetes supports specifying container image registry keys on a Pod. As I have the docker credentials like username: AWS and password: <authorization token from get-login-password>, I can create a new docker-registry (Dockercfg) secret in my namespace. docker-registry (Dockercfg) secrets are used to authenticate against Docker registries. Please note that Pods can only reference image pull secrets in their own namespace, so this process has to be done one time per namespace. This secret can be created by running the following command after substituting the corresponding values.

kubectl create secret docker-registry <name> --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL -n <namespace>

Below you can find a more specific example with the above command after assigning the authorization token to the TOKEN variable and using an AWS ECR registry in the docker-server flag.

TOKEN=`aws ecr get-login-password --region eu-central-1 | cut -d' ' -f6
kubectl create namespace my-namespace
kubectl create secret docker-registry regcred --docker-server=782135427521.dkr.ecr.eu-central-1.amazonaws.com/my-app --docker-username=AWS --docker-password=$TOKEN -n my-namespace`

After creating the above secret with the name regcred , there are different ways to use/refer to it.

One way is to specify it in a Pod definition.

apiVersion: v
kind: Pod
metadata:
  name: my-app
  namespace: my-namespace
spec:
  containers:
    - name: my-app
      image: <private-aws-ecr-registry>/my-app:v1
  imagePullSecrets:
    - name: regcred

Another way is to add it to the default ServiceAccount in the Pod’s namespace with the kubectl patch command. In this way, you can avoid the imagePullSecrets spec in each Pod’s manifest.

kubectl patch serviceaccount default -p "{\"imagePullSecrets\": [{\"name\": \"regcred\"}]}" -n my-namespace

Kubernetes CronJob / AWS authorization token expiration

By creating manually the docker-registry secret in a namespace and referring it to a Pod afterwards or patching the default ServiceAccount (as I described in the previous section), it will work but only for the first 12 hours. As I highlighted before, the authorization token is valid only for 12 hours. This means that somehow and before its expiration, the secret should be updated with a new one. Here is where a Kubernetes CronJob resource should be created and configured accordingly.

CronJobs are meant for performing regular scheduled actions. Each of those actions should be configured to recur indefinitely. The frequency of the Kubernetes CronJob is written in the familiar cron format. For example ‘0 */6 * * *’ in the cron format means at every 6 hours. You can read more here https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#cron-schedule-syntax about the cron schedule syntax.

Below, you can find a CronJob example in order to update the docker-registry secret in your applied namespace every 6 hours. Please you should replace every value inside the angle brackets <VALUE> with your values before applying it.

As you will notice, Kubernetes CronJob schedules also a container as a Pod. This is why I have prepared a minimal Docker image (gtsopour/awscli-kubectl:latest) based on Alpine Linux with AWS CLI v2 and kubectl. Here is the Github repository https://github.com/gtsopour/awscli-kubectl with the Dockerfile and it is also public available in my Docker Hub account https://hub.docker.com/r/gtsopour/awscli-kubectl ( docker pull gtsopour/awscli-kubectl).

As part of the CronJob execution part, I retrieve the authorization token with AWS CLI, create the docker-registry secret and patch the default ServiceAccount in the applied namespace. Now you are wondering probably how the AWS credentials are getting populated. I would like to mention here that the AWS CLI could be also configured via environment variables (AWS CLI is already installed in the image). Environment variables provide another way to specify configuration options and credentials, and can be useful for scripting or temporarily setting a named profile as the default. AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID are among the supported AWS CLI environment variables. You can find here https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html all the supported environment variables.

Finally, you will notice that I am creating a new ServiceAccount including the Role and RoleBinding with permissions to get, create, patch and delete secrets and serviceaccounts. I am doing this because the existing permissions of the default ServiceAccount don’t allow it to list or modify any resources. The new ServiceAccount is assigned through the property serviceAccountName: sa-ecr-token-helper.

cat <<EOF | kubectl apply -f 
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: <namespace>
  name: ecr-token-helper
rules:
  - apiGroups: [""]
    resources:
      - secrets
      - serviceaccounts
      - serviceaccounts/token
    verbs:
      - 'delete'
      - 'create'
      - 'patch'
      - 'get'
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ecr-token-helper
  namespace: <namespace>
subjects:
  - kind: ServiceAccount
    name: sa-ecr-token-helper
    namespace: <namespace>
roleRef:
  kind: Role
  name: ecr-token-helper
  apiGroup: ""
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-ecr-token-helper
  namespace: <namespace>
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: ecr-token-helper
  namespace: <namespace>
spec:
  schedule: '0 */6 * * *'
  successfulJobsHistoryLimit: 0
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: sa-ecr-token-helper
          containers:
            - command:
                - /bin/sh
                - -c
                - |-
                  TOKEN=`aws ecr get-login-password --region ${REGION} | cut -d' ' -f6`
                  kubectl delete secret -n <namespace> --ignore-not-found $SECRET_NAME
                  kubectl create secret -n <namespace> docker-registry $SECRET_NAME \
                  --docker-server=$ECR_REPOSITORY \
                  --docker-username=AWS \
                  --docker-password=$TOKEN \
                  --namespace=<namespace>
                  kubectl patch serviceaccount default -p '{"imagePullSecrets":[{"name":"'$SECRET_NAME'"}]}' -n <namespace>
              env:
                - name: AWS_SECRET_ACCESS_KEY
                  value: <AWS_SECRET_ACCESS_KEY>
                - name: AWS_ACCESS_KEY_ID
                  value: <AWS_ACCESS_KEY_ID>
                - name: ACCOUNT
                  value: <ACCOUNT>
                - name: SECRET_NAME
                  value: 'regcred'
                - name: REGION
                  value: <REGION>
                - name: ECR_REPOSITORY
                  value: <ECR_REPOSITORY>
              image: gtsopour/awscli-kubectl:latest
              imagePullPolicy: IfNotPresent
              name: ecr-token-helper
          restartPolicy: Never
EOF

This article is an effort to explain in detail several principles around the private AWS ECR registry, the IAM Permissions/Policies, the AWS authorization token and its expiration, the Kubernetes docker-registry Secrets and the Kubernetes CronJob resources. As always, your feedback is always welcome.