This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Installing To Kubenetes

Deploying Drycc on a Kubernetes Cluster.

1: Requirements
2: Specify Gateway
3: Installing Drycc Workflow
4: Configuring Object Storage
5: Configuring Postgres
6: Configuring Registry

1 - Requirements

To run Drycc Workflow on a Kubernetes cluster, there are a few requirements to keep in mind.

Kubernetes Versions

Drycc Workflow requires Kubernetes v1.16.15 or later.

Components Requirements

Drycc uses gateway as a routing implementation, so you have to choose an gateway. We recommend using istio or kong.

Workflow supports the use of ACME to manage automatic certificates, cert-manager is also one of the necessary components, if you use cert-manager EAB, you need to set the clusterResourceNamespace to the namespace of drycc.

Workflow supports stateful apps. You can create and use them through the ‘drycc volumes’ command. If you want to use this feature, you must have a StorageClass that supports ReadWriteMany.

Workflow also supports the OSB API through the ‘drycc resources’ command. If you want to use this function, you need to install service-catalog.

Storage Requirements

A variety of Drycc Workflow components rely on an object storage system to do their work, including storing application slugs, Container images and database logs.

Drycc Workflow ships with drycc storage by default, which provides in-cluster.

Workflow supports Amazon Simple Storage Service (S3), Google Cloud Storage (GCS), OpenShift Swift, and Azure Blob Storage. See configuring object storage for setup instructions.

Resource Requirements

When deploying Drycc Workflow, it’s important to provision machines with adequate resources. Drycc is a highly-available distributed system, which means that Drycc components and your deployed applications will move around the cluster onto healthy hosts as hosts leave the cluster for various reasons (failures, reboots, autoscalers, etc.). Because of this, you should have ample spare resources on any machine in your cluster to withstand the additional load of running services for failed machines.

Drycc Workflow components use about 2.5GB of memory across the cluster, and require approximately 30GB of hard disk space. Because it may need to handle additional load if another one fails, each machine has minimum requirements of:

At least 4GB of RAM (more is better)
At least 40GB of hard disk space

Note that these estimates are for Drycc Workflow and Kubernetes only. Be sure to leave enough spare capacity for your application footprint as well.

Running smaller machines will likely result in increased system load and has been known to result in component failures and instability.

2 - Specify Gateway

Choose the gateway provider that best suits your needs and platform.

Install Drycc Workflow (Specify gateway)

Now that Helm is installed and the repository has been added, install Workflow with a native gateway by running:

$ helm install drycc oci://registry.drycc.cc/charts/workflow \
    --namespace drycc \
    --set gateway.gatewayClass=istio \
    --set controller.appGatewayClass=istio \
    --set global.platformDomain=drycc.cc \
    --set builder.service.type=LoadBalancer

Of course, if you deploy it on a bare machine, you probably do not have Load Balancer. You need to use NodePort:

$ helm install drycc oci://registry.drycc.cc/charts/workflow \
    --namespace drycc \
    --set gateway.gatewayClass=istio \
    --set controller.appGatewayClass=istio \
    --set global.platformDomain=drycc.cc \
    --set builder.service.type=NodePort \
    --set builder.service.nodePort=32222

If you want to use Load Balancer on a bare machine, you can look at metallb

Where global.platformDomain is a required parameter that is traditionally not required for Workflow that is explained in the next section. In this example we are using drycc.cc for $hostname.

Helm will install a variety of Kubernetes resources in the drycc namespace. Wait for the pods that Helm launched to be ready. Monitor their status by running:

$ kubectl --namespace=drycc get pods

You should also notice that several Kubernetes gatewayclass has been installed on your cluster. You can view it by running:

$ kubectl get gatewayclass

Depending on the order in which the Workflow components initialize, some pods may restart. This is common during the installation: if a component’s dependencies are not yet available, that component will exit and Kubernetes will automatically restart it.

Here, it can be seen that the controller, builder and registry all took a few loops waiting for storage before they were able to start:

$ kubectl --namespace=drycc get pods
NAME                          READY     STATUS    RESTARTS   AGE
drycc-builder-hy3xv            1/1       Running   5          5m
drycc-controller-g3cu8         1/1       Running   5          5m
drycc-controller-celery-cmxxn  3/3       Running   0          5m
drycc-database-rad1o           1/1       Running   0          5m
drycc-fluentbit-1v8uk          1/1       Running   0          5m
drycc-fluentbit-esm60          1/1       Running   0          5m
drycc-storage-4ww3t            1/1       Running   0          5m
drycc-registry-asozo           1/1       Running   1          5m

Install a Kubernetes Gateway

Now that Workflow has been deployed with the gatewayClass , we will need a Kubernetes gateway in place to begin routing traffic.

Here is an example of how to use istio as an gateway for Workflow. Of course, you are welcome to use any controller you wish.

$ helm repo add istio https://istio-release.storage.googleapis.com/charts
$ helm repo update
$ kubectl create namespace istio-system
$ helm install istio-base istio/base -n istio-system
$ helm install istiod istio/istiod -n istio-system --wait
$ kubectl create namespace istio-ingress
$ helm install istio-ingress istio/gateway -n istio-ingress --wait

Configure DNS

User must install drycc and then set up a hostname, and assumes the *.$host convention.

We need to point the *.$host record to the public IP address of your gateway. You can get the public IP using the following command. A wildcard entry is necessary here as apps will use the same rule after they are deployed.

$ kubectl get gateway --namespace drycc
NAME      CLASS   ADDRESS         PROGRAMMED   AGE
gateway   istio   138.91.243.152  True         36d

If we were using drycc.cc as a hostname, we would need to create the following A DNS records.

Name	Type	Value
*.drycc.cc	A	138.91.243.152

Once all of the pods are in the READY state, and *.$host resolves to the external IP found above, the preparation of gateway has been completed!

After installing Workflow, register a user and deploy an application.

If your k8s does not provide public network loadblance, you need to install TCP proxy services such as haproxy on machines that can access both internal and external networks, and then expose 80 and 443.

3 - Installing Drycc Workflow

This document is aimed at those who have already provisioned a Kubernetes cluster and want to install Drycc Workflow.

If help is required getting started with Kubernetes and Drycc Workflow, follow the quickstart guide for assistance.

Prerequisites

Verify the Kubernetes system requirements
Install Helm and Drycc Workflow CLI tools

Check Your Setup

Check that the helm command is available and the version is v2.5.0 or newer.

$ helm version
Client: &version.Version{SemVer:"v2.5.0", GitCommit:"012cb0ac1a1b2f888144ef5a67b8dab6c2d45be6", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.5.0", GitCommit:"012cb0ac1a1b2f888144ef5a67b8dab6c2d45be6", GitTreeState:"clean"}

Choose Your Deployment Strategy

Drycc Workflow includes everything it needs to run out of the box. However, these defaults are aimed at simplicity rather than production readiness. Production and staging deployments of Workflow should, at a minimum, use off-cluster storage which is used by Workflow components to store and backup critical data. Should an operator need to completely re-install Workflow, the required components can recover from off-cluster storage. See the documentation for configuring object storage for more details.

More rigorous installations would benefit from using outside sources for the following things:

Postgres - For example AWS RDS.
Registry - This includes quay.io, dockerhub, Amazon ECR, and Google GCR.
Valkey - Such as AWS Elasticache
Grafana

Gateway

Now, workflow requires that gateway and cert-manager must be installed. Any compatible Kubernetes entry controller can be used.

Install Drycc Workflow

If the version of helm is 3.0 +; you need to create the namespace in advance:

kubectl create ns drycc

If you want to change it, set the variable when using helm.

$ helm install drycc oci://registry.drycc.cc/charts/workflow \
    --namespace drycc \
    --set builder.imageRegistry=quay.io \
    --set imagebuilder.imageRegistry=quay.io \
    --set controller.imageRegistry=quay.io \
    --set database.imageRegistry=quay.io \
    --set fluentbit.imageRegistry=quay.io \
    --set valkey.imageRegistry=quay.io \
    --set storage.imageRegistry=quay.io \
    --set grafana.imageRegistry=quay.io \
    --set registry.imageRegistry=quay.io \
    --set global.platformDomain=drycc.cc

Helm will install a variety of Kubernetes resources in the drycc namespace. Wait for the pods that Helm launched to be ready. Monitor their status by running:

$ kubectl --namespace=drycc get pods

If it’s preferred to have kubectl automatically update as the pod states change, run (type Ctrl-C to stop the watch):

$ kubectl --namespace=drycc get pods -w

Here, it can be seen that the controller, builder and registry all took a few loops before they were able to start:

$ kubectl --namespace=drycc get pods
NAME                                     READY     STATUS    RESTARTS   AGE
drycc-builder-574483744-l15zj             1/1       Running   0          4m
drycc-controller-3953262871-pncgq         1/1       Running   2          4m
drycc-controller-celery-cmxxn             3/3       Running   0          4m
drycc-database-83844344-47ld6             1/1       Running   0          4m
drycc-fluentbit-zxnqb                     1/1       Running   0          4m
drycc-valkey-304849759-1f35p              1/1       Running   0          4m
drycc-storage-676004970-nxqgt             1/1       Running   0          4m
drycc-monitor-grafana-432627134-lnl2h     1/1       Running   0          4m
drycc-monitor-telegraf-wmcmn              1/1       Running   1          4m
drycc-registry-756475849-lwc6b            1/1       Running   1          4m
drycc-registry-proxy-96c4p                1/1       Running   0          4m

Once all of the pods are in the READY state, Drycc Workflow is up and running!

For more installation parameters, please check the values.yaml file of workflow.

After installing Workflow, register a user and deploy an application.

Configure DNS

User must to set up a hostname, and assumes the drycc-builder.$host convention.

We need to point the drycc-builder.$host record to the public IP address of your builder. You can get the public IP using the following command. A wildcard entry is necessary here as apps will use the same rule after they are deployed.

$ kubectl get svc drycc-builder --namespace drycc
NAME              CLUSTER-IP   EXTERNAL-IP      PORT(S)                      AGE
drycc-builder     10.0.25.3    138.91.243.152   2222:31625/TCP               33m

If we were using drycc.cc as a hostname, we would need to create the following A DNS records.

Name	Type	Value
drycc-builder.drycc.cc	A	138.91.243.152

Once all of the pods are in the READY state, and drycc-builder.$host resolves to the external IP found above, Workflow is up and running!

After installing Workflow, register a user and deploy an application.

4 - Configuring Object Storage

A variety of Drycc Workflow components rely on an object storage system to do their work including storing application slugs, Container images and database logs.

Drycc Workflow ships with Storage by default, which provides in-cluster.

Configuring off-cluster Object Storage

Every component that relies on object storage uses two inputs for configuration:

Access credentials stored as a Kubernetes secret.
You must use object storage services that are compatible with S3 API.

The helm chart for Drycc Workflow can be easily configured to connect Workflow components to off-cluster object storage. Drycc Workflow currently supports Google Compute Storage, Amazon S3, Azure Blob Storage and OpenStack Swift Storage.

Step 1: Create storage buckets

Create storage buckets for each of the Workflow subsystems: builder and registry.

Depending on your chosen object storage you may need to provide globally unique bucket names. If you are using S3, use hyphens instead of periods in the bucket names. Using periods in the bucket name will cause an ssl certificate validation issue with S3.

If you provide credentials with sufficient access to the underlying storage, Workflow components will create the buckets if they do not exist.

Step 2: Generate storage credentials

If applicable, generate credentials that have create and write access to the storage buckets created in Step 1.

If you are using AWS S3 and your Kubernetes nodes are configured with appropriate IAM API keys via InstanceRoles, you do not need to create API credentials. Do, however, validate that the InstanceRole has appropriate permissions to the configured buckets!

Step 3: Configure Workflow Chart

Operators should configure object storage by editing the Helm values file before running helm install. To do so:

Fetch the Helm values by running helm inspect values oci://registry.drycc.cc/charts/workflow > values.yaml
Update the builder/storage and registry/storage parameter to reference the platform you are using.
Find the corresponding section for your storage type and provide appropriate values including region, bucket names, and access credentials.
Save your changes.

Note

Assume we are using MinIO’s play for storage, noting that it is only a test server and should not be used in production environments:

$ helm install drycc oci://registry.drycc.cc/charts/workflow \
	--namespace drycc \
	--set global.platformDomain=youdomain.com \
	--set builder.storageBucket=registry \
	--set builder.storageEndpoint=https://play.min.io \
	--set builder.storageAccesskey=Q3AM3UQ867SPQQA43P2F \
	--set builder.storageSecretkey=zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG \
	--set builder.storagePathStyle=auto \
	--set registry.storageBucket=registry \
	--set registry.storageEndpoint=https://play.min.io \
	--set registry.storageAccesskey=Q3AM3UQ867SPQQA43P2F \
	--set registry.storageSecretkey=zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG \
	--set registry.storagePathStyle=auto

You are now ready to run helm install drycc oci://registry.drycc.cc/charts/workflow --namespace drycc -f values.yaml using your desired object storage.

5 - Configuring Postgres

Drycc Workflow’s controller and passport component rely on a PostgreSQL database to store platform state.

Configuring Postgres

By default, Drycc Workflow ships with the database component, which provides an in-cluster PostgreSQL database backed up to in-cluster or off-cluster object storage. Currently, for object storage, which is utilized by several Workflow components, only off-cluster solutions such as S3 or GCS are recommended in production environments. Experience has shown that many operators already opting for off-cluster object storage similarly prefer to host Postgres off-cluster as well, using Amazon RDS or similar. When excercising both options, a Workflow installation becomes entirely stateless, and is thus restored or rebuilt with greater ease should the need ever arise.

Provisioning off-cluster Postgres

First, provision a PostgreSQL RDBMS using the cloud provider or other infrastructure of your choice. Take care to ensure that security groups or other firewall rules will permit connectivity from your Kubernetes worker nodes, any of which may play host to the Workflow controller component.

Take note of the following:

The hostname or public IP of your PostgreSQL RDBMS
The port on which your PostgreSQL RDBMS runs– typically 5432

Within the off-cluster RDBMS, manually provision the following:

A database user (take note of the username and password)
A database owned by that user (take note of its name)

If you are able to log into the RDBMS as a superuser or a user with appropriate permissions, this process will typically look like this:

$ psql -h <host> -p <port> -d postgres -U <"postgres" or your own username>
> create user <drycc username; typically "drycc"> with password '<password>';
> create database <database name; typically "drycc"> with owner <drycc username>;
> \q

Configuring Workflow

The Helm chart for Drycc Workflow can be easily configured to connect the Workflow controller component to an off-cluster PostgreSQL database.

Step 1: If you haven’t already fetched the values, do so with helm inspect values drycc/workflow > values.yaml
Step 2: Update database connection details by modifying values.yaml:
- Update the database.enabled parameter to false.
- Update the values in the [database] configuration section to properly reflect all connection details.
- Update the values in the [controller] configuration section to properly reflect platformDomain details.
- Save your changes.
- Note: you do not need to (and must not) base64 encode any values, as the Helm chart will automatically handle encoding as necessary.

You are now ready to helm install drycc oci://registry.drycc.cc/charts/workflow --namespace drycc -f values.yaml as usual.

6 - Configuring Registry

Drycc Workflow’s builder component relies on a registry for storing application container images.

Drycc Workflow ships with a registry component by default, which provides an in-cluster Container registry backed by the platform-configured object storage. Operators might want to use an off-cluster registry for performance or security reasons.

Configuring Off-Cluster Private Registry

Every component that relies on a registry uses two inputs for configuration:

Registry Location environment variable named DRYCC_REGISTRY_LOCATION
Access credentials stored as a Kubernetes secret named registry-secret

The Helm chart for Drycc Workflow can be easily configured to connect Workflow components to off-cluster registry. Drycc Workflow supports external registries which provide either short-lived tokens that are valid only for a specified amount of time or long-lived tokens (basic username/password) which are valid forever for authenticating to them. For those registries which provide short lived tokens for authentication, Drycc Workflow will generate and refresh them such that the deployed apps will only have access to the short-lived tokens and not to the actual credentials for the registries.

When using a private registry the container images are no longer pulled by Drycc Workflow Controller but rather are managed by Kubernetes. This will increase security and overall speed, however the port information can no longer be discovered. Instead the port information can be set via drycc config set PORT=<port> prior to deploying the application.

Drycc Workflow currently supports:

off-cluster: Any provider which supports long-lived username/password authentication, such as Azure Container Registry, Docker Hub, quay.io, or a self-hosted Container registry.

Configuration

If you haven’t already fetched the values file, do so with helm inspect values drycc/workflow > values.yaml
Update registry location details by modifying the values file: * Update the registry.enabled parameter to reference the registry location you are using: true, false * Update the values in the section which corresponds to your registry location type.

You are now ready to helm install drycc oci://registry.drycc.cc/charts/workflow --namespace drycc -f values.yaml using your desired registry.

Examples

Here we show how the relevant parts of the fetched values.yaml file might look like after configuring for a particular off-cluster registry:

Azure Container Registry (ACR)

After following the docs and creating a registry, e.g. myregistry, with its corresponding login server of myregistry.azurecr.io, the following values should be supplied:

builder:
  registryHost: "myregistry.azurecr.io"
  registryUsername: "xxxx"
  registryPassword: "xxxx"
  registryOrganization: "xxxx"
registry:
  enabled: false

Note: The mandatory organization field (here myorg) will be created as an ACR repository if it does not already exist.

Quay.io

builder:
  registryHost: "quay.io"
  registryUsername: "xxxx"
  registryPassword: "xxxx"
  registryOrganization: "xxxx"
registry:
  enabled: false