# Kubernetes Install

This guide walks you through a detailed set up of FlowFuse Platform on a container environment managed by Kubernetes. Typically suited for large on premise deployments or deployment in Cloud infrastructure.
By the end, you will have a fully functioning FlowFuse instance running on a Kubernetes cluster.

# Checklist

<div className="grid,grid-cols-2,gap-8">
<div className="checklist">
<label>

Prerequisites

</label>


     <div>
<checklist-item task="Domain Name">



</checklist-item>


       <checklist-item task="Kubernetes cluster">



</checklist-item>


       <checklist-item type="recommended" task="Setup Dedicated Database">



</checklist-item>


       <checklist-item type="recommended" task="Prepare TLS Certificates">



</checklist-item>
</div>
</div>


   <div className="checklist">
<label>

Installation

</label>


     <div>
<checklist-item task="Download FlowFuse">



</checklist-item>


       <checklist-item task="Configure FlowFuse">



</checklist-item>


       <checklist-item type="recommended" task="Enable HTTPS">



</checklist-item>
</div>
</div>
</div>

## Prerequisites

Before you begin, ensure you have the following:

1. **Domain Name & DNS:** A domain name that you own and can configure DNS settings for (explained in [DNS](#dns))
2. **kubectl:** To manage a Kubernetes cluster you will need a copy of the `kubectl` utility. See the [kubectl install docs](https://kubernetes.io/docs/tasks/tools/)
3. **Helm:** FlowFuse provides the Helm chart to manage platform deployment. Install it by following the instructions on [their website](https://helm.sh)
4. **Kubernetes Cluster:** The deployment has currently been tested on the following environments:

  - [AWS EKS](/docs/install/kubernetes/aws-terraform)
  - [Digital Ocean](/docs/install/kubernetes/digital-ocean)
  - MicroK8s
5. **Ingress Controller:** Install [Traefik](https://doc.traefik.io/traefik/) on the cluster.
6. **Cert-Manager:** EMQX requires the CRDs from cert-manager. See the [installation instructions](https://cert-manager.io/docs/installation/) for details.
7. **EMQX Operator:** This installs the operator that deploys the platform's MQTT broker, which is required whenever the broker is enabled. You must install exactly version 2.2.29 — later versions are not supported. Follow the [installation instructions](https://docs.emqx.com/en/emqx-operator/latest/getting-started/getting-started.html#install-emqx-operator) and pin the version by adding `--version 2.2.29` to the install command.

For a production-ready environment, we also recommend:

- **Database:** Prepare dedicated database on an external database server (see [FAQ](#how-to-use-external-database-server) for more details)
- **TLS Certificate:** Prepare a TLS certificate for your domain and configure FlowFuse platform to use it (see [Enable HTTPS](#i-would-like-to-secure-the-platform-with-https-how-can-i-do-that))

### Hardware requirements

For a Kubernetes-based deployment, resource requirements depend on the number of FlowFuse and Node-RED instances running. As a baseline, we suggest:

Control Plane: At least 2 vCPUs, 4 GB RAM
Worker Nodes: Minimum 2 vCPUs, 4 GB RAM per node, 2 nodes for high availability
Storage: 20Gb of host storage (for container images), StorageClass of your choice available for Hosted Node-RED instances (optional)

Each Node-RED instance you host will use 0.1 CPU cores and 256 MB of memory by default. These parameters can be adjusted in admin area of FlowFuse platform. Keep this in mind when sizing your hardware, especially if you plan to create multiple hosted instances.

### DNS

You will need a [wildcard DNS entry](https://en.wikipedia.org/wiki/Wildcard_DNS_record)
pointing to the domain that is used for the project instances. This will need to point
to the kubernetes Ingress controller.

For example if you want projects to be accessible as `[instance-name].example.com`
you will need to ensure that `*.example.com` is mapped to the IP address used by
your Kubernetes cluster's Ingress controller.

By default the FlowFuse application will be mapped to `forge.example.com` assuming
that you set the domain to `example.com`.

Notes on how to setup DNS can be found [here](/docs/install/dns-setup).

## Installing FlowFuse

### Add FlowFuse Helm Repository

```bash
helm repo add flowfuse https://flowfuse.github.io/helm
helm repo update
```

### Customize Helm Chart

All the initial configuration is handled by the Helm chart. This is done by creating a `customization.yml` file that will be passed to the Helm along with the chart.

To create `customization.yml` file with a minimal required configuration (replace `example.com` with your domain):

```bash
cat <<EOF > customization.yml
forge:
  entryPoint: forge.example.com
  domain: example.com
  https: false
  localPostgresql: true
EOF
```

A full list of all the configuration options can be found in the [Helm Chart README](https://github.com/FlowFuse/helm/blob/main/helm/flowfuse/README.md#configuration-values).

### Label Nodes

By default FlowFuse platform expects that Kubernetes nodes have specific labels applied. The main reason behind this approach is to separate core application components from Node-RED instances.

You will need to label at least one node to run the management application and one to run the Node-RED Projects:

List all nodes in the cluster:

```bash
kubectl get nodes
```

Label management nodes:

```bash
kubectl label node <management-node-name> role=management
```

Label project nodes:

```bash
kubectl label node <projects-node-name> role=projects
```

To override this behavior, you can remove the node selectors with the following entry in the `customization.yml` file which will mean that all pods can run on any nodes.

```yaml [customization.yml]
forge:
  projectSelector:
  managementSelector:
```

## Start FlowFuse Platform

Once you have the `customization.yml` file created, you can install FlowFuse using our Helm chart. This will automatically create all required objects and start services:

```bash
helm upgrade --atomic --install --timeout 10m flowfuse flowfuse/flowfuse -f customization.yml
```

## First Run Setup

The first time you access the platform in your browser, it will take you through
creating an administrator for the platform and other configuration options.

For more information, follow [this guide](/docs/install/first-run).

Once you have finished setting up the admin user there are some [Kubernetes specific items to consider](#common-questions).

## Upgrade

All technical aspects of the upgrade process of Flowfuse application running on Kubernetes and managed by Helm chart are maintained in our repository.
Please refer to the [Flowfuse Helm Chart documentation](https://github.com/FlowFuse/helm/blob/main/helm/flowfuse/README.md#upgrading-chart) for more details
about the upgrade process.

## Common Questions

### I would like to secure the platform with HTTPS, how can I do that?

In cloud environments, it is recommended to use a Load Balancer to terminate SSL traffic.

However, if you want to use SSL termination on the Kubernetes Ingress Controller, this is possible by using the [Cert-Manager](https://cert-manager.io/docs/) tool (not part of the FlowFuse Helm chart).

Once you have Cert-Manager installed, you can enable TLS support in the `customization.yml` file by specifying the [ClusterIssuer](https://cert-manager.io/docs/configuration/#cluster-resource-namespace) name:

```yaml [customization.yml]
ingress:
  clusterIssuer: <your-cluster-issuer>
```

Apply changes with [platform startup command](#start-flowfuse-platform).

#### Using Internal/Private Certificate Authorities

If you are issuing your own HTTPS certificates (rather than using a Public Certificate Authority) you will need to tell the Hosted Node-RED Instances to trust this CA. To do this you need to pass base64 encoded CA certificate chain to the helm chart in the `customization.yml`. Details are in the Helm chart [README.md](https://github.com/FlowFuse/helm/blob/main/helm/flowfuse/README.md#private-certificate-authority).

```yaml [customization.yml]
forge:
  privateCA:
    certs: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk...
```

Assuming the certificate chain is held in `chain.pem` the value would be created by running `base64 -w 0 certs.pem`

### I use Kubernetes Network Policies, how can I configure them?

If your cluster uses Network Policies to restrict traffic between namespaces, you'll need to create appropriate policies.

Here's an example Network Policy that allows traffic from the `flowforge` namespace (default namespace for Node-RED instances) to the `flowfuse` namespace:

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-flowforge-to-flowfuse
  namespace: flowfuse
spec:
  podSelector: {}
  ingress:
    - from:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: flowforge
  policyTypes:
    - Ingress
```

You may need to adjust this policy based on your specific network requirements and namespace configuration.

### How to use external database server?

FlowFuse platform uses PostgreSQL database to store its data. By default, the Helm chart creates and manages the internal database instance.

If you want to use an external database server, you need to edit `customization.yml` file and provide the database connection details:

```yaml [customization.yml]
forge:
  localPostgresql: false # Disable internal database
postgresql:
  host: <database-host>
  port: <database-port>
  auth:
    username: <database-username>
    password: <database-password>
    database: <database-name>
```

Apply changes with [platform startup command](#start-flowfuse-platform).

Check the [FlowFuse Helm chart documentation](https://github.com/FlowFuse/helm/tree/main/helm/flowfuse#postgresql) for more details about the parameters that can be configured for the PostgreSQL database.

### How to backup embedded database?

If you are using the internal database (value `forge.localPostgresql` set to `true`), you can use Kubernetes [CronJobs](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/) to backup the database.

Apply below `CronJob` and `PersistentVolumeClaim` definitions to create a backup job which will be executed every day at 23:05 and store the backup in a PVC named `db-backup-pvc`:

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: db-backup-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: postgres-backup
spec:
  schedule: "5 23 * * *"
  jobTemplate:
    spec:
      ttlSecondsAfterFinished: 60
      template:
        metadata:
          labels:
            app: flowforge
        spec:
          containers:
          - name: backup
            image: postgres
            env:
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: flowfuse-postgresql
                  key: postgres-password
            command:
            - /bin/sh
            - -c
            - |
              pg_dump -h flowfuse-postgresql -U postgres -d flowforge -F c -b -v -f /backup/db_backup.dump
            volumeMounts:
            - name: backup-volume
              mountPath: /backup
          restartPolicy: OnFailure
          volumes:
          - name: backup-volume
            persistentVolumeClaim:
              claimName: db-backup-pvc
```

### I would like to invite my team members to the platform with e-mail, how can I do that?

FlowFuse platform allows you to invite team members to the platform using their e-mail addresses.
To enable this feature, you need to configure the e-mail settings in the `customization.yml` file.

Check this [page](/docs/install/configuration#email-configuration) for more details about the parameters.
Check [FlowFuse Helm chart documentation](https://github.com/FlowFuse/helm/tree/main/helm/flowfuse#email) for information where configuration values should be placed in `customization.yml` file.

If you use AWS EKS (Elastic Kubernetes Service) and want to use AWS SES (Simple Email Service) for sending e-mails, you need to provide the IAM role with the required permissions to use SES.

```yaml [customization.yml]
forge:
  entryPoint: forge.example.com
  domain: example.com
  cloudProvider: aws
  aws:
    IAMRole: arn:aws:iam::<aws-account-id>:role/flowforge_service_account_role
  email:
    ses:
      region: eu-west-1
```

Apply changes with [platform startup command](#start-flowfuse-platform).

### I would like to use embedded MQTT broker, how can I do that?

<details>
<summary>

Click to expand

</summary>


The FlowFuse Helm chart provides the MQTT broker service.
To enable the MQTT broker you need to add the following to the `customization.yml` file:
```yaml [customization.yml]
forge:
  broker:
    enabled: true
```
Apply changes with [platform startup command](#start-flowfuse-platform).
Check the [FlowFuse Helm chart documentation](https://github.com/FlowFuse/helm/tree/main/helm/flowfuse#mqtt-broker) for more details about the parameters that can be configured for the MQTT broker.

</details>

### I would like to use Kubernetes Persistent storage to store data, how can I do that?

Starting with the `2.6.0` release the Pods running the Node-RED Instances have a Persistent Volume mounted on `/data/storage` in which files can be written.
These files will persist for the lifetime of the Instance including across Suspend/Resume and Stack upgrades.

To enable this feature the following configuration needs to be added to the `customization.yml` file (replace '<storage-class-name>

' with the name of the StorageClass you have in the cluster):

</storage-class-name>



```yaml [customization.yml]
forge:
  persistentStorage:
    enabled: true
    size: 5Gi
    storageClass: <storage-class-name>
```

Apply changes with [platform startup command](#start-flowfuse-platform).

### I would like to use FlowFuse File Storage to store context data, how can I do that?

To enable the FlowFuse File Storage component add the following to the `customization.yml` file:

```yaml [customization.yml]
forge:
  fileStore:
    enabled: true
```

Apply changes with [platform startup command](#start-flowfuse-platform).

Check the [FlowFuse Helm chart documentation](https://github.com/FlowFuse/helm/tree/main/helm/flowfuse#file-storage) for more details about the parameters that can be configured for the File Storage.

### I would like to run FlowFuse on AWS EKS. Do you have any guidance?

Yes, we have a dedicated guide on how to deploy FlowFuse on AWS EKS. You can find it [here](/docs/install/kubernetes/aws).
Furthermore, we also provide terraform scripts to automate the deployment process of all required AWS service. You can find the guide [here](/docs/install/kubernetes/aws-terraform).