Installation

Note

This guide is for installing the Flowcore Platform on a dedicated Kubernetes cluster. It requires a dedicated cluster license, this can be purchased from the AWS Marketplace.

To install the Flowcore Platform, you will first have to have a dedicated Kubernetes cluster.

Prerequisites

• Kubernetes cluster version 1.28 or higher
• A dedicated cluster license

Installation of Prerequisites

1. Create a kubernetes cluster using your preferred method.

   Tip

   We recommend using EKS or GKE for a dedicated cluster.

2. Once the cluster is ready, you will need to install the following services:

   • Cert-Manager
   • ArgoCD

   Note

   ArgoCD needs to be configured to have public access, this is required for GitHub to notify ArgoCD of changes, otherwise deployed changes will be delayed to the next sync cycle. It is also recommended to setup a webhook to notify ArgoCD of changes to your repository.

   • Nginx Ingress Controller

   Note

   The Nginx Ingress Controller can be swapped out for any other ingress controller, such as AWS ALB Ingress Controller.

Installation of Flowcore Platform

To install the Flowcore Platform, you will need to use the Flowcore Platform Helm Chart.

Base Credentials

Create the following credentials in the flowcore namespace.

1. Create a secret for the S3 Compatible Bucket.

   kubectl create secret generic flowcore-s3-credentials \
     --from-literal=AWS_ACCESS_KEY_ID=<your-access-key-id> \
     --from-literal=AWS_SECRET_ACCESS_KEY=<your-secret-access-key> \
     --from-literal=AWS_REGION=<your-region> \
     --from-literal=AWS_S3_BUCKET=<your-bucket-name> \
     -n flowcore

Tip

You need to create a S3 Compatible Bucket, this can be done using your preferred method.

2. Create a secret for the Cassandra database credentials.

   kubectl create secret generic platform-source-cassandra-user \
     --from-literal=cassandra-password=<your-password> \
     -n flowcore

3. Create a secret containing the Valkey HA Password.

   kubectl create secret generic platform-source-valkey-ha \
     --from-literal=password=<your-password> \
     -n flowcore

4. Create a secret containing the Valkey Single Password.

   kubectl create secret generic platform-source-valkey-single \
     --from-literal=password=<your-password> \
     -n flowcore

5. Create a secret containing the PostgreSQL credentials.

   kubectl create secret generic postgresql-ha \
     --from-literal=repmgr-password=<repmgr-password> \
     --from-literal=password=<postgres-password> \
     --from-literal=admin-password=<pgpool-password> \
     -n flowcore

Flowcore Tenant Credentials

You will also need to create a few credentials for the Flowcore Tenant that will be tied to the dedicated cluster license.

Note

You will have gotten your license key for the Flowcore Tenant when you purchased the dedicated cluster license, if you have not received this please contact support, via email or discord on our website

First you will need to create an API key to use for the Flowcore Tenant.

1. Create an API key using the Flowcore CLI.

   flowcore auth new key -t <tenant> dedicated-cluster

2. Create the following policies to be used by the API key.

   policy.yaml

   apiVersion: iam.flowcore.io/v1
   kind: Policy
   metadata:
     name: dedicated-cluster-policy
     tenant: <tenant>
   spec:
     version: "1.0.0"
     description: "Allows read and write access to all resources"
     policyDocuments:
       - resource: "frn::<tenant>:tenant/*"
         action: [
           "read",
           "write"
         ]
       - resource: "frn::<tenant>:license/*"
         action: [
           "read",
           "write"
         ]

   flowcore apply -f policy.yaml --v2

3. Bind the policy to the API key.

   policy-binding.yaml

   apiVersion: iam.flowcore.io/v1
   kind: PolicyBinding
   metadata:
     name: dedicated-cluster-policy-binding
     tenant: <tenant>
   spec:
     policy: "dedicated-cluster-policy" # References the policy name above
     subjects:
       - type: key
         id: <api-key-id>

   flowcore apply -f policy-binding.yaml --v2

4. Create a secret containing the following:

   • The API key
   • The License Key
   • The Tenant ID
   • The Tenant Name

   kubectl create secret generic flowcore-tenant-credentials \
     --from-literal=api-key=<your-api-key> \
     --from-literal=license-key=<your-license-key> \
     --from-literal=tenant-id=<your-tenant-id> \
     --from-literal=tenant-name=<your-tenant-name> \
     -n flowcore

GitHub Credentials

The Flowcore Platform Control Plane needs access to your GitHub repository to deploy your transformers. This is done by creating a GitHub App and adding the necessary permissions.

1. Create a GitHub App here.

2. Add the following permissions:

   • Read and Write Access to Metadata
   • Read and Write Access to The Contents of Repositories

3. Add the GitHub App to your repository.

4. Create a secret containing the following:

   • The GitHub App ID
   • The GitHub App Installation ID
   • The GitHub App Private Key
   • The Github Repository URL

   kubectl create secret generic flowcore-github-credentials \
     --from-literal=github-app-id=<your-github-app-id> \
     --from-literal=github-app-installation-id=<your-github-app-installation-id> \
     --from-literal=github-app-private-key=<your-github-app-private-key> \
     --from-literal=github-repository-url=<your-github-repository-url> \
     -n flowcore

Install the Flowcore Platform

Now that you have created the necessary credentials, you can install the Flowcore Platform.

1. Install the Flowcore Platform using the Helm Chart.

   helm repo add flowcore https://flowcore-io.github.io/flowcore-helm-charts/
   helm install flowcore-platform flowcore/flowcore-platform --namespace flowcore \
     --set flowcore.ingestionUrl=<url you want to use for ingestion> \
     --set flowcore.eventSourceUrl=<url you want to use for event source> \
     --set flowcore.stateUrl=<url you want to use for fetching state>

2. Wait for the Flowcore Platform to be ready.

   kubectl get pods -n flowcore

Note

It can take up to 30 minutes for the Flowcore Platform to be ready. So grab a coffee and relax.

3. Once the Flowcore Platform is ready, you should be able to access any data or state through the Flowcore Platform UI and CLI.

Tip

The ingestion endpoint and event source endpoints are used by the Flowcore Platform control plane to route your requests to the correct Flowcore Tenant when fetching data, state or ingesting data.

Troubleshooting

If you are experiencing issues with the Flowcore Platform, please contact support at support@flowcore.io.

Common Issues

1. Flowcore Platform is not ready.

   kubectl get pods -n flowcore

   If the Flowcore Platform is not ready, please check the logs of the pods in the flowcore namespace.

2. Flowcore Platform is not ingesting data.

   kubectl get ingress -n flowcore

   Make sure the ingress resource is created and is accessible, also make sure the flowcore.ingestionUrl is correct.

3. The Flowcore Platform UI or CLI is not fetching any logs from my transformers.

   kubectl get ingress -n flowcore
   kubectl get pods -n flowcore

   Make sure the pods are running and are accessible, also make sure the flowcore.stateUrl is correct.

   kubectl get pods -n org-<your tenant>-config

   Make sure your transformers are running and are accessible.

4. I have ingested data but the Flowcore Platform UI does not show any data in my data cores.

   kubectl get ingress -n flowcore

   Make sure the ingress resource is created and is accessible, also make sure the flowcore.eventSourceUrl is correct.