1. Ingestion

Ingestion is the process of sending data to Flowcore, where it is stored in a data core. The data can essentially come from any source, but in this case we will be focusing on data that it is ingested from your local machine.

Before we can start to ingest data into Flowcore, we first need to create a data core.

Likewise, for the purpose of this workshop, we need to output the data to the Flowcathon mini app, using a scenario. That way we can see the ingested data live within the app.

Note

We will touch on the concept of scenarios, in the proceeding challenges.

1. Obtain your tenant name

   Go to flowcore.io, and copy your tenant name. You can find your tenant name like this.

   Caution

   If you have given your tenant a display name, then you will find your tenant name by looking at the drop down in the top left corner of the page, or by pressing “Settings”, in your tenant profile.

2. Scaffold your data core and scenario

   create a flowcore.yaml file somewhere on your machine, with the following content:

   flowcore.yaml

   version: 1
   ## ⬇️ INSERT YOUR TENANT NAME HERE ⬇️
   tenant: <your-tenant-name>  # The name of your tenant, this can be found by visiting your tenant page on flowcore.io
   dataCore:
     name: flowcathon
     deleteProtection: false
     isPublic: false
     description: Data related to the Flowcathon challenges
     flowTypes:
       ingestion.0:
         description: Contains the data for the ingestion challenge
         eventTypes:
           ingestion.challenge-a-ingested.0:
             description: Collecting ingestions for the ingestion challenge A
           ingestion.challenge-b-ingested.0:
             description: Collecting ingestions for the ingestion challenge B
           ingestion.challenge-c-ingested.0:
             description: Collecting ingestions for the ingestion challenge C
   scenario:
     name: flowcathon
     description: >-
       This scenario is used for development of the Application Flowcathon
       application
     transformers:
       ingestion:
         description: Ingestion Challenge
         dataCore: flowcathon
         flowType: ingestion.0
         events:
           - ingestion.challenge-a-ingested.0
           - ingestion.challenge-b-ingested.0
           - ingestion.challenge-c-ingested.0
         parameters:
           - name: PROXY_ENDPOINT
             type: manual
             value: >-
               https://flowcathon.customer.flowcore.io/api/transform/ingestion.0
           - name: PROXY_HEADER_SECRET
             type: secret
             value: transformer-secret-flowcathon
         blueprint:
           artifactUrl: >-
             https://flowcore-public-runtimes.s3.eu-west-1.amazonaws.com/transformer-proxy-1.4.1.zip

   Tip

   It is also possible to create a flowcore.yaml file, by running the following commands:

   flowcore data-core init -t <your-tenant>
   flowcore scenario init -t <your-tenant>

   You can read more about it here.

3. Create a transformer secret

   To the keen eye, some of you may have noticed something called transformer-secret-flowcathon in the flowcore.yaml file above. This secret is used to give you permission to output your data to the flowcathon mini app.

   Currently, the scenario will fail, because it cannot find the secret in your tenant. Therefore, we need to create a secret in your tenant, called transformer-secret-flowcathon.

   The secret value is provided by the instructors.

   Run the following command to create the secret:

   flowcore auth new secret transformer-secret-flowcathon -t <your-tenant-name> --from-literal <the-instructor-provided-transformer-secret-value>

   You can read more about flowcore secrets here.

   Please verify that the secret has been created, by navigating to your tenant page on flowcore.io, press the “Settings” button, and scroll down until you see the secret.

4. Deploy your manifest (resulting in the creation of a data core and scenario on Flowcore.io)

   Now that you have scaffolded your entire project, you can deploy your manifest with:

   flowcore apply -f flowcore.yaml

   When asked if you want to create your data core, say yes. Afterwards you will be asked if you want to create your scenario, say yes again.

   Once the command is finished, your scaffolded project manifest will have been setup in flowcore.io, under your tenant. Verify these resources have been created by:

   • viewing your data core under your tenant page,
   • and viewing the scenario on the scenario page. (which you can visit by pressing the scenario button on the left hand side)

   Tip

   To figure out if your scenario is ready and running, you can view the transformer logs from here:

5. Ingest your data

   Firstly, generate an api key for your tenant, by running the following command:

   flowcore auth new key <your-api-key-name> -t <your-tenant>

   Caution

   Remember to save the key, as it will not be shown again.

   This key is used to confirm that you are allowed to ingest data into your tenant. Please copy the key, as you will need it in the next step. You can read more about the api keys here.

   Next, go to the ingestion page, and generate a webhook ingestion url, as shown in the first section here.

Challenge A

Send some data to your ingestion url, and observe the data showing up on the on the flowcathon mini app. This can be done from your terminal, with something like curl.

For example:

Mac OS

Using curl in a unix compatible terminal

curl -X POST "<your-ingestion-url>" \
-H "Content-Type: application/json" \
-d '{"hello": "<your-name>"}'

Windows

Using Invoke-WebRequest in Powershell

Invoke-WebRequest -Uri "<your-ingestion-url>" -Method Post -Body @{hello="<your-name>"}

Software

You can, for example, use one of the following software options:

• Postman
• Insomnia

Note

You can also verify that the data is ingested, by viewing your data core on flowcore.io.

Challenge B

Now that you have confirmed that you are able to ingest data, lets try to ingest data into a separate event type. Try sending some data to your ingestion.challenge-b-ingested.0 event type. To do so, please refer to Challenge A, as well as the prerequisites.

Challenge C

Now that you have figured out how to ingest data into multiple different event types. You now need to process a .csv file into yet another event type, called ingestion.challenge-c-ingested.0.

So, what you need to do is this:

1. Download the csv file from the flowcathon mini app
2. Write a script (or use any method you like) to read the csv file
3. Send the entries in the csv file to your ingestion.challenge-c-ingested.0 event type in the format of a json array like this:

[
  {"Name": <name>, "Age": <age>, "Email": <email>, "City": <city>, "Occupation": <occupation>, "Challenger": <your name>},
  ...
]

Tip

Flowcore also supports bulk ingestion, which means that you can ingest multiple events at once. You can read about it here.

Enter Access Code

This content is for an on-site Flowcathon. Your instructor will provide you with the access code.

Submit