2. Stream

The data core challenge shows you how to stream data from a data core in the Flowcore Platform. But first we need to setup the platform for the challenges.

Challenge A

Now that you have ingested some data into your data core.

We can try to stream the data back onto your machine. This requires you to both find the stream url, as well as stream the data onto your machine.

Please, follow this guide to stream the data back onto your machine.

Challenge B

Update your data core and scenario with a new flow type and event type.

1. Add a new flow type, dedicated to the stream challenge

   Go to your existing flowcore.yaml file, and add an additional flow type

   flowcore.yaml

       stream.0:
         description: Contains the data for the stream challenge
         eventTypes:
           stream.challenge-b-solved.0:
             description: Collecting stream data for the stream challenge B

   So that the data core manifest section can look like this:

   flowcore.yaml

   dataCore:
     name: flowcathon
     deleteProtection: true
     isPublic: false
     description: This data core contains the data for the Application Flowcathon application
     flowTypes:
       ingestion.0:
         description: Contains the data for the ingesdtion challenge
         eventTypes:
           ingestion.challenge-a-ingested.0:
             description: Collecting ingestions for the ingestion challenge A
           ingestion.challenge-b-ingested.0:
             description: Collecting ingestions for the ingestion challenge B
           ingestion.challenge-c-ingested.0:
             description: Collecting ingestions for the ingestion challenge C
       # V- Add this flow type -V
       stream.0:
         description: Contains the data for the stream challenge
         eventTypes:
           stream.challenge-b-solved.0:
             description: Collecting stream data for the stream challenge B

2. Add a new transformer to the existing scenario, dedicated to the stream challenge

   Go to your existing flowcore.yaml file, and add an additional transformer.

   flowcore.yaml

       stream:
         description: Stream transformer
         dataCore: flowcathon
         flowType: stream.0
         events:
           - stream.challenge-b-solved.0
         parameters:
           - name: PROXY_ENDPOINT
             type: manual
             value: >-
               https://flowcathon.customer.flowcore.io/api/transform/stream.0
           - name: PROXY_HEADER_SECRET
             type: secret
             value: transformer-secret-flowcathon
         blueprint:
           artifactUrl: >-
             https://flowcore-public-runtimes.s3.eu-west-1.amazonaws.com/transformer-proxy-1.4.1.zip

   So that the scenario section of your manifest can look like this:

   flowcore.yaml

   scenario:
     name: flowcathon
     description: >-
       This scenario is used for development of the Application Flowcathon
       application
     transformers:
       ingestion:
         description: Ingestion Challenge
         dataCore: flowcathon
         flowType: ingestion.0
         events:
           - ingestion.challenge-a-ingested.0
           - ingestion.challenge-b-ingested.0
           - ingestion.challenge-c-ingested.0
         parameters:
           - name: PROXY_ENDPOINT
             type: manual
             value: >-
               https://flowcathon.customer.flowcore.io/api/transform/ingestion.0
           - name: PROXY_HEADER_SECRET
             type: secret
             value: transformer-secret-flowcathon
         blueprint:
           artifactUrl: >-
             https://flowcore-public-runtimes.s3.eu-west-1.amazonaws.com/transformer-proxy-1.4.1.zip
       # V- Add this transformer -V
       stream:
         description: Stream transformer
         dataCore: flowcathon
         flowType: stream.0
         events:
           - stream.challenge-b-solved.0
         parameters:
           - name: PROXY_ENDPOINT
             type: manual
             value: >-
               https://flowcathon.customer.flowcore.io/api/transform/stream.0
           - name: PROXY_HEADER_SECRET
             type: secret
             value: transformer-secret-flowcathon
         blueprint:
           artifactUrl: >-
             https://flowcore-public-runtimes.s3.eu-west-1.amazonaws.com/transformer-proxy-1.4.1.zip

3. Deploy your manifest changes

   This is done, just like before, by running the following cli command:

   flowcore apply -f flowcore.yaml

Now that you have confirmed that you are able to stream data from your data core, lets try something a bit more challenging, here is a public data core that you can stream from. Your challenge is to find a unique event (the needle) in this haystack of events. Then you will need to send that event id back to us.

Then you can post your event back to the stream.challenge-b-solved.0 event type in the stream.0 flow type in your own data core. The data needs to be in the following format:

{
  "needleId": "<id in the needle payload>",
  "challenger": "<your name>"
}

Enter Access Code

This content is for an on-site Flowcathon. Your instructor will provide you with the access code.

Submit