Stream

The Flowcore CLI allows you to stream data from Flowcore to your local machine. This is useful for debugging, development and local exploration of your data.

Pick data to stream

Navigate to the Data Core, Flow Type and Event Type you want to stream. Click on the menu icon and select flowcore-cli stream command.

This will open a dialog with the command you need to run to start streaming data to your local machine.

Copy the command and run it in your terminal.

Wildcard

To stream all events from a Data Core, you can use the * wildcard to stream all events from the Data Core.

flowcore-cli stream "https://flowcore.io/<tenant>/<datacore>/*"

or to stream all events from a Flow Type, you can use the * wildcard to stream all events from the Flow Type.

flowcore-cli stream "https://flowcore.io/<tenant>/<datacore>/<flowtype>/*"

info

It is also possible to use the wildcard when specifying event types. For example event.user.*.0 this will stream all events that start with event.user..

Configuration

The default command will log out the data to the terminal. If you want to send data to a different location, you can use the subcommand http and specify a -d flag to specify the http endpoint to send the data to.

To specify how many hours, days, weeks, months or years of data you want to stream, you can use the -s flag to sepecify the start time to fetch events. The -s flag accepts a string in the format 1h, 1d, 1w, 1m or 1y.

JSON

You can also pipe the output to a file or another command by using the > operator.

flowcore-cli stream "stream url" -s 1h --json > output.json

Local development

If you have a development defined in the flowcore manifest, you can use the local command to start a stream thread for each of the proxy endpoints. This is useful for local development and debugging.

flowcore scenario local -f flowcore.yaml -e http://localhost:3000

This will start a stream thread for each of the endpoints specified and send the data to the specified endpoint.

development:
  proxyEndpoints:
    people:
      description: "example proxy endpoint"
      dataCore: example
      flowType: example.flow.0
      events:
        - example.flow.created.0
        - example.flow.updated.0
        - example.flow.archived.0
      endpoints:
        - "/api/transform/flow.0"

info

This is especally useful when you are development an application using for example Next.js and you deploy the transformers as api endpoints.

Payload

If you only want the payload of the event, you can use the --payload flag to only stream the payload of the event.

flowcore-cli stream "stream url" -s 1h --payload

This will only log the payload of the event to the terminal.

Only Live

If you only want to stream live data and not data in the past, you can use the -s flag with now as the value to only stream live data.

flowcore-cli stream "stream url" -s now

This will only stream data from the time you run the command and not any data in the past.

Max Events

If you only want to stream a certain number of events, you can use the -m flag to specify the maximum number of events to stream.

flowcore-cli stream "stream url" -m 100

This will only stream 100 events and then stop the stream.

JSON

If you want to stream the data in JSON format, you can use the --json flag to stream the data and output it in JSONL format.

flowcore-cli stream "stream url" --json

This will stream the data and output it in JSONL format, which is useful for piping the output to another command or file.

Destination

If you want to send the data to a different location, you can use the http subcommand and specify a -d flag to specify the http endpoint to send the data to.

flowcore-cli stream http "stream url" -d http://localhost:3000/api/transform

This will send the data to the specified endpoint.

info

If you want to stream the data to a different Data Core, then you need to use the --payload flag to only stream the payload of the event to the webhook endpoint.

info

The default output is http so you don’t have to specify the -o flag if you want to send the data to a different location. And the default endpoint is http://localhost:3000/transform so you don’t have to specify the -d flag either if you want to send the data to the default endpoint.

Headers

If you want to send headers with the data, you can use the -H flag to specify the headers to send with the data.

flowcore-cli stream http "stream url" -d http://localhost:3000/api/transform -H "Authorization: Bearer token"

This will send the data with the specified headers, you can specify multiple headers by using the -H flag multiple times.

Only stream what is there, don’t switch to live

If you only want to stream what is currently in the Data Core and not switch to live data, you can use the --no-live flag to only stream what is currently in the Data Core.

flowcore-cli stream "stream url" --no-live

This will only stream what is currently in the Data Core and not switch to live data after reaching the end of the data stream.

Stream from a specific time

If you want to stream data from a specific time, you can use the -s flag to specify the start time to fetch events. The -s flag accepts a string in the format ISO date, 1h, 1d, 1w, 1m or 1y.

flowcore-cli stream "stream url" -s 1h --no-live

This will stream data from the last hour and then stop the stream.

Stream from and to a specific time

If you want to stream data from a specific time to a specific time, you can use the -s and -e flags to specify the start and end time to fetch events. The -s and -e flags accept a string in the format ISO date, 1h, 1d, 1w, 1m or 1y.

flowcore-cli stream "stream url" -s 2022-01-01T00:00:00Z -e 2022-01-02T00:00:00Z

This will stream data from the 1st of January 2022 to the 2nd of January 2022 and then stop the stream.

info

When using the -s and -e flags, the stream will stop when it reaches the end time specified.