overwrite
If a data core already exists in the manifest file, you will be prompted to overwrite the existing data core.
pass the --overwrite
flag to overwrite the existing data core without being prompted.
The Flowcore CLI allows you to create resources in Flowcore using a manifest file. This is useful for creating multiple resources at once, or for creating resources in a CI/CD pipeline.
To create a data core, you can use the apply
command with a manifest file. The manifest file should be in YAML format.
tenant: example-tenant
dataCore:
name: example-data-core
description: "An example data core"
flowTypes:
example:
description: "An example flow type"
eventTypes:
example.event.0:
description: "An example event"
to generate a manifest file, you can use the generate
command:
flowcore data-core generate manifest -t example-tenant --placeholder
this will generate a flowcore.yaml
file, then run the following command to apply the manifest file
flowcore data-core apply -f flowcore.yaml
this will create a plan to create the data core, and then apply the plan to create the data core.
overwrite
If a data core already exists in the manifest file, you will be prompted to overwrite the existing data core.
pass the --overwrite
flag to overwrite the existing data core without being prompted.
To create a scenario, you can use the apply
command with a manifest file. The manifest file should be in YAML format.
tenant: example-tenant
scenario:
name: example-scenario
description: "An example scenario"
transformers:
example-transformer:
description: "An example transformer"
dataCore: example-data-core
flowType: example
events:
- example.event.0
parameters:
- name: PROXY_ENDPOINT
type: manual
value: "https://example.com"
blueprint:
artifactUrl: "https://flowcore-public-runtimes.s3.eu-west-1.amazonaws.com/transformer-proxy-1.1.0.zip"
to generate a manifest file, you can use the generate
command:
flowcore scenario generate manifest -t example-tenant --placeholder
this will generate a flowcore.yaml
file, then run the following command to apply the manifest file
flowcore scenario apply -f scenario.yaml
this will create a plan to create the scenario, and then apply the plan to create the scenario.
Parameters
Parameters can be configured with a type of manual
, secret
or variable
. The manual
type is used for
parameters that are manually set by the user. The secret
type is used for parameters that are sensitive and
are stored on the organization level. The variable
type is used for parameters that are set on the organization level and
are shared across all scenarios in the organization.
If you don’t have any parameters to set, you can simply write the following
parameters: []
.
You can also apply all resources in a manifest file at once by using the root apply
command:
flowcore apply -f manifest.yaml
By default, data cores are created without delete protection enabled. You can enable delete protection by setting
deleteProtection
to true
for each resource in the manifest file.
tenant: example-tenant
dataCore:
name: example-data-core
deleteProtection: true
Keep in mind that if delete protection is enabled on a resource all nested resources will also have delete protection enabled. To delete a resource with delete protection enabled, you must first disable delete protection.
By default, transformers are created with a size of SMALL
. You can set the size of the transformer by setting the size
field to SMALL
, MEDIUM
or LARGE
or XLARGE
in the manifest file.
tenant: example-tenant
scenario:
name: example-scenario
transformers:
example-transformer:
size: "medium"
This will set the resource requirements for the transformers. The size of the transformer will determine the amount of CPU and memory allocated to the transformer.
Sizes
In the table below you can see the resource requirements for the different sizes of transformers. The transformer unit indicates what the transformer will consume in terms of cost. A single unit costs $10
per month. (5
are included in the free tier)
Size | CPU (vCPU) | Memory (MB) | Transformer Units |
---|---|---|---|
SMALL | 0.25 | 512 | 1 |
MEDIUM | 0.5 | 512 | 2 |
LARGE | 1 | 1024 | 4 |
XLARGE | 2 | 2048 | 8 |
By default, transformers are created with a page size of 1000
. You can set the page size of the transformer by setting the pageSize
field to a number in the manifest file.
tenant: example-tenant
scenario:
name: example-scenario
transformers:
example-transformer:
pageSize: 5000
Available values
The page size determines how many records the transformer will pull from the data core at a time. The page size can be set to 1000
, 5000
or 10000
.
By default, transformers are created with a default start date of 7 days
. You can set the default start date of the transformer by setting the defaultStartDateTime
field to a number in the manifest file.
tenant: example-tenant
scenario:
name: example-scenario
transformers:
example-transformer:
defaultStartDateTime: "2024-04-09T11:01:57Z"
This will set the default start date of the transformer to selected ISO start time.
Accuracy
This will set the “cursor” of the transformer to the closest event to the selected start date. Do not use this field to start before specific events, this is more for setting the start date to a specific time.
By default, transformers are created with a concurrency of 1
. You can set the concurrency of the transformer by setting the concurrency
field to a number in the manifest file.
tenant: example-tenant
scenario:
name: example-scenario
transformers:
example-transformer:
concurrency: 5
This will set the number of concurrent executions of the transformer.
Available values
The concurrency can be set to any number between, but a range of 1
and 50
is recommended. This value will determine the amount of concurrent executions of the transformer, and if the transformer is not designed to handle this, it can result in unexpected behavior.
By default, transformers are created with a single replica. You can set the number of replicas of the transformer by setting the replicas
field to a number in the manifest file.
tenant: example-tenant
scenario:
name: example-scenario
transformers:
example-transformer:
replicas: 3
This will determine the number of replicas of the transformer.
Price
The number of replicas will determine the cost of the transformer. A single replica consumes a transformer unit depending on the size of the transformer. The cost of the transformer is calculated by multiplying the number of replicas with the transformer unit cost.
Experimental
Currently, Postgres and MySQL endpoints are supported as read model endpoints. More endpoints will be added in the future. Also, only the blueprint, matchKey and onMatch fields are checked for updates. If you want to update the table name or type of the endpoint, you will have to delete the endpoint and create a new one.
To add an endpoint to a transformer, you can add the this to the manifest file:
tenant: example-tenant
scenario:
name: example-scenario
transformers:
example-transformer:
endPoint:
name: example-endpoint
type: "DATA_CORE"
definition:
dataCore: some-data-core
flowType: some-flow-type
eventType: some-event-type
This will set up an endpoint for the transformer that will send data returned from the transformer to the specified data core.
To create a read model endpoint, you can add the following to the manifest file:
tenant: example-tenant
scenario:
name: example-scenario
transformers:
example-transformer:
endPoint:
id: 559d9980-0630-4d0f-8ba0-921a944310b2 # visible in the UI, info icon next to the read model name
name: example-endpoint # name of the table in the database
type: "POSTGRES" # or "MYSQL"
definition:
blueprint:
"example_column":
type: "string" # or "text" or "integer" or "decimal" or "boolean"
length: 255 # optional
matchKey: "some-key"
onMatch: "skip" # or "update"
This will set up an endpoint for the transformer that will send data returned from the transformer to the specified database.