In the Output Settings screen, users can see the various components that make up an Output, including tabs for Output Settings, Mappings, and Output History. When initially configuring an Output, this is the only visible tab.
Initial Parameters
-
Name: The name of the Output. Every output in the DataForge environment must have a unique name.
-
Description: The description of the Output.
-
Active: If set to Active, the Output will be immediately available for use, and any sources connected to the output through a Channel will automatically run Output at the end of all processes.
Output Type
It is important to decide which Output Type an Output is. There are three types described below. The parameters available will dynamically change depending on users' selections.
Batch sources can be mapped to any output type. Stream sources support mapping to output types of Event, Delta Table, and Virtual.
Event:
DataForge will publish data to an Event topic. For more information on setting up an Event output, refer to the User Manual's Kafka Events Connections documentation.
File:
DataForge will output a File using a File Connection.
There are five Output File Types: Avro, CSV, JSON, Parquet, or Text. Parameter selections will update dynamically depending on the selection.
Table:
DataForge can output and refresh data to a database Table using a Table Connection. The type of table output is defined by the Driver option on the Output Connection used.
Virtual:
Rather than push data out to a separate system, DataForge can also manage a database view on top of the Hub tables within the connected Databricks environment, otherwise known as a Virtual Output.
Output Parameters
Asterisks (*) in the Parameter Name column indicate mandatory parameters
Output
| Appears Under | Parameter Name | Default Value | Description |
| Table, File | Connection* | Name of the DataForge connection used to write the output to the desired destination | |
| Table | Table Name* | Name of output table to be written in the target DB | |
| Table | Table Schema* | Name of the schema the output table will be written to in the target DB. | |
| Table | Output Source Id Partition | TRUE | Designates whether output_source_id will be top level partition and only applies to Delta Lake outputs. |
| Table | Manage Table | TRUE | Decides if you want table to be altered if there are missing columns, or automatically updated if new columns are added |
| Table |
Reset Channels On
Delta Overwrite
|
TRUE | Reset all other output channels when Delta Lake table is overwritten due to schema changes. Only applies to Delta Lake output. |
| Table | Delete Orphan Data | TRUE | Delete data from output table that was created by a channel that no longer exists. Does not delete rows where s_output_source_id is null |
| Table | Create Cci On Table | FALSE | Creates clustered columnstore index on destination table. We recommend CCI on SQL server output tables to improve performance and avoid deadlocks. |
| Virtual | View Name* | Name of the Virtual Table/View that will appear in the Databricks environment once the output is processed. | |
| Virtual | View Database | Default | Name of database for virtual view in Databricks. |
| File Type: All | File Name* | File name for output file. For all except text, extension will be added automatically. | |
| File Type: Text | File Extension* | Extension that will be appended to the file name when DataForge writes text file outputs | |
| File Type: All | Single File |
TRUE (FALSE for Parquet) |
Toggle for whether or not the output should be written to multiple files or one single file. Multiple files is more performant |
| File Type: Avro, CSV, JSON, Parquet | Limit By Effective Range | FALSE | Output uses effective range calculations to limit data for Time Series sources |
| Event | Value Schema Type | avro_from_registry |
Schema type for selected topic. Used to associate data with schema type during output |
| Event | Key Schema Type | avro_from_registry |
Schema type for selected topic. Used to associate data with schema type during output |
| Event | Key Schema |
Json (avro) or text (json DDL) specified |
|
| Event | Value Schema |
Json (avro) or text (json DDL) specified |
|
| Event | Key Subject |
Schema registry subject for key schema |
|
| Event | Value Subject |
Schema registry subject for value schema |
Output Retention
| Appears Under | Parameter | Default Value | Description |
| File Type: CSV, Parquet | Archive Files | 1 year | How long files will remain in archive folder. |
| File Type: CSV, Parquet | Buffer Files | 0 | Interval to retain files in fast output buffer storage |
| File Type: CSV, Parquet | Temporary Files | 14 days |
Amount of time temporary files will remain stored
|
Output Alerts
| Appears Under | Parameter | Default Value | Description |
| All | Output Failure Topics | List of AWS Simple Notification Service topic ARNs to be notified upon Output Failure | |
| All | Output Success Topics | List of AWS Simple Notification Service topic ARNs to be notified upon Output Success | |
| All | Output Failure Emails | List of email addresses to be notified upon Output Failure. Requires SMTP setup. | |
| All | Output Success Emails | List of email addresses to be notified upon Output Success. Requires SMTP setup. |
Post Output Commands *
When Custom Notebook value is selected, a Custom Cluster Configuration for the Post-Output step needs to be selected.
Updated