User Manual

Sources

  • Sources Overview

    A Source represents a single schema of data. Sources are the logical grouping for Inputs, Relations, and Rules...

  • Source Settings

    The Settings tab for a Source allows a user to specify key information about the Source including input types ...

  • Raw Schema

    The Raw Schema tab for Sources allows users to view the raw database attributes as well as raw metadata. Viewi...

  • Dependencies

    Dependencies allow configurators to modify the workflow engine to introduce waits to the processing queuesIntr...

  • Relations

    Relations define intra-source connections and enable users to configure lookups and cross-source aggregates.Cr...

  • Rules

    Rules allow DataForge to modify and transform data.Rules TabThe Rules tab allows users to select, edit, remove...

See all 14 articles

Processing

  • Processing Queue

    The Processing Queue tab provides an interactive overview of all processes completed, active, errored, or othe...

  • Ingestion Queue

    Sample Ingestion QueueThe Ingestion Queue tab provides a view of all ingestions that are waiting to run, curre...

  • Workflow Queue

    Sample Workflow Queue The Workflow Queue tab provides a view of processes that have upstream dependencies they...

  • Job Runs

    The Job Runs table shows detailed information on all job runs in the current environment. Both active and term...

  • Resetting Processes

    Types of ResetsDuring the development lifecycle, users will need to reset their sources often to change source...

  • (Databricks only) Recommended Compute Configurations during Development

    For environments that are undergoing daily development, it may be beneficial to set up your compute configurat...

Outputs

  • Outputs Overview

    The Outputs screen allows users to see all Outputs, as well as search, edit, and filter them. By default, only...

  • Output Settings

    In the Output Settings screen, users can see the various components that make up an Output, including tabs for...

  • Output Mapping

    Output Mapping controls the way data is sent to its final destination. It allows a user to rename columns, app...

  • Output Process

    This tab provides a quick view into the Processing page, pre-filtered for this Output, to assist with debuggi...

Connections

  • Connections

    The connections home page enables users to quickly search and access connections already configured in the Da...

  • Generic JDBC Connection

    DataForge offers a generic JDBC connection type for ingesting data from external databases, outside of the pre...

  • Salesforce Connection

    DataForge offers a pre-built connector for Salesforce.  Before data can be ingested from Salesforce, users nee...

  • Kafka Events Connection

    DataForge integrates directly with Kafka Event Topics for batch or stream data Source ingestion and Output pub...

  • (Databricks only) Unity Catalog

    DataForge supports reading from and writing to tables stored in Databricks Unity Catalog.  RequirementsUnity C...

Schedules

  • Schedules

    A schedule uses a CRON expression to determine how often source inputs are updated. Multiple sources can be as...

Lineage

  • Lineage Graphs

    Lineage refers to a directed acyclic graph (DAG) generated by DataForge describing how data is processed, trac...

System Configuration

  • Global System Configurations

    WARNING: Changes to these settings can have a severe impact to the platform and break functionality if set inc...

  • Cleanup Configuration

    Cleanup Configuration defines retention settings for data lake objects and metadata. It is accessible via main...

Projects

  • Projects Overview

    Projects represent a group of configurations in DataForge that users control and are the primary vehicle for m...

  • Managing Projects with Github

    Managing Project configurations with GitHub allows users to efficiently merge changes from one project to anot...

Users - Access Administration

  • Manage Users

    This article explains how to manage user access to DataForge workspaces.  Overview of User Access:User access ...

Talos AI

  • Talos: Your AI data assistant

    Getting started with TalosTalos is your intelligent assistant for data management, automation, and analytics w...

Templates and Tokens

  • Templates and Tokens Overview

    Templates combined with tokens enable centralized deployment and management of re-usable Rules and Relations a...

  • Tokens

    Token management screens overviewTokens can be managed and created in the Tokens page which is found by openin...

  • Relation Templates

    Relation Templates Use CaseWhen configuring multiple Sources within DataForge, it is common for repeated patte...

  • Rule Templates

    Rule Templates Use Case When configuring multiple Sources within DataForge, it is common for repeated pattern...

  • Best Practices

    Best practices, patterns and recommendations for template configuration and managementDoWhen first starting to...

Agents

  • Agents

    OverviewAgents are a lightweight application that can be installed on a machine to allow data ingestion from a...

  • Logs

    UI LogsAgent logs can be found in the UI by clicking on the Logs icon on the Agents screen. Logs can be filter...

  • Installing a New Agent

    Details requirements and configurations to installing an Agent on a server that can access an on-premise data ...

SDK

Cloning

  • Cloning Overview

    Summary and Use Case Cloning allows users to create copies of multiple sources/outputs and the relationships t...