The DataForge interface is broken down into multiple interfaces and separate tabs within each interface as sub-pages.
Navigation Menu
Upon login, users are directed to the main Sources screen. The Navigation Menu, located in the top left corner of the screen, allows users to navigate between the primary DataForge pages.
Primary Interfaces
Sources
The main Sources page provides a view of all configured sources along with the ability to create new sources using the New + button. Open a source by clicking the name or row of the Source.
After opening a Source, users are presented with multiple tabs for configuration and monitoring of the source.
The Source interface controls loading data from external systems into DataForge, assesses quality via Validation Rules, and transforms data using Relations and Enrichment Rules. Additionally, it contains the Inputs tab to track the progress of individual file processing and enable the restart of any failed or misconfigured processing tasks. Lastly, it contains a Data View link to view and query the data for this Source stored within the Data Hub.
When thinking of the logical data flow, consider the Sources interface as the manager from raw data ingestion to output.
For more information regarding Sources, refer to the User Manual's Sources documentation.
Processing
The Processing interface shows the progress and outcome of all execution tasks through the platform, including sub-processes. Processing can be viewed environment-wide by using the navigation menu, or for a specific source or output by opening the Process tab within the source or output. In the environment-wide Processing interface, users are given tabs for viewing the Ingestion Queue, Workflow Queue, and active and past Job Runs.
For more information regarding Processing, refer to the User Manual's Processing documentation.
Outputs
The main Outputs interface provides a view of all configured outputs along with the ability to create new outputs using the New + button. Open an output by clicking the name or row of the output.
After opening a Source, users are presented with multiple tabs for configuration and monitoring of the source.
The Output interface controls loading data from the Data Hub to external systems, or the Data Warehouse layers. Each output provides tabs of Settings, Mapping, and Process for configuration and monitoring of the output. Sources are mapped to Outputs and added columns are mapped to source fields available. Use the arrows in the bottom right to navigate through pages of column mappings.
For more information regarding Outputs, refer to the User Manual's Outputs documentation.
Connections
The main Connections interface provides a view of all configured connections along with the ability to create new connections using the New + button. Open a connection by clicking the name or row of the connection. Connections are created for source systems and output systems to allow Sources and Outputs to communicate with external data systems.
After opening a Connection, users are presented with three tabs for configuring Settings, checking Connection Metadata, and viewing Applied Objects which are the sources and outputs using the connection. Users are able to create new sources directly from Connection Metadata.
For more information regarding Connections, refer to the User Manual's Connections documentation.
Templates
Templates provide users the ability to templatize objects within the application which can be quickly and easily applied to multiple Sources, Relations, Rules, Outputs, and Connections. Templates are a key building block to allow users to mass clone groups of Sources, Outputs, and Connections that require similar configurations.
For more information regarding Templates, refer to the User Manual's Templates documentation.
Schedules
Schedules are used to dictate when data ingestions and processing occur. They are also used to allow users to schedule system-led activities such as Automatic Upgrades and Cleanup processing. Schedules are attached to Sources to control the timing and frequency of job runs.
For more information regarding Schedules, refer to the User Manual's Schedules documentation.
Lineage
Lineage provides users the ability to trace origin and destination objects connected to each other within the application. A graph legend is available in the top right corner to assist with symbol and coloring dictation.
Users start a new lineage graph or add to an existing one by clicking the Lineage symbol where it appears on each interface.
For more information regarding Lineage, refer to the User Manual's Lineage documentation.
System Configuration
System Configuration pages give the ability to manage cluster and process configurations used for all processing within the DataForge environment. Users will also be able to manage cleanup configurations and system wide configurations such as Auto Upgrade and Global Ingestion Disable/Enable. DataForge recommends avoiding updating the parameters in the Service Configurations page unless advised to do so as incorrect changes can create environment-wide issues. Please reach out to DataForge support if a question needs answering on how the service configurations work or should be updated.
For more information, refer to the User Manual's Cleanup Configuration, Cluster Configuration, Process Configuration, or Service Configuration documentation.
Projects
Projects are a top-level container and provide the ability to group sets of configurations within an environment and easily migrate configurations. Users can export and import project contents from one project to another. Users have the ability to create, delete, and manage projects within the DataForge Environment Projects page.
For more information on how to use Projects, refer to the User Manual's Project documentation.
Users
The Users page is available in the main navigation menu of DataForge. Depending on the type of deployment, users will experience different options. SaaS users use this page to make access changes to users. For private deployments, the Users page is a visual representation of who has access to the environment through Auth0 and should be managed in Auth0.
Updated