The DataForge workspace is broken down into multiple pages and separate tabs within each page.
Navigation Menu
Upon login, users are directed to the main Sources screen. The Navigation Menu, located on the left side of the screen and collapsible, allows users to navigate between the primary DataForge pages.
User Menu
The User menu in the top-right allows for switching themes between light mode and dark mode, managing the DataForge account for account owners, and logging out.
Primary Interfaces
Sources
The main Sources page provides a view of all configured sources along with the ability to create new sources using the New + button. Open a source by clicking the name or row of the Source.
After opening a Source, users are presented with multiple tabs for configuration and monitoring of the source.
The Source page controls loading data from external systems into DataForge, assesses quality via Validation Rules, and transforms data using Relations and Enrichment Rules. Each source contains an Inputs tab to track the progress of Alloy data refinement and enable the restart of any failed or misconfigured refinement tasks. Lastly, it contains a Data View link to view and query the data for this Source stored within the Data Hub.
When thinking of the Alloy refinement process, consider the Sources page as the manager from raw data ingestion to output.
For more information regarding Sources, refer to the User Manual's Sources documentation.
Processes
The Processes page shows the progress and outcome of all Alloy refinement through the platform. Processes can be viewed by project or across the whole workspace by using the navigation menu Processing page, or for a specific source or output by opening the Process tab within the source or output. In the Processes page, users are given tabs for viewing the Ingestion Queue, Workflow Queue, and active and past Job Runs.
For more information regarding Processing, refer to the User Manual's Processing documentation.
Outputs
The main Outputs page provides a view of all configured outputs along with the ability to create new outputs using the New + button. Open an output by clicking the name or row of the output.
After opening an Output, users are presented with multiple tabs for configuration and monitoring of the output.
The Output page controls loading data from the hub tables to external systems. Each output provides tabs of Settings, Mapping, and Process for configuration and monitoring of the output. Sources are mapped to Outputs and added columns are mapped to source fields available. Use the arrows in the bottom right to navigate through pages of column mappings.
For more information regarding Outputs, refer to the User Manual's Outputs documentation.
Connections
The Connections page provides a view of all configured connections along with the ability to create new connections using the New + button. Open a connection by clicking the name or row of the connection. Connections are created for source systems and output systems to allow Sources and Outputs to communicate with external data systems.
After opening a Connection, users are presented with three tabs for configuring Settings, checking Connection Metadata, and viewing Applied Objects which are the sources and outputs using the connection. Users are able to create new sources directly from Connection Metadata.
For more information regarding Connections, refer to the User Manual's Connections documentation.
Talos AI
Talos AI is your AI assistant, helping you develop data models, operate pipelines, and manage complex dependencies using natural language prompts. Quickly turn requests and requirements into fully productional data workflows. Open Talos AI via the main menu or the quick link in the top right corner.
Templates
Templates provide users the ability to templatize objects within the application which can be quickly and easily applied to multiple Sources, Relations, Rules, Outputs, and Connections. Templates are a key building block to allow users to mass clone groups of Sources, Outputs, and Connections that require similar configurations.
For more information regarding Templates, refer to the User Manual's Templates documentation.
Schedules
Schedules are used to dictate when data ingestions and processing occur. They are also used to allow users to schedule system-led activities such as Automatic Upgrades and Cleanup processing. Schedules are attached to Sources to control the timing and frequency of job runs.
For more information regarding Schedules, refer to the User Manual's Schedules documentation.
Lineage
Lineage provides users the ability to trace origin and destination objects connected to each other within the application. A graph legend is available in the top right corner to assist with symbol and coloring dictation.
Users start a new lineage graph or add to an existing one by clicking the Lineage symbol where it appears on each page.
For more information regarding Lineage, refer to the User Manual's Lineage documentation.
Compute and Process Config
Compute and Process Configuration pages give you the ability to create, manage, and hyper-tune the compute used for all data processing. Compute configurations allow you to set the size, type, and settings of the compute running behind the scenes for each job. Process Configurations allow you to set a default compute and overrides for specific compute configurations for specific process types. Process configurations are attached to sources in source settings.
Managed compute and process configs provide easy to use options to switch source processing to, but will limit your ability to change config settings. Clicking on a row/config will open it's settings and allow you to see applied objects where the config is used throughout DataForge.
For more information, refer to the User Manual's Compute Configuration and Process Configuration documentation.
Cleanup
The Cleanup page allows you to manage the default and create new cleanup configurations. These are used to enforce retention policies on your configurations and data in DataForge. Cleanup configurations are attached to sources in source settings. Clicking on a row/config will open it's settings and allow you to see applied objects where the config is used throughout DataForge.
For more information, refer to the User Manual's User Manual's Cleanup Configuration documentation.
Projects
Projects are a top-level container and provide the ability to group sets of configurations within an environment and easily migrate configurations. Users can export and import project contents from one project to another. Users have the ability to create, delete, and manage projects within the DataForge Environment Projects page.
For more information on how to use Projects, refer to the User Manual's Project documentation.
Users
The Users page is available in the main navigation menu of DataForge. Depending on the type of deployment, users will experience different options. SaaS users use this page to make access changes to users. For private deployments, the Users page is a visual representation of who has access to the environment through Auth0 and should be managed in Auth0.
For more information on how to manage Users, refer to the User Manual's User documentation.
System
The System page contains workspace-wide settings and configurations such as Auto-Upgrade/Maintenance and Global Ingestion Disable/Enable options. DataForge recommends avoiding updating the parameters in the System page unless advised to do so as incorrect changes can create workspace-wide issues. Please reach out to DataForge support if a question needs answering on how the service configurations work or should be updated.
For more information, refer to the User Manual's System documentation.
Updated