The Processing Queue tab provides an interactive overview of all processes completed, active, errored, or otherwise in the system, filtered by day.
Sample Processing Queue
By default, the process tab is collapsed only to show "process chains." A process chain is a set of processes grouped by their initializing process and executed as a series, with one process in the chain causing the next to be triggered via the DataForge workflow engine.
Each process chain can be expanded to show that chain's sub-processes to help with detailed investigation and log analysis.
To cancel a currently running or launching process, click the Job Run ID link and select the Cancel Job Run option.
The fields displayed for each process are as follows:
- Process ID: A globally unique ID for this process. Hyperlinks can be copied for easy sharing.
- Timeline: A rough sketch overview of what sub-processes happened in relation to the other visible timelines on the page.
- Clear/empty portions represent cluster startup or other infrastructure-related activity where no data processing is being performed, and no cloud service charges are accrued
- Blue/filled portions represent active processing by the system
- Scope: Each sub-process has a scope of operation
- Input: This process only operated on a single input
- Source: This process operated on all data stored in the hub table across all inputs
- Output Channel: This process used data from a specific Channel within a Source to Output Channel mapping
- Id: For this specific Scope, the ID of the affected configuration elements and data
- Operation: Process type for each process record
- Job Run ID: Id of the job run the process was executed in. Click hyperlink to access a job run in Databricks, review job run details, or cancel a job that is either launching or currently running.
- Status: An icon and help-text indicating the result of the Processing Step
No errors or warnings. Processed successfully
A failure has occurred in the process chain
The process is currently running
The process is launching a new cluster
The process chain has encountered an error but recovered successfully
- #: Processed record counts popup to access and display metadata about the data processed
- Log: Provides a popup to the indexed logging stored for this specific process. Export logs to CSV and open support tickets as required from this screen.
- Parameters: Provides a popup with the details of the parameters passed into the DataForge job executable, which uses these parameters to execute that processing stage properly.
Updated