Sample Ingestion Queue
The Ingestion Queue tab provides a view of all ingestions that are waiting to run, currently running, or have completed their run. Similar to the Job Runs table, the Ingestion Queue can be filtered with multiple values.
Opening the Filter Type dropdown gives users several filter options: Status, Agent, Connection, Schedule, and Source. Once a type has been selected, users can then either search for the desired value or manually scroll through the populated dropdowns. Once a filter is applied, it appears as a filter tag on the screen. Multiple filter types can be used but only one filter of each type is currently allowed.
The Ingestion Queue automatically refreshes its data. If there are more rows than can be displayed on screen, the Load More... option will be available.
Users can also navigate to other areas of DataForge user interface from the Ingestion Queue. The Source, Connection, Agent, and Schedule columns are all linked to their appropriate pages. Clicking a link will open a new tab to the desired page.
Detailed breakdown of each header:
- ID: Job Run ID of the ingestion record
- Status: Current status of the ingestion. Queued if in the future and Completed if in the past.
- Initiation Type: Reason for the ingestion record to be in the queue. Options include: Scheduled, Pull now ui, and Retry. Pull now ui is triggered from a user using "Pull Data Now" option in source. Retry is based on number of retries set in the Performance & Cost parameters of the source settings.
- Source: Name of the Source for the ingestion record.
- Connection: Name and link to the Connection tied to the source for the ingestion.
- Schedule: Name and link to the Schedule set on the source for the ingestion.
- Agent Code: Agent used for the ingestion tied to the Connection. Local or on-premise set up in the Agents page.
- Scheduled Start Date: Date and Time the ingestion is/was set to run in local time.
The Ingestion Queue is a good tool to use to identify jobs that may have issues and didn't run at the time they were meant to. If there are ingestion records that are still queued with a scheduled start date in the past, it may be that the agent is dead and needs to be reviewed or that core is dead and needs to be restarted.
Updated