Overview
Agents are a lightweight application that can be installed on a machine to allow data ingestion from a system that DataForge may not be networked into already. For more information on the agent installation process, visit the Installing a New Agent documentation.
Agent configurations can be managed or created by navigating to the Agents page from the main menu.
Settings Tab
The Agent Settings Tab can be found after clicking on an agent and allows a user to configure metadata that the Agent uses for system configuration. The parameters control heartbeat intervals, auto-updating, security, Agent concurrency, and plugin configurations.
After the Agent settings are saved, the settings can be accessed at any time through the Agent screen. Note that changing certain settings and saving will prompt a new config file to be downloaded and the agent will stop working until the config file is updated on the target machine. For more information on changing settings, visit the installation guide.
Base Parameters
Asterisks (*) mean the Parameter is mandatory and must be specified by users.
- Name*: Name of the Agent. The name must be unique. It will be displayed on the Agents screen when browsing Agents, and will be shown on Connections that the Agent is connected to.
- Description*: Description of the Agent
- Code*: Agent code of the Agent. This will be the backend identifier of the Agent and is needed during MSI install. Recommended to be lowercase, alphanumeric, and with underscores instead of whitespace.
- Default Agent*: If set to True, the Agent will be auto-populated when configuring a new Connection.
- Active*: If set to Active, the Agent will trigger Ingestions that it's configured to run.
- Region*: AWS region for AWS cloud environments. Azure region for Azure cloud environments
- Machine Guid*: Machine GUID for the host server that the Agent is being installed on. More information on finding Machine GUID can be found in the Installing a New Agent guide.
- Authentication Protocol*: Controls how DataForge Agent authenticates with Workspace and changes config bin file during installation.
- IAM Access Key* (AWS only): AWS IAM access key generated from IAM user that has access to the datalake, encrypted on save and stored in an encrypted config file. Not stored in the metastore.
- IAM Secret Key* (AWS only): AWS IAM secret key generated from IAM user that access to the datalake, encrypted on save and stored in an encrypted config file. Not stored in the metastore.
- Storage Account Name* (Azure only): Name of the Azure storage account that hosts the datalake container.
- Storage Account Key* (Azure only): Access key for the Azure storage account that hosts the datalake container.
- Datalake Container Name* (Azure only): Name of the Azure datalake container.
- AkkaStreamTimeout: JDBC Connection timeout in seconds
- AutoUpdate: If set to True, the on-premise Agent will auto-update when a new DataForge version is deployed in the environment
- MaxResources: Controls how many Table Ingestion processes the Agent can run at once. Default is 4.
- HeartbeatInterval: Main heartbeat of the Agent in seconds. The interval in which Agent checks file locations and Ingestion processes to pick up and run. It is not recommended to set this lower than 30 seconds.
- Launch Retries: Controls whether the Agent launches ingestion retries after restarting if ingestions were in progress and marked failed due to a restart.
Restarting Agent
If an Agent is alive and heartbeating, the Agent can be restarted from the UI. Navigate to the main Agent screen and click on the triple dots to the right of the Agent that needs a restart. A "Restart Agent" button will appear - click it to trigger a restart of the Agent. Please be aware that restarting an agent will mark all in-progress ingestions using the agent as Failed.
The Agent should restart within 30 sec - 1 minute of clicking the button. Check the Agent logs to make sure it restarts successfully.
If the Agent isn't heartbeating, it will most likely need a manual restart. Either restart the Agent service on the machine it is installed on or through the UI option above. Local agent is special in that it is a service running directly in the Cloud Provider. To restart the Local agent, either stop the Agent container task (AWS) or restart the Agent container instance (Azure).
Updated