Installing a New Agent

Details requirements and configurations to installing an Agent on a server that can access an on-premise data source.

DataForge Agents are a means of transporting data from a source data source to a cloud storage location DataForge can manage (either in AWS or Microsoft Azure). Agents are managed through the DataForge interface once installed. This section walks through the process of installing a new DataForge Agent on a Windows server. Currently, the Agent can only be installed on Windows.

Prerequisites (Requirements for Installation)

Prior to installing the DataForge Agent, the following requirements must be met. Please ensure that the machine hosting the DataForge Agent meets the requirements prior to installation.

The machine or VM must be running a Windows operating system:
- Windows 7 / Server 2008 R2 or later
The latest version of the Amazon Corretto JDK 21 should preferably be installed on the destination machine. This is recommended for users with an Azure environment as well since the java version is agnostic of the cloud provider DataForge is used with. Oracle JDK 21 is also sufficient. If neither is installed on the destination machine, navigate to the site link and download and install the Corretto JDK that is compatible with the target architecture and OS of the machine that the Agent will be installed on.
- https://docs.aws.amazon.com/corretto/latest/corretto-21-ug/downloads-list.html
Since the Agent initiates all its own connections, outbound connections to various cloud resources on the public Internet are required. If a firewall is limiting outbound internet access, the following resources should be allowed through the firewall (exact domain names will vary by environment). Note that these are all secure endpoints, so if SSL inspection is enabled at the firewall, special consideration is needed to ensure that the certificate presented to the machine hosting the Agent is trusted.
- AWS S3 / Azure Data Lake Storage via HTTPS (port 443)
- DataForge API endpoint via HTTPS (port 443)
- Auth0 via HTTPS (port 443)
File and database sources that will be accessed through the DataForge Agent must be accessible from the machine the Agent is being installed on. In the case of database sources, many customers install the Agent software directly on the database server - network connectivity from the machine that the Agent software is being installed on is all that is required. Note that the Agent will be performing data pulls and uploading to AWS / Azure, so the recommendation is to not segment off the Agent machine from the sources being accessed in a way that traffic needs to cross a limited capacity network segment to reach those sources.
The SQL user account intended for the Agent to use to access source databases needs to be set up with native database engine authentication. In the case of SQL Server, Windows / Azure AD authentication is not supported for the Agent user and only SQL Server authentication can be used.
- Mixed-mode authentication can still be enabled to allow for integrated authentication for non-DataForge related loads.
A Windows user account should be created to run the agent with username and password. This user must have the ability to access any folders containing source data as well as the ability to run services. The command whoami can be run in a command line to determine the current userid. If a user is not created, there is an option to run the Agent as local system, but is not recommended.
If installing with Authentication Protocol 2.0, access to IAM in AWS or access to the Azure Storage Account that hosts the datalake is recommended.

Machine Guid Command

Your machine Guid will be needed to complete the installation. To get the Machine Guid, run the following command in the terminal on the server that the Agent will be installed on.

reg query HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography /v MachineGuid

Setting up a New Agent Configuration

New Agent Creation

In the DataForge UI, from the left hand menu navigate to Agents, and click New. From this settings interface input the required inputs as seen below. Additional parameters exist for more detailed installations. Note that you will need information about the machine the agent will be hosted on. For more information on setting up the 2.0 protocol specific fields (IAM, Azure Storage), navigate to the section of the guide for setting up 2.0 protocol.

Agent Inputs	Detail
Name	Agent name on the backend, and name reference for sources and other DataForge elements within the UI.
Description	Necessary clarification and detail.
Code	Agent code on the backend - this ideally should be the same as Name
Machine Guid	A specific key to identify the machine the agent is running on, see below for the terminal command to obtain.
Authentication Protocol	1.0 protocol uses Auth0 for agent authentication and will be deprecated in a future release. 2.0 uses DataForge managed authentication and is recommended for all new agent installs. 1.0 is not selectable in SaaS environments.
IAM Access Key (AWS Databricks cloud only)	Authentication protocol 2.0 parameter - the IAM access key for the user that has access to write files to the datalake bucket
IAM Secret Key (AWS Databricks cloud only)	Authentication protocol 2.0 parameter - the IAM secret key for the user that has access to write files to the datalake bucket
Storage Account Name (Azure Databricks cloud only)	Name of storage account that hosts datalake container
Storage Account Key (Azure Databricks cloud only)	Access key for storage account that hosts datalake container
Datalake Container Name (Azure Databricks cloud only)	Name of datalake container

Save the Agent and keep the config file available

When the Agent is being saved, a prompt will appear to Save and Download Config File. Be sure to save this config file as it will be needed after the agent MSI installation below.

Setting up 2.0 Protocol for AWS Databricks Cloud

If the cloud that Databricks is running in is AWS, a user will need to be created in IAM that has access to the S3 Datalake that Databricks uses to process Dataforge files. It is recommended to create one user per agent, and to only give it write IAM permissions, no deletes. A recommended policy to attach to the user would be as follows (replace <datalake-bucket> with your datalake name)

{
"Statement": [
     {
        "Action": [
            "s3:ListBucket",
            "s3:GetBucketLocation"
    ],
   "Effect": "Allow",
   "Resource": "arn:aws:s3:::<datalake-bucket>",
    "Sid": ""
   },
   {
        "Action": [
           "s3:PutObjectAcl",
           "s3:PutObject",
           "s3:GetObjectVersion",
           "s3:GetObject"
   ],
 "Effect": "Allow",
 "Resource": "arn:aws:s3:::<datalake-bucket>/*",
  "Sid": ""
  }
  ],
"Version": "2012-10-17"
}

Once the user is created, go to Security Credentials on the IAM user, and generate Access Keys. Enter the IAM Access Key and IAM Secret Key in the agent parameters page with 2.0 selected.

Setting up 2.0 Protocol for Azure Databricks Cloud

If the cloud that Databricks is running in is Azure, access to the Azure Storage Account and Datalake storage container that Databricks uses to process Dataforge files will need to be configured using a storage account key.

Storage account name will be the name of the storage account

Datalake container name will be the name of the datalake container that is mounted for Databricks access

Storage account key can be found in the "Access Keys" section

Make sure the key selected is "Key" and not "Connection string.

Finally, the Networking on the storage account needs to be set up in a way that the Agent will be able to access the container.

Finding the Agent MSI Install File

The Agent MSI install file is downloadable from the Agents page. In the DataForge UI, from the left hand menu navigate to Agents, and click Download MSI.

Download MSI button

Installing the DataForge Agent (Windows)

Download the appropriate version of the MSI installer. Run the installer and fill in the appropriate prompts.

Initial MSI screen

A specific DataForge Agent Code will be required, and this is the Agent Code that was configured in the DataForge UI. If this is the first DataForge agent, then the port should be left as the default value of 25530. Only one DataForge Agent can be installed on each machine. Do not alter the configuration key unless directed to by a DataForge team member.

If installing the Agent with the service log on as USER, click the button labeled "Enter USER credentials" to progress the installation. If installing the Agent with service log on as LOCALSYSTEM, select LOCALSYSTEM in the "Service Log On As" dropdown and then click the button labeled "Next".

Agent Setup Screen 1

Agent Setup Screen 2 - Service User Credentials

The service account username MUST include the domain or local server name. Example: <domain>\username or <local server name>\username

This can be easily found by opening a command prompt and running "whoami" in the terminal.

The user must have the ability to log on as and run services. The user must also be able to access any folders containing source data if the Agent will be performing file Ingestions. An alphanumeric password is recommended.

After selecting localsystem install or entering user credentials, the last screen of the installer will allow you to choose the install setup type. We recommend using the "Typical" option. Once you've made a selection and clicked Next, proceed to the final step of the installation below.

Place the Agent-Config.bin file and restart the Agent service

The final step of the installation pertains to providing the appropriate credential configuration to the application. When the Agent was created and saved previously, a "agent-config.bin" file was downloaded. Locate this file. If the config file was already deleted, the agent will need to be changed slightly to trigger a new config file on save. Move this file to the bin folder within the Agent program files location: C:/Program Files/DataForge/bin

Once the file has been placed, navigate to the "Services" explorer on the server and restart the service named "DataForgeAgent_[agent-code]". For example, if your agent code was "Dev", the service would be named "DataForgeAgent_Dev".

Example of Services App explorer using agent with agent code of aj-agent

Once the service is restarted, navigate to the DataForge UI and open the Agents page from the main menu. A green check should appear in the Agent status that indicates the agent is communicating. Open the Agent and check the Agent logs and status to make sure it is online and heartbeating.

Troubleshooting installation

If the Agent does not come online with a green status icon within 5 minutes, please double-check all of the above steps to make sure that configuration was set up properly. If the above steps all look correct, please check the log files on the server that the Agent has been installed on. These files will be located at: C:/logs/dataforge/. The current log file will be called "agent.log".

Agent Log Files

Another common error is the misplacement of the agent-config.bin file. If the following message is in the logs, please make sure that the agent-config.bin file is placed in the appropriate location, and the service is restarted after the placement of the file. The file needs to be named "agent-config.bin" as well.

Agent Config error message

If these troubleshooting steps are followed and the Agent is still not running, please reach out to your client team for support.

This should complete the setup of the Agent on the client machine. All further configuration should be able to be completed via the DataForge UI, such as locating file paths or database/data warehouse credentials.

Common Errors with Agent Install

If the service is running and no log files are being created in the C:/logs/dataforge directory, there is a good chance that Java is not installed correctly on the server. Please consult the Java requirements at the beginning of the guide. Run "java -version" in a terminal to see if Java is installed, you should get back a message that looks similar to:
- openjdk version "21.0.3" 2024-04-16 LTS
  OpenJDK Runtime Environment Corretto-21.0.3.9.1 (build 21.0.3+9-LTS)
  OpenJDK 64-Bit Server VM Corretto-21.0.3.9.1 (build 21.0.3+9-LTS, mixed mode, sharing)
The service account doesn't have access to Program Files location or service account doesn't have permissions to create a service on the server.
Service account username doesn't include the domain, i.e. user123 was entered instead of productionserver\user123. Make sure to run "whoami" in a terminal on the server to get the fully qualified username with a domain for the service account.
Service account password has special characters. It is recommended to use an alphanumeric password at this time
There is already a service with the same name on the server from a previous installation attempt. Use "sc delete" command in a terminal window to remove the existing service before trying again. The service will be named IntellioAgent_<agent-code>. An example of the command is below in the next section regarding uninstalling the DataForge Agent.
Machine GUID is incorrect or is unable to be acquired by the agent. Check the agent logs (C:/logs/agent.log) and see if the Machine GUID is populated in the logs. If it is empty, the agent will need to be uninstalled and reinstalled with the Machine GUID value entered in the configuration-key prompt in the MSI installer.
Config file is misplaced or incorrect. Please make sure that the agent-config.bin file is placed in the appropriate location, and the service is restarted after the placement of the file. The file needs to be named "agent-config.bin" as well.

If the install is failing with a message similar to the following image, navigate to this location on the server: C:\Users\<username>\AppData\Local\Temp. Sort by date modified in your file explorer and look for the most recent file that starts with "MSI". The contents of the file will give more information about the MSI install error.

Uninstalling the DataForge Agent

Uninstalling the DataForge Agent can be triggered in two ways.

The first way (recommended) is to re-run the initial MSI file that was used to install the Agent. Double-click the MSI and click Remove when prompted. If the original MSI is no longer available, open "Add or remove Programs" and choose to Modify the DataForge Agent which will open the MSI. After clicking Remove you will enter the agent code and complete the removal.

The second way to remove the agent is from "Add or remove Programs" in the Windows control panel. The program will be called DataForge Agent or DataForge Agent Dev if it is the second service on the server. Manually remove this program.

When the uninstall is complete using the second method, there is a manual step needed to remove the service. Open a command prompt in Administrator mode and run the command (replace "DataforgeAgent_Dev" with the service name that was installed on the server.)

Remove Service Command

Updated August 13, 2024 18:06

Was this article helpful?

0 out of 0 found this helpful