Custom processing steps (Custom Ingestion, Custom Parse, Custom Post-Output) require users to select a compute configuration for each custom step. This custom compute configuration defines compute parameters and specifies the custom notebook that will be executed.
Configure Compute for Custom Ingestion Source
Sources configured to use Custom Ingestion (Custom connection type) on the Source Settings page require users to select Compute Configuration for the Custom Ingestion step:
After the Custom connection type is selected, a Custom Ingest Compute Config dropdown will appear. Simply click on the dropdown to select an existing compute configuration for that source.
Users can also create a new compute configuration by click on the open in new icon. This will open a separate tab to a new compute configuration. After creating the new compute, users have the option to refresh the compute list on the Source Settings page. Refreshing the list will show their newly created compute configuration
Users may also edit an existing compute by selecting a compute configuration from the dropdown and then clicking the edit icon. This will open a new tab to the existing compute. After edits are saved, users are once again prompted to refresh the compute list.
Configure Compute for Custom Parse Source
Sources configured to use Custom Parse on the Source Settings page require user to select a Custom Parse Compute Config for the Custom Parse step:
Configure Compute for Custom Post-Output
Sources configured to use Custom Post Output on the Output Settings page require user to select a Custom Post Output Compute Configuration for the Custom Post Output step:
Updated