Types of Resets
During the development lifecycle, users will need to reset their sources often to change source settings, apply rules, update output tables, and restart failed processes. The following is a "reset cheatsheet" that should be referenced when deciding how to reset data on a source.
The following buttons are considered the "source level reset" buttons and will be referred to throughout this guide.
The following buttons are considered the "input level reset" buttons and will be referred to throughout this guide.
Ingestion Parameters Have Changed
If the source query is changed, there are a couple options that may need to happen on the source. If a completely new dataset is being pulled with the updated source query, it is recommended to delete all source data and delete the metadata on the source. Another common tactic is to have a source query that pulls all data in a historic load, and afterward, the source query is parameterized to pull deltas. In this scenario, there is no need to clear data or metadata on the source.
Parsing Parameters Have Changed
For sources with the File Connection Type, a parsing process will run for every input on the source. The parse process is what parses the data from flat file to a DataForge usable format. Changing any parsing parameter should require a reset of the parse process on the input level reset buttons. If the change to the parsing parameter could alter the raw metadata on the source, it's recommended to delete all source data, clear the metadata on the source, and reload data for the source.
Refresh Type or Change Data Capture Parameters Have Changed
If the Refresh Type on the source is changed, the source will automatically show a pop up on save that will ask you to reset CDC. It is required to follow this prompt and reset CDC to save the change to prevent data integrity issues. Once you click save and reset CDC, a process will be enqueued on the source to reset CDC for all inputs. If you are still in development and making changes, it is typically easier and quicker to delete all source data before making the change and then pulling new data after the source changes are saved.
A common processing misstep that users find themselves in is to reset CDC after making changes to Rules. This results in extra processing time for the source and is unnecessary.
Validation/Enrichment Rules Have Changed
If any Validation or Enrichment rules are updated or created, there are two options to propagate the changes to existing input data.
- Use the source level reset button called "Recalculate". Recalculate will then run and check the rules and current hub table and run any rules that have been updated or are new, resulting in an up-to-date hub table. The benefit of recalculate is that it will run on the entire source, and not each individual input, saving the user processing time. Recalculate will also skip the "refresh" process, which can have long and expensive processing time.
- Reset enrichment for an individual input. The enrichment process will run and the normal following processes will run on the input.
Output Parameters/Mappings Have Changed
If any changes have been made to the Output Settings or Output Mappings, then there are two options to reset the Output.
- Use the source level reset button called "Reset All Output". The benefit of resetting output this way is that refresh types are optimized to run in this fashion, especially when running deletes on the target output table.
- Use the input level reset button called "Reset Output". This will reset the output for the individual input's data. This is generally useful for unit testing changes to output settings before running all inputs on the source through output.
Ingestion Has Failed
A failed Ingestion process can not be reset. Delete the failed input using the input level "Delete" button and use the source level button "Pull Data Now" or the "Pull Now" button above the input table to kick off a new Ingestion process.
Parse Has Failed
Once the error has been resolved, use the input level "Reset Parse" button to enqueue a new Parse process for the failed input.
Capture Data Changes Has Failed
Once the error has been resolved, use the input level "Reset CDC" button to enqueue a new CDC process for the failed input.
Enrichment Has Failed
Once the error has been resolved, use the input level "Reset Enrichment" button to enqueue a new Enrichment process for the failed input.
Refresh Has Failed
Once the error has been resolved, use the source level "Retry Last Failed Process" button to enqueue a new Refresh process for the failed input.
Output Has Failed
Once the error has been resolved, use the input level "Reset Output" button to enqueue a new Output process for the failed input. The source level reset button can be used here as well if multiple inputs have failed.
Updated