6.0.0 Major Release Blog

The DataForge 6.0.0 major release is here with several bug fixes, new features, and Beta lineage.

1. Persist Processing Unit weights and calculations and display in UI 
2. Detailed tracking and automated management of raw schema 
3. Add "Max Parallel Clusters" parameter to cluster configuration 
4. Stop idling sparky job runs when core is not running 
5. Upgrade Postgres version of metadata database to 14 
6. Ingestion and scheduling refactor for all ingestion types 
7. Global workflow visualization: Lineage  

Persist Processing Unit weights and calculations and display in UI 

By: Vadim Orlov 

DataForge now has transparent PU usage, data persistence, visualization, and collection in the DataForge user interface, UI. Centi PU, cPU or 1/100th of PU, values are now visible in various DataForge tables; sources, rules, outputs, output mappings, and processes.  

Users will also find total PU and detailed breakdowns in history.process. This is done using the following formula with PUs and data volume factor: 

meta.enrichment, meta.source (sum of enrichment cIPUs), 
meta.output_source_column, meta.output_source (sum of mapping cIPUs)  

Usage history can now be used in queries and reports using meta.process_history table in databricks. Here is an example of the query returning total usage for each process record:   

select from_json(ipu_usage, 'base decimal(7,2), cdc_volume decimal(7,2), 
refresh_type decimal(7,2), rules decimal(7,2), refresh_recalc decimal(7,2), 
output decimal(7,2), total decimal(7,2)').total from meta.process_history 

Detailed tracking and automated management of raw schema 

By: Vadim Orlov 

Users will find that the previous pain points associated with stale raw attributes have been greatly mitigated with DataForge version 6.0.0. 

For instance, detailed tracking of raw attributes for each input in the input.raw_attribute_ids column has been added to DataForge.  

In addition, two new flags have been added to the Raw Schema table that help visual raw attribute usage. The first is the Target flag which indicates whether a raw attribute is referenced in either a rule or mapping. The second is the Inputs flag. This shows whether a raw attribute exists in at least one or more inputs.  

If at least raw attribute has a value of “false” for both flags, the option to delete the orphaned attribute is available. Simply click “Delete Orphan Attributes” to remove them from the Raw Schema table. 

And finally, on the Inputs tab, the new option titled, “View Raw Schema”, is available within the triple dot menu. Clicking “View Raw Schema”, opens a new tab to show a Raw Schema table filtered by attributes that are included in the current input. 

Add "Max Parallel Clusters" parameter to cluster configuration 

By: Vadim Orlov 

The new “Max Parallel Clusters” parameter limits the number of concurrent Job Runs for each cluster configuration. When active, queued jobs stay in the process queue until an active Job Run is available. This allows for more efficient processing since redundant startups are minimized.  

However, if the “Max Parallel Clusters” parameter is set too low, users might see lower speeds. This is due to clusters being forced to go through a small number of Job Runs. The default value is 1,000 in order to mimic the previous DataForge cluster state, unconstrained. 

For best practices, it is recommended for environments with multiple sources ingesting data on the same schedule to use a cluster configuration with a “Max Parallel Clusters” value of one half of the number of sources. If users are seeing reduced speeds due to idle job starts, reduce the parameter value. If processes have long queue times, it is recommended to increase the parameter value to reduce time spent waiting in queue. 

Stop idling sparky job runs when core is not running 

By: Vadim Orlov 

Previously, there was an issue with spark jobs idling indefinitely when the core was stopped. In DataForge v6.0.0, this issue has been resolved. This change primarily affects non-production environments, such as development environments, that are not always up and running. 

Upgrade Postgres version of metadata database to 14 

By: Vadim Orlov 

Previously, in DataForge v5.2.0 Postgres version 10 was used in both Azure and AWS environments. Now, since upgrading to the AWS Aurora version 2 engine, DataForge is able to run Postgres version 14 in both environments.  

These version upgrades are exciting for DataForge users for multiple reasons: 

• Additional granular scaling options 
  • Aurora version 1 (v1) was able to scale in geometric proportion, from 2 to 4, 8, 16, etc. ACUs 
  • Aurora v2 scales in 0.5 ACU increment and has lower minimum ACU of 0.5 

• Faster scaling 
  • Aurora v1 moves the workload to different computed nodes as part of scale operation. This prevented scaling from occurring while active transactions were present 
  • Aurora v2 scales up and down in place allowing transactions to continue to run.  
    • As a result, v2 scale operations typically take just a few seconds and are virtually unnoticeable 

• Improved User Experience (UX) 
  • Processing resiliency 
  • Cost savings for AWS customers 

Azure deployments have also been upgraded to run Postgres version 14 with Azure Database for PostgreSQL - Flexible Server. 

Ingestion and scheduling refactor for all ingestion types 

By: Joe Swanson 

Scheduling has been refactored to use two new tables; meta.ingestion_queue and history.ingestion_queue. This will affect all ingestion types; custom, sparky, loopback, table, file. 

Now, when users attach a schedule to a source, a record will be inserted into the Ingestion Queue table with the next time the Ingestion process should run. Core will check this table every 30 seconds and create a process if the scheduled time is ready to run. If the process is run by Sparky, a Databricks job will be launched and ran in Databricks workflow. Instead, if the process is run by an Agent, the Agent will pick up the queued process on a configurable heartbeat interval and run the process.  

“Pull now” has also been reworked. It now creates an Ingestion process immediately after clicking. File watcher in the Agent has been refactored too. It will now pick up all files it finds on its heartbeat interval, create processes for them, then immediately begin processing as well.  

These updates optimize file ingestions to run faster and more efficiently during large file and several multiple file uploads.  

Agents on 5.2.0 will be able to auto-update to 6.0.0 using the normal process, but they will no longer to be able to run Ingestion processes until they update to 6.0.0.   

Global Dataflow Visualization: Lineage  

By: Mary Adams EC  

The dataflow visualization in DataForge has arrived. Known as Lineage, this interactive graph allows users to see full dataflows from a source container to its output from an output column to its original source node and so much more. Due to the technology of this feature, it is currently in beta and will have many updates in future releases. 

Lineage starts with a node. Users can kick off a lineage graph by navigating to an DataForge table with lineage nodes. Container nodes, or parent nodes, can be found in the following locations: Sources, Relations, Outputs, and Output Source Channels. Child nodes are found in the following views on DataForge: Raw Schema, Rules, Data Viewer, and Output Mapping Columns. 

Users have the option, when clicking on a node, to either start a new lineage instance or add the node to their current session. Users can access their current session by clicking on the Beta Lineage option located in the main menu. 

A lineage session will always start with a single node. As stated earlier, the node may either be a parent container or child node. The selected node will be highlighted on the graph. Users can navigate the dataflow related to any node visible on the graph by right-clicking on the desired node. A submenu will appear with the option to Add or Remove nodes in the dataflow. For child nodes, the option to Isolate will also be available. 

The different types of nodes are represented by combinations of colors and symbols. This is true for dataflow arrows or edges as well. Users will find the legend key in the upper right corner of the lineage screen. It can be toggled at any time in order to differentiate between the nodes. 

Edges show the dataflow starting from the left, origin, to the right, destination. Edges that are thicker and colored magenta represent relations. Users can left-click on these relation edges in order to view the relation graph. Note, the relation graph from lineage does not currently support editing relations and displays relations in the source_id order returned by the database. 

In order to navigate lineage dataflows, users can select origin or destination. Origin shows the previous node in the dataflow. Destination shows the next node. Both have the recursive option which shows the start node for origin and the final node for destination. Users can also remove origins and destinations in order to simplify the dataflow. 

Users can expand or collapse parent nodes by clicking the left-hand corner of a container. Since some dataflows can have hundreds of edges and nodes, the option to Collapse All is also available. Expand All is an option as well, though its use is recommended for smaller graphs. 

Currently, lineage does not support link sharing between user accounts. Instead, users may export a PNG of their lineage session by clicking the Export button available at the top of the screen. Users can look forward to link sharing in a future release.

As a starting point for learning lineage, users can find the legend key below.

Full Changelog

Updated May 14, 2024 17:37