What’s New

Lineage History Retention

The platform has added a new function, which is the Lineage history retention, removing unused data lineage history older than 30 days.

JAVA/Scala and Python to Import the Data Asset SDK via Notebook

In the past users cannot import the data asset from Notebook to Data catalog. We have enhanced a library to import the data asset including internal data and Delta table into the Data Catalog via Notebook. This feature supports only Spark(Scala) and Python language.

The Spark example:

notebook SDK

The Python example:

SDK python

The Spark with Delta table example:

notbook delta SDK

Support Run Parallel in Workflow Management

With this version, we have introduced the Notebook run parallel service type in the Workflow Management menu. This feature helps users run notebooks in parallel with configurations. Users can set up configurations provided by the platform. Before using this service, the Notebook needs to be set in isolated mode first.

run parallel

The parameter in the configuration part supports both Spark and Python. For Spark, users can only use standard data types such as string, integer, float, list, and map, while Python supports all data types, including objects.

Users can also run parallel log SDK in Notebook. This library is designed to store the result history as a publish Notebook URL, which helps convert old message results into a table format. Users can view these results within 30 days.

  • description parameter is a boolean value of true or false. If set to true, the result will display the result message after the notebook has finished running. If set to false, it will only show the result status.

Note: This run Parallel log is available for Python and PySpark only.

Encrypt User SDK in Notebook

We have created an Encrypted User configuration to support Notebook SDK running using the Workflow Management menu and Run Scheduler option. Users can configure this in Zeppelin or Blendata Management. If the configuration is run as normal via the Notebook run option, it will use the details of the currently logged-in user; otherwise, it will use the details of the encrypted user.

Limitations for Workflow Management and Run Scheduler: 1.) For the Run Parallel Log SDK, if an encrypted user cannot access the Notebook, an error will be displayed. 2.) When importing a Data Asset via the Notebook SDK, the result will be displayed in the encrypted user’s Data Catalog page.

Rest API Importing

We have improved the REST API import type to enable putting dynamic parameters in the URL, in case it requires a Job Schedule to run this option on a schedule. Using ${…} to replace fixed date-time values with dynamic expressions. The units can be day(d), hour(h), minute(m), and second(s).

For Example: https://localhost:9999/api/get?start=${now-30m,yyyy-MM-dd’T’HH:mm:ssX}&end=${now,yyyy-MM-dd’T’HH:mm:ssX}. This means the start time is equal to 30 minutes before the current time.

Moreover, this version also supports HTTP request methods including GET, POST, and PUT.

  • GET: Used to retrieve information from the server using a specified URI.
  • POST: Used to send data to the server.
  • PUT: Replaces all current representations of the target resource with the uploaded content.

Note: The parameter in the configuration part supports both Spark and Python. For Spark, users can only use standard data types such as string, integer, float, list, and map, while Python supports all data types, including objects.

Fixed Bug

Design

  1. BDEN-1460 Fixed the height of content and tab dialog in the Blendata Enterprise.

Login

  • BDEN-627 Fixed the error where users could still log in to Blendata after entering the wrong password more than six times.
  • BDEN-688 Fixed the bug where the dark theme style was displayed incorrectly when changing a password.

Data Catalog

  • BDEN-1173 Fixed the Vacuum Table icon position to align with other options.

Import Data

  • BDEN-324 Fixed the limitation of Google profiles issue; users can only create up to four profiles.
  • BDEN-326 Fixed a bug that prevented users from proceeding to the next step with the error notification when activating SQL mode without entering an SQL command.
  • BDEN-1006 Fixed the error where the import job didn’t support a plaintext password.
  • BDEN-407 Resolved the issue where the user was deleted, but the key of HDFS in the Google Sheet was not deleted.
  • BDEN-253 Fixed the error message to make it more understandable when the user changes the access of the Google Sheet profile “User cannot update the data assets with the old version user should refresh first”.
  • BDEN-770 Resolved the error where the user could not select the Zeus group during RDBMS importing.
  • BDEN-787 Fixed the bug where the menu icon displayed incorrectly for no permission of user or role.
  • BDEN-1110 Fixed the error where the imported record number showed incorrect number.
  • BDEN-1020 Resolve the incorrect menu(Delta Optimization) on the Table Service Customization, when the user imported the Kafka streaming type.
  • BDEN-1497 Resolved the issue where the user could not import data from an RDBMS while setting up the reconciliation function.

Export Data

  • BDEN-629 Edited the mismatch information text of Zip File Output Option to be “The information attached to Zip File Output Option belongs to Number of File Output option”.
  • BDEN-1225 Fixed the error where the password parameter is empty. Then, the platform encrypted this password parameter value.
  • BDEN-1457 Fixed the Error 600 Null Input Buffer, that a user exported the data to external storage.

Delta Table

  • BDEN-252 Resolved the issue where the Delta table imported from Google Sheet did not have the ‘Change access profile’ option.
  • BDEN-595 Fixed the error where the action button appeared in the result part after users proceeded the CRUD successfully and the platform showed the number of affected record on the result part.
  • BDEN-794 Fixed the bug where a user saved an aggregate table with Delta table type, but the platform displayed an incorrect data source.

Data Lineage

  • BDEN-789 Fixed the data lineage menu should not have been shown for Kafka data assets.
  • BDEN-1166 Fixed the bug of export job service did not create a data lineage.

Data Processing

  • BDEN-312 Fixed the error where users could not export the result in Simple Query menu.
  • BDEN-406 Resolved the error where the Workflow list style could not load occasionally.
  • BDEN-571 Fixed the bug where the encrypt data asset was created but the encrypt data icon did not show in the Data Exploration and SQL Editor menu.
  • BDEN-591 Fixed the bug where user cannot drag and drop the query condition on Simple Query.
  • BDEN-592 Resolved the issue where the number per page displayed incorrectly in the Simple Query menu.
  • BDEN-634 Fixed the issue where the user couldn’t save the query with the same name as a previously deleted saved query in the Data Exploration menu.
  • BDEN-636 Resolved the issue when the user did a merge with all time range option.
  • BDEN-1073 Resolved the issue where the Record limit could fill in with text.
  • BDEN-1158 Fixed the error where the user with management role deleted a Notebook asset but the Notebook still appeared on the interface when using original or dark theme.
  • BDEN-1241 Fixed the server IP issue in the advanced options.
  • BDEN-1306 Fixed the bug where the Workflow cannot change to advanced option.
  • BDEN-1316 Fixed the error where the SQL query consisted of ‘\\’ in View Table cannot be used when users used the JDBC connection.

Visualization

  • BDEN-250 Fixed the bug where the user applied the split option with a True/False value, and the visualization only displayed the True value.
  • BDEN-313 Resolved the Map/Leaflet Map/Gauge/Word Cloud/Sankey visualization issue that occurred when switching the view to table style.
  • BDEN-316 Fixed the error where the visualization displayed an incorrect color when the user set a transparency color to the border color.
  • BDEN-320 Fixed the bug where users encountered an issue when selecting the scatter visualization type and using it with other visualization types for multi-trace.
  • BDEN-906 Fixed the error when users add a text in the visualization, the applied visualization in the Global filters setting are unselected.
  • BDEN-1004 Resolved the issue when users deleted a text in the visualization, the applied visualization in the Global filters setting are unselected.

Service Management

  • BDEN-841 Fixed the error where the Import data selected the Zeus group, but some services were not applied to the Zeus group.
  • BDEN-345 Fixed the error where the Component Name of Import Data and Services Management were not displayed as list format in role details.
  • BDEN-600 Fixed the error where the alert service was running abnormally.
  • BDEN-1343 Fixed the bug when the user did rerun the service job, the platform used the incorrect configuration version.

License

  • BDEN-255 Fixed the error when the past billing is empty and Billing history showed an error.
  • BDEN-1018 Revised the error message that was displayed when the user uploaded a license file before the start period.

Improvement

User Interface

  • BDEN-683 Improved the dropdown list dialog location on Data Exploration menu.
  • BDEN-1164 Improved the Blendata Enterprise user interface to align with the design and to be symmetrical.

Import Data

  • BDEN-376 Added the input_file_name field as a default column for Excel import data.
  • BDEN-408 Adjusted the Google key storage from cache to be called from the server database instead.
  • BDEN-912 Improved the Kafka import data by adding a ‘key’ value to default field.

Data Catalog

  • BDEN-219 Added the notification message, when there is an update of default column in the Column Customization.
  • BDEN-1181 Added a Delta Compaction and Vacuum in the related service for Delta table asset.
  • BDEN-920 Improved the reconcile number of records to support streaming data importing, and enabled viewing the history for the last 3 months.

Data Processing

  • BDEN-319 Improved the loading format for all data processing, which enables users to kill the process when it takes too long for waiting or there are some error.
  • BDEN-1241 Improved the email notification feature with multiple email input and performance across the multiple features.

Notebook

  • BDEN-913 Improved notebook and SDK to connect Hera api by notebook token. (Notebook token generate by Session Id, Notebook Id and bdeToken)

Visualization

  • BDEN-916 Added the default value setting for Combo box in Dashboard filter.

Share