What’s New?

1. Specific Pool

Introducing an advanced task execution option that allows you to allocate execution resources within designated pools. With this new feature, you can now easily select the precise pool for your processing tasks directly through the platform’s user interface. Please note that this functionality is exclusively available for Zeus stand-alone configurations.

2. API call from Dashboard

Able to share the data from the dashboard via Rest API to do the notification or share the data between the different software applications and communicate with each other.

3. Data Lineage

Data Lineage involves tracking and visually representing the flow of data from its source to downstream usage. It facilitates understanding of data movement through system processes and transformations. This includes details about data origin, transformations, and final destinations. The visual representation helps in understanding data journeys, ensuring transparency and traceability in complex data workflows.

image 17 1
  • Upstream: The data origin.
  • Downstream:The final destination data.
  • Unlinked Asset: The unused source that has severed any connection with the selected asset.

Moreover, users can navigate directly to related data origin or downstream through this page. For example, clicking on the Asset Name will take them directly to that specific data asset.

4. License Management

We have enhanced License Management to efficiently handle client licenses in Blendata Enterprise, supporting both online services (hosted by Blendata) and offline services through the License Management Service (BDL).

  • Online License has transformed the client license processing to the new service(BDL).
  • Offline Licenses, it is necessary to use the new format of the license file provided by the new License Management Service. Consequently, old license files will no longer be usable in the future.

Also the platform provided a real time the CPU usage from Compute Engines. If the CPU is over the limit, the license should be invalid and stop doing data processing.

5. Advance Mode for Workflow

The Workflow Management feature now supports running Workflow tasks using Python scripts. In the Workflow Configuration tab, users can switch to Advanced mode, and the platform provides a Python template for creating and uploading Python scripts to the server. In this mode, the workflow configuration displays the Python code instead of the diagram.

image 18

Users can only select either the simple or advanced mode.

6. Import/Export the Workflow

We have enhanced the Workflow feature to enable the export of your own workflow as a JSON file for sharing with team members and you can import workflow files into your workspace. Users can choose a Workflow with only one version.

7. Sync BDE-Spark Asset

Considering there are advanced users who train machine learning or perform advanced analytics via the Notebook menu, the data is created on the Spark Meta-store, and they cannot import this Spark Meta-store into the Data Catalog page. Thus, we have improved this feature to enable users to sync this Spark Meta-store data into the Data Catalog, with Super-Admins being the only role capable of performing this action.

This version, users cannot select some Spark Meta-store files so the platform will import all Spark Meta-store files into Data Catalog page.

8. Reconcile with No. of Lines

To display the imported record number when the user imports the data via Upload File or File Path methods into our Blendata Enterprise for the first time and also to show the total imported record number and the latest updated date in the Basic Information details of each data asset on the Data Catalog page excepts a streaming data asset type.

image 19

Additionally, the user can view historical records, including the total number of records and the number of processed records, within the last three months.

image 20

9. Specific trash path for HDFS

Normally, HDFS has its own trash feature, but it needs to be enabled before use. If this feature is not enabled and a file is deleted, it is permanently removed. To prevent this issue, we introduced a feature to specify the trash path for HDFS, similar to the local protocol. When the user deletes a data asset without selecting backup, the platform will move this data asset to the trash path.

10. Table Size Statistic

Users import data every day, so viewing the data size statistics helps them plan storage or identify peak times for data import. Thus, we have created the table size information in the catalog page. The users are able to see the current total size of data and the statistic number of table size in specific time range.

image 22

11. Delta Compact and Vacuum

These two services are used to manage the Delta data asset in our Blendata platform. Because the Delta table can use CRUD commands, it generates many versions of Parquet files, affecting the performance of data processing. Thus, the platform allows users to optimize Delta tables with Delta compaction and vacuum.

  • Delta Compaction: Used to compact the data files(parquet file) to reduce the number of files.
  • Vacuum: Used to delete the old data files(parquet file) based on the specified period setting. The default is 7 days.

Moreover, users can use these two service in the Workflow Management menu.

These two services cannot run in parallel; they must run sequentially.

12. Workflow API Calling Job

We have added the ‘Call API’ service type to the Workflow Management menu, allowing users to select and create a job with an API.

Fixed Bug:

Log in

  • BDEN-537 Resolved the issue where the popup was displayed multiple times if the user was inactive for more than 15 minutes.

Import Data

  • BDEN-556 Resolved an error when the user imported the Delta table and there was a blank field in the data asset.
  • BDEN-768 Resolved the issue where the platform did not utilize the selected Zeus settings for the recovery partition job and refresh table job service during Kafka importing.
  • BDEN-798 Hided the REST API import option for the external table type as the platform doesn’t support this API option now.
  • BDEN-1060 Fixed the issue with importing streaming data with partitions.

Data Catalog

  • BDEN-280 Resolved the problem where the confidential column value did not match the source table.
  • BDEN-587 Fixed the error of data assets that user cannot see update table options excepted internal data asset such as RDBMS and Scheduled File Path.
  • BDEN-588 Fixed the error about tag search using number as keyword. The search result now show the tag containing the number instead of order of the tag.
  • BDEN-806 Fixed the error where the user couldn’t delete the delta table due to a different backup protocol. As a result, for the delta data asset, the platform won’t backup the data and will permanently delete the table instead.
  • BDEN-1025 Fixed the bug where the partition management popup does not display the partition in each state.

Data Exploration

  • BDEN-772 Resolved the issue where the toast bar displayed ‘User has no permission to access any table’ during the loading of the table, even though the user had access to the data asset.
  • BDEN-744 Fixed the issue where the platform displayed the Encrypt (Aggregate Table) symbol, but the data asset was not an encrypted table on the Data Exploration page.

Export Data

  • BDEN-303 Added a warning message, “Update Preview Result” to the Export File popup when users edit the Job Triggering part, reminding the user to refresh the data in the preview section.
  • BDEN-1051 Masked a password from Zip file when creating the export job in Service Chain and Workflow Management menu.

SQL Editor

  • BDEN-628 Resolved the error where multiple workspace tabs remained active while editing the tab name.

Workflow Management

  • BDEN-405 Fixed the bug when the user used zoom in option on Workflow Configuration and cannot see the task service box.
  • BDEN-443 Notified the user only once after they set a permission and click save.
  • BDEN-447 Fixed a bug that prevented users from editing a workflow service in the Workflow Configuration after deployed the workflow.
  • BDEN-610 Adjusted the text color of the information tab to align with the color scheme of other tabs.
  • BDEN-663 Added the message “Workflow Environment cannot be reached” for situations where the platform is unable to connect with the Airflow server, resulting in the disabling of all actions.
  • BDEN-675 Fixed the error that caused the platform to display a loading popup for an extended period when the error occurred.
  • BDEN-684 Fixed the issue where the workflow task log on the overview tab displayed a blank log for the first time.
  • BDEN-748 Resolved the issue where the version history was not visible when there were no tasks in the Workflow Configuration.
  • BDEN-837 Resolve the notification code to be compatible with MariaDB type.
  • BDEN-788 Fixed the error message to display on the user interface when the duplicated workflow name is quite long, and when the user attempts to save the edited workflow without an internet connection.
  • BDEN-918 Fixed a bug where duplicating the version resulted in the platform using the deploy version name instead of the edited version name.

Notebook

  • BDEN-461 Resolved the permission error that occurred when users created or uploaded data in the Notebook, allowing them to manage the notebook immediately.
  • BDEN-488 Fixed the error when the Data Assets on notebook are deleted but they still displayed on the page and cannot select to delete again.
  • BDEN-489 Resolved the issue about the permission setting. Each data asset need to remember the permission setting.
  • BDEN-490 Fixed the error that user with View/Edit permission can delete the data asset on a notebook.

Dashboard

  • BDEN-518 Fixed the bug related to the local filter using combo box(Enable-Multi-option select). Now, when the user selects a value from the dropdown list, it is displayed at the correct position.
  • BDEN-840 Fixed the error that occurred when the user applied a filter condition with an equal condition and needed to enclose the value in double quotes.
  • BDEN-825 Fixed the issue where the filter combo box with multiple options cannot apply the comma symbol in the value.

Service Management

  • BDEN-70 Fixed the error where users with management permissions were unable to access the chained job.
  • BDEN-123 Resolved the slow loading issue when the user clicked the chain logs tab for the first time.
  • BDEN-293 Corrected the chain log ordering to align with the order ID.
  • BDEN-491 Fixed the error where user cannot search the Service Chains on previous page. Corrected the error preventing users from searching for Service Chains on the previous page.
  • BDEN-103 For alert notification Removed “/opt/noti-server/” from the CSV file name when the user receives a notification email with an attachment.
  • BDEN-769 Fixed the error where the system service(compact) did not use the selected pool as per specific pool setting.
  • BDEN-795 Resolved the error where the user cannot save the edited service chain and showed error message “Update job failed”.
  • BDEN-822 Fixed the error where the custom service cannot be created in the service chain.

Improvement:

Information

  • BDEN-611 Improved the information page. Users now only need to click either ‘Skipped’ or ‘Get Started’ to close the tutorial information.

Import Data

  • BDEN-681 Supported the AVRO format file on Upload file type option.
  • BDEN-803 Improved the export/import data asset and service to use encryption methods. Users export data and import it into another Blendata environments without having to input the password again.
  • BDEN-808 Added decimal type when users customize the fields.

Export Data

  • BDEN-120 Protected a data when it’s exported from the system. Users should be able to add a password to extract the zip file.

SQL Editor

  • BDEN-677 Fixed the location of the “Add” button when the user opens the workspace tab at maximum capacity.
  • BDEN-678 Adjusted the cursor style to be a hand pointer when hovering over ‘How Can I Save SQL to My Query?’

Workflow Management

  • BDEN-448 Aligned a size of email notification dialog in the Workflow Management.
  • BDEN-451 Revised the notification message about the duplicated name issue, “Duplicate Workflow Failed: workflow name is too long“.
  • BDEN-503 Revised the conditions dropdown location in case there are many parent tasks.
  • BDEN-612 Added an detail in the result logs instead of blank box in case there are no logs available for the selected workflow service.
  • BDEN-783 Added an API calling option to create the task in the workflow management.
  • BDEN-811 Configured the platform to utilize default settings if the user does not specify the realm ID or Zeus ID.
  • BDEN-819 Added a custom service option to create the task in the workflow management.

Notebook

  • BDEN-327 Synchronized data between Blendata Enterprise and the Notebook environment when users refresh, or recovery data or partitions in the Enterprise environment.
  • BDEN-826 As of now, Notebooks do not support parallel runs. Therefore, we have introduced a workaround for parallel runs by duplicating a notebook and removing it once the user obtains the desired result.
  • BDEN-902 Controlled the permission to access table path from storage, which use to protect data store with Operation System ACL.

Service Management

  • BDEN-124 Resolved the issue with notification attachments by setting the encoding type to UTF-8-BOM, supporting Thai characters.
  • BDEN-589 Provided the Zeus options for the alert notification processing.
  • BDEN-103 Updated the alert scheduler for notification emails to include a timestamp with both date and time in the attachment name.

Spark Monitoring

  • BDEN-781 Added the Job details including Job name and Job start date and time in Spark Job Description, which helps to monitor the job with ease.

Administrator

  • BDEN-121 Administrators are able to search for user activity on data access.

Share