Please, Endorse Enhancement Request - 109566

unknown

Request for Indexing Data Lake Tables

The objective of this project is to create indexes for the Data Lake tables. This initiative aims to improve the performance of external applications that query the data stored in the Data Lake.

Description:

• Review of the tables present in the Data Lake.
• Identification of the most queried tables by external applications.
• Evaluation of the most frequent queries to understand usage patterns.
• Development of specific indexes for each table based on query analysis.
• Implementation of indexes on tables identified as priorities.
• Ensuring that the created indexes meet performance needs without compromising storage efficiency.

Ken Miller

Hi Leonardo,

While I agree querying data directly in Data Lake can become a long process after lots of data in the table is updated, because there are no indexes. But this is because Data Lake is not a database, and not meant to be. It runs / operates using metadata because it is a data repository of various data types.

And this is why you can connect Data Lake to either on-prem or cloud-base databases, which you would then be able to index and have quick queries. And in the very near future Infor will offer Lakehouse, an Infor data warehouse product.

This concept of Data Lake also (so I'm told) keeps the cost down as it is just a repository.

Thanks,
Ken