Dashboard to monitor data flow from applications, datalake and on-premise dwh

Hi,

we have in place a simple ETL process that export data from the Data Lake to an on-premise SQL server (we use the spoon ION ETL client for about 100 tables).

From time to time, for different reasons, the number of instances/records exported differs from the numbers we have in the data lake and we need to recover them.

What we are figuring out is a monitoring dashboard that checks the flow from the erp (in our case LN), data lake, and dwh at least in terms of records count.

Now the issues we encounter to realize it are:

-erp table count check is possible only with a ST cloud. How can we check for MT?

-counting Data Lake table row in Compass takes a very long time especially when skip reformatting option is not used. We refresh the extraction every 5 minutes while compass sql count takes hours)

-ION ETL client, with both compass and payload query mode, seems not to list the compass view we save in ION Desk (while a DB management tool like DBever do it).

I'd like to know if you have some suggestions about this topic.

Thanks in advance.

Alberto

Find more posts tagged with

Infor Data Fabric

Accepted answers

Legacy Contributor

Hi Alberto,

Data Ledger will give the details but the feature is not available right now. You can watch the below video to understand the feature.

https://www.youtube.com/watch?v=FU90Y3arulc

This feature is particular for M3 but you can rise ER for Infor LN too. Hope it helps.

All comments

Rene Bottern

- In MT you can open the "Business Engine Data Management" from the M3 Admin Menu, it shows the Record Counts for each Table.

I agree that Compass Queries against the DL is really slow, and hopefully Infor will come up with an alternative to this (maybe even an ODBC/OLE DB option for local SQL Server integration instead of only JDBC).

Since you are extracting to a Local SQL instance with ETL Tool, you should be able to do fast Record Counts from there at least.

Legacy Contributor

Hi Alberto,

Data Ledger will give the details but the feature is not available right now. You can watch the below video to understand the feature.

https://www.youtube.com/watch?v=FU90Y3arulc

This feature is particular for M3 but you can rise ER for Infor LN too. Hope it helps.

apiazza

Thanks Rene, our currently need is for Ln but also M3 will follow. Do you know if we can publish the Business Engine Data Management session tables directly into DL?

Alberto

apiazza

Hi Mohan this is definitely what is needed (for M3 and Ln). A solid monitorig and handling system of packages is as important as the publishing process itself; debug differences between the 3 data layer ERP vs DL vs on-premise DWH is often a daunting task and a restriction to use the DL itself. The capability to integrate this tool with external information (in this case the results from on premise DWH) will be a good complement. Thanks. Alberto

Copyright © 2025 Infor. All rights reserved.