Looking at the DataLake ETL Tool to update a local database. Wha are your experience with the ETL t

We are looking at purchasing the ETL tool and using it to update a local database from the Datalake. We're just looking for feedback on your experience with the tool. Is it a reliable tool for updating a local database? The demonstrations we've see make it look like it is relatively easy to use.

Your thoughts and experiences would be very helpful.

Find more posts tagged with

Infor Data Fabric

Accepted answers

All comments

unknown

For the last 4 years we have been doing a CSV replication to local. I have written a ton of code around it to handle errors, timing, etc but it still requires manual intervention at least once per week. It is a very time and resource intensive process.

After a proof of concept, we have decided to go with the ION ETL tool as we need more than two replications per day. We have over 400 integrations with other applications and outside entities that just cannot be handled via IPD's, compass queries, or JDBC calls. We simply cannot achieve the response time required to provide these interfaces with the data using INFOR supplied tools.

From my POC, the setup was rather straightforward, however I can foresee several months of tweaking and developing error handling before I have nailed it down. I'm waiting for my application servers to be spun up now before I start on the working with the INFOR Test server. I'll let you know in another 6 months how I feel about it.

unknown

Thank you for the reply. We've been doing the same thing for the last 4 years and agree that it requires a lot on manual intervention.

tdouglass

I am curious for those who have this established, do you typically have one job for each table, or to keep data in sync one job with multiple tables? I see pros/cons to both. If multiple transformations/tables in a job, then if one fails, they all do...but if not then chance that some data might be out of sync...aka a detail w/o a header. Feedback?

Rene Bottern

We started using ETL Tool for Replication via DataLake in 2021, and it have worked fine.

Replications are very fast, after initial replication, and is an excellent Tool if you need local data (SQL Server) that can be easily queried and used for other Tools (i.e. Cognos, PowerBI etc.).

The data is normally available in the local Database within minutes.

NEW ETL Processes promises that Data will be available within seconds now instead, so it should be a viable alternative in most scenarios (as long as Ad-Hoc is not required, in which case you should use API's)..

tdouglass

Rene, do you use one job per table, or one job for multiple tables? or do you bulk them together to evenly distribute the load?

Kevin Heiman

You may want to consider the recent function of Streaming Pipelines.

Covered in the Product Roadmaps, Release Notes, YouTube and Community Posts.

unknown

Hello, I couldn't find this function in the Data Fabric roadmap. The only reference to "Open data platforming" refers to Snowflake.

Could you provide the link to the document corresponding to your screenshot ?

Thank you.

christophedaeninck

Our on premis M3 version ran on a AS/400. A lot of interfaces were written using an ODBC connection. For the migration to the cloud we had no time and budget to redesign these interface, instead we decided to use the ION ETL. The only thing to change in the interfaces was the ODBC connector and parameters. At the end we have 140 tables to be synchronized. Most of the data is required on a realtime base so the ETL runs every 5 minutes. Due to the high volumes to sync (yes MOATTR and MIAATR are also needed) we had to split the sync of tables in 3 parts: every 5 minutes, every hour and daily. Initally there were a lot of problems but since we changed to using the documentdate it is rare to have failures.

unknown

Hi Mike

There are a few different ways you can achieve this, you also need to be aware of Infor OS limits for things like API calls and compute minutes.

The Data Lake ETL tool is used for slow moving data, Infor also have streaming pipelines which is for fast moving data. So refreshing every couple of hours vs every few minutes. Depending on your requirements you may have a combination of the two.

Happy to discuss in more detail if needed, we’ve also created a tool internally at PCG that synchronizes the data lake so should be able to answer any questions you have or just give you more of an overview. I’ll drop my email in case: mhenson@pcgservices.com

Thanks

Mark

Bojan Rafajloski

https://www.youtube.com/watch?v=oOYhGbQlVrw&t=6s

https://docs.infor.com/inforos/2024.x/en-us/useradminlib_cloud/default.html?helpcontent=datafabrug/jez1678807772006.html

Stream Pipelines currently supports connectivity to cloud destinations PostgreSQL and Snowflake, and not on-prem systems. Additional support for other destination technologies are evaluated for our roadmap, for example SQL Server is in 2025 roadmap.

Roy Es van

We created a Data Lake Synchronization product (Syynchronizer) that replicates the data into a sql server database. From there we do all bulk integrations and exports. The tool is available to everyone under license

s-grabner

Hi Christophe,

how you managed the initial load process when talking about 140 tables? how long have it took to migrate them into the local DB?

Thx.

BR
Stefan

christophedaeninck

It took 96 hours mainly because of MIATTR, MOATTR en some other large tables. Last friday i added MITSTA and it took 4 hours.

Andrew Sowden

Sounds interesting ... could I have more information?

Andrew Sowden

Sounds interesting ... could I have more information?

s-grabner

Thanks that´s quite fast from my POV.

I made a test run last time with OSBSTD - 24 mio. records. It made 250 rec. per second and it took more than 24 hours to fully import it.

The question now is if it is getting very slow when running some tables in parallel.

How did you manage that? Did you run this in batches so for example 5 tables at once?

You have any tips how to tune this?

Roy Es van

Hi Andrew, I have send you a PM.

Copyright © 2025 Infor. All rights reserved.