ETL Tool Initial Load

Dear all,

I want to ask into the community what are the experiences with the ETL tool initial load.

I am asking because we will have for sure 30+ tables that we want to replicate into an On Prem data warehouse needed for Infor dEPM v12 and also other use cases (we extensivly used Application studio with the adhoc queries today that we need to somehow change).

What I am a bit worried about when talking about the ETL process is not the daily jobs that will for sure run quite fast. Instead I see a bit of a problem with the initial load cause of the data volume.

Does anyone know or has experience with it how long it will/can take to export the tables:

1. To datalake via the API

2. to the data warhouse using the ETL tool

Thanks in advance for the input.

Best regards

Stefan

Find more posts tagged with

General Discussions

Accepted answers

All comments

Rene Bottern

I think the question should be how much history you need in your local database.
If you are OK with only the last couple of years, don't replicate anymore, if nothing else, because of space.
The initial load is pretty fast as well, but it will take a while to get the initial load done, so run on a weekend if there's a lot of data.

s-grabner

so it would be good to make the API call with the Initiate transaction which sends the data to datalake already with a condition on date/time?

Or is it better to push everything to Datalake and then filter in the ETL tool that´s what I am thinking about, because we had discussions with Infor consultants and then it was not quite clear for us ...

Rene Bottern

If you don't need all of the History, use a Start Date for sure in the ETL Tool.

After initial load it will keep this updated and only load net-changes, so it will be fine.

s-grabner

okay thx so far

Kevin Heiman

Consider the types of files you are loading. For referential purposes, you may want more completed files, to ensure referential elements are not broken. Tables that are Codes or Master data versus Transactional tables.

Some projects go a step further and have a separate load process for 'Initial' loading of code/master tables, vs day to day. That allows them to reset if they feel the DW is out of sync. This initial load usually uses Compass views of a DW start date to quickly get to the starting point in time of the DW. Then uses all transactions via the incremental loading.

If you find loading the initial load to long, you could consider that approach..

Kevin Heiman

As to you question #1 "To DataLake", (as with #2) there are a lot of variables involved. For #1, Network capacity, ION Enterprise Connecter capacity, proximity to the Database, database capacity, and method used in loading. Width of data and volume of data are other considerations.

What is your initial approach on the method? Data Loader and Any SQL or are you writing to APIs. The ION provided Data Loader and AnySQL are the most common and are good about batching the data to optimal payloads for data lake and ION.

unknown

My client use etl tool for exporting over 100 tables.
We can export 400 records per 1 second
(short length data: 1000 records per 1 second, long length data(like MITMAS) 100 records per 1).
Considering CPUcs count, I am running 4-6 transformations multiply(by JOB scheduler).
(when differencial update but I remember same performance in initial export time)
Multiple running relates H/W resource (cpu memory in local machine) and DBMS capability.
To export huge table (like MITBAL) needs many hours(36 hours on my experience) in initial load , so you may run holiday better.

is this your answer?

Copyright © 2025 Infor. All rights reserved.