Replication to DataLake - Red Alert Flag removed but processing not complete?

For my replication from the DataLake to local server I do the following:

I use an IPA to kick off my replication to the Datalake.
Within that IPA I loop, checking for: RedAlert (the flag on the replication set stating the replication is still running)
If the RedAlert flag is not equal to 'true" then I write a completion file to the FTP server and the IPA ends.
I monitor for the completion file via VisualCron and once the file arrives, I kick off the IONETL process to import to my local SQL server.
After the ETL process is complete, I kick off a record count for each table in the DataLake and my local database, creating a comparison report.
The whole process takes about 25 minutes, more than half of that is taken up by the record count of the DataLake.

I have observed my comparison report continually shows I have less records in my local than the DataLake, for 10 or 15 tables out of 500.

However, if I run the single IONETL transaction for one of those tables again, say 5 to 10 minutes after my comparison report is complete, I receive the "missing" records and the count matches between the DataLake and my local database.

The RedAlert != "true" does not occur until all work units have been processed through the Queue as shown in the Asynch monitor.

Is there a cache or a process working in the background that has not finished when the RedAlert flag is removed from the replication set? If there is, then is there a way to monitor it so I don't kick off my download until after it has completed?

In the meantime, I'm going to add a hard coded 5 minute delay between when the RedAlert is removed and I kick off the IONETL job(s). I'll be working on finding the "correct" wait time.

Thanks,

Polar Bear

Find more posts tagged with

Infor Data Fabric

Accepted answers

All comments

unknown

A five minute delay helped, moving to a 10 minute delay to see if that catches all the updates.

Note, there is no other process updating the tables in the DataLake, these are not INFOR delivered. These tables and the replication was specifically created to use for our download to local, purposes.

unknown

10 minute delay catches more of the changes, it really depends upon the load. If I see in my record count comparison there is a large difference, I run the specific transaction to get the remainder of the data.

unknown

I've opened a ticket with INFOR on this.

unknown

Results from INFOR ticket:
There is a delay between when the replication set states it is complete (red flag is removed) and when the data lands in the DataLake.

Regardless of your selection of IMS or DataLake, the data does not go directly to the DataLake. An interim container holds the data, either IMS or Atlas(?). Either way the red flag is removed when the replication set has completed, it has no knowledge, or doesn't care, where or if the data reached its ending point.

To validate the records have reached the DataLake you must check if all the records have been indexed in the DataLake. You would do this by: executing a SQL Statement using the DATAFABRIC API to check the record count for all RepSetBC by this Variation ID, when the counts of all replicated classes match the RepSetHistory for this Variation ID the data has been indexed.

Enrico Menotti

I have something to say here which may be interesting for you, but first let me ask a couple of questions.

Are you replicating from LN?
What is IPA? In particular, what does the acronym stand for? (Pardon my ignorance.)

Now my two cents. What I have learnt is that the Data Lake is made up of two parts, Atlas and Compass. The former is a collection of raw files, either json or dsv (or maybe other formats as well, I don't know). The latter contains what Infor calls the "Queryable Area", and this is where Compass retrieves data from when you write a query in the Data Fabric web interface (OS Portal -> Data Fabric -> Compass). The Queryable Area is not built up until at least one query is raised against the raw data; this first query triggers the process and does not yield any result until the process has ended (usually this first query simply times out). My understanding is that the QA is like an indexed cache. There are commands in Compass to clean it up (table by table, at a couple of levels).

The classic method for moving data from LN to the Data Lake is through the Infor Messaging Service, or in other words through ION. Data go anyway into Atlas first. They have recently introduced a direct publication from LN to Atlas, and the classic method is about to be deprecated.

I don't know whether the API you're using queries Compass or Atlas directly. Probably it's Compass and your issue is due to the fact that when you ask for data you're triggering the replication from Atlas to the Compass Queryable Area, and you only get data which have already been replicated. (I may be wrong, as I said this is my understanding.)

I also know that the old ETL tool to move data from the Data Lake to a local SQL server used to query Atlas directly. I don't know about newer versions.

Another example is Birst. There was a connector to Atlas, which had very limited functionalities and was very slow. Some time ago a new connector to Compass has been introduced, much more advanced (it understands almost all the Microsoft Transact SQL language, except some things like e.g. recursive queries) and fast. However, when you replicate a new table from LN to Atlas and then try to query it from Birst by using the new connector, you're likely to time out. You're in fact triggering the replication to Compass and data will only be available once this replication is over.

Enrico Menotti

BTW, what is the Case number? If I can, I'd like to take a look at it.

unknown

To view the case online, please use the view case link below.
CS0236862

Copyright © 2025 Infor. All rights reserved.