Data Sync service to transfer data from Infor data lake to external object storage

Hi,

Problem: With the introduction of micro-batching in Data publish management module in ERP LN the messages are delayed sync to the external applications. To overcome this publishing the delta load by Infor data lake ingestion is the possible solution. As we have our primary data platform/ data lake is outside the Infor ecosystem, we avoided the Infor data lake ingestion and used the ION messaging service to publish the data and used ION API connection to send the data to our primary data lake. Now we understand that IMS messages are published with delay due to micro-batching and also Infor is going to deprecate IMS in the near future, we want to check the possibility of any data sync service to push the data from Infor data lake to external object storages in the cloud like AWS S3.

Looking for the expert comments

Find more posts tagged with

Infor Data Fabric

Accepted answers

All comments

Kevin Heiman

Hi. I am not clearly following your subject but feel there are some misunderstandings.

When an ERP such as Ln (or any data producer) is publishing the table change data (live or initial load), it can do so through IMS API or directly via Data fabric API (using ingestion or streaming ingestion). There are other methods via ION to pull data into data lake but I will skip that for now.

If IMS API is used then a ION Data flow routes the data to data lake.
If Data Fabric Ingestion API is used , the data is written directly to data lake.
If Data Fabric streaming ingestion is used, the data is available for pipeline to consumers, and will be micro batched into data Lake as well.

I am unclear on the problem case in your first and second statement. Are you trying to overcome the micro batching delay from Ln? by running initial data load frequently?

The model where you have data platform/ data lake outside of Infor is covered by the Pipeline consumer feature.

I am unclear why you would avoid the Infor Data Lake, we would see it as servicing both with data.

When you state "used ION messaging service to publish the data and used ION API connection..." this is an acceptable consumer / outbound data mode. It is unclear as to if you are using both to build the pipe, as that would seem contradictory as IMS is an API, or are you using IMS as your connection to your data store architecture?

How are you acquiring the data to feed the integration? Pull from Data Fabric or push via a data flow (from ERP IMS). Direct ingestion and streaming ingestion would require a pull from data lake. Pipelines would be ideal choice there.

Your mention of Infor deprecating IMS messages, I presume is in respect to Ln publishing approach. Infor OS is not deprecating IMS.

My takeaway is there are variable options in Ln publishing causing time delays in data for you. What are your time requirements? I would suggest pipelines is one solution but there are considerations there. The consulting teams have built data flows to external consumers and likely has some skills to offer in the design.

You can contact me via email and we can have a call if needed.

unknown

InforOS is one of the ERP applications we have in our Application portfolio.
Our primary data platform/data lake is in our own AWS account.
All the data producers in our application landscape pushes the data to the primary data platform and it is then utilized for different operational purposes.

We have certain datasets maintained in ERP LN data publish management which are triggered on ERP LN events.
In Infor ION, we have configured a data flow which uses source connection point as IMS and AWS S3 API as the target connection point.
We are thus pushing the data from IMS source to S3.

Now that the IMS will be deprecated in the future what will be the possible solution to sync these data sets to the external AWS account?

Hope the problem statement is more clear now.

Kevin Heiman

Thank you, that was helpful. The Ln publishing using IMS is being deprecated in a move to doing streaming ingestion, therefore you will not be able to use the ION Data Flow to collect this data and forward to S3.

Data Fabric Pipelines is a method to collect the streaming data and persist it to a Data Store in your account. Currently PostgreSQL and other options coming such as Snowflake and even roadmap plans for your own consumer.

In the new model for Ln streaming, there is a Data Fabric service that will collect the Ln stream and persist the Ln data into Infor Data Lake in a micro batching based on size or time limits.

Once the data is are rest in Data Lake, the existing methods for acquiring that data are still there to push the data.

using ION AnySQL connection to pull incremental sets (files or Queries) and push to your IMS / S3 connection point.
Using Data Fabric APIs to request the data as raw files or as compass results
Using the ION ETL Tool to fetch the data to your store
Using BaaS to orchestrate complex data scenarios.

I suspect you want #1 using files, but there are other choices.

unknown

Thanks Kevin,

That was helpful. Yes I suppose #1 is something we could make use without much changes in the current setup. We are also thinking of making use of kinesis streaming but also considering the cost involved.

#3 & #4 sounds interesting too.

#3 - Can the ETL tool route the data to object store?

#4 - Is this already available to the customers? Can you share some more details on Backend as a service ( BaaS ) ?

unknown

Hey Niraj,

In addition to the options outlined by Kevin, transitioning your Data Lake Flow in ION to utilize the retrieval mechanism is the easiest and best recommended approach if keeping things generally like-for-like. You can read more on retrieval steps here.

In short, as applications evolve to utilize more native Data Fabric ingestion services, we recommend using services that directly interface with the Data Lake and/or Data Fabric services as part of a greater solution architecture. Specific to those apps that are streaming data, events follow a service level objective (SLO) where information will arrive in Data Lake within 10 minutes (or 5 MB of streamed data). This should provide a more predictable overall footprint and content designs/expectations for data consumers. I'd recommend checking out the documentation for more information.

Updating your Data Lake Flow to the retrieve step and placing it on a schedule will provide general continuity of your designs you have in place but over time, I'd recommend evaluating some of the new Pipelines features to see if there's better alignment with your functional needs. We're also looking at adding an S3 destination for Pipelines in the next year so if there's interest here, add in an Enhancement Request via ERS

unknown

Thank you Mike & Kevin

This is helpful.

Data Lake Flow with ION is an easy option for now but its not event triggered and works based on scheduled retrieval.

S3 support for the stream pipelines will be a good option for us. We will raise an ER for this

Copyright © 2025 Infor. All rights reserved.