Infor M3 Events to Azure Data Lake

Hi All,

I am evaluating how can we replicate infor m3 data (events) to azure data lake ? Have you come across similar scope of work. If yes, could you shed some light on the technical approach taken ?

The below options seems to be PULL scenarios and start with Infor Data Lake as the data source, however, I am exploring the possibility of having one data lake (lets say Azure DL) for our organisation where m3 events can replicate (event based or scheduled) for real time streaming and BI usecases. This data lake will serve as the single place for data converge across the enterprise.

1. Infor ETL Client

2. Compass API

3. Payload API

Find more posts tagged with

Infor Data Fabric

Comments

per-melander

Hi, are you looking for feeding data via Azure DL to a Power BI or similar? Would you build a Kafka solution to stream data from Infor (AWS) Datalake to Azure? What do you find as the benefits with MS over AWS? Would you want to see a scenario where M3 was streaming data directly (or finally) into Azure DL for BI consumption? Sorry for all questions - just want to know what you ultimately would want as your solution for M3

Legacy Contributor

Hi, I am looking for streaming M3(cloud) data into Azure DL for BI consumption.

On a side note, I was looking into ION Desk and seems like Connect + Dataflow/Document Flow -> can help me to push M3 table data to external system.

Let me know your views and if you have ever run into similar use cases.

Cheers,

Vaib

tdouglass

Hi Sharma, was this a succesfull approach? Just wondering for our future. How did you estimate the amount of data that you would be pulling out of the Data Lake to know if you are within the limits or how far over and above the limits you would be?

Legacy Contributor

Hey all,

Interesting question & topic. Data Lake is a big data platform without real-time tendencies so if building for real-time use cases, I would recommend being cautious about expectations. While M3 replicates in near real-time @ every 20 seconds or so per table, the Data Lake is "eventually consistent" and so it's not effective to govern with real-time SLAs at this layer.

While you can technically use ION to bifurcate the flow of data objects from M3 to an Azure API, for example, ION is an asynchronous integration platform and your client would also need to re-implement processing logic on how to handle parsing, variation sorting, and so on. However, using storage and Compass APIs are probably the more effective routes in addressing the spirit of the question.

All this is to say that the business & use case are on point - and that we're building new solutions to layer in real-time capabilities within Data Fabric We have a new streaming data platform that allows applications to stream their events in real-time and we're introducing a new feature later this year called Data Fabric Pipelines that can simultaneously process those events and push them to data systems (e.g. cloud-based data warehouses) to support operational use cases.

Let us know what kind of connectors, technologies, and destinations ya'll want to see as we build up our roadmap to expand

tdouglass

Mike, can't wait. Thank you for this information as operational reporting is a big struggle with us. I do have a question i am curious on, which i am sure is just terminology. When you mention Real time Data replication into Cloud Data Warehouses, I am assuming you are not referring to 'BI' data warehouses, and just using data warehouses in a general term? I ask that because Operational Reporting and Star Schema type data warehouses are contradictory? Thoughts?

Legacy Contributor

Hey TD,

I'm using data warehouses as a general catch-all just to maintain brevity and sanity.

Pipelines will have connectors to Infor products - think, d/EPM, Mongoose, Birst, and others over time. However, there's a need to support independent technologies as well. We're starting with AWS Aurora and plan to expand to other AWS RDS instances.

However, pipelines is about addressing operational use cases and not all operational use cases involve relational stores. So, while we will expand to add destinations around Azure, GCP, and other relational instances, we're also looking at opportunities to support technologies like object stores (S3, Azure Blob), managed streaming services (AWS Kinesis, Azure Event Hub), and others.

One thing to keep in mind is that real-time processing isn't necessarily a replacement for complex ETL - capabilities that you can manage and orchestrate with Compass, for example.

Operational use cases are going to be a sweet spot but there are some advantages that could present themselves when you look at solving star schema use cases in terms of process flow (e.g. reduction in load times prior to BI workload execution). There are other advantages that are contingent on some visionary roadmap topics but I'll save that for a live stream or conference

Legacy Contributor

There's no like button for replies, so ...

Legacy Contributor

A word of caution regarding using the data lake for operational reporting. Our ERP is CSD, and for several months now, we've been running into instances where transactions are either not replicating into the DL, or there are issues with the proper variation ID. Both of these issues have resulted in inaccurate data appearing in our operational reports when compared to the status in the ERP. In our case, the primary result has been transactions such as orders or transfers that are showing as being in one stage code (received, canceled, etc.) in the ERP, but in another stage code (entered, picked, shipped) in the DL. We've been working with Infor support to correct these issues, but continue to encounter discrepancies.

tdouglass

Tom, just curious on what tool you use for operational reporting. Do you extract the data from the data lake into Mongoose or do you have something using the API or Connector? Just curious

Legacy Contributor

Hey Tom,

I wouldn't ordinarily recommend Data Lake for operational reporting use cases given it's an eventually consistent data platform but it depends on one's definition of 'operational,' of course.

If this is less to do with data delivery timing and more so incorrect data, the ERP team would need to address that, unfortunately. I have seen some customers reporting duplicate variation IDs which can be annoying but I would also recommend utilizing the variation ID to determine handling of how you process records downstream. Specifically, avoiding inserting or updating records in a target system unless the variation being processed is > than the one already stored.

Copyright © 2025 Infor. All rights reserved.