Looking for an example of using Data Lake Bulk ingestion/ION IMS APIs to create a custom data load for a table in our ERP (CSD) that is not already part of the pre-built Object Schema definitions. Absent an example, process steps would be helpful.
If you are intending to build row level changes as your data set to data lake, you should include database deletes and also some variation ID in the row. The optimal data is ND-JSON as you note.
What do you mean by prebuilt content? If you intend to populate an Infor application schema, with specific data, for the intent of using Content such as Infor Analytics in Birst, I don't think that would be a good idea.
As to micro-batching, the Data Lake prefers larger batches over micro. It is a question of how frequently you want to send your schema row changes. We have some that range form 5 seconds to ones that you Schedule hourly - daily. If you are capturing every DB transaction, and collecting for 1 minute or until the file is at a size under your license, that would be a good start.
Note that if you are using Bulk Ingestion, then the size can vary based on your network capacity. You should keep it under 10 mg though.
IMS API needs to be at your license limit size.
There will be a new API planned that will support streaming, but you may want to build before that.
Be sure to add as many properties to the send, and at a minimal 'instances' and 'source' . If it is a batch then add those properties.
These are two different APIs for different use cases:
1. Bulk Ingestion is where you create compressed files to send to data lake directly, bypassing ION
2. IMS is where you would build payloads to send to ION to then Document Flow to Data Lake. Payloads need to meet your size restrictions. Batch identification should be used.
Also consider using the features built into ION; Data Loader and/or AnySQL, to load data to Data Lake.
Are you looking for example in a particular language ?
Hi,
Our team (Platform Technology Group) have built a couple of utilities and a Widget through which you can perform bulk load data into Datalake using the latest Datafabric APIs. This will be in Infor's marketplace soon but i can showcase the utilities. Please reach out to me at raghavender.hariharan@infor.com
One such Widget is already in Marketplace. https://market.infor.com/en-US/apps/359766/csvexcelxmljson-to-ion-via-file-import-infor-widget
Our use-case is to replicate as best we can the ingestion process for pre-built content that already has an object schema definition. As we understand, that process micro-batches data to optimize data publishing and query performance - sending data objects in flat Newline-delimited JSON format.
I would invite you to create an Enhancement Request for the CSD product management to include this table as part of the replication set. You can find instructions on how to do that here: https://community.infor.com/infor-ion/w/wiki/154/how-to-enter-enhancement-requestsYou might want to search if the request already exists and endorse it instead since there are quite a few requests of this type in the system.
