Hi,
What is the max flat file size a file connection point can consume?
I need to process files of about 160Mb in size, but it looks like ION does not like it.
Thanks,
KK
Hi Kirill, Maximum file size depends on the license tier: https://docs.infor.com/inforosulmt/xx/en-us/usagelimits/default.html?helpcontent=nkb1720800864258.html . The recommended size is 5 mb: https://docs.infor.com/inforosulmt/xx/en-us/usagelimits/default.html?helpcontent=nkb1720800871140.html We are working on increasing the maximum file size that ION can handle. What kind of file is it? Do you need to transfer this file (Lift & Shift) or do you want to process (Mapping, Scripting, Routing, Monitor, etc.) this file?
Kind regards, Danil.
Krill, there is are Batch and Streaming APIs to write the data to data lake if you have that option in AirFlow.
https://docs.infor.com/inforos/2024.x/en-us/useradminlib_cloud/default.html?helpcontent=datafabrug/llv1631199543458.html
If you require an ION Connection point then the the limits apply as it will need to go through a Flow.
For your 'upload files to dataLake ' … as Danil asked, nothing in the ION flow except connection point to collect the file and first step to write to data lake (or other destination). There is no filter, mapping, scripting, … Generally referred to as Lift and Shift. What type of connection point? and what type of type (XML, JSON, DSV, binary..)? Read Files, so an application will use the Data Fabric File APIs to read them.
https://docs.infor.com/inforos/2024.x/en-us/useradminlib_cloud/default.html?helpcontent=datafabrug/pba1631199548960.html
And then write them back as newer version or different file? Using the Batch or Streaming API.
Hi Danil,
I need to do two things:
I can upload raw files to DataLake with Airflow as well, is there a limit for file size in DataFabric API?
Thanks
Hi Kevin,
It's file connection point (SFTP). Files are just text files.
Hi Kirill,
This is an interesting one as although the above documentation is referring to 5mb regardless of license tiers, I was under the impression file sizes scaled up accordingly with licensing.
I haven't seen anything in the documentation stating that 5mb is a compressed file size, or uncompressed file size. 5mb uncompressed seems low for modern day cloud architecture.
There might be the possibility of ingesting the file as a compressed.zip file, then running an ION Script over it to uncompress, extract what you need and output a much smaller file size if possible - provided you stay within the ION Scripting allowances https://docs.infor.com/inforosulmt/xx/en-us/usagelimits/default.html?helpcontent=avj1720800864622.html and ~15 seconds execution time limit (from memory)
Hi, Related to this max message file size and Service OS limits, we are using InforOS enterprise so should be able to process larger files without ion batching them (we have a 20mb file and Enterprise allows upto 50mb it says here: https://docs.infor.com/inforosulmt/xx/en-us/usagelimits/default.html?)helpcontent=nkb1720800864258.html.
However, we are still seeing ion batching into 5 chunks , then passing to a python script to process 5 times and then outputting 5 split files. This is causing 2 issues , the main one is that the entire dataset needs to be processed by script ,otherwise we get invalid and duplicate data output as related data is being split across files and processed separately. We need a single file so are using the connection point , document "append" option as a workaround. A Single native file would be better. Support say 5mb is a python hard limit , unrelated to OS service limits. So we are not sure of the expected behaviour. Can someone explain ?
The Maximum Message Size only applied when a flow is direct application connection to another application connection. This is mentioned in the Glossary and examples. There are roadmap topics on enlarging the message size passed into other nodes/ components that could be added to a flow. AnySQL modeler setting on size recommend 5mb due to this downstream document flow components limit on size.
As messages sizes growth, there will eventually be a limit to handle size of file, micro batching vs large file processing is desired in most integrations. Follow the Integration Roadmap for future possibilities.
