Hi, I have an issue where I a datalake query that returns about 60k rows, This appears to be being batched by ion into smaller files (4 batches), and then each file processed by a python script. We are then appending the output to the same file. We have a few issues as below and would like to ask if anyone has this issue, or has a workaround inside Cloudsuite, or do we need to process the larger files outside somehow . Thanks for any input / help !
- We need python to process the whole file as it needs to see the whole file to perform the script correctly, breaking it up results in errors in the data
- We are using the Connection point "if file exists" option to append the output from each iteration of the script. This is resulting in a large no of duplicates.
- We have the same issue with infor OS Essentials and Enterprise tiers , the Enterprise tier should allow file sizes of 50mb (https://docs.infor.com/inforosulmt/xx/en-us/usagelimits/default.html?helpcontent=sni1720800863492.html)
We have logged this with infor and they cannot assist much because they said:
- the higher license tiers allow larger message sizes but python scripting has a hardcoded 5mb file size limit
- they don't recommend larger than 5mb anyway due to potential platform issues
