Brand new to IPA and need some assistance. Trying to construct a flow that reads a file, changes it, then writes a new file. I seem to be able to read the file in with file access node, but then not sure how to use the file once I get it.
Good morning Todd, Welcome to Infor Process Automation Community. I will take a shot at answering your question.
For this example, I created fake csv data which is just a file with:
two,one,three,four,five
...
I want to build a flow that reads this file, modifies it to be:
one,two,three,four,five
The flow uses the Data Iterator node with an Input Method of "File" to parse my file by line.
IMPORTANT: Using the Data Iterator to iterate through the file is the Infor Support Recommended path. If you use a File Access node, you would then need to use the output from the file access and change the Data Iterator to Input Method = Data. Until a VERY recent Landmark CU there was a performance issue when using the Input Method = Data to loop through a file.
Inside the loop of the Data Iterator, I am leveraging a javascript split command to send each field into a variable and I am also incrementing a counter to count up to 100, when I reach 100 I goto a file access to Append my output file, then clear out the stored data. This prevents you from building an excess large file in memory and impacting system performance.
IMPORTANT: Java strings are immutable; if I build output of a file in a string and keep changing its value for every record in my input file without ever clearing it out, the system has to use extraordinary amounts of memory to do what you are asking it to do. It is imperative when building output from within a flow, to periodically cleanse the variable and append the file.
For example: Consider a file that writes 100,000 lines on the way to creating a file that is 3mb's in size. A simple way to view this would be, at least half 50,000 lines would be appending a string that is at least 1.5 mb's in size. Since EACH modification to the string creates a new object in memory and throws the old one out. Your flow would be generating a minimum of 1.5 x 50,000 = 75,000 Mb = 75Gb of memory objects to write the last 50,000 lines.
If instead you were to clear out the output variable every 100 times you modify it; you would never be creating a variable/object larger than 3mb / 100,000 * 100 = .003 mb ... writing those last 50,000 lines would take .003 * 50,000 = 150mb of memory objects
I have included an example flow for your convenience that illustrates these concepts, I hope it helps.
Russel Cornelson
Principal Technical Support Analyst for IPA @ Infor.
NOTE: The following file is an IPA .lpd file; but Community forum won't let me upload it with that extension so I renamed it to .xml. If you download this file rename it back to an .lpd file extension.
[View:/cfs-file/__key/communityserver-discussions-components-files/57/exampleDataReadModifyWrite.xml:320:240]
I am getting "File "C:UsersADMNAYAKHEmployee" does not exist." error when using DataIterator in IPA to read from file. Even though file exist in the location above.
Make sure that the file "Employee" exist in the following directory "C:UsersADMNAYAKH"
Yes, the file exist in that location.
So are you trying to run the flow from your local computer or from the Landmark server? If you uploaded the flow to Landmark and you're running it from there, it's trying to find that directory on the server. Based on the directory name, I'm assuming this file is located on your computer. The access can also depend on how you have the File Access property setup under SystemConfiguration.
Yes I am trying to access from the uploaded flow to Landmark. I have the File Access - Property setup for main and system to use local under SystemConfig.
