Memory Error on Reading a file from pfiFileStorage (65k lines)

Hi,

In our process flow, we cycle through the lines of four Infor-generated HIPAA834 files. With 5,000 employees and 15 lines per employee, each file is approximately 65k lines. We do this to look for any errors prior to sending the file. The outer loop is just looping through each of the four files. Within that, there is a Data Iterator loop. We are using File as the input method and parsing by Line. Basically, cycling through all lines of four 65k files.

The only thing we are storing is a few Bad Lines in the MsgBuilder. We simply use a javascript "indexOf" to see if there is any bad data. If we find bad data, we set a flag to send an email notification with the lines that we saved in the MsgBuilder.

It seems strange that the Assign6340 node would have 94,310 MiB. The only thing it's doing is doing an "indexOf" on each of the lines and setting a flag if it finds a particular value. It's not storing any data other than the flag.

[#1 - |Assign6340|Assign|295442x|94310.33 MiB|149020 ms|125710 ms|424661 ms]
[#2 - |Branch5890|Branch|295442x|47909.04 MiB|68648 ms|57680 ms|185064 ms]
[#3 - |End-FileLines|IterEnd|295446x|33345.39 MiB|57599 ms|49310 ms|105296 ms]
[#4 - |FileLines|DataIterator|295446x|32951.44 MiB|50245 ms|44050 ms|99994 ms]
[#5 - |End|End|1x|9.71 MiB|22 ms|30 ms|56 ms]
[Memory Alert - Final] WU Alloc [182914.85 MiB] > [100000.00 MiB] - See KB 2148373

Are there any hints out there for reading large files to check for bad data? Here is our flow:

dEPM-665 - Module Activation.mp4

Find more posts tagged with

Process Automation (IPA) - General Discussions

Accepted answers

russel-cornelson

var lines = String(FileAccess_outputData).split(/
|
|
/).filter(Boolean)
var lastRow = lines.splice(-1);

lines.forEach(function (line) {
if(line.indexOf('INS*Y') != -1) {
SubOrDep = '(Subscriber)';
} else if (line.indexOf('INS*N') != -1) {
SubOrDep = '(Dependent)';
} else if(line.indexOf() != -1) {
..
}
});

This should run quite a bit faster, and should use far less memory.

NOTE: I haven't tested this but ...

string.indexOf() is not supposed to be the most efficient way to look for a string inside a string because i believe it creates a new array as it accepts a starting position that it loops through. But in your example you are not using it that way as you are not sending a starting position so it might be smart enough to not create a new array.

string.includes("string") is more efficient; but we don't have access it yet on our current version of Javascript Engine. Although we are working on upgrading that engine at some point in the near future. which would then allow the includes function along with ES6 javascript.

An alternative way would be to put an assign node outside of your loop that created regex expressions like this...

var Sub = new RegExp("INS*Y)
var Dep = new RegExp("INS*N)
var EMP = new RegExp("NM1*IL")
var DTP = ...
var DMG = ...

These would need to be created outside the loop so you don't recreate them every iteration through the loop but then your javascript could be...

if(Sub.test(line)) {
SubOrDep = Subscriber
} else if (Dep.test(line)) {
SubOrDep = Dependent
}

I haven't tested these .... so I am not sure if there is a difference in your use case or not. But if you are looking to reduce memory I think it might we worth a shot if its not alot extra work. Normally I would test and let you know my results but I am short on time this week.

All comments

unknown

...also, #2 above (the Branch5890 node) is only checking a flag to see if it's true or false. I don't see how that Branch node can use 47,909 MiB in the Work Unit. That's almost half of what is allowed for the whole Work Unit. It's checking the flag about 260,000 times (four files with 65k lines each) but I don't see how that is related to memory. There must be a more efficient way to do this, but I don't want to have to create lots of separate IPA flows just to lower the memory count.

russel-cornelson

@david-wilson

I have a few questions:
- How big are the individual files?
- When you say you are looking for bad data what exactly are you looking for?

If I am making some guesses I would say that those files are probably under 10Mb of data; which means I would do something like this.

ForEachLoop to loop through files

FileAccess node read file
Assign node to look through the data.

var baddata = [];
var lines = String(FileAccess_outputData).split(/
|
|
/).filter(Boolean)
//splits the file into an array of individual lines and removes any blank lines

lines.forEach(function(line) {
//loops through each line and in the loop you can reference a variable called line.

var [field1,field2,field3] = line.split(",") //not familiar with file format but this splits each line by comma into fields

if (field1 = "somebaddata") {

baddata.push("this is a bad line" + line)
//push the line to an array to hold the bad lines;

} // end if

}); // end of for each line loop

Without specifics of what is in your assign node or how you are determining what is good/bad I can't give any more specifics. But the FileAccess > AssignNode to loop through is very efficient in most use cases. You should probably avoid this strategy if the file is over 100MB as you are loading the entire file into memory at once where the data iterator node streams it in chunks.

russel-cornelson

If your file has a header row:

var lines = String(FileAccess_outputData).split(/
|
|
/).filter(Boolean)
var header = lines.shift()

now you have a variable called "header" that contains the headers; but the "lines" array no longer has the header record.

russel-cornelson

if your file has "totals" on the last line of the file.

var lines = String(FileAccess_outputData).split(/
|
|
/).filter(Boolean)
var totals = lines.splice(-1)
now you have a variable called totals; and the last line containing the totals is stripped out.

0712131256598360.doc

unknown

Hi Russel,

Thanks for responding. Here are the answers to the questions:

You are correct...each file (there are four of them) is about 2Mb at the most, so quite a bit less than 10Mb.
By looking for Bad Data, I mean that I am looping through each line of a HIPAA 834 file (using a Data Iterator) and doing an "indexOf" on the line to find something like "D8*00000000". That would mean a date is empty. Here is an example of a line. The DMG indicates that this line is a birthdate. If birthdate was eight 0's, it would be bad data.
- DMG*D8*19520801*F~
So I look for FileLines_outputData.indexOf('DMG*D8*00000000') > -1
The file does not have a header, but it does need to include a count of lines at the end of the file as a check.

I should note that when cycling through the files, very few lines have bad data. So the message builder I have might only have a few lines in it.

Also, I am saving a name and indication of Subscriber or Dependent when I come across them while looping through the file, but the variables get overwritten each time I get to the next person. This is so that if I do find a bad record, I have the name associated with it. This is my Assign Node within the Data Iterator loop.

Within the file, there are line types and 10 lines make up the set of data for a person. Whenever I come across the "INS" type, that's the start of a new set of records.

Looking over your previous message, it looks like you are in favor of using javascript to cycle through the records. I was using a Data Iterator. Maybe the javascript uses less memory or manages it better? I can try that instead of this Data Iterator.

Thanks!

David

Copyright © 2025 Infor. All rights reserved.