Special Characters in Python Script reading CSV

Nick Stoltzfus

Hello,

I have a python script that is working very well for the process of reading csv files and converting them to BODs. The issue I'm having is that occasionally the csv contains characters that seem cause problems with the input paramter of the python script. They are spanish language characters such as Ñ Á Ó Í. When these characters occur anywhere in the CSV file the input string is completely unrecognizable from what the file actually contained.  The input string is invalid before I try to do anything with it in the script.

For example, the csv file begins like this:
HDR,WHSE02,820598,11320970,RDY,

The value as it is in the script begins like this:
eF6dldtunDAQhu8j5R38AAMa2/iUO7Kwh4bYkYF

Oddly enough, this problem is not evident when using the script Testing function and I paste the CSV contents into the input. It only occurs when it runs through a data flow.

Would there be some kind of work-around to handle these characters before they arrive at the scipt in a corrupted state?

unknown

Can you please check if the input csv file is properly encoded using one of the ION supported formats? Top of my head, I think it needs to be either UTF-8 or ISO8859-1, but please check the file templates documentation to confirm. 

And also please check there are no BOMs (Byte order marks) at the end of your file. Its tricky to check this, but if present they are known to corrupt the file processing in some cases as well.