Removing Duplicates using ION

Hello,

We have remote telemetry units in the field that transmit readings to a vendor's cloud. The region's climate dictates the frequency of the readings being sent to the vendor to conserve battery life. There are only 3 fields that we will be retrieving; ReadingDate, AssetID, and Reading. We only retrieve the data on the first day of the month. So, there may be multiple readings for an Asset on different days. We only want to input one reading per Asset. Vendor will not be massaging the data for us. We will be using ION to retrieve the data. We are only interested in the last reading for any duplicate Asset reading. Is there a way for ION to remove all duplicate Asset readings? Would that processing be in the dataflow/workflow? Or, through scripting to remove duplicate readings? Not sure how to even proceed in either cases. The file format that is available to us can either be a CSV or json file. Not sure which one would be easier for ION to handle.
Any guidance would be appreciated.

Thank you in advance.

Find more posts tagged with

General Discussions

Accepted answers

unknown

Hello Mike,

I found some sample code on the internet and tested it out with my sample dataset on an online python. It works great. But, now I just need to know where I would place it within ION. Any guidance would be appreciated. I have never attempted using script before.

Thanks.

All comments

unknown

Hey there, RST -

So, it sounds like you have a 3rd party vendor generating a CSV/JSON file with a number of readings but you're interested in sorting the readings by most recent and selecting only the most recent reading itself, correct?

Beyond that. what's the rest of your requirement look like? Are you trying to process that data into an Infor application, store that reading into a database, or something else entirely?

Some additional details could be helpful in evaluating what might be the best tool for the job here

unknown

Hello Mike,

Sorry about that. Yes, you are correct. The 3rd party vendor is able to supply us a CSV/JSON file with readings for the month. Once we grab it from their site using ION, we need to remove duplicates and then add the meter readings into our Infor EAM r5readings table. We are able to update the r5readings table using ION. But, we need to remove the duplicates prior to doing so. Not sure whether that can be done in dataflow, workflow, or script stage. I would rather not have to manually filter out the duplicates during the process. Just wondering if ION can do that.

Thanks.

unknown

How are you updating your EAM readings table - API call or sending a BOD to EAM?

If you're ultimately going the BOD route, you could take the vendor CSV and use a file template to convert that to a BOD and apply some mapping logic in the ION Mapper to potentially order and/or select the row you'd want. That depends on the data itself and how amenable it is this kind of XSLT for processing.

However, you could just as well use a Python script embedded into your flow and process the data that way by evaluating the incoming file and removing any duplicate or stale data before passing on the output to the next step of your flow.

unknown

Hello Mike,

We are currently sending a BOD to EAM. We used ION Mapper and converted their JSON file. We ventured the XSLT and found it quite daunting and exhausting. Spent numerous sleepless nights playing around with it and got so confused. Kinda scared to go back into it unless someone has some code to offer me.

Like the Python script embedded into the flow. Will need to figure out the code and then figure out where I need to place it in the flow. Do you think the internet would have some sample remove duplicate Python code?

Thank you for your suggestions.

unknown

Hello Mike,

Thanks.

unknown

Found some of Keith's scripting videos. Will use that as a guide.

Thank you Mike and Thank you Keith. Keep those informative videos coming.

Copyright © 2025 Infor. All rights reserved.