Choosing Timestamp Field for Data Lake

Hello,

We have several million records in the R5EVENTS table. The EVT_LASTSAVED was introduced May 2019. So, records prior to that date have null in the EVT_LASTSAVED field. I do understand that InforSupport can populate the EVT_LASTSAVED field on prior records.
I am wondering if I can do initial download using EVT_CREATED as the Timestamp Field first. Once I have most of the records ingested into the Data Lake, I will switch the Timestamp Field to EVT_LASTSAVED. Does anyone see any problem with this method?
My biggest concern is that we operate 24x7 globally. And, if I was to have InforSupport start updating the EVT_LASTSAVED starting on a Friday afternoon (EST), that script may continue to run throughout the weekend onto Monday morning (CET).

Thanks.

Find more posts tagged with

Infor Data Fabric

Accepted answers

Legacy Contributor

Hello Syd!

There are some limitations you need to understand with EAM and DataLake sync.

1) Only 15,000 records may have the same EVT_LASTSAVED value.

2) A single DataLake sync may not be larger than 500MB (maybe it is 512MB).

If you have Alert Management I recommend you create an alert that runs every minute and update 15k records. With several million records this will take some time to process through. Also, rather than setting the value of EVT_LASTSAVED from EVT_CREATED I would use a COALESCE(EVT_UPDATED, EVT_CREATED). That way you only need to push the records once.

Thank you,

James Crow

All comments

Legacy Contributor

Hey Syd,

By timestamp, I think you mean the variation ID - correct?

In your scenario, do you mean updating the definition or selection of which field functions as the variation in the EAM Data Lake Upload utility? If so, changes to the schema there would cascade over to Data Catalog to reflect the definition provided by EAM.

I believe you'd still end up with EVT_LASTSAVED as having a null value regardless of which column you selected as the variation ID.

Legacy Contributor

Hello Syd!

There are some limitations you need to understand with EAM and DataLake sync.

1) Only 15,000 records may have the same EVT_LASTSAVED value.

2) A single DataLake sync may not be larger than 500MB (maybe it is 512MB).

Thank you,

James Crow

Legacy Contributor

Thanks James. The coalesce() was a great idea to update the lastsaved field.

Copyright © 2025 Infor. All rights reserved.