Hi All,
How to implement incremental load in ANY SQL Modeler?
we have tried using incremental keys ,but it is always doing full load. PFB for the same.
it would be great help if anybody can help us on this.
Regards,
Anish k.
Hi Anish,
Looks like you're in the right direction here. AnySQL can leverage columns within your table(s) to run incremental data loads but there must be some criteria met in order to successfully work. In short, your incremental key must be either:
The first time AnySQL runs, it'll complete a full initial load. At the end of that run, it'll read whatever you've defined as the incremental key and save that to memory so that the next time AnySQL is scheduled to run, it appends a "silent" filter to look for any records after that key.
AnySQL doesn't store anything beyond that incremental key or cross-check to see if records were captured during previous runs.
In this case, it looks like perhaps you may be okay to just use your DateModified column unless that's a NULL value? Right now, AnySQL is capturing your DateCreated + DateModified and without knowing some more about how your data and tables are configured, it's a bit of a guessing game as to what's preventing incremental from running as you'd expect
You can try to submit a support incident to review the tables and see if there's some sort of bug or a configuration issue.
Best of luck
