The Infor Communities is globally accessed system by Infor and its customers, please do not post or upload any content that could be deemed as sensitive, confidential or subject to any data regulation requirements. If you are unsure whether or not the content contains sensitive, confidential or has any data regulation requirements, please seek the advice of your internal security, legal, and compliance experts.

  • State Suggested Answer
  • Replies 1 reply
  • Answers 1 answer
  • Subscribers 827 subscribers
  • Views 171 views
  • Users 0 members are here
Related

Any SQL Incremental Load

Anish Kulkarni
2 months ago

Hi All,

How to implement incremental load in ANY SQL Modeler?

we have tried using incremental keys ,but it is always doing full load. PFB for the same.

it would be great help if anybody can help us on this.

Regards,

Anish k.

Parents
  • 0 2 months ago

    Hi Anish,

    Looks like you're in the right direction here. AnySQL can leverage columns within your table(s) to run incremental data loads but there must be some criteria met in order to successfully work. In short, your incremental key must be either:

    • Some auto-incrementing ID/value
    • Datetime field
    • Date field
    • Time field
    • Concatenated date + time field split across multiple columns

    The first time AnySQL runs, it'll complete a full initial load. At the end of that run, it'll read whatever you've defined as the incremental key and save that to memory so that the next time AnySQL is scheduled to run, it appends a "silent" filter to look for any records after that key.

    AnySQL doesn't store anything beyond that incremental key or cross-check to see if records were captured during previous runs.

    In this case, it looks like perhaps you may be okay to just use your DateModified column unless that's a NULL value?  Right now, AnySQL is capturing your DateCreated + DateModified and without knowing some more about how your data and tables are configured, it's a bit of a guessing game as to what's preventing incremental from running as you'd expect Disappointed

    You can try to submit a support incident to review the tables and see if there's some sort of bug or a configuration issue.

    Best of luck

Reply
  • 0 2 months ago

    Hi Anish,

    Looks like you're in the right direction here. AnySQL can leverage columns within your table(s) to run incremental data loads but there must be some criteria met in order to successfully work. In short, your incremental key must be either:

    • Some auto-incrementing ID/value
    • Datetime field
    • Date field
    • Time field
    • Concatenated date + time field split across multiple columns

    The first time AnySQL runs, it'll complete a full initial load. At the end of that run, it'll read whatever you've defined as the incremental key and save that to memory so that the next time AnySQL is scheduled to run, it appends a "silent" filter to look for any records after that key.

    AnySQL doesn't store anything beyond that incremental key or cross-check to see if records were captured during previous runs.

    In this case, it looks like perhaps you may be okay to just use your DateModified column unless that's a NULL value?  Right now, AnySQL is capturing your DateCreated + DateModified and without knowing some more about how your data and tables are configured, it's a bit of a guessing game as to what's preventing incremental from running as you'd expect Disappointed

    You can try to submit a support incident to review the tables and see if there's some sort of bug or a configuration issue.

    Best of luck

Children
No Data