ETL Client for Data Lake

The following slide was shown on TechEd last December, which says that some ETL client exists to get data from Infor Data Lake.
Pentaho was used as example, but it was shown too briefly.

I am wondering if there an additional information or documentation available?

Thank you,

Ruslan

Find more posts tagged with

Infor Data Fabric

Accepted answers

All comments

unknown

Hi Ruslan

there are a number of ways to get data out of the Data Lake, depending really on your use case and if you want to build your own solution or use a pre-built one. There are a number of API options that you can use to build your own tool, either "raw" object API's that return the object exactly as it was put in to the DL or Compass API's that provide an SQL interface over the data and return data sets - there is also a JDBC driver that can be used for SQL in the right use case as well.

But all of these are tools to build your own solution and included in the normal CloudSuite licencing - the ETL tool is an add-on (so additional licence) that uses these methods to pull data from the DL and write it to a local DB. It understands thinks like variation IDs and deleted flags, as well as the Data Catalog etc... - so in other words makes it a lot easier to just replicate some DL objects ("tables") from MT to a local DB - allowing you to then just use the local DB as you did before. Of course Infor maintain this tool and so as and when there are new features or changes to things, it will be updated and so you don't have to worry about maintaining your ow solution

For full disclosure regardless of the method you use to pull the data out of the DL to OP you will also need the add-on External Transfer licence

I hope this helps

unknown

Ooh I think that is the ETL data shredder referenced in another post. Any user guide or developer guide available for that by chance? I saw a slide detailing the pricing, but would love to understand functionality a little more. Thanks!

unknown

The tool is really easy to use, but also very flexible and powerful, let me just explain the basics. First on your MT instance you would go in to ION API and create an authorised app, this will create a set of secure credentials you can use with the ETL tool to connect to you tenant. Once you have that go to the ETL tool and drag drop the Data Lake icon on to the design area, then drag the upsert DB option - open the DL one and select the ION API credentials you created, click the drop down to show you the list of objects you have defined, select the object ("table") you want, press the button to get the objects details - columns, keys, variation id indicator etc... all just pulled from the data catalog for you. as the easiest option just select the "all" option and you are about done. You could look at a preview and adjust things but it is not normally needed. now open the upsert option, point it to your DB, enter the table name and click a couple of buttons to pull in the key details, variation ID, full list of columns - you can them use an inbuilt option to create the correct table, with indexes etc... - you are done, save the whole thing and hit run, it will pull the data down and populate the table for you. It will understand the different variation's of the object and rows in the data, set the delete flags and anything else you need.

Of course you can then take that and automate the pull, or set rules around which other tables are in the same batch, etc.. You can do a little extra work in the first step to convert it to do incremental loads of data, then you only pull what you need each time - or you can switch from the "all" mode to an SQL mode where you can specify an SQL statement of your choice, if you feel you need to manipulate the data before it is pulled down as a result set. The right option will depend on your use cases and can be a mix of all things available.

I have just scratched the surface here really, although the above is what the majority of people actually need - the tool can do all manner of other things if needed.

unknown

Does this tool support connection to MS-SQL databases? Is it possible for it to transfer only the changed records on a schedule?

tdouglass

Hi, this references a local DB as the target, however is there any reason why this couldn't be an azure sql database that is in the cloud but outside of the Infor Suite?

tdouglass

Hi, Is there any feedback on this question?

unknown

I don't think I have seen Azure DB as a default option but as long as there is a standard driver like a JDBC driver then I expect it is possible

Copyright © 2025 Infor. All rights reserved.