Looking for best way to read data from Data Lake.

Legacy Contributor

What is the best way to read data from datalake to a Flat files or to update on-prem database? I tried creating a data flow where dalalake retrieve as the source and a file connection point as the target. Should the file gets created or database gets updated when I activate the Data flow? My data flow is doing anything, just wondering if there something I have to do.

Legacy Contributor

Hey there pmahapatra -

Hope you don't mind, I've moved your thread over to the Data Lake forum. I didn't originally catch the question.

Today, the best way of producing a flat file (e.g. CSV) from a Data Lake query is by using the Data Lake's Compass APIs. You can reach that through the App Switcher > ION API > "Infor ION" suite > ../datalakeapi (page 2).

You can submit your query through the Compass' jobs API (POST method), check the status of the query, and retrieve the result set. When you're submitting the query, you can request the result to be generated as CSV (text/csv) or ndjson (application/x-ndjson).

Data Lake Flows are typically used for either:

• Data ingestion into Data Lake
• Raw file retrieval out of Data Lake (batch-based integration scenarios)

For raw-object retrieval scenarios, activating a Data Lake Flow tells ION to retrieve a series of raw data objects (aka the original files) and send them to some application location like a file server. This is typically useful if you have integration scenarios where a 3rd party might not want event-driven integration publishing but rather batch-based integrations - such as all files once per night @ 10pm.

We're extending ION AnySQL to provide a drag-and-drop interface on top of Data Lake early next year so you'll start to see new modeler scenarios within Data Lake Flows very soon

Hope this helps!