Data model between tables from Live connection and Data Lake

Sethupathy T

we have WMS live connection in SCE_WM* spaces and there are few reports that require blending of data between tables in data lake and Live WMS. the option available right now is using LookupRow function but that is too slow even if the report has 50 records.

Alternatively, I have used both source tables from data lake, but the data publish from WMS to data lake sometime is delayed for 8 hours or more. that is not useful. we expect the space is refreshed every 2 hours with latest data.

It will be good if we get an option to data model between Live connection and data lake table.

Matthew Nye

just a quick question, why is that data taking 8 hours to publish to the data lake? The easiest solution here is to increase the frequency of the data lake pushes. If thats not available you have some other options.

The first and most reliable would be to see if you are licensed Streaming Pipelines. If so, you can bypass the data lake all together and publish to data fabric. From there you should be able to connect to it with live data access.

The kicker is that Streaming Pipelines is an additional cost. So if your organization hasnt already purchased it, it will be an additional cost.

Alternatively you could also set up exports from WMS to an SFTP and consume the data from there in Birst, though Id question if youre going through that trouble why not just post to Data Lake and have one common source.

My guess is theres more to this use case around data load and performance, curious to hear more details.

hth,
Matt