Hi,
What is the best way to transfer large volume of M3 data to Mongoose?
I've heard it is M3 -> DataLake -> Mongoose.
Is it correct? If so, what triggers dataflow from DataLake to Mongoose?
Any tips are appreciated.
Regards,
Kirill
One option would be to not move the data , but rather use the APIs to pull the data , typically using SQL through Compass APIs, or optionally raw data and object through the File APIs.
my concern is speed
it took me 370 seconds to pull first 10 records from FPLEDG table in Compass Query
I think if I have all the date in Mongoose available it will be way faster
Hi Kirill,
That is correct.
As for the dataflow, assuming you already have the prerequisite setup in datalake - you can make use of the Datalake IDO Wizard in Mongoose (Runtime builder > External Data Source > Datalake IDO Wizard).
Exploring this just a little further.
Do you see that duration consistently? approx. how many FPLEDG files / rows are you adding to data lake hourly?
How frequently are you accessing, as the delta reformat time is a component of overall time. i.e new data since last SQL has to be processed into the reformatted data, then surfaces in SQL result.
What is the impact if the query does take 300+ seconds? is there a consumer / user waiting?
Thanks
Hi Kevin,
I don't have details yet. I just have an vague concept of a report I want to build.
Report consumer can wait, it will not be a problem.
Regards
Thanks JM!
I run DataLake IDO Wizard for DataLake table MMSVAH.
It created IDO in Mongoose. It has properties = DataLake table fields.
However there is no table in IDO. So is this IDO like a view of DataLake data?
I created a form with this IDO as data source.
When I open form and click of "Filter in place" I get error message - method GetIONAPIData not found.
What I want to do is to have data automatically flow from M3 (or datalake) to my Mongoose table.
The wizard version you have currently doesn't have that button. Setup a meeting with me and I can walk you through the process.
Hi Yao,
Are you available for a quick teams call now?
