Hello everyone:
We are Cloud Suite 10 hosted with the monthly updates.
I have a few questions about best practices on Streaming Pipelines. We did buy it and are learning as we go. Any links to articles.. KB's..
Is there campus training on this. I can't find it.
Initial loads take a LOT of time. We have a contract that allows 165,000 transactions per day from my understanding. FAR from optimal for Initial Loads of 1M plus record tables. Not sure about contracts but we pay xxxx/yr for that threshold and am hoping to know what it costs for more bandwidth would be.
And the throttling is like. 165K./day. then stop. At least that's what it seems like. is that true? Anyone else trying this that can compare notes.
That throttling makes loading up a database that has more than 165K records a very time-consuming process. We have some ideas on alternatives to the pipelines for the initial loads but I would be curious to here about how others are doing initial loads and what tools they are using?
Also regarding monitoring? Based on my experience to date it seems like I need to check each pipeline separately to see if we have errors and them and make necessary changes and replay the pipeline? Is there a more global (BI Like) option to show errors without going thru each pipeline.
Also, How do I use the pipelines to move SCD stuff. I know the data its there in the Lake and it can be exposed . But how do I choose to move that to Azure?What is an Exclusion? Where do I get details around Live Data Exclusions? Ive looked thru the infor Docs and really couldnt find anything.
I want to share some notes from us just getting started that may or may not help others:
a) How we create our tables: Ask Co-Pilot to convert Json data schemes extracted from Compass.. to SQL CREATE table syntax.
Than we perfect the layout in Azure SQL in the sense that the JSON contains a lot of NOT NULL fields and on Initial Load this can be an issue in that older data will not conform to current table scheme as its loaded and it will error older records out on the NOT NULLS . We change the NOT NULLS to NULL in AZURE
Thx,
Joe
