Hey all,
One of the areas where we're looking to improve the Data Lake is in providing performant access to data. It was great to release the initial Data Lake Compass platform including the browser-based query editor as well as the APIS back in September 2019. While they gave you access, it was more like the speed of smell rather than the speed of sound.
As of this past weekend, the 12.0.42 update has gone GA for all customers and you should see some noticeably awesome improvements to performance. Not all queries and datasets are alike so your mileage may vary but for some fairly common query profiles like the ones outline below, we're seeing queries run 200-500% faster than Data Lake up to v12.0.41 (September to December).
- SELECT * FROM MyTable
- SELECT count(*) FROM MyTable
- SELECT FROM MyTable WHERE lastModified >= '2020-01-01...'
You'll see faster queries across the board. The bigger & more complex the query, the advantage will scale slightly differently, of course.
It's good to remember that data lakes aren't purpose-built data stores with performant access in the same way as data warehouses fit the bill. They're general purpose big-data storage & query platforms that can be used to feed any number of data-driven use cases. That said, I was quite tired to see a simple row count query run for 45-90 seconds as much as you were 
The better news is that this was the first in several iterative and really never-ending performance & data tuning optimizations we'll continue to make in the cloud. Over time, you will gradually see your same queries perform faster across updates. In this case, we've improved our caching and reduced the number of calls to our data catalog for metadata stashes. We'll continue to keep posting and letting you know when to expect new changes and updates like this.
In the meantime, let us know how things are going!
