[Feature Preview] Atlas Object Browser

Hey everyone,

Amongst other topics & features on our roadmap for 2020, one project we're quickly advancing is our new Atlas browser that would allow you to explore the Data Lake's file system and better understand, preview, manage, and download content you or your applications are putting in the Data Lake. Previously, interacting with files including object metadata search and single & multi-object payload streaming was limited to the Payload APIs within the ION API Gateway.

With Atlas, objects are organized by object name with an overview of indexed content associated with each object. The right-most context panel provides file previews as well as some light editor capabilities to make it easier to search for data in specific files. From here, you can easily download, purge, or mark objects as corrupt which would preserve your data objects in the Data Lake but omits it from processing with interfaces like Compass or Birst.

The goal of previewing the feature with the community is to help your teams see what's coming down the pike but also generate feedback from your use cases and ideas. Let us know what you like, what you think could be improved, and new ideas you might not see represented here. Gathering your feedback helps improve Data Lake and I can't get enough of when you're all excited for these kinds of features!

Final note - we're still doing a lot of polishing so you'll notice changes between when Atlas is delivered this year (hint: another thing to be thankful for).

Key Features

Overview & management panes organized by object name
Identify orphaned data objects in storage that no longer have a corresponding metadata model in Data Catalog
Download one or more objects directly from the Atlas for local storage
Mark objects as corrupt to prevent data processing in Compass
Preview and search files in the preview panel with configurable features to fold, wrap, and improve contents' visibility including a high contrast mode

Vision & Roadmap

Our first release of Atlas will be the foundation for a lot of new innovative features within the Data Lake to improve and introduce management of content.

Statistical dashboards and ingestion graphs
Sync validation with Infor applications to ensure Data Lake has all the data ERPs have replicated
Virtual buckets for creating logical containers of objects with shared characteristics based on data and metadata
Deeper configuration options for how to influence Compass querying dynamics and partitioning behaviors (performance)
Security management framework

Infor Data Lake Atlas

Find more posts tagged with

Infor Data Fabric

Comments

Legacy Contributor

[quote userid="16617" url="~/infor-ion/f/infor-data-lake/14467/feature-preview-atlas-object-browser"]Sync validation with Infor applications to ensure Data Lake has all the data ERPs have replicated

Hi Mike, do you have a specific time frame for when this feature will be released? This would solve a big issue were currently dealing with to ensure validation from CSI10 to a replica SQL database using the ETL Tool.

Legacy Contributor

Hey there,

We're actively working on story development and planning but we're not yet ready to provide estimates and roadmap commitments just yet

This is an acutely challenging area in terms of not only identifying data drift and temporal agreement of how many rows of data existed between systems at a particular time but also identifying when and how that delta drift transpired and identifying a narrow window of time where it's likely to have taken place.

We're taking the approach of reporting on data across systems to begin with but we plan to grow this to more useful and intuitive than simply KPIs reporting on row numbers.

Stay tuned!

Legacy Contributor

Thanks for the quick response Mike.

Do you know if there is an alternative method that we could potentially look at? Perhaps using a compass query or ION APIs? Our biggest issue is with CSI10, we don't have direct access to the database so we can't get a definite count, but we know the amount that is processed through on an initial load. Other team members using the ETL Tool when running a count query through Compass and MSSQL notice that the counts don't match. I know Variation Numbers and Delete Indicators are associated properties for each record instance in the Data Lake, so more than one of the same instance record may exist in the Data Lake. And the Upsert Staging in the ETL Tool only uploads the record instance with max variation.

Could you perhaps assist me in this or point me to the right direction, just need to find a way to verify and validate?

Thanks,

Matt Castaneda

Legacy Contributor

Hey there Matt -

You may want to touch base with support & CSI product management to determine whether retrieving statistics from the CSI application through some Mongoose service or API might be possible. Some applications do provide methods to run SQL or generate statistics on a given table within their API layers.

With respect to the ETL tool, you might end up with a scenario where records that were deleted in CSI still persist in your target data warehouse but the delete flag was set to positive. You still retain the row in the target system. This might result in you having "more" rows than Compass reports, for example, because you're returning back a row count of all active rows in addition to deleted ones. You could orchestrate a process that reads that delete indicator and drops the row from the table to reflect CSI.

One quick thing I would take a look at is running a query against the data warehouse (target system) and filter out all rows where the delete indicator is true/yes/1/whatever your delete indication value is.

E.G. SELECT COUNT(*) FROM Products WHERE deleted = 0

Obviously, you would replace your table name, the column used to flag deletion and the actual values therein to reflect CSI. I'd wonder if the results of this query line up with what you see in Compass.

Copyright © 2025 Infor. All rights reserved.