Null Values in Data Fabric Compass

Hello All,

Pulled a few ICSW records from CSD tenant in CSV format and converted the CSV file to JSON format.

Then compressed the JSON file to .zlib format using deflate compression technique through Python script.

Uploaded the .Zlib file to Data Fabric using Data Fabric Ingestion API (DATAFABRIC/ingestion/v1).

Data got uploaded successfully in Data Fabric and I can also see the entry in Atlas (below is the screenshot), but I am getting no ICSW records when querying in Compass.

Getting NULL values when running this query.

SELECT

"icsw"."cono",

"icsw"."prod"

FROM

infor.AllVariations('icsw')

Please help in correcting the mistakes and how to improve the entire approach.

Thanks,

Find more posts tagged with

Infor Data Fabric

Accepted answers

Bojan Rafajloski

You are getting that error because you've uploaded a CSV file instead of a NDJSON, which is what Compass expects to find in Data Lake based on the Data Catalog object schema for ICSW.

In case you already have samples on another tenant, you can download the data object via Atlas. That will ensure you are working with raw example as sent from the source application, in this case CSD. Otherwise you should recreate the data in a ndjson format manually.

The second screenshot you show is a CSV format which I assume you exported from the Compass UI data grid. The Compass UI exports the data in a CSV format and not in the original format.

Here's how to download from Atlas a raw data object:

After you have the file locally and you do your edits, just upload the file via Atlas Upload Widget (no need to compress it yourself).

User Guide for Atlas Upload Widget: docs.infor.com/.../default.html

Kevin Heiman

By default, Compass only returns Max Variation ie latest non deleted record. You can use options in your query to see other sets including deleted and or multiple variations.

The documentation is worth a look through as it has examples. https://docs.infor.com/inforos/2024.x/en-us/useradminlib_cloud/default.html?helpcontent=datafabrug/gve1631199565990.html

select * from infor.allvariations(&#39;Products&#39;)

All comments

Bojan Rafajloski

1. There should not be need to export ICSW data to CSV and convert it back to NDJSON. CSD ICSW tables are already published in NDJSON format in Data Lake, so you can either download a data objects sample from Atlas or query the data via Compass API and ask for response in ndjson format

2. For uploading local files to Data Lake you can use the Atlas Upload Widget. Not sure if your use case is to upload local data, but if your are doing that the widget helps avoid the compression step so you can just select local files. Otherwise if you are integrating a client application via API Gateway, you are doing the correct thing. Please note that usually you should not include your own generated data in the tables the ERPs are using to replicate their data to Data Lake.

3. Can you show us the uploaded ndjson payload including the field values for cono and prod?

raghav-verma

Hi Bojan,
Thanks for helping.

At present ICSW CSD table data is not in sync with Data Fabric so there is no data to query in Data Fabric. Right now I just need to push some ICSW data to Data Fabric for validation.

I downloaded the JSON data from Atlas and it does look weird, please see below.

Today, I pulled the ICSW data again from CSD and converted it to .Zlib format using python and again uploaded through Ingestion API. Today the data in Atlas looks better, please see below.

In the file data, Cono is 1 and Prod is "000009999"

But I am getting below error today while querying the ICSW data in Compass.

Data Storage ERROR 513

Error message: Error parsing raw data line of type JSON:

Should I be using a different approach for pulling ICSW data from CSD and uploading through Ingestion API ?

Bojan Rafajloski

You are getting that error because you've uploaded a CSV file instead of a NDJSON, which is what Compass expects to find in Data Lake based on the Data Catalog object schema for ICSW.

The second screenshot you show is a CSV format which I assume you exported from the Compass UI data grid. The Compass UI exports the data in a CSV format and not in the original format.

Here's how to download from Atlas a raw data object:

After you have the file locally and you do your edits, just upload the file via Atlas Upload Widget (no need to compress it yourself).

User Guide for Atlas Upload Widget: docs.infor.com/.../default.html

raghav-verma

Thank you. I was able to pull some data from another tenant and upload it to the target tenant.

Post uploading, I was able to query the ICSW record in Compass.

For testing purpose - I modified the VariationID and ICSWRowID and baseprice in JSON file and again uploaded it to Data Fabric and now I had two entries in Atlas.

I was expecting two ICSW records with different baseprice but I got only one record with the latest base price.

Is n't Data Fabric supposed to record and show changes being done in the tables by users in CSD for audit purposes?

For example - Initially the base price was 100 and that record logged in Data Fabric. User came and changed the base price to 200 - Should n't we have one more ICSW record in Atlas showing baseprice as 200?

Audit flag is ON for icsw in CSD.

Copyright © 2025 Infor. All rights reserved.