Using Custom Object Schema

unknown

We are using Data Lake API calls to populate an external PIM.  There are a couple fields where the format needs to be transformed from Decimal to Integer.  We thought we could create a custom Object Schema for the tables/fields in question through ION Desk then Purge and reload the tables from our ERP.

 For example, we thought changing this:

     "keyno": {

      "$id": "#/properties/keyno",

      "type": "number",

      "title": "The Sequence # Schema",

      "default": 0.0

    },

to this:

     "keyno": {

      "$id": "#/properties/keyno",

      "type": "integer",

      "title": "The Sequence # Schema",

     "default": 0,

      "maximum": 9223372036854775807,

      "minimum": -9223372036854775808

    },

Newly loaded data still retained the 'number' data type, so seems we missed a step.

Is this approach a valid way to transform data as it is coming into Data Lake?

Is transforming as part of the API call a better method?

Kevin Heiman

The schema represents the data at rest and being processed on Compass request.   It does not affect any transformation of the data from the producer to at rest.  If you look in Atlas at any specifc file at rest, you would only see the format that the producer built it as.

Is this schema for a custom file or Infor delivered file.  (we would not recommend changing an Infor one)  If custom, is it via Data Loader / AnySQL or your own pipeline?

The consumer side, if you are using the File APIs, is giving you a copy of what is at rest, no data alterations in that.

Only Compass request would look at the schema and apply data formatting.