Deflating doesn't work in ZappySys

I use a popular add-on SSIS PowerPack from ZappySys to simplify connecting to, first datalake APIs, and now Data Fabric APIs to download tables. I have no trouble using identity, but the deflate does not work. I get errors like

"Error: System.Exception: MalformedDocumentException: Invalid document format detected - Make sure document data is in the correct format. Unexpected character encountered while parsing value: x. Path '', line 0, position 0.. RowNumber=1"

I have done extensive troubleshooting with the vendor. I am learning that Microsoft did a bad job with its deflate library. There's no way to find out exactly what's happening on the Infor side, so the vendor cannot reproduce or troubleshoot further. I am stuck using identity. There should be no issue using GZip with this tool, which I understand is more reliable with a small price for performance due to its additional content. If deflate can actually work, I would love to see actual code samples that connect to the Data Fabric API to get results, not the Microsoft samples. I saw another post that said that .NET 6 will fix the issues. Surely Infor must have tested this scenario. Better yet, please add support for GZip! Thanks very much.

Find more posts tagged with

ION API Gateway

Accepted answers

All comments

Kevin Heiman

You could verify in Python. Here is a excerpt of code that should be most of what you need.

=====================================================================

import zlib, sys

filename_out = "someDirectory/_location_of_your_file_"

# Extra section to read DEFLATE file back in and verify it.

# Not normally needed, unless needed for receiving DEFLATED files

print('Reading file back in to verify: ' + filename_out)

with open(filename_out, mode="rb") as verify_in:

decompress = zlib.decompressobj(

# -zlib.MAX_WBITS # see above

)

verify_file_data = verify_in.read()

# compressed_file_data = zlib.decompress(file_data)

decompressed_verify_file_data = decompress.decompress(verify_file_data)

print(f" Compressed size: {sys.getsizeof(verify_file_data):,}")

print(f"Decompressed size: {sys.getsizeof(decompressed_verify_file_data):,}")

# wrap up

Legacy Contributor

Hi, Kevin,

Interesting - Thanks for that. However the problem is occurring during the processing the output stream of the API, not the reading of a file. I also tried running the curl generated by the swagger, but it had several errors I couldn't resolve.

Kevin Heiman

I am unsure I understand the problem. I interpret your initial message as there is a problem in the ZappySys being able to decompress due to MS ZLIB problems.

The Python Decompress can verify that the DEFLATED file you have downloaded is able to be decompressed. It would verify that the API is returning a valid file for you.

If you can provide more clarity on what fails you, we can reproduce and try to help. Or you can contact me directly and we can update this thread.

If it helps, the code below contains the settings to DEFLATE for use in ingesting API. these are the flag set on ZLIB

# Compress using a obj w/ settings

# strings of file input and output

ffilename_out = "someDirectory/_location_of_your_file_"

filename_out = filename_out = "someDirectory/_location_of_your_file_" + ".zlib"

# open files in binary with buffered read

print('-- reading Input file : ' + filename_in)

with open(filename_in, mode="rb") as file_in,

open(filename_out, mode="wb") as file_out:

# File read

file_data = file_in.read()

# Compress

# Method 1 DEFLATE using compress

# compressed_file_data = zlib.compress(file_data, zlib.Z_BEST_COMPRESSION)

# Method 2 DEFLATE using compressobj

compress = zlib.compressobj(

9, # level: 0-9

zlib.DEFLATED, # method: must be DEFLATED

# -zlib.MAX_WBITS, # window size in bits:

# -15..-8: negate, suppress header

# 8..15: normal

# 16..30: subtract 16, gzip header

# zlib.DEF_MEM_LEVEL, # mem level: 1..8/9

strategy=0 # strategy:

# 0 = Z_DEFAULT_STRATEGY

# 1 = Z_FILTERED

# 2 = Z_HUFFMAN_ONLY

# 3 = Z_RLE

# 4 = Z_FIXED

)

compressed_file_data = compress.compress(file_data)

compressed_file_data += compress.flush(zlib.Z_FINISH)

Bojan Rafajloski

I assume you are using GET? /dataobjects?/byfilter to retrieve multiple deflated data objects. Is there any chance that you are inflating the whole message instead just the deflated data? The multipart message headers are not deflated in the response.

Legacy Contributor

Hi, Kevin, I checked about that, and the ZappySys tool is not able to output the incoming stream as a file. It passes the stream on to its other processing down the dataflow. Further, the data fabric get results API pages its output. You set the rows per batch and the row offset, and ZappySys loops through that internally, incrementing the offset after each batch with a new request. That's why I didn't think I could use your Python code. The ION swagger page lets you set the deflate header, but they inflate the result before it shows up on the page. There's probably a way to capture the result another way, but I am not aware of one.

Legacy Contributor

I see what you mean, but I have no visibility into the internals of the ZappySys tool. I know it works with Gzip because I've seen that demonstrated. And it works fine with identity, so I know that it is handling the request batches fine.

I am using the new Data Fabric APIs, which if you haven't seen them yet, is a set of three APIs: One to submit the query job, one to check the status for knowing when it has finished, and one to get the results, which has this format:

https://mingle-ionapi.inforcloudsuite.com/tenantname/DATAFABRIC/compass/v2/jobs/{{User::queryId}}/result/?offset={{User::offset}}&limit={{User::limit}}

QueryID, offset, and limit are of course variable placeholders here.

Raghavender Hariharan

@ssaari : We have working concepts being implemented in JAVA and Angular to perform deflate using zlib an end to end solution from uploading a File into Datalake. We also built few utilities which will be released into Infor marketplace soon. For Angular we are releasing a Widget for which @bojan-rafajloski and his team is working on to release as part of Standard Homepage Widget. So you will have the Widget available to upload the file into Datalake.

JAVA WORKING SAMPLE:

public void compress(Logger log, File raw, File compressed) throws Throwable {
    byte[] buf =new byte[2048];
    if (!compressed.exists()){
        compressed.mkdir();
    }
    FileInputStream fileInputStream = new FileInputStream(raw);
    FileOutputStream fileOutputStream = new FileOutputStream(compressed+File.separator+raw.getName());
    DeflaterOutputStream deflaterOutputStream = new DeflaterOutputStream(fileOutputStream);
    int data;
    try {
        while ((data = fileInputStream.read(buf)) != -1) {
            deflaterOutputStream.write(buf, 0, data);
        }
        log.info("************* File is compressed successfully ****************");
    } catch (Exception ex){
        throw new Exception("File Compression Unsuccessful: " + ex);
    } finally {
        fileInputStream.close();
        deflaterOutputStream.close()
}
}

Legacy Contributor

That sounds great. However I am only extracting from data lake. Thanks.

Copyright © 2025 Infor. All rights reserved.