Hello,
I have created a web application that uses compass API calls. When querying the datalake directly the queries do indeed take about 10-12 seconds each, which has caused some latency issues in getting pages to load on my web app. I am using python and flask. I was wondering if anyone knows of a way to speed up the API calls. Specifically, the latency is caused by my code looping to check for the result url via get /jobs/{queryId}/result once a query.
I am submitting the sql query to the API via the following line of code:
response = requests.post(url, headers=aheaders, data=sql_query), (this runs fine)
Here is the most complex SQL query in my code:
SELECT LN_tisfc001.pdno, LN_tisfc001.qrdr, LN_tisfc001.cwar, LN_tisfc001.mitm, LN_tcibd001.dsca, LN_tcibd001.cdf_draw, LN_ticst001.sitm,(SELECT LN_tcibd001.dsca FROM LN_tcibd001 WHERE LN_tcibd001.item = ln_TICST001.SITM) AS material_desc FROM LN_tisfc001 INNER JOIN LN_tcibd001 ON LN_tisfc001.mitm = LN_tcibd001.item LEFT JOIN LN_ticst001 ON LN_tisfc001.pdno = LN_ticst001.pdno AND LN_ticst001.cwar = 'ERAW' WHERE LN_tisfc001.pdno = '{}'".format(production_order)
Below is the loop that is causing the latency:
while True:
# Send GET requests to both URLs
status_response = requests.get(status_url, params={"offset": 0, "limit": 100}, headers=aheaders)
result_response = requests.get(result_url, params={"offset": 0, "limit": 100}, headers=aheaders)
# Check if both responses are HTTP status code 200
if status_response.status_code == 201 and result_response.status_code == 200:
print('Both status and result response codes are in 200, breaking out of loop')
break
else:
print("Result responses not ready")
time.sleep(.5).
I have 4 queries that follow this procedure and it adds up to a wait time of about a minute. Looking to cut this down siginifcantly.
