add_request with same url but different payload
do you have any example of adding a request to the requestQueue with using use_extended_unique_key because I am using the same url for every request the only change is the payload ? I have this error TypeError: Unicode-objects must be encoded before hashing with this code
for i in range(1,max_index + 1) :
# Update the page number in params_query
#json_data['indices']['products']['paging']['index'] = int(next_index)
json_data = json.dumps(json_data)
await rq.add_request(request={'url': url, 'payload': json_data, 'method': 'POST'}, use_extended_unique_key=True)
2 Replies
sunny-green•13mo ago
use_extended_unique_key is a boolean. So everything should be handled automatically.
https://docs.apify.com/sdk/python/reference/class/RequestQueue#add_request
Please provide full error trace, so we can analyze it. Could be some bug.
Also, check your payload, maybe it has some unusual chars / format ? Is there an example ?
cc @Vlada Dusekeager-peach•13mo ago
Hi, looking into the source code -> https://github.com/apify/apify-sdk-python/blob/master/src/apify/_utils.py#L477:L484 -> payload is expected to be a
bytes type, not a string (ouput of json.dumps)
You should probably use an encode method of string.
json_data = json.dumps(json_data).encode(encoding='utf-8')