CA
xenial-black
Customize which values return and order of table columns for each run
I'd like to customize the order of the columns I get from each run of the YouTube scraper.
So JSON return would be:
"title":
"text":
"date":
"viewcount":
"duration":
"url":
And let's just say that's all I want. Just that, and that's all I want.
I can't seem to find a really crystal clear, simple example of how to customize my scrape to get specifically a table of records with only these values in this specific order each time.
Can anyone point me in the right direction?
Thanks!
4 Replies
uninterested-red•3y ago
You would need to post process it with e.g. https://apify.com/lukaskrivka/dedup-datasets
Why does order of JSON fields matter?
Apify
Merge, Dedup & Transform Datasets · Apify
The ultimate dataset processor. Extremely fast merging, deduplications & transformations all in a single run.
xenial-blackOP•3y ago
Hi Lukas thanks much for getting back to me. Is there an example of this preDedupTransformFunction method that I can model my effort from?
The reason that column order matters is that let's say I scrape 10 competitors YouTube videos and want to upload all the records into a database. Well... it would save me time, and be easier to eventually automate, if I could return the results from each run in the exact column order of my target database table.
Alternatively, I'm doing 10 runs, downloading each to CSV, uploading to a staging table, reordering the data and eliminating whatever fields are unnecessary, and then uploading to the final table.
I'm guessing someone in the community has figured out how to do this much more elegantly.
uninterested-red•3y ago
If you are downloading CSV, you can adjust the order with
fields dataset parameterxenial-blackOP•3y ago
Thx Lukas... Is there anywhere that I can find an example of this being used in action?
Ah, I think I see now. Thx 🙂