CA
solid-orange
Integrating webhook with Fivetran?
Hi! I am a one-man data team at a startup, so we are leveraging Fivetran to do a lot of our heavy lifting for social accounts. One thing they are missing is TikTok social data, since no API.
I like the Apify TikTok social scraper you have here, and would like to store it via webhook in my BigQuery instance via the webhook connector in Fivetran.
Does anybody have any experience to know if this use-case is compatible with what's currently on offer? I'm not seeing data flow in Fivetran, but I could be making a mistake somewhere.
If this isn't the right route, is there a way of forcing a format on Google Drive storage? I ask because when I was going to do Google Drive > BigQuery, there was no defined format (.csv, etc), which required me to download the file and add it myself. I'm hoping to take an automated approach to this.
Thank you.
1 Reply
dependent-tan•3y ago
Hey there! I am not familiar with Fivetran. But it should be possible one way or another. The "worst" case scenario is that you would need to add a second actor, which will get the TikTok dataset, do some post processing, and either send the data via some Fivetran client, or maybe to google drive. Then you'll need to figure out which format you need for the google drive storage (or Fivetran). Once you know it - it would be possible to give a more precise answer, but well - you could send pretty much anything - send json, xls, csv, create some other format, etc