CA
fair-rose
How to have multiple crawler on the same repo ?
I currently have 5 crawlers on my repo and I would like to create 5 apify actors. Is there a way to make to use the same github repo for all of them but specifiy which file to run ?
I found out how to setup continuous integration with apify and build the crawlers in their own separate actors on apify. The issue is that, I can't specify which file to run.
Also, is it possible to build all the files on github, then only push a single javascript file to apify ?
Thank you very much
15 Replies
You can specify a folder for your actor source like this:
each folder could have separate Dockerfile, input_schema.json, etc.
Source code | Apify Documentation
Learn all about an Apify actor's code. Compare deployment options, create a custom Dockerfile, set environment variables and metamorph your actor.
fair-roseOP•3y ago
Thank yo Joefree!
:perfecto:
Hello @Vince
Also, is it possible to build all the files on github, then only push a single javascript file to apify ?No, unfortunately all the builds have to be done on the Apify platform, if you plan your actors to be used there.
fair-roseOP•3y ago
Thank you for your answer. One last question:
I tried your solution JoeFree, but it doesn't seem to work with my setup. The issue is that my all my crawler use functions utils and packages in a shared folder at the root. I think I might have to setup the build to copy the shared folder in each crawler folder, then do some magic to make it work. Is there a better solution?
Thanks again guys
@Vince It is currently in experimental testing and it is not documented yet. But there is a repository which could give you an idea how to solve it https://github.com/apify/actor-monorepo-example it uses relative dockerContext, so you need to do some changes to do
.actor/Dockerfile and .actor/apify.json you may compare it to the files from the clean template.
It currently does not support pushing code via apify push.fair-roseOP•3y ago
Great thank you so much! I’ll try that out ! 😄
@Vince We would be glad for any feedback 🙂
fair-roseOP•3y ago
Hey @Pepa J, thanks for sending me the link. It gave me the information that I needed ! I used the repo as a template, and added typescript, all my crawlers, a shared folder with packages. I'm using yarn workspace for my crawlers and shared. Everything works great, after 55 build on apify lol! I'm so happy that it works now.
If you want, I can cleanup my code and post my template (monorepo, typescript, shared functions and packages) on my github. I can make a readme as well.
:perfecto:
@Vince Wow! Thank you for your time, while dealing with this! I can imagine, it was not really straight forward task, but I am glad to hear, that you nailed it!
I think since we have not documentation about this yet, then any repository with examples would be very welcomed! Especially with TS, since our monorepo example is missing it 🙂
fair-roseOP•3y ago
Here it is ! I will post in in the chat channel as well ! 🙂 https://github.com/vincepaquette/apify-monorepo-shared-ts-example
GitHub
GitHub - vincepaquette/apify-monorepo-shared-ts-example
Contribute to vincepaquette/apify-monorepo-shared-ts-example development by creating an account on GitHub.
I was also facing some problem. I end up downloading file manually (using
RUN curl inside Dockerfile). Fortunately I am only needed 2 library files.
I have question, wonder if it's possible to have 1 deployment-key for all sub actor, instead I have to set each actor a different deployment-key, all actor under same root repository.@!!!Joefree!!! 👑 just advanced to level 8! Thanks for your contributions! 🎉
@!!!Joefree!!! 👑 You may not have same deployment key for different Actors on the platform.
@Vince Very good work, thank you! 🙂