Scrapy does not save images in Django during the crawl
Project ID: 499333f9-ca0b-4491-b644-2c6eeb389b5c
Actually, I have a Scrappy implementation on my Django app to get some job post data across the internet. I was testing the crawling on production but noticed the spider was not saving the images in the volume. I tried to upload an image via admin manually and I see all it works fine. Also, I tried locally and all works fine there.
So, I have some questions:
1. Is the volume available during the deployment process?
2. Can Scrappy download images from the service during the deploy?
scrapper/pipeline.py
Notice it's calling object_field.save(...)
(in this case is an ImageField
from the model) to save the images using the FileSystemStorage
from Django.
settings.py
nixpack.toml
Please notice in the attachment images what happened in each caseSolution:Jump to solution
1. that is not the same path you set in
MEDIA_ROOT
the correct mount point would be /app/media
2. if the crawl script downloads media to the volume, you would need to run the crawl script during deployment and not during build`...11 Replies
Project ID:
499333f9-ca0b-4491-b644-2c6eeb389b5c
Is the volume available during the deployment process?do you mean to ask "Is the volume available during the build stage?" ?
Yeah
what is the mount point of your volume
also, no the volume is not available during build
if it was, there would be a
/data
mount since that is where i mounted my volume toSolution
1. that is not the same path you set in
MEDIA_ROOT
the correct mount point would be /app/media
2. if the crawl script downloads media to the volume, you would need to run the crawl script during deployment and not during build`ok, I will try these changes
It works, thanks
awesome!