Crawlee & Apify•16mo ago

Not sure if this is a feature request,

Not sure if this is a feature request, but it's an idea I implemented and there doesn't seem to be a better channel for general chatting. When using Scrapy, there's this concept of Items. I was able to extend Items so that they contain declarative specification of Apify input types (see code):

class ExchangeRate(Item):
    code = Field()
    rate = Field(apify_format="number")

class ExchangeRate(Item):
    code = Field()
    rate = Field(apify_format="number")

Then I go through these classes and generate output schemas for Apify datasets automatically: - CLI - schema generation Perhaps this is too specific to my project, perhaps this can inspire someone to simplify their workflow.

13 Replies

correct-apricot•16mo ago

cc @Vlada Dusek

flat-fuchsia•16mo ago

Hi, thanks Honza for the input, we were already discussing this in DMs. Unfortunately, currently, we don't have the capacity to work on adding new features in our Scrapy integration, at least in this quarter.

flat-fuchsia•16mo ago

Yup, thanks for all the work done nevertheless!

correct-apricot•16mo ago

I was wondering, is there any way to get user's input with scrapy cuz last time I tried it didn't end well

flat-fuchsia•16mo ago

The Scrapy template works with actor inputs, if you mean those E.g. the proxy settings are there, but anything can be there

flat-fuchsia•16mo ago

@Kirada , sure, check this part of the actor template - https://github.com/apify/actor-templates/blob/dc5e68805dcf630f35d112a7e113e4f388bbf30a/templates/python-scrapy/src/main.py#L79:L82

GitHub

actor-templates/templates/python-scrapy/src/main.py at dc5e68805dcf...

This project is the :house: home of Apify actor template projects to help users quickly get started. - apify/actor-templates

correct-apricot•16mo ago

I ran the template and it goes on an infinite run I think it's bugged

correct-apricot•16mo ago

correct-apricot•16mo ago

it's been running for a few minutes even tho it finished exact same issues I had before

flat-fuchsia•16mo ago

@Kirada Hi, are you using the newest version of Apify SDK (1.5.3) and the newest version of our Scrapy template (https://github.com/apify/actor-templates/tree/master/templates/python-scrapy)? If not, please do an upgrade of your Actor. I'm aware of this infinite run bug, however, I believe it should be fixed now!

correct-apricot•16mo ago

I cloned the repo and used the template from there

correct-apricot•16mo ago

(https://github.com/apify/actor-templates/tree/master/templates/python-scrapy

GitHub

actor-templates/templates/python-scrapy at master · apify/actor-tem...

This project is the :house: home of Apify actor template projects to help users quickly get started. - apify/actor-templates

correct-apricot•16mo ago

I cloned this

Gaming

Programming

Not sure if this is a feature request,

Did you find this page helpful?