Twenty•6d ago

Workflow triggered but stays as "Not started"

I set a workflow to be triggered every x minutes. everything worked very well but all of a sudden the workflows were not executed although my worker is operational and I don't see any issues with the worker status. Any idea why?

23 Replies

Prastoin•6d ago

Hello Could you please review your admin panel and double check that your crons jobs have been correctly started ? You can also try hitting yarn command:prod cron:register:all in your worker

RonRonRonOP•6d ago

The cron looks fine is it the right metric to look for?

RonRonRonOP•6d ago

I am going to run register:all command. Also this is what I see in the twenty-server log when the job wants to start:

PromiseMemoizer Event A WorkspaceDataSource for workspace c2c1bcefa005431990d4f8152b996c29 is being cleared Actual pool closure managed by PgPoolSharedService Not calling dataSourcedestroy
Computing new Datasource for cacheKey c2c1bcefa005431990d4f8152b996c2938 out of 0
32mNest 1   39m11222025 84047 PM 32m    LOG39m 3853mNestboxAiAgentController 39m32mNestbox AI Agent callback payload 
32mNest 1   39m11222025 85336 PM

PromiseMemoizer Event A WorkspaceDataSource for workspace c2c1bcefa005431990d4f8152b996c29 is being cleared Actual pool closure managed by PgPoolSharedService Not calling dataSourcedestroy
Computing new Datasource for cacheKey c2c1bcefa005431990d4f8152b996c2938 out of 0
32mNest 1   39m11222025 84047 PM 32m    LOG39m 3853mNestboxAiAgentController 39m32mNestbox AI Agent callback payload 
32mNest 1   39m11222025 85336 PM

The job processor itself never began execution - because it blocked BEFORE the first log line . and the only place that can happen in Nest is: Workspace datasource acquisition / memoizer PromiseMemoizer Event: A WorkspaceDataSource ... is being cleared Computing new Datasource ... Right when the worker tries to get the datasource, the memoizer is wiping it and rebuilding it.

Prastoin•6d ago

Double checked the cloud monitoring cron workflow triggers work as expected What twenty version are you using ? Please share both installed twenty version and workspace version in database core.workspace.version

RonRonRonOP•6d ago

core.workspace.version twenty version 1.10, can you educate me what is "cloud monitoring cron workflow triggers work as expected"? When I run the workflow manually it works, the problem is only when the triggers are the "Record updated" or "Schedule", the job is registered but not started (keeps on "Not started" status) I also ran cron:register:all - still the same.

Prastoin•6d ago

core.workspace.version twenty version 1.10, can you educate me what is "cloud monitoring cron workflow triggers work as expected"?

I can attest that cron trigger worklfow jobs works successfully in latest twenty version What twenty version are running 1.10 ? Please check TAG= value in your .env Lets try to get you to the latest twenty version before proceeding to deeper investigations

RonRonRonOP•6d ago

my TAG that I pulled from twenty is "1.8.13" which is latest imo... Still trying to understand why "manually" starting a workflow works but "DB triggered" not starting the job... thank you for your time appreciate it

Prastoin•6d ago

my TAG that I pulled from twenty is "1.8.13" which is latest imo...

Not sure to understand here, is your TAG value to latest or 1.8.13 ?

Still trying to understand why "manually" starting a workflow works but "DB triggered" not starting the job... thank you for your time appreciate it

It's not the same trigger funnel logic, no worries !

RonRonRonOP•6d ago

You are right, it's not the latest it's 1.8.13 . I thought it's "pretty latest" tag 🙂 because the problem started "out of the blue" I think it's related to some db corrupted issue...

Prastoin•6d ago

I think it's related to some db corrupted issue...

Yep I think so too Could you please run echo $APP_VERSION in your twenty-server container please ? I'm trying to understand how you achieved to have a workspace version 1.10 with a twenty instance 1.8 ( seeing config ) Have you been manually editing your workspace version in database ? are you sure your workspace version in database is 1.10 it follow complete semver x.y.z ?

RonRonRonOP•6d ago

So I pulled tag 1.8.13, and I haven't set the "APP_VERSION" when I deployed it to my containers.

docker ps
CONTAINER ID   IMAGE                                                                   COMMAND                  CREATED       STATUS                 PORTS                                                                                             NAMES
d29a63053d08   us-central1-docker.pkg.dev/nestboxai/twenty:1.8.13c   "/app/entrypoint.sh …"   2 weeks ago   Up 2 weeks                                                                                                               twenty-worker
e8f6d3a4141e   caddy:2.8                                                               "caddy run --config …"   2 weeks ago   Up 2 weeks             0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:443->443/tcp, [::]:443->443/tcp, 443/udp, 2019/tcp   twenty-caddy
7730e6a93608   us-central1-docker.pkg.dev/nestboxai/twenty:1.8.13c   "/app/entrypoint.sh …"   2 weeks ago   Up 2 weeks (healthy)   0.0.0.0:3000->3000/tcp, [::]:3000->3000/tcp                                                       twenty-server
ec043bf35ac5   redis:7                                                                 "docker-entrypoint.s…"   2 weeks ago   Up 2 weeks (healthy)   6379/tcp                                                                                          twenty-redis
root@twenty-crm-instance-131:/home/roy# echo $APP_VERSION
<empty>

docker ps
CONTAINER ID   IMAGE                                                                   COMMAND                  CREATED       STATUS                 PORTS                                                                                             NAMES
d29a63053d08   us-central1-docker.pkg.dev/nestboxai/twenty:1.8.13c   "/app/entrypoint.sh …"   2 weeks ago   Up 2 weeks                                                                                                               twenty-worker
e8f6d3a4141e   caddy:2.8                                                               "caddy run --config …"   2 weeks ago   Up 2 weeks             0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:443->443/tcp, [::]:443->443/tcp, 443/udp, 2019/tcp   twenty-caddy
7730e6a93608   us-central1-docker.pkg.dev/nestboxai/twenty:1.8.13c   "/app/entrypoint.sh …"   2 weeks ago   Up 2 weeks (healthy)   0.0.0.0:3000->3000/tcp, [::]:3000->3000/tcp                                                       twenty-server
ec043bf35ac5   redis:7                                                                 "docker-entrypoint.s…"   2 weeks ago   Up 2 weeks (healthy)   6379/tcp                                                                                          twenty-redis
root@twenty-crm-instance-131:/home/roy# echo $APP_VERSION
<empty>

Prastoin•6d ago

Are you building your own twenty image ?

RonRonRonOP•6d ago

the twenty image is the tag I pulled

Prastoin•6d ago

Do you mean git tag from the repo or are you consuming official built twenty docker image from docker hub ?

RonRonRonOP•6d ago

git tag

Prastoin•6d ago

Ok you're building your own twenty image

RonRonRonOP•6d ago

yep

Prastoin•6d ago

Are you adding any layer over the fork or any custom infra ? Just double checking your motiviation as it might not be required in the end As building your own twenty image requires a bit more configuration Please refer to https://discord.com/channels/1130383047699738754/1372831640920789032/1372833279610196039

RonRonRonOP•6d ago

I automated some configuration (like workspace init) but other than this it's the tag, the workspace worked for a month now, and now had a hiccup. with workflows not starting ("Not started") I thought it's a general issue that I am not sure I am capable to backtrack because it's in the workflow level, wondering if other people saw the same issue

Prastoin•6d ago

I automated some configuration (like workspace init) but other than this it's the tag,

If it's for seeding purpose I recommend either using the new twenty-apps ( still in dev though ) or having external seeding script over the api )

wondering if other people saw the same issue

Not that I'm aware of One of the tradeof of building your own twenty image is the reproducibility of the issues as it's pretty hard to tell if the regression comes from the soft of contextual configuration and migrations Depending on the complexity of your instance, the most worth it might be to restart from a latest twenty instance from scratch by either exporting everything as cvs + pg dump and pg restore operations

RonRonRonOP•5d ago

Just solved it, the "Failed" jobs were too high so the new processes were not triggered. Will be nice to have an indication in the Admin page. So the "getRemainingRunsToEnqueueCountFromCache()" was 0 because I had too many failed processes I found the constant that can be configured (no env parameter for it): export const WORKFLOW_RUN_QUEUE_THROTTLE_LIMIT = 100;

martmull•5d ago

Indeed we may make those configurable with env variables

martmull•5d ago

https://github.com/twentyhq/twenty/issues/16011

GitHub

Add environment variable to configure workflow throttling mecanism ...

Some self hosted users want to have control on the WORKFLOW_RUN_QUEUE_THROTTLE_LIMIT and other variables related to workflow throttling Indeed it depends on the worker configuration Lets create env...

Gaming

Programming

Workflow triggered but stays as "Not started"

Did you find this page helpful?