PG Boss Concurrent Jobs
Hi everyone 👋
I’m using Pg-Boss jobs in Wasp (0.16.x).
Right now only one contentGenerationJob runs at a time, even when multiple users queue drafts.
I’d like:
- A global concurrency of 3 jobs per worker process.
- A per-user limit of 1 active job (token-bucket in the worker).
Docs say to set options through PG_BOSS_NEW_OPTIONS, but I keep tripping dotenv / JSON parsing.
Could somebody confirm the correct .env.server lines and any extra code needed?
This is what I think should work in .env.server:
After restarting wasp start I expect to see pg-boss: subscriptions online | teamConcurrency=3 in the logs.
But I get:
Is there a better pattern recommended by the Wasp team for per-user fairness + global parallelism?
Any sample snippet / repo link would be super helpful.
Thanks! 🙏
22 Replies
Hi @InterestEmpire. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:
----------------------
You're on the right track with using the
PG_BOSS_NEW_OPTIONS environment variable to configure global concurrency for pg-boss in Wasp. However, the error you’re seeing (Environment variable PG_BOSS_NEW_OPTIONS was not parsable by JSON.parse()!) is likely due to how the JSON string is formatted in your .env.server file.
Correct .env.server Syntax
When using .env.server, you should not wrap the JSON value in single quotes. The value should be a valid JSON string, and if you need to include quotes inside the value, you should escape them. For example:
If you need to include more options (like the connection string), you must include them as well, since setting PG_BOSS_NEW_OPTIONS overwrites all Wasp defaults, including the database connection info. For example:
Do not use single quotes around the JSON value in .env.server—this is a common source of JSON parsing errors in Node.js environments that use dotenv, as dotenv does not strip the quotes and passes them directly to the application, causing JSON.parse() to fail see docs.
Per-User Concurrency
Wasp and pg-boss do not provide a built-in, declarative way to enforce per-user concurrency limits (like a token-bucket per user) out of the box. The knowledge sources do not mention a recommended pattern for this in Wasp. You would need to implement this logic yourself, likely by:
- Checking in your job handler if the user already has an active job (e.g., by querying the pgboss.job table for jobs with the same user ID and a status of "active" or "created").
- If a job is already running for that user, you could skip or reschedule the new job.
There is no sample snippet or official repo link for this pattern in the current Wasp documentation or issues.
Summary
- Use unquoted, valid JSON for PG_BOSS_NEW_OPTIONS in .env.server.
- Include all required pg-boss options, including the connection string, if you override defaults.
- Per-user concurrency must be implemented manually in your job logic; Wasp does not provide a built-in pattern for this yet.
Example .env.server:
For more details, see the Wasp docs on pg-boss considerations.
If you need help with the manual per-user concurrency logic, let me know!
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------
Sorry @InterestEmpire, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist:
Perhaps try again in 15mins?That doesn't work:
wasp start
🐝 --- Starting compilation and setup phase. Hold tight... ------------------------
🐝 --- Compiling wasp project... --------------------------------------------------
wasp-bin: user error (Failed to parse dot env file: ../app/.env.server:62:22:
|
62 | PG_BOSS_NEW_OPTIONS={"teamConcurrency":3}
| ^
unexpected '"'
expecting "$(", "${", '$', end of input, end of line, escaped character, or unescaped character
)
@kapa.ai
Let me check the knowledge base for you again. One moment! :books:
Sorry @InterestEmpire, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist:
Perhaps try again in 15mins?
@kapa.ai
That doesn't work:
wasp start
🐝 --- Starting compilation and setup phase. Hold tight... ------------------------
🐝 --- Compiling wasp project... --------------------------------------------------
wasp-bin: user error (Failed to parse dot env file: ../app/.env.server:62:22:
|
62 | PG_BOSS_NEW_OPTIONS={"teamConcurrency":3}
| ^
unexpected '"'
expecting "$(", "${", '$', end of input, end of line, escaped character, or unescaped character
)
Let me check the knowledge base for you again. One moment! :books:
----------------------
Thank you for sharing the error message. According to the Wasp documentation, environment variables in
The knowledge sources do not contain enough information to confirm that setting
.env.server should be defined as simple key-value pairs, and there is no explicit mention of supporting JSON objects as values in the .env.server file. The docs do mention setting PG_BOSS_NEW_OPTIONS for Heroku, but the example is given as a stringified JSON, which may not be directly compatible with Wasp's dotenv parser see docs.
The error you are seeing:
suggests that Wasp's dotenv parser does not accept double quotes or JSON objects as values in .env.server.
Based on the knowledge sources, there is no documented way to set complex JSON values in .env.server for Wasp at this time. The documentation does not provide an example of using PG_BOSS_NEW_OPTIONS in .env.server outside of Heroku, and even then, it is not clear if this works outside of Heroku deployment.
What you can do:
- You may need to set PG_BOSS_NEW_OPTIONS as a shell environment variable when starting Wasp, rather than in .env.server. For example:
This approach is mentioned in the docs as a way to override env vars see docs.
- For per-user concurrency, the knowledge sources do not provide a built-in or recommended pattern in Wasp. You would need to implement this logic manually in your job handler.
Summary:The knowledge sources do not contain enough information to confirm that setting
PG_BOSS_NEW_OPTIONS as a JSON object in .env.server is supported by Wasp. The error you encountered suggests it is not. Setting the variable via the shell is the recommended workaround according to the available documentation.
If you need further help, you may want to reach out to the Wasp community or check for updates in the documentation regarding support for complex env vars in .env.server.
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------
Sorry @InterestEmpire, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist:
Perhaps try again in 15mins?Hm that is weird @InterestEmpire , I agree that it looks as if wasp should be able to correctly parse that env var.
We are using this library in the background for parsing dotenv files: https://github.com/stackbuilders/dotenv-hs .
Based on its docs,
PG_BOSS_NEW_OPTIONS='{"teamConcurrency":3}' should be fine, but it seems that doesn't work.
@miho is it possible that our relatively newer additions to env vars validation are the cause of this? That they can't JSON.parse env var with value like this because JSON expects " and not '?
If not, do you maybe have any idea why this would not be working?
What you can try @InterestEmpire is using " instead of '`':
let us know if this helps!
Does not work, have alredy tried that
@martinsos I have tried passing it in ahead of wasp start too to no avail
Which error exactly do you get in this case?
Let's wait for @miho he will have a good idea probably of what is going on here, it is quite unusual.
Alright, I will wait to see what he says
Any ideas @miho ?
You have any other ideas @martinsos ?
@kapa.ai any other ideas on what to do, nothing seems to work
Let me check the knowledge base for you again. One moment! :books:
----------------------
I'm sorry you're running into this issue. Based on the available knowledge sources, there is no documented, reliable way to set complex JSON values (like
- Setting complex JSON in
PG_BOSS_NEW_OPTIONS) directly in .env.server for Wasp. The dotenv parser used by Wasp appears not to support JSON objects as values in .env.server, as evidenced by your error and the lack of examples in the docs.
The only alternative mentioned in the documentation is to set such environment variables via the shell when starting your Wasp app, for example:
This method is suggested for overrides and is not specific to Wasp, but rather how environment variables work in the shell Defining Env Vars in Development.
Unfortunately, the knowledge sources do not provide any other officially supported patterns or workarounds for this scenario. If this approach does not work for your use case, you may need to reach out to the Wasp community or open an issue for further support, as this appears to be a limitation of the current dotenv parsing in Wasp.
If you need to implement per-user concurrency, you will have to handle it manually in your job logic, as there is no built-in support for this in Wasp or pg-boss integration at this time pg-boss considerations.
Summary:- Setting complex JSON in
.env.server is not supported.
- Use shell environment variables as a workaround.
- No other documented solutions are available in the current Wasp docs.
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------
Sorry @InterestEmpire, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist:
Perhaps try again in 15mins?@kapa.ai
Does not work either
Let me check the knowledge base for you again. One moment! :books:
----------------------
Thank you for clarifying that setting
The knowledge sources do not contain enough information to provide a working solution for setting
PG_BOSS_NEW_OPTIONS via the shell (e.g., PG_BOSS_NEW_OPTIONS='{"teamConcurrency":3}' wasp start) does not work either.
Based on the knowledge sources, there is no explicit documentation or example confirming that complex JSON values for PG_BOSS_NEW_OPTIONS can be set successfully in Wasp, either via .env.server or via shell environment variables. The documentation does mention the need to set PG_BOSS_NEW_OPTIONS for Heroku deployments, but it does not provide a working example for local development or clarify how to handle JSON parsing issues with Wasp's dotenv parser pg-boss considerations.
Additionally, the error you encountered with .env.server and the lack of success with the shell approach suggest that Wasp's current environment variable handling does not support this use case.
In summary:The knowledge sources do not contain enough information to provide a working solution for setting
PG_BOSS_NEW_OPTIONS with a JSON value in Wasp, either via .env.server or the shell. There is no alternative pattern or workaround documented for this scenario. For per-user concurrency, there is also no built-in support or recommended pattern in the current Wasp documentation.
If you require this functionality, the best next step would be to reach out to the Wasp community or open an issue for further clarification and support.
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------
Sorry @InterestEmpire, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist:
Perhaps try again in 15mins?Looking into this today
Yep, we have a bug with parsing JSON in the env files
We have a weird case of parsing the
.env.server with Haskell and then writing it down in .env without properly escaping values (it seems).
Also, when you provide PG_BOSS_NEW_OPTIONS you override the connection string - which is not something you'd want to do (I feel like).
I'll encode this into an issue for us to fix ASAP @martinsosI've written down the issue here: https://github.com/wasp-lang/wasp/issues/2846
GitHub
Passing JSON in env variables is broken · Issue #2846 · wasp-lang...
The way we parse and output the values in .env.server is broken. It prevents users from customizing the PgBoss options via the PG_BOSS_NEW_OPTIONS env variables. Reproduction You'll need an app...
Created one extra issue to allow for customisation of PGBoss options in a more humane way 😄 https://github.com/wasp-lang/wasp/issues/2847
GitHub
Provide
setupFn for PgBoss options · Issue #2847 · wasp-lang/waspMotivated by this issue: #2846 Currently we ask users to pass JSON via env variables to set up PgBoss which is not the best DX. It's also fragile since the current implementation overrides the ...
Nice @miho ! Anyway for @InterestEmpire to work around this at the moment @miho ?
I'd say, maybe mock the concurrency like we did with Mage - https://github.com/wasp-lang/wasp/tree/main/mage/src/server/jobs @InterestEmpire please check this repo out and how we did jobs
GitHub
wasp/mage/src/server/jobs at main · wasp-lang/wasp
The fastest way to develop full-stack web apps with React & Node.js. - wasp-lang/wasp
Thanks @miho and @martinsos
No worries @InterestEmpire and sorry we couldn't give a better answer this time! Thanks a lot for feedback though
This is actually a better way.
I had planned to eventually move to redis down the road but will copy your mage method for the time being.