What is Egress and why am I suddenly paying it?
Hi there,
I just noticed I have something like 600GB of Egress usage and I have no idea what that means. My app has never used any egress up until now (again, I did not even know that field existed), and the amount seems to be absolutely insane.
Could you please tell me where it is coming from and what can I do to avoid this?
139 Replies
Project ID:
b80fbea0-3f24-4fa5-ab1b-3d267f761495b80fbea0-3f24-4fa5-ab1b-3d267f761495
Had exactly the same thing this morning. I'm assuming this is a billing error. Luckily my prject was stopped by my billing limit.
I hope so. It's currently $60 on a project where I've barely paid more than $20 per month. There seem to be absolutely nothing out of ordinary.
make sure you get a billing limit put in place!
Same here
yesterday my costs were $52.36
today $107.80
should we send an email to billing?
But this does look like more a web error than a billing issue
And it's impossible for me to have used that amount of GB
I would rather not put an arbitrary limit since this is used by a client in a different timezone and I wouldn't want the app to stop working when I'm sleeping 😆
Same here, looks like the metrics changed from bytes to MB/MB to GB. Everything has been multiply by 1000. When I see 1GB it should be 1MB, when I see 1MB it should be 1B and so on
I'm pretty upset with this ngl
Same here. There is literally nothing that could explain that
what if I had a hard limit for whatever reason
and it hit the limit and the service stopped for no reason
I noticed this error seems to be related to MongoDB in my case
Hope they fix it soon. Dont want to see the 445GB convert to 4500GB in a few hours
curiously mine is postgresql, which supposedly egressed 100GB
perhaps it only affects the provided db services. no other containers seem to be affected
I'm presuming a graph error too; My node websites are seeing much higher egress than normal (though no where near what you all are seeing thankfully), even with no users on the site
Edit: From comparing old results to now, I'd estimate my egress is showing 10x higher than it should
Any updates?
Also waiting for an update!
I'm waiting to hear from them too...
Thread has been flagged to Railway team by @Adam.
Oh no- this is not good.
Hi! Egress is outbound traffic for your application
We were charging for it before but weren't calculating it correctly. We fixed a bug with it yesterday and will be throwing it into the changelog
Happy to look into it if it's not accurate
Hmm I'm pretty sure isn't accurate...
Project and serviceID?
40e44f3c-0508-45c6-b255-a4558921d6f5
This is a yesterday screenshot that I sent to a friend
this is today:
geez
So, that appears to be accurate according to the instance itself
Are you making requests via the external network or internal private network?
external
everything external
Yea that's an issue. So you're transfering payloads over the public network which we pay downstream to our provider :/
We've been charging for this for a while (since early/middle of the year) but recently fixed a bug where we were undercounting it by a LOT. Cause it was brought up internally that we were billing X for egress, but paying GCP almost 50x that
I'm happy to apply credits for this months bill on the above difference, but you're gonna want to switch to using the internal network
Does that seem reasonable?
(BTW if you use the internal network, the cost, at least for the PG instance, should disappear entirely)
We're also moving to our own hardware early/middle next year because, well, GCP are awful for this + a littiny of other reasons (At which point we can bill MUCH cheaper for that SKU)
Honestly this is a no no for me. This is basically a killing thing for me that won't make it possible to run my service on Railway. I run an n8n container image that just receives plain text data and re-directs to other endpoints... how I'm supposed to use private network in my case?
Oh lemme clarify; I just mean the n8n -> db connection
public -> n8n is like, $10 in your screenshot, and you're also probably paying for egress from the DB to the n8n container (since you're round tripping outside the cluster's intranet)
you mean the connection between the database and the n8n controller?
Ye
I though by calling it directly by the var
it would use internal network?
Unfortunately the plugins dont' have an internal network :/ (nor do they let you update the image)
Which is why we released the Service+Volume version of plugins, and internal is the default for those
(I see how this is frustating. Talking internally about ways to remedy this at least for this month
Proposed solutions include voiding egress for V1 plugins for this month)
question, until what date the plugins wont be supported?
I mean you guys released a new thing for these ok
but when it will be "unsupported" to use old plugins? I think the egress thing should be voided until that date bcs I installed the plugin when it was the default thing
V1 plugins are deprecated as of last month (new ones aren't able to be provisioned)
They'll be unsupposed as of Jan 2024 (3mo+ to cutover)
in my opinion the egress should be voided until unsupported date, cause a night-update change literally can increase costs of an app by 100% from one day to next... now I have to investigate how to pass from plugin to native postgres db service
to use the internal network
otherwise my project can't run on Railway
This request doesn't seem unreasonable FWIW
Regardless we'll have resolution in the next hour BTW
I'd love to hear from @jeremy and @domb84 since y'all saw this issue as well
Is it just with the DB instance or are you seeing crazy egress with others?
Ok, can you please point me a guide or documentation (if exists) how to migrate from postgres plugin to native postgres?
We're gonna enable a 1-Click today for it
Thanks to god
(Assuming we haven't already)
and you guys lol
This might be admins only ATM
Okay, that will help
a lot
Ye
ok, cool
then it makes sense
So, 1-click, waive all egress for this month, maybe 50% next month?
to just void it for this month and next 50% yeah
Kinda a little sliding scale as they go out. Good forcing function and all that
(And we'd apply it as a Plugin_V1_Egress discount on the billing, so that people don't see it as a massive surprise bill that they have to pay immediately)
Okay, last question Cooper as I have your attention here. So per my image my "egress" is in the postgress plugin, which as you mentioned is between n8n and db. But when I respond to a webhook that "egress" that is sent to a public url (https) will still count as external right?
So you pay for the in not the out, if that makes sense
Or, out not in. I'd have to check
Regardless, that JSON payload should probably be TINY in comparison to doing a ton of selects consistently and passing that out/in
yeah okay
I'm seeing it for all my services (441e4d9c-c378-48c1-b974-77aa4aeee907, 9bda1f3e-10b6-4bcc-882c-f4345264c168, 43a0cabc-1194-4e24-858c-55e9cab88488 and a redis plugin). I would love to get more information on what the issue was, and how I can debug it on my side.
My bill going from 20$/month to 60$/month is a steep overnight patch fix 😅
ProjectID plz?
34849d22-d685-4e73-8858-fbd4fe42ea65
I bet you like 80% of that is the redis plugin, and 10% is actually "backwash" from the instances connecting to the plugin over public network
@jeremy can you pull this view for me?
Top right -> Icon -> Billing -> hello aurora -> click to expand
Sorry to bother you, this 1-click migrate will only apply to postgresql or all the plugins? I figured out that my Redis instance is a plugin too....
All plugins
ok cool
Not gonna leave you high and dry 
The 1-click migrate is not released yet. We will announce when it is available
yea I know, cooper explained it's a soon-release feature...
@Cooper just DB. theres no way that 100GB of traffic has been sent from that DB
Yea so, 80% of your egress is from that redis plugin, 10% backwash into your other services
100GB egress on a tiny zabbix instance?
transfers happen a LOT
but all the egress has happened in one night, and no additional monitoring added
Alright here's the resolution:
- We will revert the egress PR for today
- We will announce it in the changelog today + email
- We will add a coupon for 100% discount on egress for plugins this month, 50% next
- We will merge the coupon PR + unrevert the cost calc on Monday
refund is good, but i'm not sure where redis enters my issue with a standard postgres db
What do you mean sorry?
Can you post this for your project?
i dont have a billing tab?
Yea so 50:1 egress for PG vs zabbix services
Any plan to have an included amount of egress by plans? 50Go included (hobby) and more for Pro for example. Or any egress will always be billable?
106GB of egress from the standard postgresdb in a couple of hour window?
also how is it egres
its local between the containers
egress will always be billable, but when we go bare metal early/mid next year we'll cut the billing drastically
Cause it uses the external URL
its not mapped externally?
its not accessible via the wan
or are you saying the DBs are by default and you don't route traffic internally for services?
We don't route it for legacy plugins
We do for the PluginV2
legacy in what sense... theres one option when creating a db?
V1s no
I built them almost 3 years and 2 countries ago
As of middle last month all Postgres instances created are V2
I think you got, unfortunately, one of the last legacy plugins
it still makes no sense that 100GB of data was egressed
have you got a historical egress graph?
also how would anybody know what version of what plugin is used in their environment?
We will tell you as of later today (or latest next week)
Link me your projectID?
so we migrate to new "v2" database, and egress charges stop to services within the project?
This should be your historical graph
97a6977d-11e5-4ec8-aef0-9b42b6449d4f
Should drastically provided you use the private URL
(Almost entirely)
i get cpu and ram, no egress tracking
if your database does have a volume, it's a database v2.
if your database doesn't have a volume, it's a legacy database
This has egress on it?
its just a number
OH
yesterday egress was pennies
then this morning it was 100GB and $15
Shit, sorry you want a graph on it
ideally!
because all the egress charges occured in a few hours, hence my confusion
my spend for the month was predeicted to be sub $10, then over night i supposedly egressed more than that
do we get the option to migrate from a v1 to a v2 later in the week? currently there is no option @Cooper
Yea we're working on rolling it out either this week or next
(Admins only ATM while we make sure the migration workflows are as ezpz as possible/make sure data safely migrates in various situations)
(Hence the "Waive 100% for a month")
also i can see from the network graph the egress i think, but its between containers, so i would have taken this to mean zero egress
Can you define "between containers"
please 🙏 $0.10/GB is so overpriced compared to other clouds tbh
I think it's what other people charge? Other than say, Fly
And that's cause they have their own racks
This is actually the lowest egress cost on GCP (costs 0.20/GB for China egress but we eat that cause we want it to be easy to understand ;_;)
between the containers in the project and the db in the project, i would have assumed a zero cost as nothing SHOULD be egressed
but you're saying that traffic between the containers in the project and the db are actually egressed over the internet
This is what I tought too but seems V1 plugins really don't do that
They pass the data through public network which is why we see that ton of GBs
but with the new V2 native db that data should run internal network
Cooper explained to me if you scroll up
where $0.10/GB is the lowest egress?
thats between regions
Yea the V1 plugins were built really fast, 3 years ago, by a sub par engineer (See: me)
not from google cloud to the internet
yeah you're right
in premium tier networking is the expensive thing
Yup it's this one IIRC
We want the best stuff for y'all
Hey @Cooper , appreciate that you're helping everyone here, could you enlighten me on what is going on in my case? I see the usage has gone back down substantially, but I'm still confused by the egress
Project ID b80fbea0-3f24-4fa5-ab1b-3d267f761495
Could you do this for me?
CC @Omeganex9999
Hey, I'm in billing but I don't see any "Hello xxx" to expand
I just see my plan, payment method, billing email and billing history
Usage on the left?
Here's a screen with everything
What is today just 10.07 GB of Egress, was 600+ this morning
Is it just a Mongo instance?
If so, that's also Plugin V1 egress stuff
Yes, just that one. Anything I should do?
does your mongo instance have a volume?
I think it's a plugin, don't see "volume" anwyhere
migrate to a new mongo database, and then connect to the database over the private network, i was told there is no network costs when using the private network
Brody, quick question I created this testing postgresdb to play a bit and understand concepts, I can see there is two urls, one private one public, ok. But the host uses the public url... so if I have to connect my container to use the private network I put the private network database url and the host that contains the public url? Does it make sense?
half way though it stopped making sense
does your app require the use of
PGHOST and PGPORT ?this is what my app require:
can you open a new thread titled "help me use the private network for database conenctions" and ill help you there
ok 😆
Thinking about it now, Redis is being used by the 3 services. Can you give me a percentage of how much egress is going to each services? I believe the Redis itself shouldn’t haven’t any egress, but the others services should be the one having the egress. Also, what does “10% backwash mean?”
Your redis instance is "serving the data back"
To 3 services. You can probably estimate how much by looking at that screenshot breakdown of the egress of all 3 services summed then divided by the service itself
Hey, just updated my N8N template to use workers and run off docker images rather than GitHub repo’s, more reliably, in addition to using an internally networked Postgres service to significantly reduce a huge chunk of egress fees; noticed this could be of help to you. Saw you were using a modified version of mine with a worker service added, not sure if there’s any way to transfer database contents but just wanted to let you know if you were interested.
https://docs.n8n.io/workflows/export-import/
https://railway.app/template/r2SNX_
Export and import | n8n Docs
Export and import workflows
^ Brody approved template
Hey thanks @Jack really useful for sure I'll use it for other side-project I have. My prod project need the ability to install custom modules using npm install on the dockerfile which I think yours doesn't give yet the ability to. However I installed this template 1 month ago before yours had a queue mode option which now I think it's really really helpful!