MongoDB suddenly increased 10 GB and decreased 20GB
MongoDB suddenly increased 10 GB and decreased 20GB
36 Replies
Project ID:
b51e42c0-8a37-4633-8e1d-8f14fc187052b51e42c0-8a37-4633-8e1d-8f14fc187052
can anyone please help?
Can you check if any update/delete operation was performed in the logs?
there is none
a lof of valueable data were lost
data was deleted down to 2GB for no reason
after the screenshot
Im not sure what it could be, but did you make a backup of your database? In Postgres you have pg_dump, mongoDB probably also has something like it
we didn't make any backup
we have to refund all users' assets and close down the business for couple of days...
we use a different mongodb now, not railway's anymore
our service is trying to get back to the orignial data by scanning the blockchain... but it will take couple of days
Ah alright, yeah railway is not that reliable it seems for database stuff
it will only recover part of our data
Lets hope you get your data back
didn't know it would downsize itself for no reason
data is gone
can we get some railway support here to see what is wrong with my mongodb?
Yeah its weird, havent seen something like this either
Not sure if i can ping someone on this server, im just a customer as well
Brody is pretty good with solving bugs though, but hes offline now
Railway’s databases aren’t fantastic, but they won’t just randomly delete data or drop tables. If that’s happened, I encourage you to verify in the logs. This is very likely a case of user error where you or one of your teammates accidentally deleted a table.
Or your database is exposed to the public web without proper authentication
they won’t just randomly delete data or drop tables.I wish that is true. I cannot see any logs related to this issue. can someone please help. We are devastated right now
please send your project/service id and I will flag in the team
Thread has been flagged to Railway team by @Adam.
Hey there @adeg - we haven't accessed your DB nor do we see any signs of data corruption. This is likely a code issue on your part.
I have another service with the same code and connect to a different database, that database is fine, nothing has happened to it. If there is something is wrong with my service code, both databases should be impacted.
can you pull out the logs?
?
The logs you have there are the same logs we have.
I am seeing a bunch of
failed to auth and disconnect logs, so it must have meant that you didn't close your connections properly.how do I know what caused the deletion?
I don't know, it's Mongo... usually you set up replica sets for this reason.
also there is a sudden data increase for 10GB
within couple of minutes
Again, we didn't do that.
This is an error with your code.
how would that happen? I want to prevent that from happening in the future? how would I do that?
If you are running an indexer, you should have local state of the chain queried via redis and then you persist the state of the chain onto S3 backed up or on your Mongo.
Anyway, locking thread. If you are a company who is operating a mission critical service, you may want to consider Business Class.
I'm on Pro plan...
Yep- realized that. Sorry, misqueried.
Point stands.
I am terribly sorry about your business, but there is nothing we else can do here.
is there anyway to have the mongodb query logs?
it didn't record what is going on with the data incease and decrease
I want to have more logs for that matter
just in case that happened again
I can debug
now I don't have anything to debug with
the logs you see are the only logs that exist
how do I get more logs if I continue use your service?
or there is no way to get more logs in the future?
How can I download the volume?
https://railway.app/template/guiCBv I can only click the file one by one
can someone please help?
?
anyone please?
You can alter the image to add debug logs