theodor
theodor
Explore posts from servers
RRailway
Created by theodor on 2/26/2024 in #✋|help
Railway incident ?? Ghost deployment without our interaction
This is incredibly serious. We just found out that railway created a deployment out of nothing out of thin air, taking our site down, using a very old random commit.
25 replies
RRailway
Created by theodor on 2/2/2024 in #✋|help
Railway Networking issues
Hi! This is a bit broad as we're still debugging this, but we've been noticing that certain webhook events don't make it to our servers. This is really bad, since we don't really get an error when that happens (since they don't make it to the server!i ) We can also see that some requests fail because the TLS fails, e.g. request to https://api.stability.ai/v1/generation/stable-diffusion-xl-1024-v1-0/text-to-image failed, reason: Client network socket disconnected before secure TLS connection was established From our research, this means something may be still wrong with the proxy. Any ideas what we can do? Should we look into cloudflare? And yes, we do sleep 3s before we start the services...
36 replies
RRailway
Created by theodor on 1/30/2024 in #✋|help
Setting shared env var doesn't wrok
Tonight it looks like setting shared env vars doesn't work. Setting and redeploying seems to work, but then on startup the env var doesn't exist...
11 replies
RRailway
Created by theodor on 12/13/2023 in #✋|help
Issue accessing env vars
No description
7 replies
DTDrizzle Team
Created by theodor on 11/28/2023 in #help
Index not being created in table
Hi! I'm using drizzle-kit push:pg to update my local dev database. When I use this command, indices with shared names (e.g. updated_at_idx) only get created on the first table in the schema.ts. For example, if I have:
export const users = pgTable(
'user',
{
id: uuid('id').primaryKey().defaultRandom().notNull(),
name: text('name'),
email: text('email'),
createdAt: timestamp('created_at', { precision: 3 }).notNull().defaultNow(),
updatedAt: timestamp('updated_at', { precision: 3 }).notNull().defaultNow(),
},
(table) => ({
emailIdx: index('email_idx').on(table.email),
nameIdx: index('name_idx').on(table.name),
createdAtIdx: index('created_at_idx').on(table.createdAt),
updatedAtIdx: index('updated_at_idx').on(table.updatedAt),
}),
);
export const users = pgTable(
'user',
{
id: uuid('id').primaryKey().defaultRandom().notNull(),
name: text('name'),
email: text('email'),
createdAt: timestamp('created_at', { precision: 3 }).notNull().defaultNow(),
updatedAt: timestamp('updated_at', { precision: 3 }).notNull().defaultNow(),
},
(table) => ({
emailIdx: index('email_idx').on(table.email),
nameIdx: index('name_idx').on(table.name),
createdAtIdx: index('created_at_idx').on(table.createdAt),
updatedAtIdx: index('updated_at_idx').on(table.updatedAt),
}),
);
and:
export const conversations = pgTable(
'conversation',
{
id: uuid('id').primaryKey().defaultRandom().notNull(),
name: text('name'),
userId: uuid('user_id')
.notNull()
.references(() => users.id),
createdAt: timestamp('created_at', { precision: 3 }).notNull().defaultNow(),
updatedAt: timestamp('updated_at', { precision: 3 }).notNull().defaultNow(),
},
(table) => ({
userIdIdx: index('user_id_idx').on(table.userId),
createdAtIdx: index('created_at_idx').on(table.createdAt),
updatedAtIdx: index('updated_at_idx').on(table.updatedAt),
}),
);
export const conversations = pgTable(
'conversation',
{
id: uuid('id').primaryKey().defaultRandom().notNull(),
name: text('name'),
userId: uuid('user_id')
.notNull()
.references(() => users.id),
createdAt: timestamp('created_at', { precision: 3 }).notNull().defaultNow(),
updatedAt: timestamp('updated_at', { precision: 3 }).notNull().defaultNow(),
},
(table) => ({
userIdIdx: index('user_id_idx').on(table.userId),
createdAtIdx: index('created_at_idx').on(table.createdAt),
updatedAtIdx: index('updated_at_idx').on(table.updatedAt),
}),
);
The created_at and updated_at indices on conversations will NOT be created. I noticed that the SQL gets generated correctly if I generate the migration.
1 replies
RRailway
Created by theodor on 9/12/2023 in #✋|help
nginx revere proxy 503 error
Hi! I wanted to set up a reverse NGINX proxy with mixpanel. I used the dockerfile provided by them:
FROM nginx
COPY nginx.conf /etc/nginx/nginx.conf
FROM nginx
COPY nginx.conf /etc/nginx/nginx.conf
And this configuration:
events {}
http {
server {
listen 80 default backlog=16384;
listen [::]:80 default backlog=16384;

location /lib.min.js {
proxy_set_header X-Real-IP $http_x_forwarded_for;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
proxy_pass https://cdn.mxpnl.com/libs/mixpanel-2-latest.min.js;
}

location /lib.js {
proxy_set_header X-Real-IP $http_x_forwarded_for;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
proxy_pass https://cdn.mxpnl.com/libs/mixpanel-2-latest.js;
}

location /decide {
proxy_set_header Host decide.mixpanel.com;
proxy_set_header X-Real-IP $http_x_forwarded_for;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
proxy_pass https://decide.mixpanel.com/decide;
}

location / {
proxy_set_header Host api.mixpanel.com;
proxy_set_header X-Real-IP $http_x_forwarded_for;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
proxy_pass https://api.mixpanel.com/;
}
}
}
events {}
http {
server {
listen 80 default backlog=16384;
listen [::]:80 default backlog=16384;

location /lib.min.js {
proxy_set_header X-Real-IP $http_x_forwarded_for;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
proxy_pass https://cdn.mxpnl.com/libs/mixpanel-2-latest.min.js;
}

location /lib.js {
proxy_set_header X-Real-IP $http_x_forwarded_for;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
proxy_pass https://cdn.mxpnl.com/libs/mixpanel-2-latest.js;
}

location /decide {
proxy_set_header Host decide.mixpanel.com;
proxy_set_header X-Real-IP $http_x_forwarded_for;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
proxy_pass https://decide.mixpanel.com/decide;
}

location / {
proxy_set_header Host api.mixpanel.com;
proxy_set_header X-Real-IP $http_x_forwarded_for;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
proxy_pass https://api.mixpanel.com/;
}
}
}
I also enabled a custom domain which is verified (reverse-proxy.rizzgpt.app). However, I'm getting a 503 when I try to load it. Any idea why?
15 replies
RRailway
Created by theodor on 9/4/2023 in #✋|help
Restore old logs?
Hi! Is there a way to restore old log feature while you figure out the kinks in the new one? The new one is very slow and makes it hard to figure things out during an emergency. I'd rather not use it for now.
20 replies
RRailway
Created by theodor on 7/21/2023 in #✋|help
Issues setting up datadog with docker containers
Hi - I'm trying to use private networking to set up datadog. I have two docker files that start normally: server and datadog. Both seem to run fine. I enabled private networking for the project already. 1. What are the environment variables I can use to refer to the hostnames of the services? I had to hard code them in. 2. It doesn't seem like the services are able to talk to one another. I don't see any errors, but I onyl get silence. Is there something off about the way I defined the hostnames below? 3. When I run railway run echo $DD_SERVICE or any similar command to check the environment variables of the service, I don't get anything. Is this ok? Here are the dockerfiles:
# Start from the official Node.js 18 image
FROM node:18

# Define environment variables
ENV DD_ENV=prod \
DD_LOGS_INJECTION=true \
DD_SERVICE=cami-server \
DD_AGENT_HOST='datadog.railway.internal'

# Create app directory
WORKDIR /usr/src/app

# Copy package.json and package-lock.json
COPY package*.json ./

# Install app dependencies
RUN yarn install --frozen-lockfile

# Copy app source code
COPY . .

# Build the app
RUN yarn run build

# Start the app
CMD [ "yarn", "run", "start" ]
# Start from the official Node.js 18 image
FROM node:18

# Define environment variables
ENV DD_ENV=prod \
DD_LOGS_INJECTION=true \
DD_SERVICE=cami-server \
DD_AGENT_HOST='datadog.railway.internal'

# Create app directory
WORKDIR /usr/src/app

# Copy package.json and package-lock.json
COPY package*.json ./

# Install app dependencies
RUN yarn install --frozen-lockfile

# Copy app source code
COPY . .

# Build the app
RUN yarn run build

# Start the app
CMD [ "yarn", "run", "start" ]
Datadog:
# Start from the official Datadog agent image
FROM datadog/agent:latest

# Copy your Datadog configuration to the correct location
COPY datadog.yaml /etc/datadog-agent/datadog.yaml

# Set the hostname and port
ENV DD_HOSTNAME='datadog.railway.internal' \
DD_LOGS_ENABLED=true \
DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true \
DD_BIND_HOST=::

# Start the Datadog agent
CMD ["/init"]
# Start from the official Datadog agent image
FROM datadog/agent:latest

# Copy your Datadog configuration to the correct location
COPY datadog.yaml /etc/datadog-agent/datadog.yaml

# Set the hostname and port
ENV DD_HOSTNAME='datadog.railway.internal' \
DD_LOGS_ENABLED=true \
DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true \
DD_BIND_HOST=::

# Start the Datadog agent
CMD ["/init"]
81 replies
RRailway
Created by theodor on 7/2/2023 in #✋|help
redis connection fails
Hi! Having some issues with redis. I'm getting
Error: connect ECONNREFUSED 34.82.163.149:7679
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1300:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '34.82.163.149',
port: 7679
}
Error: connect ECONNREFUSED 34.82.163.149:7679
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1300:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '34.82.163.149',
port: 7679
}
I already tried restarting Redis or restarting/redeploying the project. Any suggestions? everything was working fine previoously
40 replies
RRailway
Created by theodor on 5/30/2023 in #✋|help
urgent: horizontal scaling limited at 5 replicas
Hi! We're on the teams plan. We're trying to scale as fast as we can, but the scaling is limited at 5 replicas. is there anything we can do to get past that?
127 replies
RRailway
Created by theodor on 5/29/2023 in #✋|help
"horizontal-scaler" missing on railway
Hi! We're being hit hard by a lot of traffic and our server is having trouble scaling. I wanted to use the 'horizontal-scaler' option described in the docs, but it's missing. Anyone know why?
29 replies
RRailway
Created by theodor on 3/13/2023 in #✋|help
Environment Variables missing from project
The environment variables seem to be missing from the project. Loading the view also takes much longer than it used to. I noticed this after modifying one of them in Vercel , but I'm not sure if that's related.
8 replies