I'd like my pages.dev to be crawled by search engines like Google

L

LambdaOP•9/26/23, 5:45 AM

How can I do that please.

LLambda I'd like my pages.dev to be crawled by search engines like Google

H

Hello, I’m Allie!•9/26/23, 8:30 AM

Maybe add a

robots.txt

robots.txt

robots.txt

robots.txt? Other than that, I don't think I do anything for Flareutils, and it is indexed.

R

Rich•9/26/23, 12:50 PM

Does anybody have any advice on setting up different environments with Cloudflare Pages. We want a production + preview as well as development + preview environments so that we can point prod to our prod APIs and development to our dev APIs.

Given an app in Cloudflare Pages only offers two envs (production and preview) I was thinking we would need to create a separate app for development vs production.

However, the Cloudflare UI is saying the repo is "In use" with another application, so I can't select it

RRich Does anybody have any advice on setting up different environments with Cloudflar...

S

Shados•9/26/23, 2:18 PM

I dont know if there's a better way, but we're doing that but publishing directly from a github action instead of using Page's own build.

SShados I dont know if there's a better way, but we're doing that but publishing directl...

R

Rich•9/26/23, 2:22 PM

That's exactly what I'm researching now

https://github.com/cloudflare/pages-action/

One thing I can't figure out is how to define which branches are preview vs production when using direct upload rather than connecting it to Git -- seems the dashboard loses the ability to configure it.

GitHub

GitHub - cloudflare/pages-action

Contribute to cloudflare/pages-action development by creating an account on GitHub.

B

bbs_samuray•9/26/23, 4:10 PM

Hello, my next.js project needs to download a zip file from external source and process some data at build time

B

bbs_samuray•9/26/23, 4:11 PM

Since this processed data won't change, I think doing this with every commit will be wasteful

B

bbs_samuray•9/26/23, 4:12 PM

Do my build artifacts remain after the build is completed

B

bbs_samuray•9/26/23, 4:13 PM

Or does it start from scratch every time

Bbbs_samuray Do my build artifacts remain after the build is completed

H

Hello, I’m Allie!•9/26/23, 4:14 PM

From scratch, unless they are in

git

git

git

git, but there is the build-caching-beta, which should hopefully be able to help you there, in the future.

B

bbs_samuray•9/26/23, 4:22 PM

I am guessing it is in closed beta

B

bbs_samuray•9/26/23, 4:22 PM

Thanks, I will find a way that is both convenient and not a resource hog

Eeddie soooo how can i www. a cloudflare pages site with a custom domain

C

Chaika•9/26/23, 8:48 PM

If you already have your apex (example.com) as a custom domain, it's recommended for SEO/etc purposes to just redirect www to apex, ex:
https://developers.cloudflare.com/pages/how-to/www-redirect/
Otherwise, if you're just asking how you can add it, just add it as a custom domain, it's no different then any other subdomain:
https://developers.cloudflare.com/pages/platform/custom-domains/

S

Samaritan•9/27/23, 2:19 AM

Hi, we have a fairly large Gatsby site that we are porting across from Gatsby Cloud as it shuts down. The site build times are taking between 15-30 minutes to run and I wanted to look at how we can implement caching of the public, node_modules and a .cache folder between build to try and speed up the builds. Is that possible with the Cloudflare pages build system?

SSamaritan Hi, we have a fairly large Gatsby site that we are porting across from Gatsby Cl...

C

Cyb3r-Jak3•9/27/23, 2:21 AM

There is a beta of build caching, see build-caching-beta. More info at build-caching-beta

CCyb3r-Jak3 There is a beta of build caching, see <#1144554260311326830>. More info at https...

S

Samaritan•9/27/23, 2:40 AM

Oh awesome, I'll check it out thank you!

A

addison•9/27/23, 2:43 AM

No Access does anyone know? is there a known issue with the stripe library that causes this? only happens in workers/pages

R

Rich•9/27/23, 7:53 AM

Does anybody know if it's possible to get the commit hash in a worker or perhaps pass env vars specific to a single deploy?

I'd like to have a /version endpoint in my Remix app which indicates exactly what code is deployed.

RRich Does anybody know if it's possible to get the commit hash in a worker or perhaps...

H

Hello, I’m Allie!•9/27/23, 8:08 AM

Like this? https://kian.org.uk/cloudflare-workers-build-time-constants-with-wrangler/

Kian's Blog

Cloudflare Workers - Build time constants with Wrangler

Insert Git hashes, Sentry releases or remove debugging code in production environments from your Cloudflare Worker with Wrangler!

HHello, I’m Allie!Like this? https://kian.org.uk/cloudflare-workers-build-time-constants-with-wran...

R

Rich•9/27/23, 9:12 AM

Yeah I think that might be the solution. TBH I'd rather it be a runtime variable as I've always avoided coupling a build artifact with a deployment... but CF Pages already does a new build per deployment so not really a big deal

K

kian•9/27/23, 9:13 AM

I don't think Pages can use

define

define

define

define

R

Rich•9/27/23, 9:15 AM

Ah... is that because Pages doesn't have a wrangler.toml file? (At least I don't have one as geneated CF Pages project from Remix CLI)?

R

Rich•9/27/23, 9:15 AM

Maybe I just need to do a dumb manual token replacement in a postbuild step

RRich Ah... is that because Pages doesn't have a wrangler.toml file? (At least I don't...

H

Hello, I’m Allie!•9/27/23, 9:19 AM

Now I'm wondering, if Remix uses Vite or something similar, you might be able to do the replacement at the Remix level...

HHello, I’m Allie!Now I'm wondering, if Remix uses Vite or something similar, you might be able to...

R

Rich•9/27/23, 9:20 AM

Remix uses esbuild but it's not super extensible, but will do some digging

RRich Ah... is that because Pages doesn't have a wrangler.toml file? (At least I don't...

K

kian•9/27/23, 9:29 AM

Basically

K

kian•9/27/23, 9:30 AM

Plus, if Remix builds your project, it won't be Wrangler's esbuild that does it

R

Rich•9/27/23, 9:48 AM

I've asked in the Remix discord here https://discord.com/channels/770287896669978684/770287896669978687/1156524276686528572

But have a feeling they will just say I should define a runtime env var.

It does seem a shame CF Pages doesn't let you define a deployment-level env var.... I guess it's because env vars are request-level in workers so perhaps not coupled to a deployment.

RRich I've asked in the Remix discord here https://discord.com/channels/77028789666997...

H

Hello, I’m Allie!•9/27/23, 10:12 AM

Env Vars on Pages anyway are coupled to a deployment. For example, if you change vars in the dash, they will only be added on subsequent deploys. This means that you could theoretically get a commit hash or deploy date for each deployment, you would just have to edit the vars on the dash for every deploy

HHello, I’m Allie!Env Vars on Pages anyway **are** coupled to a deployment. For example, if you ch...

R

Rich•9/27/23, 10:14 AM

I did wonder this, but given they're defined on

context.env

context.env

context.env

context.env which is bound to a request figured they must change dynamically?

Otherwise I'd love to just have them on

global.env

global.env

global.env

global.env so I don't have to pass them around to various modules all the time.

R

Rich•9/27/23, 10:14 AM

If we were to update the env vars via the API in our build script, there'd be a race condition between deploys I expect, it'd need to be passed in as part of the wrangler deploy cmd I expect and not impact other deployments?

K

kian•9/27/23, 10:16 AM

I did wonder this, but given they're defined on context.env which is bound to a request figured they must change dynamically?

context.env

context.env

context.env

context.env is a Pages/Functions wrapper over the

fetch(req, env, ctx)

fetch(req, env, ctx)

fetch(req, env, ctx)

fetch(req, env, ctx) that all Workers use. Your resulting built Worker will use

export default { async fetch(...) }

export default { async fetch(...) }

export default { async fetch(...) }

export default { async fetch(...) }.

K

kian•9/27/23, 10:17 AM

Otherwise I'd love to just have them on global.env so I don't have to pass them around to various modules all the time.

AsyncLocalStorage is a nicer way to do this, but this is fine. If you use it for arbitrary per-request things (i.e doing

globalThis.reqId = "123"

globalThis.reqId = "123"

globalThis.reqId = "123"

globalThis.reqId = "123" then other requests will be able to see and mutate that.

K

kian•9/27/23, 10:17 AM

If we were to update the env vars via the API in our build script, there'd be a race condition between deploys I expect, it'd need to be passed in as part of the wrangler deploy cmd I expect and not impact other deployments?

wrangler deploy

wrangler deploy

has no way to do this, unfortunately. It is all done in the dashboard/API.

K

kian•9/27/23, 10:18 AM

It does seem a shame CF Pages doesn't let you define a deployment-level env var.... I guess it's because env vars are request-level in workers so perhaps not coupled to a deployment.

Changing variables doesn't do anything until you do a new deployment, which is a new Worker.

K

kian•9/27/23, 10:18 AM

A given deployment will always have the same variables.

Kkian > I did wonder this, but given they're defined on context.env which is bound to ...

R

Rich•9/27/23, 10:30 AM

Interesting, I did briefly look at

AsyncLocalStorage

AsyncLocalStorage

but felt it was a bit complex / overkill, just wanted to shove it on

globalThis

globalThis

globalThis

globalThis :

export const onRequest = async (
  requestContext: EventContext<any, any, any>
): Promise<Response> => {
   globalThis.env = requestContext.env;
   handleRequest(requestContext.request);
}

export const onRequest = async (
  requestContext: EventContext<any, any, any>
): Promise<Response> => {
   globalThis.env = requestContext.env;
   handleRequest(requestContext.request);
}

export const onRequest = async (
  requestContext: EventContext<any, any, any>
): Promise<Response> => {
   globalThis.env = requestContext.env;
   handleRequest(requestContext.request);
}

export const onRequest = async (
  requestContext: EventContext<any, any, any>
): Promise<Response> => {
   globalThis.env = requestContext.env;
   handleRequest(requestContext.request);
}

However these docs (https://developers.cloudflare.com/workers/runtime-apis/handlers/fetch/#parameters) made me nervous as it says:

env
The bindings assigned to the Worker. As long as the environment has not changed, the same object (equal by identity) is passed to all requests.

But maybe bindings != env vars? (tbh I think I have a bit of a knowledge gap given my first usage of Workers is through Pages)

Fetch Handler · Cloudflare Workers docs

Incoming HTTP requests to a Worker are passed to the fetch() handler as a request object. To respond to the request with a response, return a Response …

K

kian•9/27/23, 10:30 AM

Bindings are env vars

K

kian•9/27/23, 10:30 AM

Those docs are for Workers, a given Worker

K

kian•9/27/23, 10:30 AM

Pages Functions are a different Worker per deployment

K

kian•9/27/23, 10:30 AM

One cannot affect the other

K

kian•9/27/23, 10:31 AM

Section 1 on this issue might help you visualise it in the context of Pages

https://github.com/cloudflare/next-on-pages/issues/1#issuecomment-1403845325

K

kian•9/27/23, 10:31 AM

We were hesitant to add this on day 1 because we re-use isolates as a performance optimization wherever possible. This means that as we process one request with set of environment bindings, another request also start to be processed, potentially with a different set of environment bindings. In the typical Cloudflare Worker written in ESM format, this isn't an issue since the request and env come in together. However, if you were to access the environment from a global scope like Next.js does, there's a risk that the new set of environment bindings will override the previous set as that previous request is still processing.

To be clear, we do not share isolates across accounts (or even, across different Workers), so there's never as risk of your environment leaking across accounts or projects. Instead, if you updated your environment variables of your project as it served live traffic, we were concerned that potentially some requests would be 'polluted' with the newer set of environment variables as they were still being processed.

It turns out, that after some investigation, the way that Pages does deployments means that this won't happen anyway. It took us some time to confirm this, but we're now confident that there will be no accidental 'leakage' of environment bindings across deployments.

K

kian•9/27/23, 10:33 AM

In the context of Workers, you can have a single Worker that has two requests with a different

env

env

if you change the bindings.

In the context of Functions, with the way that a given deployment can't have the bindings changed, it's irrelevant.

K

kian•9/27/23, 10:33 AM

Treat two Pages deployments as entirely separate Workers, basically.

K

kian•9/27/23, 10:35 AM

That first paragraph, as a bit of trivia, is why Module Workers are faster than Service Workers. They don't need a global scope to be bootstrapped on every request.

K

kian•9/27/23, 10:39 AM

In the case of AsyncLocalStorage, it's like this

import { AsyncLocalStorage } from "node:async-hooks";

const asyncLocalStorage = new AsyncLocalStorage();

export const onRequest = async (
  requestContext: EventContext<any, any, any>,
): Promise<Response> => {
  return asyncLocalStorage.run(requestContext, () => {
    return handleRequest();
  });
};

function handleRequest() {
  const requestContext = asyncLocalStorage.getStore();
  const req = requestContext.request;
  await requestContext.env.KV.put("foo", "bar");
}

import { AsyncLocalStorage } from "node:async-hooks";

const asyncLocalStorage = new AsyncLocalStorage();

export const onRequest = async (
  requestContext: EventContext<any, any, any>,
): Promise<Response> => {
  return asyncLocalStorage.run(requestContext, () => {
    return handleRequest();
  });
};

function handleRequest() {
  const requestContext = asyncLocalStorage.getStore();
  const req = requestContext.request;
  await requestContext.env.KV.put("foo", "bar");
}

import { AsyncLocalStorage } from "node:async-hooks";

const asyncLocalStorage = new AsyncLocalStorage();

export const onRequest = async (
  requestContext: EventContext<any, any, any>,
): Promise<Response> => {
  return asyncLocalStorage.run(requestContext, () => {
    return handleRequest();
  });
};

function handleRequest() {
  const requestContext = asyncLocalStorage.getStore();
  const req = requestContext.request;
  await requestContext.env.KV.put("foo", "bar");
}

import { AsyncLocalStorage } from "node:async-hooks";

const asyncLocalStorage = new AsyncLocalStorage();

export const onRequest = async (
  requestContext: EventContext<any, any, any>,
): Promise<Response> => {
  return asyncLocalStorage.run(requestContext, () => {
    return handleRequest();
  });
};

function handleRequest() {
  const requestContext = asyncLocalStorage.getStore();
  const req = requestContext.request;
  await requestContext.env.KV.put("foo", "bar");
}

K

kian•9/27/23, 10:42 AM

Eventually, you might be able to just do this instead

import getRequestContext from 'cloudflare:request-context';

const [ req, env, ctx ] = getRequestContext();

import getRequestContext from 'cloudflare:request-context';

const [ req, env, ctx ] = getRequestContext();

import getRequestContext from 'cloudflare:request-context';

const [ req, env, ctx ] = getRequestContext();

import getRequestContext from 'cloudflare:request-context';

const [ req, env, ctx ] = getRequestContext();

https://github.com/cloudflare/workerd/pull/1213

R

Rich•9/27/23, 10:45 AM

Ok interesting. So sounds like it'd be totally safe to add env to

globalThis

globalThis

globalThis

globalThis within CF Pages Functions (but not CF Workers). But maybe if

AsyncLocalStorage

AsyncLocalStorage

is as simple as above, I might try that as would be super handy to access the req too.

I'd like my pages.dev to be crawled by search engines like Google

Similar Threads

Similar Threads

Similar Threads