Memory (RAM) issues

I have a workflow where one step is to enrich every page on a website (sometimes 600 or more) with wider business context information, then after that, a

forEach

forEach

step that handles each page of the website and examines the HTML / attributes.

With a very large site, I got this error in the Mastra logs (attached).

I have a few questions:

1) Is there any way around this (apart from "buy a server with higher spec")?

I'm wondering especially from the perspective of workflow architecture/orchestration.

2) If I do plan to load in the entire HTML of a given web page (along with business context) and then process it, are there any best practices that I might not be doing?

The Workflow is basically:

export const postOnboardingWorkflow = createWorkflow({
  id: "post-onboarding-workflow",
  description: "Rank, deduplicate, and research keywords with rankings-first approach",
  inputSchema: PostOnboardingWorkflowInputSchema,
  outputSchema: PostOnboardingWorkflowOutputSchema,
})
  .then(pageRankingStep)
  .then(pageDeduplicationStep)
  .then(enrichPagesWithContextStep)processing
  .foreach(singlePageKeywordStep, { concurrency: RANKING_KEYWORDS_CONFIG.MAX_CONCURRENCY })
  .then(keywordCannicalizationCleanupStep)
  .commit();

export const postOnboardingWorkflow = createWorkflow({
  id: "post-onboarding-workflow",
  description: "Rank, deduplicate, and research keywords with rankings-first approach",
  inputSchema: PostOnboardingWorkflowInputSchema,
  outputSchema: PostOnboardingWorkflowOutputSchema,
})
  .then(pageRankingStep)
  .then(pageDeduplicationStep)
  .then(enrichPagesWithContextStep)processing
  .foreach(singlePageKeywordStep, { concurrency: RANKING_KEYWORDS_CONFIG.MAX_CONCURRENCY })
  .then(keywordCannicalizationCleanupStep)
  .commit();

(

MAX_CONCURRENCY

MAX_CONCURRENCY

and it still does this)

The key part from the attached file:

<--- Last few GCs --->

[1:0x739912b6a650] 96509996 ms: Mark-Compact 254.7 (283.1) -> 254.2 (283.1) MB, 83.91 / 0.00 ms  (average mu = 0.803, current mu = 0.315) allocation failure; scavenge might not succeed
[1:0x739912b6a650] 96510117 ms: Mark-Compact 258.2 (283.1) -> 257.7 (291.1) MB, 100.70 / 0.00 ms  (average mu = 0.663, current mu = 0.169) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

<--- Last few GCs --->

[1:0x739912b6a650] 96509996 ms: Mark-Compact 254.7 (283.1) -> 254.2 (283.1) MB, 83.91 / 0.00 ms  (average mu = 0.803, current mu = 0.315) allocation failure; scavenge might not succeed
[1:0x739912b6a650] 96510117 ms: Mark-Compact 258.2 (283.1) -> 257.7 (291.1) MB, 100.70 / 0.00 ms  (average mu = 0.663, current mu = 0.169) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

message.txt7.2KB

Memory (RAM) issues

I have a workflow where one step is to enrich every page on a website (sometimes 600 or more) with wider business context information, then after that, a

forEach

forEach

export const postOnboardingWorkflow = createWorkflow({
  id: "post-onboarding-workflow",
  description: "Rank, deduplicate, and research keywords with rankings-first approach",
  inputSchema: PostOnboardingWorkflowInputSchema,
  outputSchema: PostOnboardingWorkflowOutputSchema,
})
  .then(pageRankingStep)
  .then(pageDeduplicationStep)
  .then(enrichPagesWithContextStep)processing
  .foreach(singlePageKeywordStep, { concurrency: RANKING_KEYWORDS_CONFIG.MAX_CONCURRENCY })
  .then(keywordCannicalizationCleanupStep)
  .commit();

export const postOnboardingWorkflow = createWorkflow({
  id: "post-onboarding-workflow",
  description: "Rank, deduplicate, and research keywords with rankings-first approach",
  inputSchema: PostOnboardingWorkflowInputSchema,
  outputSchema: PostOnboardingWorkflowOutputSchema,
})
  .then(pageRankingStep)
  .then(pageDeduplicationStep)
  .then(enrichPagesWithContextStep)processing
  .foreach(singlePageKeywordStep, { concurrency: RANKING_KEYWORDS_CONFIG.MAX_CONCURRENCY })
  .then(keywordCannicalizationCleanupStep)
  .commit();

(

MAX_CONCURRENCY

MAX_CONCURRENCY

and it still does this)

The key part from the attached file:

<--- Last few GCs --->

[1:0x739912b6a650] 96509996 ms: Mark-Compact 254.7 (283.1) -> 254.2 (283.1) MB, 83.91 / 0.00 ms  (average mu = 0.803, current mu = 0.315) allocation failure; scavenge might not succeed
[1:0x739912b6a650] 96510117 ms: Mark-Compact 258.2 (283.1) -> 257.7 (291.1) MB, 100.70 / 0.00 ms  (average mu = 0.663, current mu = 0.169) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

<--- Last few GCs --->

[1:0x739912b6a650] 96509996 ms: Mark-Compact 254.7 (283.1) -> 254.2 (283.1) MB, 83.91 / 0.00 ms  (average mu = 0.803, current mu = 0.315) allocation failure; scavenge might not succeed
[1:0x739912b6a650] 96510117 ms: Mark-Compact 258.2 (283.1) -> 257.7 (291.1) MB, 100.70 / 0.00 ms  (average mu = 0.663, current mu = 0.169) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

message.txt7.2KB

Memory (RAM) issues

Memory (RAM) issues

Similar Threads

Similar Threads

Similar Threads