Advice on approach for my trending feature in my website
Hey everyone! I’m working on a trending pain points feature that shows recurring posts with issues over time (today / last 7 days / last 30 days).
The plan:
/trends route displays trending pain point labels. Clicking a label shows all posts under that trend.
Backend workflow:
Normalizing post text (remove markdown, etc.)
Generating embeddings with an LLM (OpenAI text-embedding)
Cluster embeddings (using
)
Using ChatGPT to generate a suitable label for each cluster
I’m new to embeddings and clustering, so I’d love some guidance on whether this approach makes sense for production, best clustering packages (HDBSCAN, etc, ive been told ml-kmeans is for toy data so i went with
Right now, whenever new posts come in, I normalize text and save them in the DB and run a cron every 2 hours to fetch posts from the DB and run the buildTrends.js that embeds, clusters the posts and generates the labels!
Here’s the gist with relevant code
https://gist.github.com/moahnaf11/a45673625f59832af7e8288e4896feac
– includes cluster.js, embedding.js(helpers that i import into buildTrends.js), buildTrends.js, cron.js, and prisma.schema
please feel free to go through my code files and let me know if im on the right track. Ive done tons of research and this is what ive been able to come up with and im kinda scared LOL as ive never worked with embeddings and clustering before!
Any advice or pointers would be amazing!
The plan:
/trends route displays trending pain point labels. Clicking a label shows all posts under that trend.
Backend workflow:
Normalizing post text (remove markdown, etc.)
Generating embeddings with an LLM (OpenAI text-embedding)
Cluster embeddings (using
const clustering = require("density-clustering");Using ChatGPT to generate a suitable label for each cluster
I’m new to embeddings and clustering, so I’d love some guidance on whether this approach makes sense for production, best clustering packages (HDBSCAN, etc, ive been told ml-kmeans is for toy data so i went with
density-clusteringRight now, whenever new posts come in, I normalize text and save them in the DB and run a cron every 2 hours to fetch posts from the DB and run the buildTrends.js that embeds, clusters the posts and generates the labels!
Here’s the gist with relevant code
https://gist.github.com/moahnaf11/a45673625f59832af7e8288e4896feac
– includes cluster.js, embedding.js(helpers that i import into buildTrends.js), buildTrends.js, cron.js, and prisma.schema
please feel free to go through my code files and let me know if im on the right track. Ive done tons of research and this is what ive been able to come up with and im kinda scared LOL as ive never worked with embeddings and clustering before!
Any advice or pointers would be amazing!
A friendly place for developers to meet other devs, ask questions, get help, and just have a good time 🙂.
36,263Members
Resources
Was this page helpful?
