How to integrate Neon with search vendors?
I'm evaluating Neon for its ability to integrate with search vendors such as Typesense. I especially need full text search for languages that don't delimit words with spaces, such as Chinese. Neon doesn't support the PGroonga extension, which seems to be the popular way to do FTS for Chinese, Japanese, and other such languages.
A concerning thing is that in Typesense Cloud documentation, the integration with Supabase is based on a cron job, and it requires these Postgres extensions, all of which are not supported by Neon.
https://typesense.org/docs/guide/supabase-full-text-search.html#enabling-relevant-postgresql-extensions
PG_NET (opens new window): allows the database to make asynchronous http/https requests with JSON
HTTP (opens new window): allows the database to make synchronous http/https requests with all data formats
PG_CRON (opens new window): gives the database the ability to double as a CRON server.
Is there any docs/guides/walkthroughs/etc that could help a Neon developer set up search through a vendor such a Typesense, Algolia, Meilisearch, etc.?
Typesense Documentation
Documentation for Typesense Search
6 Replies
plain-purpleOP•2y ago
Would this be accomplish-able once logical replication is ready? It just seems it would be financially expensive because the data would need to go to Debezium, Confluent, and then to TypeSense
The front end of my app is using Next.js, hosted on Vercel
implicit-lime•2y ago
@Matt Luo I don't know much about TypeSense, but you don't necessarily need to run the cron in the db.
1. Run a VM with a cron on Fly.io or similar hosting
2. Read records that have changed, and POST to TypeSense
You could also post them realibaly to TypeSense in real-time using a Postgres queue like Graphile worker.
plain-purpleOP•2y ago
I see, thanks @ShinyPokemon. Any future docs/blog posts/ case studies on this topic would be much appreciated.
implicit-lime•2y ago
I think you're right. I have one in the backlog for queues like Graphile and River!
Cron would be another good topic 👍
plain-purpleOP•2y ago
@ShinyPokemon - once Neon logical replication is available, would you recommend pushing data to the search vendor via logical replication or one of the methods you mentioned? I’m trying to understand if you had suggested those other methods as workaround alternatives to logical replication
implicit-lime•2y ago
Either is probably fine. Logical Replication allows it to be real-time, whereas with a cron your data isn't always fresh. Logical Replication has storage cost implications that you might prefer to avoid, especially if the cron is frequent enough for your needs.