Does Neon support full text search for non-English languages?
Are Neon users able to access the filesystem (in order to implement custom dictionaries, stemmers, synonyms, etc).
6 Replies
harsh-harlequinOP•2y ago
Looks like users cannot access the file system. How could I implement full text search for Chinese?
inland-turquoise•2y ago
Hey @Matt Luo, can you explain the apporach you're taking. I'm not familiar with this personally, and it seems like there's a few extensions designed to help.
Maybe Neon should look into supporting one of these extensions?
harsh-harlequinOP•2y ago
THanks @ShinyPokemon , I was using Meilisearch docs as insipiration of how to accomplish full text search for Chinese.
https://www.meilisearch.com/docs/learn/what_is_meilisearch/language
In two separate questions in Neon's Discord server, I asked
1) Is the PGroonga extension supported? Answer was no (not yet).
2) Is the zhparser extension supported? No reply yet
conscious-sapphire•2y ago
@Matt Luo PGroonga does not use Postgres’ storage and resource managers for permanent storage of the index, so unfortunately this is not an extension we can support with our architecture.
Regarding zhparser, I honestly don't know if we can support this extension.
I had a look at the documentation on http://www.xunsearch.com/scws/index.php and on GitHub (https://github.com/amutu/zhparser?tab=readme-ov-file), but this doesn't allow me to understand if this extension can be supported in our infra.
May I kindly ask you to raise a support case for this?
I will liaise with our engineering team to validate the feasibility of this request.
Please note that such a request will take a bit of time, but at least you will have a definitive answer.
Thanks!
GitHub
GitHub - amutu/zhparser: zhparser is a PostgreSQL extension for ful...
zhparser is a PostgreSQL extension for full-text search of Chinese language - amutu/zhparser
harsh-harlequinOP•2y ago
Thanks @Yanic - I am currently on the free plan, so I don't readily see how I can create a support case. The Neon Console support link in the sidebar has this dialog box message:
"Got questions? Reach out to our Discord Community or check our Documentation. For direct support from our team, Upgrade your plan."
Is there an alternative way to make a support case?
Anyway, thanks for this info. I am starting to think that a realistic path forward is to pay an external vendor for full text search for languages that cannot be segmented by the Neon's supported plugins. Does Neon have a recommendation for an external vendor for FTS for Chinese and such languages?
Is Vector search that same predicament as FTS? e.g. Would I need an external vendor for vector search for Chinese?
conscious-sapphire•2y ago
Ah indeed, we disabled the support case creation for free users.
It's possible for free users to raise support case by sending an email to support@neon.tech.
Alternatively, you can drop me an email and I can raise the case for you (yanic@neon.tech).
However, please note that this should be exceptional: we really intend to support our free customers over discord.
Regarding Chinese FTS, I honestly don't know which third party vendor I could recommend.
In all honesty, I have no exposure at all to chinese and never faced this situation before.
I don't know any provider which could provide such a service unfortunately.
Regarding vector search, pgvector (https://github.com/pgvector/pgvector) is the way to go.
It's fully supported and fully functional in Neon!
GitHub
GitHub - pgvector/pgvector: Open-source vector similarity search fo...
Open-source vector similarity search for Postgres. Contribute to pgvector/pgvector development by creating an account on GitHub.