Comments on: The State Of Full-Text Indexing For The Poor Man http://joseph.randomnetworks.com/archives/2004/08/09/the-state-of-full-text-indexing-for-the-poor-man/ cat /dev/random Sat, 04 Jul 2009 08:07:38 +0000 http://wordpress.org/?v=2.9-rare hourly 1 By: Peufeu http://joseph.randomnetworks.com/archives/2004/08/09/the-state-of-full-text-indexing-for-the-poor-man/comment-page-1/#comment-161206 Peufeu Thu, 19 Apr 2007 19:01:48 +0000 http://joseph.randomnetworks.com/archives/2004/08/09/the-state-of-full-text-indexing-for-the-poor-man/#comment-161206 swish-e does not support Unicode. End of story. > In tsearch2 you end up adding a field to you table and then creating an index on that field. Yes, and that is why it works much better. When you search and rank results, mysql reads the index, then has to reparse the text, whereas tsearch2 uses the canned pre-parsed tsvector in your table. Besides, since postgres compresses the text and vector, the resulting table size is the same as mysql. Postgres also does lexemization so your search results are much better. Some benchmarks for you. Read on ;) http://peufeu.free.fr/ftsbench/ swish-e does not support Unicode. End of story.

> In tsearch2 you end up adding a field to you table and then creating an index on that field.

Yes, and that is why it works much better. When you search and rank results, mysql reads the index, then has to reparse the text, whereas tsearch2 uses the canned pre-parsed tsvector in your table. Besides, since postgres compresses the text and vector, the resulting table size is the same as mysql. Postgres also does lexemization so your search results are much better.

Some benchmarks for you. Read on ;)

http://peufeu.free.fr/ftsbench/

]]> By: heri http://joseph.randomnetworks.com/archives/2004/08/09/the-state-of-full-text-indexing-for-the-poor-man/comment-page-1/#comment-40479 heri Thu, 27 Jul 2006 01:57:19 +0000 http://joseph.randomnetworks.com/archives/2004/08/09/the-state-of-full-text-indexing-for-the-poor-man/#comment-40479 i use innodb for mysql and i replicate all varchar/text data to a search index table each time there is an update or create. i think it is the best way because you get innodb and text search...until mysql release full text for innoDB i use innodb for mysql and i replicate all varchar/text data to a search index table each time there is an update or create. i think it is the best way because you get innodb and text search…until mysql release full text for innoDB

]]> By: Jian Chen http://joseph.randomnetworks.com/archives/2004/08/09/the-state-of-full-text-indexing-for-the-poor-man/comment-page-1/#comment-2069 Jian Chen Thu, 07 Jul 2005 04:58:57 +0000 http://joseph.randomnetworks.com/archives/2004/08/09/the-state-of-full-text-indexing-for-the-poor-man/#comment-2069 I am currently using Lucene plus either MySql or Postgres. However, I must say, it would be lot better if MySql could integrate CLucene into its implementation. I am currently using Lucene plus either MySql or Postgres. However, I must say, it would be lot better if MySql could integrate CLucene into its implementation.

]]> By: Simon Willison http://joseph.randomnetworks.com/archives/2004/08/09/the-state-of-full-text-indexing-for-the-poor-man/comment-page-1/#comment-44 Simon Willison Mon, 09 Aug 2004 22:14:44 +0000 http://joseph.randomnetworks.com/archives/2004/08/09/the-state-of-full-text-indexing-for-the-poor-man/#comment-44 You can import data in to swish-e by writing a script that pulls things from your database and spits them out to standard output one at a time as simple HTML files - you then pipe that straight in to the swish-e indexer. All of our sites are dynamic / database driven, but we use the swish-e indexer to build an index of our content once a day. Then we use the Python swish-e binding to query the index for serving up actual search results (there are bindings for other languages as well). You can import data in to swish-e by writing a script that pulls things from your database and spits them out to standard output one at a time as simple HTML files – you then pipe that straight in to the swish-e indexer. All of our sites are dynamic / database driven, but we use the swish-e indexer to build an index of our content once a day. Then we use the Python swish-e binding to query the index for serving up actual search results (there are bindings for other languages as well).

]]> By: Joseph Scott http://joseph.randomnetworks.com/archives/2004/08/09/the-state-of-full-text-indexing-for-the-poor-man/comment-page-1/#comment-43 Joseph Scott Mon, 09 Aug 2004 20:30:20 +0000 http://joseph.randomnetworks.com/archives/2004/08/09/the-state-of-full-text-indexing-for-the-poor-man/#comment-43 I don't remember coming across swish-e before, it certainly looks interesting. I'm not sure how exactly this would integrate into a purely dynamic database drive site though. I'm glad you mentioned this though because I've got another project at work where this will come in very handy. I don’t remember coming across swish-e before, it certainly looks interesting. I’m not sure how exactly this would integrate into a purely dynamic database drive site though.

I’m glad you mentioned this though because I’ve got another project at work where this will come in very handy.

]]> By: Simon Willison http://joseph.randomnetworks.com/archives/2004/08/09/the-state-of-full-text-indexing-for-the-poor-man/comment-page-1/#comment-42 Simon Willison Mon, 09 Aug 2004 19:54:35 +0000 http://joseph.randomnetworks.com/archives/2004/08/09/the-state-of-full-text-indexing-for-the-poor-man/#comment-42 Have you looked at swish-e? It's an open source full text indexing engine: http://www.swish-e.org/ I absolutely love it - it's what I used to build the search engine for http://www2.ljworld.com/search/ and http://www.visitlawrence.com/search/ , among others. You need to run the indexer periodically (we do it once a day) so it doesn't search your "live" data, rather a snapshot from last time you ran the indexer. Have you looked at swish-e? It’s an open source full text indexing engine: http://www.swish-e.org/

I absolutely love it – it’s what I used to build the search engine for http://www2.ljworld.com/search/ and http://www.visitlawrence.com/search/ , among others. You need to run the indexer periodically (we do it once a day) so it doesn’t search your “live” data, rather a snapshot from last time you ran the indexer.

]]>