MeiliSearch: Answers aren't found reliably

[heubeck]

Hey team,

thank you for providing us with the MeiliSearch plugin.

It's working well when searching for text or title of a question, but there are flaws in finding text in answers, haven't identified the pattern yet.

For instance, the answer in this question:

cannot be found:

where others can be:

Furthermore, it seems, that, when searching for the first word of an answer text, it can never be found.

No idea, how to provide you with more insights, the MeiliSearch log seems unspectacular (http 202 when creating/updating posts, http 200 on searches).

Running the latest answer:all-in-one image, and the getmeili/meilisearch:v1.3.0 image.

[LinkinStars]

@heubeck Thanks for the feedback. Let me check first. Can you pull the latest answer:all-in-one image again and then resave the meilisearch plugin configuration in the backend admin page? Since there was a recent update, we'd like to be sure to exclude if the most recent update fixes the issue first.

[heubeck]

Thx, @LinkinStars,

re-tested with the latest answer:all-in-one image, and also created a new meilisearch index.

The issues persist, and

Furthermore, it seems, that, when searching for the first word of an answer text, it can never be found.

seems not to be limited to answers, also searching for the first word of a question doesn't find it.

[LinkinStars]

@heubeck Got it.

[sivdead]

Thx, @LinkinStars,

re-tested with the latest answer:all-in-one image, and also created a new meilisearch index.

The issues persist, and

Furthermore, it seems, that, when searching for the first word of an answer text, it can never be found.

seems not to be limited to answers, also searching for the first word of a question doesn't find it.

hi, would you try to search directly in meilisearch's own admin page, see if it can be searched?

[heubeck]

hi, would you try to search directly in meilisearch's own admin page, see if it can be searched?

thx @sivdead - didn't know about this UI ;)

It shows the issue very well:
meilisearch doesn't find "word parts", and because there's always a <p> element at the start of each post, the first or last word of a question or answer cannot be found.
same applies for different other html tags used in the raw text:

[sivdead]

hi, would you try to search directly in meilisearch's own admin page, see if it can be searched?

thx @sivdead - didn't know about this UI ;)

It shows the issue very well: meilisearch doesn't find "word parts", and because there's always a <p> element at the start of each post, the first or last word of a question or answer cannot be found. same applies for different other html tags used in the raw text:

maybe the search plugin should remove all html tags,use only pure words to index, or just use markdown? @LinkinStars does elasticsearch plugin have the same problem?

[heubeck]

There are some discussions and issues around html content in the meilisearch github project.
common sense: insert only, what you'd like to search... because of what we found, but also that html tags itself should not necessarily be searchable (searching for <p> finds everything ;))

are there libraries out there that reliable remove all formatting and styling elements from html content?

[LinkinStars]

hi, would you try to search directly in meilisearch's own admin page, see if it can be searched?

thx @sivdead - didn't know about this UI ;)
It shows the issue very well: meilisearch doesn't find "word parts", and because there's always a <p> element at the start of each post, the first or last word of a question or answer cannot be found. same applies for different other html tags used in the raw text:

maybe the search plugin should remove all html tags,use only pure words to index, or just use markdown? @LinkinStars does elasticsearch plugin have the same problem?

@sivdead @heubeck

Sorry for responding to this issue so late. We have been discussing a suitable solution to this problem. Here is the reason for this issue.

Answer itself will give the content of the Q&A to the plugin to process, now the content that Answer gives to the plugin is parsed HTML format content. Therefore, after the split words are used to build the inverted index, the search engine will treat the tag as a keyword that can be searched for such as <p>. As you said, this kind of problem occurs in all search plugins.

However, the problem is that Answer currently has only two types of data, one is markdown text and the other is html. If it is passed as markdown text, then it will result in a search for "#" which will return all the results.

So, we need some other solutions to filter out HTML tags in Answer in a reasonable way, and to keep the blocks of code in markdown that the user would normally type in. This is a bit complicated to implement, and we are still discussing the solution.

[heubeck]

Thx for the feedback.
Yes, that's indeed challenging.
I assume, Answers build-in search is also affected, as querying the raw_text fields.

[heubeck]

found this for html: https://github.com/microcosm-cc/bluemonday