Add support for MULTISEARCH

[mlorant]

Some RFC defines new command that can improve usability of IMAP for developers. For example, the RFC 6237 defines a command ESEARCH that allows to search for messages through all mailboxes (instead of only the one currently selected).
That allows to search for all messages without iterating over each mailbox, which can be long with some people who have hundreds of folders in their account 📬

The user can still check the feature is supported through .capabilities() (and provide a fallback for servers which does not support it). I don't know if other commands like that could be implemented. Today, we need to use _raw_command and its friends, which withdraws interest in the lib :)

[mjs]

Cool I see that both Gmail and Dovecot support ESEARCH so IMAPClient certainly should too.

[mlorant]

I've dug a little into this one this afternoon... And this is not really a success for now. I've tried with three differents servers, and neither of them seem to handle ESEARCH !

I made a SEARCH and ESEARCH command each time to show the difference in the error message.

ssl0.ovh.net (OVH, one of the biggest cloud provider in Europe)

? SEARCH
? BAD Error in IMAP command SEARCH: Missing search parameters
? ESEARCH
? BAD Error in IMAP command ESEARCH: Unknown command.
? CAPABILITY
* CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS QUOTA

imap.laposte.net (post service in France with some digital services)

? SEARCH
? BAD parse error: unexpected end of line; expected ' '
? ESEARCH 
? BAD parse error: command not implemented
? CAPABILITY
* CAPABILITY IMAP4rev1 ACL BINARY CATENATE CHILDREN CONDSTORE ENABLE ESEARCH ESORT I18NLEVEL=1 ID IDLE LIST-EXTENDED LIST-STATUS LITERAL+ LOGIN-REFERRALS MULTIAPPEND NAMESPACE QRESYNC QUOTA RIGHTS=ektx SASL-IR SEARCHRES SORT THREAD=ORDEREDSUBJECT UIDPLUS UNSELECT WITHIN XLIST
? OK CAPABILITY completed

imap.gmail.com (do I need to specify this one?)

? SEARCH
? BAD Could not parse command
? ESEARCH
? BAD Unknown command: ESEARCH
? CAPABILITY
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT LIST-EXTENDED LIST-STATUS LITERAL- SPECIAL-USE APPENDLIMIT=35651584
? OK Success

Did I miss something? Do I need to activate somehow ESEARCH before using it? I tried to use ENABLE before, but the result is the same:

? OK m****.l*****@gmail.com authenticated (Success)
? ENABLE ESEARCH
* ENABLED
? OK Success
? ESEARCH ALL
? BAD Unknown command: ESEARCH

[mjs]

When a server advertises the ESEARCH capability it means that SEARCH and UID SEARCH support extra features. See here: https://tools.ietf.org/search/rfc4731

That said, to make things confusing, if the server advertises the MULTISEARCH capability then a ESEARCH command is available but this is a different thing. It's about searching multiple mailboxes at once. See here: https://tools.ietf.org/search/rfc6237. It doesn't appear to be widely supported.

In order for IMAPClient to support ESEARCH responses (of the first type) there will need to be a small change so that it expects both SEARCH or ESEARCH responses when search() is called. This shouldn't be too hard.

[mlorant]

Alright, I read too fast and missed the "A server that supports this extension includes "MULTISEARCH" in its IMAP capability string" part indeed... So all those servers handle MIN/MAX/COUNT in SEARCH syntax but none of them is supporting ESEARCH since I don't see MULTISEARCH in their capabilities list.

So your comment in December is not relevant because of this "confusing" naming, right?

[mjs]

Sorry about my previous comment. I didn't realise the difference between the 2 uses of ESEARCH until today :)

Do you know of any servers that support MULTISEARCH?

[mlorant]

I've just tried at work, where I have tests accounts on more services. Not a single of them has "MULTISEARCH" in their capabilities. The RFC has been amended in 2014 (RFC 7377) to be considered final, so the late adoption might explains why there is no implementation.

Stéphane Bortzmeyer, a French guy who loves writing articles about RFC, wrote in this article http://www.bortzmeyer.org/7377.html (translated here):

Is there any implementation of this standard? Apparently there is one, in a closed-source software at Oracle. This extension seems rare (and this was noted in IETF debates: should they pursue in invest time in (IMAP) standards?). I tried some servers and have not seen the capability MULTISEARCH in recent Zimbra, neither in Debian' Courier. Nothing in cloud services laposte.net or Gmail.

Even though it could be implemented and tested with a server, would it be valuable to maintain this if it is not really used in the real world...

[mjs]

I don't think it's worth supporting MULTISEARCH if there's no server support.

@mlorant & @NicolasLM : are you guys interested in IMAPClient supporting the ESEARCH capability? Seems useful. If so, I might open a separate ticket about that.

[mlorant]

I don't have any need of ESEARCH (yet?) personally but it should not be that hard to implement (looking quickly to the formal syntax defined in the RFC, it is just four more search options to implement in both query and response parsing).

If someone is interested to use it, it can be done easily for the 2.0.0 I guess. Plus, it is widely supported, so there's no problem in finding real use cases for this one.

[NicolasLM]

• MULTISEARCH: as there is virtually no support for it on common servers I think it's not worth supporting.
• ESEARCH: as we've seen that most servers support that, it will probably end up being useful to someone if we support it.

[mjs]

I've opened another ticket. Closing this one.