Unix socket support

[sausix]

Any chance or easy hint to get Unix sockets working for a connection?

ArangoDB does support it.

requests doesn't want to support it: Unix domain sockets #2355

Good reasons:

• Less overhead. No IP stack needed. Connections through sockets are known to be faster compared to TCP.
• More secure. The application can talk via a local socket in the file system which is not exposed as fast as a listening port by misconfiguration.
• In Docker you can switch off networking for a container stack if the only IP use is the db connection.

Thanks!

[aMahanna]

Hey @sausix!

This should be possible given that the ArangoClient constructor supports user-defined HTTP Clients, so we might be able to get away with creating a UnixSocketsHTTPClient that overrides the create_session() and send_request() methods to handle a Unix connection.

Although not natively supported in requests, you could consider the requests-unixsocket package as a workaround.

Here is what the DefaultHTTPClient in python-arango looks like, and here is what I can suggest as a draft for the UnixSocketsHTTPClient:

from typing import MutableMapping, Optional, Tuple, Union
from urllib.parse import urlparse
​
import requests_unixsocket
from requests import Session
from requests_toolbelt import MultipartEncoder
​
from arango.http import DefaultHTTPClient
from arango.response import Response
from arango.typings import Headers
​
DEFAULT_REQUEST_TIMEOUT = 60
​
​
class UnixSocketsHTTPClient(DefaultHTTPClient):
    """Unix Socket HTTP client implementation.
​
    :param unix_socket_url: Unix socket URL.
    :type unix_socket_url: str
    :param request_timeout: Timeout in seconds for each individual connection.
    :type request_timeout: int | float
    :param retry_attempts: Number of retry attempts.
    :type retry_attempts: int
    :param backoff_factor: Backoff factor for retry attempts.
    :type backoff_factor: float
    :param pool_connections: The number of urllib3 connection pools to cache.
    :type pool_connections: int
    :param pool_maxsize: The maximum number of connections to save in the pool.
    :type pool_maxsize: int
    :param pool_timeout: If set, then the pool will be set to block=True,
        and requests will block for pool_timeout seconds and raise
        EmptyPoolError if no connection is available within the time period.
    :type pool_timeout: int | float | None
    """
​
    def __init__(
        self,
        unix_socket_url: str,
        request_timeout: Union[int, float, None] = DEFAULT_REQUEST_TIMEOUT,
        retry_attempts: int = 3,
        backoff_factor: float = 1.0,
        pool_connections: int = 10,
        pool_maxsize: int = 10,
        pool_timeout: Union[int, float, None] = None,
    ) -> None:
        self.unix_socket_url = unix_socket_url
        self.request_timeout = request_timeout
        self._retry_attempts = retry_attempts
        self._backoff_factor = backoff_factor
        self._pool_connections = pool_connections
        self._pool_maxsize = pool_maxsize
        self._pool_timeout = pool_timeout
​
    def create_session(self, host: str) -> Session:
        """Create and return a new session/connection.
​
        :param host: ArangoDB host URL.
        :type host: str
        :returns: requests session object
        :rtype: requests.Session
        """
        # retry_strategy = Retry(
        #     total=self._retry_attempts,
        #     backoff_factor=self._backoff_factor,
        #     status_forcelist=[429, 500, 502, 503, 504],
        #     allowed_methods=["HEAD", "GET", "OPTIONS"],
        # )
​
        unix_adapter = requests_unixsocket.UnixAdapter(
            self.request_timeout,
            self._pool_connections,
            # connection_timeout=self.request_timeout,
            # pool_maxsize=self._pool_maxsize,
            # pool_timeout=self._pool_timeout,
            # max_retries=retry_strategy,
        )
​
        session = Session()
        session.mount("http+unix://", unix_adapter)
​
        return session
​
    def send_request(
        self,
        session: Session,
        method: str,
        url: str,
        headers: Optional[Headers] = None,
        params: Optional[MutableMapping[str, str]] = None,
        data: Union[str, MultipartEncoder, None] = None,
        auth: Optional[Tuple[str, str]] = None,
    ) -> Response:
        """Send an HTTP request.
​
        :param session: Requests session object.
        :type session: requests.Session
        :param method: HTTP method in lowercase (e.g. "post").
        :type method: str
        :param url: Request URL.
        :type url: str
        :param headers: Request headers.
        :type headers: dict
        :param params: URL (query) parameters.
        :type params: dict
        :param data: Request payload.
        :type data: str | MultipartEncoder | None
        :param auth: Username and password.
        :type auth: tuple
        :returns: HTTP response.
        :rtype: arango.response.Response
        """
        url_path = urlparse(url).path
        unix_url = f"http+unix://{self.unix_socket_url}/{url_path.lstrip('/')}"
​
        response = session.request(
            method=method,
            url=unix_url,
            params=params,
            data=data,
            headers=headers,
            auth=auth,
            timeout=self.request_timeout,
        )
        return Response(
            method=method,
            url=response.url,
            headers=response.headers,
            status_code=response.status_code,
            status_text=response.reason,
            raw_body=response.text,
        )
​
from arango import ArangoClient
​
# Instantiate the custom Unix socket HTTP client
unix_socket_http_client = UnixSocketsHTTPClient('path/to/unix/socket')
​
# Create an ArangoDB client instance with the custom HTTP client
db = ArangoClient(http_client=unix_socket_http_client).db(verify=True)

[sausix]

@aMahanna Wow. Didn't expect such a positive and detailed answer. Thank you!

I found the http_client immediately and I was already subclassing DefaultHTTPClient. I also found requests-unixsocket while digging in the issues of requests.
But I had concerns about stability. What about timeouts? What about multithreading?
An official and tested solution would be nice. One that has been tested and used by many people.
Don't get me wrong. You code will probably work and I appreciate it. But it's new and requests-unixsocket is not heavily used too.

I know it's complicated for this project by mostly being dependent of the requests package.
That's basically ok because I trust requests to be grown up and stable. python-arango simply uses requests without any hacky ways so I trust it too. And python-arango has probably a lot of use cases in the world already.

If just requests would allow Unix sockets we had no problems here and other packages would benefit too.

Using Unix sockets is just not common or well known enough. What about making it official for python-arango?
For testing and contributing to the community, I would be in!

[apetenchea]

Hi @sausix,

Our primary aim with this driver is to maintain platform independence to the greatest extent possible. While Unix domain sockets offer advantages in terms of efficiency for inter-process communication, they are inherently platform-specific. Adding them would introduce a feature that is not natively supported on Windows environments.
The fact that Unix domain sockets are designed for inter-process communication on the same host makes it a relatively niche requirement for our driver. Not saying it's not useful (could be great for testing, for example), but it's way more common for the driver and the database server to be running on different hosts.
In our experience, very specific implementations that are not widely used by a significant portion of our user base can become dusty and challenging to maintain over time ("rotting code").

Don't get me wrong, I'm all for people using Unix domain sockets on top of our driver. We strive to make it as customizable and flexible as possible. I believe the approach suggested by @aMahanna , which involves extending the driver for specific needs, is the most practical solution. This way, we can keep the core driver platform-agnostic while still providing the means for users like yourself to implement more specific functionalities.

I would tend towards closing this as "Won't fix", but if you have any further thoughts or ideas, please feel free to share them with us.

[sausix]

Hey @apetenchea

I fully agree with your arguments. Except running a database should imply it's primarily run on Linux hosts. Maybe there's a statistics? My personal use case will be Docker.
Don't want to argue anymore because I basically fully understand and agree to not break or fork for platform compatibility at too much cost.

I would totally be fine if we close this issue but at least keep @aMahanna 's wonderful snippet anywhere. Maybe here in the docs: HTTP Clients?
Could be marked as experimental and unmaintained. Using sockets would be a good real world use case for an HTTPClient implementation.
I would definitely share my experience by pull requests if Unix sockets would be a seperate docs page for example.

Thank you.

[apetenchea]

Definitely, we should keep @aMahanna's example! It's already available on this issue, which is going to be indexed by search engines, since the project is public. People will be able to get here through a search.

The HTTP Clients page already contains a generic example. We should keep it as simple as possible, therefore I would be hesitant to add the UnixSocketsHTTPClient on top of that.

Nevertheless, you are more than welcome to share your expertise with our community. I'll be moving this thread into a "Show and tell" discussion here.
Please feel free share more info, gists or any relevant code snippets. Or, if you have a blog post where you wrote in more detail about Unix sockets in the context of ArangoDB, please do share it. All contributions are much appreciated!