volumes data folder not working

[nathan-zhu]

Hi, I use official file of es 2.3

but I wannna use local data folder instead of /usr/share/elasticsearch/data folder in container,

this is my docker-compose file

elasticsearch_master:
    image: elasticsearch:latest
    command: "elasticsearch -Des.cluster.name=workagram -Des.node.master=true -Des.node.data=true"
    volumes:
      - "/opt/data/elasticsearch:/usr/share/elasticsearch/data"
    environment:
       - ES_HEAP_SIZE=512m
    ports:
      - "9200:9200"
      - "9300:9300"

and /opt/data/elasticsearch folder i give chmod 777 for it, after running docker-compose up -d,

docker ps show it running

ONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                                            NAMES
6f565f00f935        elasticsearch:latest   "/docker-entrypoint.s"   5 minutes ago       Up 5 minutes        0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   est_elasticsearch_master_1

and i can create new index there with
curl -XPUT 192.168.99.100:9200/abctest

but i check data folder there is nothing to show, what's wrong with my settings?
please help thanks.

[bhavintandel]

@nathan-zhu Data on elasticsearch is stored in /var/lib/elasticsearch

[yosifkit]

No, the data for the image is configured by the elastic.co apt package to be in /usr/share/elasticsearch/data. Which is also why the image defines this as a docker volume.

$ docker run -d --name elsa -v /home/docker/test/esdata/:/usr/share/elasticsearch/data elasticsearch:2.4 -Des.cluster.name=workagram -Des.node.master=true -Des.node.data=true
29e6b9c886aa8e2eaac1ab29c8431434eb54184215d7ac5518e21ab568c46e19
$ curl -XPUT ip-of-container:9200/abctest
$ docker exec -it elsa ls -l /usr/share/elasticsearch/data/workagram/nodes/0/indices/abctest
total 24
drwxr-sr-x 5 elasticsearch elasticsearch 4096 Oct 19 21:40 0
drwxr-sr-x 5 elasticsearch elasticsearch 4096 Oct 19 21:40 1
drwxr-sr-x 5 elasticsearch elasticsearch 4096 Oct 19 21:40 2
drwxr-sr-x 5 elasticsearch elasticsearch 4096 Oct 19 21:40 3
drwxr-sr-x 5 elasticsearch elasticsearch 4096 Oct 19 21:40 4
drwxr-sr-x 2 elasticsearch elasticsearch 4096 Oct 19 21:40 _state
$ ls -l ../test/esdata/workagram/nodes/0/indices/abctest/
total 24
drwxr-sr-x 5 105 108 4096 Oct 19 14:40 0
drwxr-sr-x 5 105 108 4096 Oct 19 14:40 1
drwxr-sr-x 5 105 108 4096 Oct 19 14:40 2
drwxr-sr-x 5 105 108 4096 Oct 19 14:40 3
drwxr-sr-x 5 105 108 4096 Oct 19 14:40 4
drwxr-sr-x 2 105 108 4096 Oct 19 14:40 _state

I think the root of the problem is the /opt/data/elasticsearch. From the IP address you used to hit elasticsearch I deduce that you are on OSX or Windows. In order to mount folders from your host to containers in the VM, they need to be from your /Users directory on OSX.

[Tjorriemorrie]

I'm having same issue trying to mount the config and data folders on 2.4:

docker run -a STDOUT -v "$PWD/config":/usr/share/elasticsearch/config -v "$PWD/esdata":/usr/share/elasticsearch/data -P elasticsearch:2

I ssh'd into the image and can see the directories mounted, but elasticsearch fails to start properly: just the log4j:WARN No appenders could be found for logger (bootstrap). error on starting the container.

I literally copied the logging.yml from the image, but it does not work. What am I missing?

[yosifkit]

@Tjorriemorrie, You may need to have a scripts/ directory in the config directory, but I am not sure. Maybe double check the spacing in the logging file you have?

[jbrinnand]

Folks - this may be a workaround but if anyone is running on a Mac, the reason why we can't see the persistent data in any subdirectory outside of /Users is because it is in the mounted directories in docker-machine. Here is how elasticsearch was started:

docker run --name els -d -v "$PWD/config:"/usr/share/elasticsearch/config -v "/usr/local/data":/usr/share/elasticsearch/data elasticsearch

I added data using the TransportClient in a test in Eclipse. But this is not necessary. You can use curl as provided above to generate the indices. Now - if you go to /usr/local/data (which must exist) on your Mac, you will see nothing! No data. However, if you start and stop the image, the data will persist. Where is this data? It appears to be in the mounted directories in docker-machine. So if you ssh into docker-machine you can see your data.

docker-machine ssh default // substitute your machine name here.
......
ls -l /usr/local/data/nodes/0/indices/X-nMLl0JSuywoVDOrL2k9g
total 0
drwxr-xr-x 5 105 108 100 Feb 2 00:40 0/
drwxr-xr-x 5 105 108 100 Feb 2 00:40 1/
drwxr-xr-x 5 105 108 100 Feb 2 00:40 2/
drwxr-xr-x 5 105 108 100 Feb 2 00:40 3/
drwxr-xr-x 5 105 108 100 Feb 2 00:40 4/
drwxr-xr-x 2 105 108 60 Feb 2 00:41 _state/

[tianon]

This image is officially deprecated in favor of upstream's images (see https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html).

For more information, please see #160, docker-library/docs#842, and docker-library/docs#945.

This image has been deprecated in favor of the official elasticsearch image provided and maintained by elastic.co. The upstream images are available to pull via docker.elastic.co/elasticsearch/elasticsearch:[version] like 5.4.2. The images found here will receive no further updates once the 5.6.0 release is available upstream. Please adjust your usage accordingly.

Elastic provides open-source support for Elasticsearch via the elastic/elasticsearch GitHub repository and the Docker image via the elastic/elasticsearch-docker GitHub repository, as well as community support via its forums.

Thanks!