Can't process PGA due to nested folder structure

[eiso]

Friction log for extracting all Python UASTs with Gitbase from a subset of PGA

pga list -l python -f json | head -n 10 | jq -r '.sivaFilenames[]' | pga get -i -o repositories

docker run -d --name bblfshd --privileged -p 9432:9432 -v /var/lib/bblfshd:/var/lib/bblfshd bblfsh/bblfshd

docker exec bblfshd bblfshctl driver install python docker://bblfsh/python-driver:v2.2.1

docker run -d --rm --name pilosa -p 10101:10101 pilosa/pilosa:v0.9.0

docker run -it --rm --name gitbase -p 3306:3306 --link pilosa:pilosa --link bblfshd -e PILOSA_ENDPOINT="http://pilosa:10101" -e BBLFSH_ENDPOINT="bblfshd:9432" -v $(pwd)/repositories/siva/latest:/opt/repos srcd/gitbase:latest

Errors
DEBU[0000] git repositories directory added              dir=/opt/repos
ERRO[0000] repository could not be added                 error="the repository could not be opened: /opt/repos/28: repository does not exist" id=28 path=/opt/repos/28
ERRO[0000] repository could not be added                 error="the repository could not be opened: /opt/repos/36: repository does not exist" id=36 path=/opt/repos/36
ERRO[0000] repository could not be added                 error="the repository could not be opened: /opt/repos/5b: repository does not exist" id=5b path=/opt/repos/5b
ERRO[0000] repository could not be added                 error="the repository could not be opened: /opt/repos/63: repository does not exist" id=63 path=/opt/repos/63
ERRO[0000] repository could not be added                 error="the repository could not be opened: /opt/repos/7c: repository does not exist" id=7c path=/opt/repos/7c
ERRO[0000] repository could not be added                 error="the repository could not be opened: /opt/repos/7f: repository does not exist" id=7f path=/opt/repos/7f
ERRO[0000] repository could not be added                 error="the repository could not be opened: /opt/repos/84: repository does not exist" id=84 path=/opt/repos/84
ERRO[0000] repository could not be added                 error="the repository could not be opened: /opt/repos/b7: repository does not exist" id=b7 path=/opt/repos/b7
ERRO[0000] repository could not be added                 error="the repository could not be opened: /opt/repos/ec: repository does not exist" id=ec path=/opt/repos/ec
ERRO[0000] repository could not be added                 error="the repository could not be opened: /opt/repos/ef: repository does not exist" id=ef path=/opt/repos/ef
ERRO[0000] repository could not be added                 error="the repository could not be opened: /opt/repos/f5: repository does not exist" id=f5 path=/opt/repos/f5
ERRO[0000] repository could not be added                 error="the repository could not be opened: /opt/repos/fa: repository does not exist" id=fa path=/opt/repos/fa
DEBU[0000] registered database to catalog                db=
DEBU[0000] registered all available functions in catalog 
DEBU[0000] created index storage                        
DEBU[0000] established connection with pilosa           
DEBU[0000] registered pilosa index driver               
INFO[0000] squash tables rule is enabled                
INFO[0000] server started and listening on 0.0.0.0:3306 

[ajnavarro]

If you run gitbase using the binary, you can load siva files from folders using path expressions: --siva=/path/to/siva/files/nested/folders/**

We need to fix the Dockerfile to allow that kind of expressions.

[mcarmonaa]

I guess #416 will solve this

[erizocosmico]

Is this solved with current master? #416 was already merged

[kuba--]

@erizocosmico I'm checking ...
If it's done, I'll close the issue.

[kuba--]

It's fixed. We can update gitbase image after release a new version

[bzz]

🎉 Looking forward a new release

[kuba--]

I'm closing so far, because master should be fixed, now. If we are able to reproduce it with current master, we can re-open the issue