Skip to content

Refactoring WIP #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 157 commits into from
May 17, 2016
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
157 commits
Select commit Hold shift + click to select a range
24d8cf3
initial sketch
alimanfoo Apr 18, 2016
68cd799
some refactoring and documentation
alimanfoo Apr 18, 2016
2e39ddd
doco
alimanfoo Apr 18, 2016
07f634b
added implementation of low-level blosc functions with simple test; c…
alimanfoo Apr 18, 2016
7b043fb
implementation of memory store; cc #21
alimanfoo Apr 18, 2016
3555c1b
tested array getitem and setitem; cc #21
alimanfoo Apr 18, 2016
07a5379
fix test runners
alimanfoo Apr 18, 2016
697ed40
fix error
alimanfoo Apr 18, 2016
231b1b9
directory store WIP
alimanfoo Apr 19, 2016
bce6c28
work on directory store
alimanfoo Apr 19, 2016
f7a791a
add array creation functions
alimanfoo Apr 19, 2016
c9a4b1c
implement resize and append
alimanfoo Apr 19, 2016
1d0b5f5
implement repr
alimanfoo Apr 20, 2016
5a0cf9e
logging
alimanfoo Apr 20, 2016
6c90554
fix blosc init
alimanfoo Apr 20, 2016
2403b27
initial sync implementation
alimanfoo Apr 20, 2016
4458268
synchronized array resize and append; synchronize attributes
alimanfoo Apr 20, 2016
c90e254
cache configuration metadata
alimanfoo Apr 20, 2016
207fd93
PY2 compatibility; update README
alimanfoo Apr 20, 2016
2a44cee
implement persistent attributes
alimanfoo Apr 20, 2016
3f7ffc4
refactor persistent metadata
alimanfoo Apr 20, 2016
090c7e7
rerun benchmark notebook to verify dask fix
alimanfoo Apr 20, 2016
4b8e32a
add zarr.store to packages in setup.py
mrocklin Apr 21, 2016
62a172f
Add DictStore
mrocklin Apr 21, 2016
cee5e56
Merge pull request #23 from mrocklin/refactor-2
alimanfoo Apr 22, 2016
aaf2944
store refactoring WIP
alimanfoo Apr 22, 2016
55ca1da
some refactoring
alimanfoo Apr 22, 2016
c622132
rename zarr.array to zarr.core to avoid name clash
alimanfoo Apr 22, 2016
11c9238
simplify; handle read-only stores
alimanfoo Apr 22, 2016
e768325
fix package imports
alimanfoo Apr 22, 2016
b1bfc8c
get tests working again
alimanfoo Apr 22, 2016
ba4784f
doco; fix setup
alimanfoo Apr 22, 2016
20af669
fix doctests
alimanfoo Apr 22, 2016
1e46f02
fix tests
alimanfoo Apr 22, 2016
e488b84
Add meta.dumps/loads functions
mrocklin Apr 22, 2016
43b6da6
Store meta and attrs in data
mrocklin Apr 22, 2016
d4453f1
flush metadata on resize
mrocklin Apr 26, 2016
38b9209
pull compression out to separate dict
mrocklin Apr 26, 2016
4ab998e
Merge pull request #24 from mrocklin/refactor-3
alimanfoo Apr 27, 2016
1f03c68
WIP refactoring storage
alimanfoo Apr 27, 2016
b11b561
refactoring of storage and compression
alimanfoo Apr 27, 2016
3e87849
clean up legacy; make tests work
alimanfoo Apr 27, 2016
5d1b2eb
minor argument handling
alimanfoo Apr 27, 2016
aada795
add store kwarg to creation funcs
alimanfoo Apr 28, 2016
b12cd54
use clear()
alimanfoo Apr 28, 2016
03430be
add coverage
alimanfoo Apr 28, 2016
e48761f
added bz2 compression
alimanfoo Apr 28, 2016
f7dc49c
add lzma filter
alimanfoo Apr 28, 2016
b9a385c
add lzma compression
alimanfoo Apr 28, 2016
f6c7872
Merge branch 'refactor' of github.com:alimanfoo/zarr into refactor
alimanfoo Apr 28, 2016
bd8ca09
lzma not available in PY2
alimanfoo Apr 28, 2016
cb44318
attribute tests
alimanfoo Apr 28, 2016
6e9cd20
compression tests
alimanfoo Apr 28, 2016
e415079
flake8
alimanfoo Apr 28, 2016
48049ad
tweak tox
alimanfoo Apr 28, 2016
843bcf1
redundant
alimanfoo Apr 28, 2016
2c21cb8
WIP improve test coverage in zarr.core
alimanfoo Apr 29, 2016
61b6382
add tests on zarr.core
alimanfoo Apr 29, 2016
5366f76
fix tox for PY2
alimanfoo Apr 29, 2016
9a20326
tweak test setup
alimanfoo Apr 29, 2016
86714a8
fix test coverage; surface zarr.create
alimanfoo Apr 29, 2016
287e783
avoide name clash
alimanfoo Apr 29, 2016
877d3e1
tests for zarr.creation
alimanfoo Apr 29, 2016
ba5d159
mapping tests
alimanfoo Apr 29, 2016
68782c3
WIP meta tests
alimanfoo Apr 29, 2016
a039437
meta tests
alimanfoo Apr 29, 2016
30f4d10
fix test errors in docs
alimanfoo Apr 29, 2016
fbd3bb3
tests for zarr.util; coverage to 100%
alimanfoo Apr 30, 2016
e8c3791
add pickle tests; ensure arrays can be pickled/unpickled
alimanfoo Apr 30, 2016
6454f5c
resolves #7 adding support for Fortran order within chunks
alimanfoo May 1, 2016
1e879a3
support numpy arrays in *_like
alimanfoo May 2, 2016
a711757
WIP documentation
alimanfoo May 2, 2016
f3210cd
add docs to tox; fox socks box knox.
alimanfoo May 2, 2016
f18c78a
rtfd integration
alimanfoo May 2, 2016
df8f465
get rtfd working
alimanfoo May 2, 2016
3bd6577
unmock
alimanfoo May 2, 2016
c73238a
make blosc import optional
alimanfoo May 2, 2016
3bce30d
add status note
alimanfoo May 2, 2016
2b134c7
fix tests
alimanfoo May 2, 2016
398e670
rework __getitem__ and __setitem__ selection machinery for partial sl…
alimanfoo May 3, 2016
eb29fd0
doco
alimanfoo May 3, 2016
8eaf865
fix default compression selection
alimanfoo May 3, 2016
176e895
document compressors
alimanfoo May 3, 2016
950b1b4
document storage
alimanfoo May 3, 2016
7fd64d9
document storage, some refactoring
alimanfoo May 3, 2016
731a07a
get rtfd working?
alimanfoo May 3, 2016
2db75f2
work on tutorial
alimanfoo May 3, 2016
f69ccd4
tutorial doc resize append
alimanfoo May 3, 2016
c935500
document compression
alimanfoo May 3, 2016
f96c2a8
finish tutorial first draft
alimanfoo May 3, 2016
a5a7415
storage spec first draft complete
alimanfoo May 4, 2016
c2cd288
refactor sync classes; finish docs
alimanfoo May 4, 2016
b1b2558
minor doc tweak
alimanfoo May 4, 2016
d0332da
update zip file example
alimanfoo May 4, 2016
c292588
minor zipfile change
alimanfoo May 4, 2016
9d269aa
clarify order and chunks
alimanfoo May 4, 2016
dd9d8a4
benchmarking
alimanfoo May 4, 2016
803e598
implement __len__
alimanfoo May 4, 2016
549dc89
more profiling
alimanfoo May 4, 2016
da0fe79
update release instructions
alimanfoo May 4, 2016
ed5dbfe
minimise array state for pickling/unpickling
alimanfoo May 5, 2016
11299fa
thrash about with setup.py
alimanfoo May 5, 2016
de09a1a
doco tweak
alimanfoo May 5, 2016
58f10ac
release instructions
alimanfoo May 5, 2016
31aae66
correct manifest
alimanfoo May 5, 2016
9c66bc1
msg tweak
alimanfoo May 5, 2016
3a31515
add Array.attrs to docs
alimanfoo May 6, 2016
52e460f
tweak logging
alimanfoo May 6, 2016
8177101
workaround rtfd issues
alimanfoo May 6, 2016
33e4ca5
resolves #27 by auto switching blosc behaviour
alimanfoo May 8, 2016
089ce2b
fix memory leak
alimanfoo May 8, 2016
687a15f
update notebooks after blosc api change
alimanfoo May 8, 2016
87a192c
clean up C extension
alimanfoo May 8, 2016
fc9ba17
consistify variable names
alimanfoo May 8, 2016
7dd018c
decomment
alimanfoo May 8, 2016
6980e11
rebuild c extension
alimanfoo May 8, 2016
c8ba8ae
profiling
alimanfoo May 9, 2016
761358e
revert line trace; add simple bench
alimanfoo May 9, 2016
dedc5de
use "none" for no compression
alimanfoo May 9, 2016
325df56
rework the blosc extension to use the buffer protocol and avoid an un…
alimanfoo May 9, 2016
059e525
profile
alimanfoo May 9, 2016
bafab2a
use array module to avoid memory copies
alimanfoo May 9, 2016
41af4eb
clean up blosc extension; remove numpy build requirement
alimanfoo May 10, 2016
bacae68
delete old tests; adapt zict.Zip test
alimanfoo May 10, 2016
b30b5a6
fix PY2 compat; workaround zict.Zip example
alimanfoo May 10, 2016
ea9e9d7
bench results
alimanfoo May 10, 2016
0ef7595
numpy is no longer a build dependency
alimanfoo May 10, 2016
4bac0ec
minor doco edits
alimanfoo May 10, 2016
3479ecd
add release notes to docs
alimanfoo May 11, 2016
6edd945
typo
alimanfoo May 11, 2016
864d80a
doco
alimanfoo May 12, 2016
4682c7d
minor doco
alimanfoo May 12, 2016
4f0df75
ensure bytes for blosc name
mrocklin May 12, 2016
79b8f71
Merge pull request #28 from mrocklin/refactor-5
alimanfoo May 12, 2016
46721d1
Revert "Merge pull request #28 from mrocklin/refactor-5"
alimanfoo May 12, 2016
3496462
compile blosc extension on windows
alimanfoo May 14, 2016
41a7123
fixes to get tests passing on windows
alimanfoo May 14, 2016
2fcb54d
add dtype to docstring examples
alimanfoo May 14, 2016
4ba0aec
appveyor config
alimanfoo May 14, 2016
8cef5f2
add missing files for appveyor
alimanfoo May 14, 2016
2a8727a
add numpy
alimanfoo May 14, 2016
e0a1788
fix doctests
alimanfoo May 14, 2016
61b9a51
remove unused import
alimanfoo May 14, 2016
ed176cc
make sure appveyor inits submodules
alimanfoo May 14, 2016
da4edba
try simpler appveyor config
alimanfoo May 16, 2016
ca1c3e9
init submodules
alimanfoo May 16, 2016
b017522
fix build
alimanfoo May 16, 2016
7ce2f65
fix appveyor environment setup
alimanfoo May 16, 2016
19f78be
more appveyor fixes
alimanfoo May 16, 2016
939de78
appveyor tweak 1
alimanfoo May 16, 2016
9a89a78
appveyor fix 2
alimanfoo May 16, 2016
f3e84f9
inline msvc
alimanfoo May 16, 2016
76b20ef
try out whole appveyor matrix
alimanfoo May 16, 2016
aaa935b
appveyor fixes for py35
alimanfoo May 16, 2016
4fd1757
build wheels
alimanfoo May 16, 2016
3460242
tweak install instructions
alimanfoo May 16, 2016
5adbe30
upgrade c-blosc to 1.9.0
alimanfoo May 16, 2016
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ install:
- python setup.py build_ext --inplace

script:
- nosetests -v
- nosetests -v zarr
4 changes: 2 additions & 2 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,8 @@

# define extension module
ext_modules = cythonize([
Extension('zarr.ext',
sources=['zarr/ext.pyx'] + blosc_sources,
Extension('zarr.blosc',
sources=['zarr/blosc.pyx'] + blosc_sources,
include_dirs=include_dirs,
define_macros=define_macros,
extra_compile_args=extra_compile_args,
Expand Down
2 changes: 1 addition & 1 deletion tox.ini
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ setenv =
PYTHONHASHSEED = 42
commands =
python setup.py build_ext --inplace
nosetests -v
nosetests -v zarr
py34: python -m doctest -o NORMALIZE_WHITESPACE README.rst PERSISTENCE.rst
deps =
nose
Expand Down
23 changes: 11 additions & 12 deletions zarr/__init__.py
Original file line number Diff line number Diff line change
@@ -1,21 +1,20 @@
# -*- coding: utf-8 -*-
from __future__ import absolute_import, print_function, division


import atexit
import multiprocessing
import atexit


from zarr.create import empty, zeros, ones, full, array, open, empty_like, \
from zarr.array import Array, SynchronizedArray, set_blosc_options
from zarr.store.memory import MemoryStore
from zarr.store.directory import DirectoryStore
from zarr.sync import ThreadSynchronizer, ProcessSynchronizer
from zarr.create import array, empty, zeros, ones, full, open, empty_like, \
zeros_like, ones_like, full_like, open_like
from zarr.ext import blosc_version, init as _init, destroy as _destroy, \
set_blosc_options
from zarr import defaults
from zarr import constants
from zarr.version import version as __version__
from zarr import blosc


ncores = multiprocessing.cpu_count()
_init()
set_blosc_options(use_context=False, nthreads=ncores)
atexit.register(_destroy)
blosc.init()
# diminishing returns beyond 4 threads?
set_blosc_options(use_context=False, nthreads=min(4, ncores))
atexit.register(blosc.destroy)
Loading