Skip to content
This repository was archived by the owner on Sep 26, 2022. It is now read-only.

Commit e820da6

Browse files
HunterAP23HunterAP23
committed
Major refactor / cleanup
1 parent 683319a commit e820da6

File tree

84 files changed

+699
-608
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

84 files changed

+699
-608
lines changed

‎README.md

+55-172
Original file line numberDiff line numberDiff line change
@@ -1,35 +1,9 @@
1-
# Cython_test_multiprocess
1+
# Python Performance Optimization Test Suite
22
Runtime comparison of different implementations of calculating Prime numbers.
33

44
- [Setup](#setup)
55
- [Tests](#tests)
6-
- [Default Test](#default-test)
7-
- [N / 2 Bounding Test](#half-of-n-bounding-test)
8-
- [sqrt(N) Bounding Test](#square-root-of-n-bounding-test)
9-
- [Implementations](#implementations)
10-
- [CPython Variants](#cpython-variants)
11-
- [Default through function calls](#cpython-through-functions)
12-
- [Default inline](#cpython-inline)
13-
- [Lambdas within function calls](#cpython-with-lambdas-through-functions)
14-
- [Lambdas inline](#cpython-with-lambdas-inline)
15-
- [Function Calls with LRU Caching](#cpython-with-lru-caching)
16-
- [Numpy Arrays](#cpython-with-numpy-arrays)
17-
- [Numpy Arrays & Lambdas](#cpython-with-numpy-arrays-and-lambdas)
18-
- [Numpy Arrays & LRU Caching](#cpython-with-numpy-arrays-and-lru-caching)
19-
- [Cython](#cpython-cython)
20-
- [Cython w/ Lambdas](#cpython-cython-with-lambdas)
21-
- [Cython w/ LRU Caching](#cpython-cython-with-lru-caching)
22-
- [Cython w/ Numpy Arrays](#cpython-cython-with-numpy-arrays)
23-
- [Cython w/ Numpy Arrays & Lambdas](#cpython-cython-with-numpy-arrays-and-lambdas)
24-
- [Cython w/ Numpy Arrays & LRU Caching](#cpython-cython-with-numpy-arrays-and-lru-caching)
25-
- [Cython Optimized](#cpython-optimized-cython)
26-
- [Cython Optimized w/ Lambdas](#cpython-optimized-cython-with-lambdas)
27-
- [Cython Optimized w/ LRU Caching](#cpython-optimized-cython-with-lru-caching)
28-
- [Cython Optimized w/ Numpy Arrays](#cpython-optimized-cython-with-numpy-arrays)
29-
- [Cython Optimized w/ Numpy Arrays & Lambdas](#cpython-optimized-cython-with-numpy-arrays-and-lambdas)
30-
- [Cython Optimized w/ Numpy Arrays & LRU Caching](#cpython-optimized-cython-with-numpy-arrays-and-lru-caching)
31-
- [Anaconda Variants](#anaconda-variants)
32-
- [PyPy Variants](#pypy-variants)
6+
- [Optimizations](#optimizations)
337

348
## Setup
359
1. Download the newest version of PyPy (skip if you're not planning on using PyPy).
@@ -49,150 +23,59 @@ the different runtimes as needed.
4923
which will use Python 3.7 and install cython and numpy automatically.
5024

5125
## Tests
52-
Each one of these programs has three different versions of calculating primes.<br>
53-
Using variable `N` as the number being tested for primality:<br>
54-
For prime numbers, we already know the following:
55-
* All prime numbers are odd numbers, except for 2.
56-
* All even numbers are divisible by 2, and therefore not prime.
57-
* We only need to start with 2 as the first prime number, then loop from 3
58-
to `N`, incrementing by 2 for each iteration to test only odd numbers.
59-
60-
#### Default Test
61-
* Check divisibility of `N` by each number less than `N`.
62-
63-
#### Half of N Bounding Test
64-
* Check divisibility of `N` by each number less than `math.floor((N / 2))`:
65-
* This method should theoretically be 2x as fast as the
66-
[default test method](#default).
67-
68-
#### Square Root of N Bounding Test
69-
* Check divisibility of `N` by each number less than `math.floor(sqrt(N))`:
70-
* This method should theoretically be log(N) as fast as the
71-
[default test method](#default).
72-
73-
# Implementations
74-
### CPython Variants
75-
#### CPython through Functions
76-
* Defined in [Find_Nth_Prime_Python.py](Find_Nth_Prime_Python.py)
77-
* Loops over all numbers from `0` to `max_num` as variable `N`, calls
78-
the test method functions for each of those numbers.
79-
* Since we're calling the each test method functions `N` times each, this means
80-
we have function call overhead of `max_num` calls per test method.
81-
82-
#### CPython Inline
83-
* Defined in [Find_Nth_Prime_Python_Inline.py](Find_Nth_Prime_Python_Inline.py)
84-
* Loops over all numbers from `0` to `max_num` as variable `N`, calls
85-
the test method functions for each of those numbers.
86-
* Since we're calling the each test method functions `N` times each, this means
87-
we have function call overhead of `max_num` calls per test method.
88-
89-
#### CPython through Functions with Lambdas
90-
* Defined in [Find_Nth_Prime_Python_Lambda.py](Find_Nth_Prime_Python_Lambda.py)
91-
* Only calls the calculation function once with a `max_num` as a parameter.
92-
* The calculation function handles all the logic that `Find_Nth_Prime_Python.py`'s the calculation functions have, but all using a single lambda function inside the calculation function.
93-
* This reduces the function call overhead, which can lead to significant time
94-
savings.
95-
96-
#### CPython through Functions with Lambdas
97-
* Defined in [Find_Nth_Prime_Python_Inline_Lambda.py](Find_Nth_Prime_Python_Inline_Lambda.py)
98-
* Only calls the calculation function once with a `max_num` as a parameter.
99-
* The calculation function handles all the logic that `Find_Nth_Prime_Python.py`'s the calculation functions have, but all using a single lambda function inside the calculation function.
100-
* This reduces the function call overhead, which can lead to significant time
101-
savings.
102-
103-
#### CPython with LRU Caching
104-
* Defined in [Find_Nth_Prime_Python_LRU.py](Find_Nth_Prime_Python_LRU.py)
105-
* Similar to the [Python](#cpython) in that the
106-
test method functions are each called `max_num` amount of times, but this
107-
implementation caches the results of previous function calls to speed up the
108-
calculation of the current function call.
109-
* Despite the supposed function call overhead
110-
111-
#### CPython with Numpy Arrays
112-
* Defined in [Find_Nth_Prime_Python_Numpy.py](Find_Nth_Prime_Python_Numpy.py)
113-
* Same as [Python](#cpython), but uses Numpy arrays instead of Python lists.
114-
115-
#### CPython with Numpy Arrays and Lambdas
116-
* Defined in [Find_Nth_Prime_Python_Numpy_Lambdas.py](Find_Nth_Prime_Python_Numpy_Lambdas.py)
117-
* Same as [Python with Lambdas](#cpython-with-lambdas), but uses Numpy arrays
118-
instead of Python lists.
119-
120-
#### CPython with Numpy Arrays and LRU Caching
121-
* Defined in [Find_Nth_Prime_Python_Numpy_LRU.py](Find_Nth_Prime_Python_Numpy_LRU.py)
122-
* Same as [Python with Lambdas](#cpython-with-lambdas), but uses Numpy arrays
123-
instead of Python lists.
124-
125-
#### CPython Cython
126-
* Defined in [Find_Nth_Prime_Cython.pyx](Find_Nth_Prime_Cython.pyx)
127-
* Same as [Python](#cpython), but compiled using Cython.
128-
129-
#### CPython Cython with Lambdas
130-
* Defined in [Find_Nth_Prime_Cython_Lambda.pyx](Find_Nth_Prime_Cython_Lambda.pyx)
131-
* Same as [Python with Lambdas](#cpython-with-lambdas), but compiled using Cython.
132-
133-
#### CPython Cython with LRU Caching
134-
* Defined in [Find_Nth_Prime_Cython_LRU.pyx](Find_Nth_Prime_Cython_LRU.pyx)
135-
* Same as [Python with LRU Caching](#cpython-with-lru-caching), but compiled using Cython.
136-
137-
#### CPython Cython with Numpy Arrays
138-
* Defined in [Find_Nth_Prime_Cython_Numpy.pyx](Find_Nth_Prime_Cython_Numpy.pyx)
139-
* Same as [Cython](#cython), but uses Numpy arrays instead of Python lists.
140-
141-
#### CPython Cython with Numpy Arrays and Lambdas
142-
* Defined in [Find_Nth_Prime_Cython_Numpy_Lambda.pyx](Find_Nth_Prime_Cython_Numpy_Lambda.pyx)
143-
* Same as [Cython with Lambdas](#cython-with-lambdas), but uses Numpy arrays
144-
instead of Python lists.
145-
146-
#### CPython Cython with Numpy Arrays and LRU Caching
147-
* Defined in [Find_Nth_Prime_Cython_Numpy_LRU.pyx](Find_Nth_Prime_Cython_Numpy_LRU.pyx)
148-
* Same as [Cython with Lambdas](#cython-with-lambdas), but uses Numpy arrays
149-
instead of Python lists.
150-
151-
#### CPython Optimized Cython
152-
* Defined in [Find_Nth_Prime_Optimized.pyx](Find_Nth_Prime_Optimized.pyx)
153-
* Uses manual static typing for variables for slightly better performance.
154-
155-
#### CPython Optimized Cython with Lambdas
156-
* Defined in [Find_Nth_Prime_Optimized_Lambda.pyx](Find_Nth_Prime_Optimized_Lambda.pyx)
157-
* Uses manual static typing for variables for slightly better performance.
158-
159-
#### CPython Optimized Cython with LRU Caching
160-
* Defined in [Find_Nth_Prime_Optimized_LRU.pyx](Find_Nth_Prime_Optimized_LRU.pyx)
161-
* Uses manual static typing for variables for slightly better performance.
162-
163-
#### CPython Optimized Cython with Numpy Arrays
164-
* Defined in [Find_Nth_Prime_Optimized_Numpy.pyx](Find_Nth_Prime_Optimized_Numpy.pyx)
165-
* Uses manual static typing for variables for slightly better performance.
166-
167-
#### CPython Optimized Cython with Numpy Arrays and Lambdas
168-
* Defined in [Find_Nth_Prime_Optimized_Numpy_Lambda.pyx](Find_Nth_Prime_Optimized_Numpy_Lambda.pyx)
169-
* Uses manual static typing for variables for slightly better performance.
170-
171-
#### CPython Optimized Cython with Numpy Arrays and LRU Caching
172-
* Defined in [Find_Nth_Prime_Optimized_Numpy_LRU.pyx](Find_Nth_Prime_Optimized_Numpy_LRU.pyx)
173-
* Uses manual static typing for variables for slightly better performance.
174-
175-
### Anaconda Variants
176-
There are no differences in the code between these implementations and the CPython ones.
177-
The only difference is that the exact same CPython tests were run with the
178-
Anaconda Python runtime rather than the default CPython runtime.
179-
* <b>Anaconda</b>: [CPython](#cpython)
180-
* <b>Anaconda with Lambdas</b>: [CPython w/ Lambdas](#cpython-with-lambdas),
181-
* <b>Anaconda with LRU Caching</b>: [CPython w/ LRU Caching](#cpython-with-lru-caching)
182-
* <b>Anaconda with Numpy Arrays</b>: [CPython w/ Numpy Arrays](#cpython-with-numpy-arrays)
183-
* <b>Anaconda with Numpy Arrays and Lambdas</b>: [CPython w/ Numpy Arrays & Lambdas](#cpython-with-numpy-arrays-and-lambdas)
184-
* <b>Anaconda Cython</b>: [CPython Cython](#cpython-cython)
185-
* <b>Anaconda Cython with Lambdas</b>: [CPython Cython w/ Lambdas](#cpython-cython-with-lambdas)
186-
* <b>Anaconda Cython with LRU Caching</b>: [CPython Cython w/ LRU Caching](#cpython-cython-with-lru-caching)
187-
* <b>Anaconda Cython with Numpy Arrays</b>: [CPython Cython w/ Numpy Arrays](#cpython-cython-with-numpy-arrays)
188-
* <b>Anaconda Cython with Numpy Arrays and Lambdas</b>: [CPython Cython w/ Numpy Arrays & Lambdas](#cpython-cython-with-numpy-arrays-and-lambdas)
189-
* <b>Anaconda Optimized Cython</b>: [CPython Cython Optimized](#cpython-optimized-cython)
190-
* <b>Anaconda Optimized Cython with Lambdas</b>: [CPython Cython Optimized w/ Lambdas](#cpython-optimized-cython-with-lambdas)
191-
* <b>Anaconda Optimized Cython with LRU Caching</b>: [CPython Cython Optimized w/ LRU Caching](#cpython-optimized-cython-with-lru-caching)
192-
* <b>Anaconda Optimized with Numpy Arrays</b>: [CPython Cython Optimized w/ Numpy Arrays](#cpython-optimized-cython-with-numpy-arrays)
193-
* <b>Anaconda Optimized with Numpy Arrays and Lambdas</b>: [CPython Cython Optimized w/ Numpy Arrays & Lambdas](#cpython-optimized-cython-with-numpy-arrays-and-lambdas)
194-
* <b>Anaconda Optimized with Numpy Arrays and LRU Caching</b>: [CPython Cython Optimized w/ Numpy Arrays & LRU Caching](#cpython-optimized-cython-with-numpy-arrays-and-lru-caching)
195-
26+
| Test | Description |
27+
|------------------|-------------------------------------------------------|
28+
| Primes | Find every Prime Number up to a given integer |
29+
| Fibonacci | Calculate all Fibonacci numbers up to a given integer |
30+
| Database Usage | Run different database operations |
31+
| Image Processing | Apply modifications to images |
32+
| Video Processing | Apply modifications to videos |
33+
## Optimizations
34+
- Python Runtimes:
35+
- __CPython__: The standard C-based Python runtime
36+
- __Pypy__: An alternative implementation of the CPython runtime, utilizing a JIT compiler, but lacks support for some CPython extensions
37+
- __Cinder__: Meta's internal CPython implementation, highly unsupported for external use cases
38+
- __Pyston__: Fork of the CPython 3.8 runtime with a claimed 30% performance improvement
39+
- Compilation:
40+
- __Interpreted__: All code is interpreted by the Python runtime
41+
- __Compiled Cython__: Base code is compiled with Cython without any Cython optimizations
42+
- __Optimized Cython__: Base code is compiled with Cython and is modified to utilize Cython optimizations (such as static typing)
43+
- __Numba JIT__: Base code is compiled just-in-time with Numba
44+
- __Numba AOT__: Base code is compiled ahead-of-time with Numba
45+
- Code Branching:
46+
- __Inline / Imperative__: Code is run within the main function
47+
- __Function__: Code is put into it's own function
48+
- Caching:
49+
- __Least-Recently-Used Cache__: In-memory caching with Python built-ins `functools.lru_cache`
50+
- __JSON Cache__: File-based caching with JSON files
51+
- __SQLite Cache__: Database caching with local SQLite
52+
- __Redis Cache__: Database caching with in-memory Redis service
53+
- Data Storage Objects:
54+
- __Lists__: Store results inside mutable Python lists
55+
- __Tuples__: Store results inside immutable Python tuples
56+
- __Numpy Arrays__: Store results inside mutable in-memory Numpy arrays
57+
- __Pandas DataFrames / Series__: Store results inside mutable Pandas DataFrames / Series
58+
- Data Comprehension:
59+
- __For Loops__: The "standard" method of iterating over data.
60+
- __List Comprehension__: Create a mutable list directly from generators
61+
- __Tuple Comprehension__: Create an immutable tuple directly from generators
62+
- __Numpy Array Methods__: Apply functions to Numpy arrays
63+
- __Pandas DataFrame / Series Methods__: Apply functions to Pandas DataFrames / Series
64+
- Miscellaneous Python Tricks:
65+
- __Generators__: Yield results as they come, rather than return an entire container of the results
66+
- __Lambdas__: Anonymous functions rather than defined named ones
67+
- __Maps__: Apply a function over an iterable rather than handle iterating manually
68+
69+
### Prime Number Tests
70+
For some given number `N`, find every number below `N`<br>
71+
Since this is a mathematical test, we can have some shortcuts.
72+
For prime numbers, we already know the following:
73+
* All prime numbers are odd numbers, except for 2.
74+
* We only need to start with 2 as the first prime number, then loop from 3
75+
to `N`, incrementing by 2 for each iteration to test only odd numbers.
76+
* Instead of checking every number up to `N` we can stop at `floor(N/2)`
77+
* Instead of checking every number up to `floor(N/2)` we can stop at `floor(math.sqrt(N))`
78+
* Instead of doing pure math, we can use bitshifting.
19679

19780
## Related Files:
19881
#### Compiling

‎TODO.md ‎TODO/General.md

+2-91
Original file line numberDiff line numberDiff line change
@@ -44,99 +44,10 @@
4444
- [ ] Loops: Number of compilations loops to use for measuring compilation time (requires `Measure Time` to be true)
4545
- [ ] Add memory usage measurements for each test
4646
- [ ] Move to using cli2gui for a GUI and CLI fallback
47-
- [ ] Add Nvidia CUDA tests for prime number calculations
48-
- [ ] Have check to see if CUDA compatible GPU is in the system
49-
- [ ] Inline
50-
- [ ] Lambda
51-
- [ ] LRU Caching
52-
- [ ] Internal
53-
- [ ] External
54-
- [ ] Both
55-
- [ ] Lambda + LRU Caching
56-
- [ ] Internal
57-
- [ ] External
58-
- [ ] Both
59-
- [ ] Numpy Lambda + LRU Caching
60-
- [ ] Internal
61-
- [ ] External
62-
- [ ] Both
63-
- [ ] Function
64-
- [ ] Lambda
65-
- [ ] LRU Caching
66-
- [ ] Internal
67-
- [ ] External
68-
- [ ] Both
69-
- [ ] Lambda + LRU Caching
70-
- [ ] Internal
71-
- [ ] External
72-
- [ ] Both
73-
- [ ] Numpy Lambda + LRU Caching
74-
- [ ] Internal
75-
- [ ] External
76-
- [ ] Both
77-
- [ ] Add Numpy array tests for prime number calculations
78-
- [ ] Inline
79-
- [ ] Lambda
80-
- [ ] LRU Caching
81-
- [ ] Internal
82-
- [ ] External
83-
- [ ] Both
84-
- [ ] Lambda + LRU Caching
85-
- [ ] Internal
86-
- [ ] External
87-
- [ ] Both
88-
- [ ] Numpy Lambda + LRU Caching
89-
- [ ] Internal
90-
- [ ] External
91-
- [ ] Both
92-
- [ ] Function
93-
- [ ] Lambda
94-
- [ ] Numpy Lambda (`numpy.fromfunction(lambda i: n % i)`)
95-
- [ ] LRU Caching
96-
- [ ] Internal
97-
- [ ] External
98-
- [ ] Both
99-
- [ ] Lambda + LRU Caching
100-
- [ ] Internal
101-
- [ ] External
102-
- [ ] Both
103-
- [ ] Numpy Lambda + LRU Caching
104-
- [ ] Internal
105-
- [ ] External
106-
- [ ] Both
107-
- [ ] Add Numba JIT tests for prime number calculations
108-
- [ ] Inline
109-
- [ ] Lambda
110-
- [ ] LRU Caching
111-
- [ ] Internal
112-
- [ ] External
113-
- [ ] Both
114-
- [ ] Lambda + LRU Caching
115-
- [ ] Internal
116-
- [ ] External
117-
- [ ] Both
118-
- [ ] Numpy Lambda + LRU Caching
119-
- [ ] Internal
120-
- [ ] External
121-
- [ ] Both
122-
- [ ] Function
123-
- [ ] Lambda
124-
- [ ] Numpy Lambda (`numpy.fromfunction(lambda i: n % i)`)
125-
- [ ] LRU Caching
126-
- [ ] Internal
127-
- [ ] External
128-
- [ ] Both
129-
- [ ] Lambda + LRU Caching
130-
- [ ] Internal
131-
- [ ] External
132-
- [ ] Both
133-
- [ ] Numpy Lambda + LRU Caching
134-
- [ ] Internal
135-
- [ ] External
136-
- [ ] Both
13747

13848
### Done ✓
13949
- [x] Add Cython Compilation
14050
- Build app with `python .\src\setup.py build_ext`
14151
- Fix build locations
142-
- [x] Migrate off of `pipenv` and onto `poetry`
52+
- [x] Migrate off of `pipenv` and onto `poetry`
53+

‎TODO/Primes/CUDA.md

+34
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
### Tests to Add
2+
- [ ] Add Nvidia CUDA tests for prime number calculations (requires Numba)
3+
- [ ] Have check to see if CUDA compatible GPU is in the system
4+
- [ ] Inline
5+
- [ ] Lambda
6+
- [ ] LRU Caching
7+
- [ ] Internal
8+
- [ ] External
9+
- [ ] Both
10+
- [ ] Lambda + LRU Caching
11+
- [ ] Internal
12+
- [ ] External
13+
- [ ] Both
14+
- [ ] Numpy Lambda + LRU Caching
15+
- [ ] Internal
16+
- [ ] External
17+
- [ ] Both
18+
- [ ] Function
19+
- [ ] Lambda
20+
- [ ] LRU Caching
21+
- [ ] Internal
22+
- [ ] External
23+
- [ ] Both
24+
- [ ] Lambda + LRU Caching
25+
- [ ] Internal
26+
- [ ] External
27+
- [ ] Both
28+
- [ ] Numpy Lambda + LRU Caching
29+
- [ ] Internal
30+
- [ ] External
31+
- [ ] Both
32+
- [ ]
33+
34+
### Done ✓

0 commit comments

Comments
 (0)