Skip to content

Commit 0f14687

Browse files
MatthewMiddlehurstMatthewMiddlehurst
committed
testing and refactor
1 parent a788a7d commit 0f14687

File tree

80 files changed

+165
-24867
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

80 files changed

+165
-24867
lines changed

‎.gitattributes

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
*.ts linguist-detectable=false

‎.github/workflows/cancel.yml

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
name: Cancel
2+
3+
on:
4+
workflow_run:
5+
workflows: ["Install"]
6+
types:
7+
- requested
8+
9+
jobs:
10+
cancel:
11+
runs-on: ubuntu-latest
12+
steps:
13+
- uses: styfle/[email protected]
14+
with:
15+
workflow_id: ${{ github.event.workflow.id }}

‎.github/workflows/tests.yml

+44
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
name: Tests
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
pull_request:
8+
branches:
9+
- main
10+
11+
jobs:
12+
pre-commit:
13+
runs-on: ubuntu-latest
14+
steps:
15+
- uses: actions/checkout@v3
16+
17+
- uses: actions/setup-python@v4
18+
with:
19+
python-version: 3.7
20+
21+
- id: file_changes
22+
uses: trilom/[email protected]
23+
with:
24+
output: " "
25+
26+
- uses: pre-commit/[email protected]
27+
with:
28+
extra_args: --files ${{ steps.file_changes.outputs.files }}
29+
30+
pytest:
31+
needs: pre-commit
32+
runs-on: ubuntu-latest
33+
steps:
34+
- uses: actions/checkout@v3
35+
36+
- uses: actions/setup-python@v4
37+
with:
38+
python-version: 3.7
39+
40+
- name: Install
41+
run: python -m pip install -r requirements-dev.txt
42+
43+
- name: Tests
44+
run: python -m pytest

‎.gitignore

+8-8
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,13 @@ ipython_config.py
8484
# pyenv
8585
.python-version
8686

87+
# pipenv
88+
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
89+
# However, in case of collaboration, if having platform-specific dependencies or dependencies
90+
# having no cross-platform support, pipenv may install dependencies that don't work, or not
91+
# install all needed dependencies.
92+
#Pipfile.lock
93+
8794
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
8895
__pypackages__/
8996

@@ -120,13 +127,6 @@ dmypy.json
120127

121128
# Pyre type checker
122129
.pyre/
123-
/sktime_estimator_evaluation/experiments/temp/
124-
125-
# Testing result and output
126-
out/
127-
ignore-results/
128130

129-
# pycharm
131+
#IDE
130132
.idea/
131-
132-
.DS_Store

‎.gitmodules

-4
This file was deleted.

‎.pre-commit-config.yaml

+54
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
repos:
2+
- repo: https://github.com/pre-commit/pre-commit-hooks
3+
rev: v4.3.0
4+
hooks:
5+
- id: check-added-large-files
6+
args: ["--maxkb=10000"]
7+
- id: check-case-conflict
8+
- id: check-merge-conflict
9+
- id: check-symlinks
10+
- id: check-yaml
11+
- id: debug-statements
12+
- id: end-of-file-fixer
13+
- id: fix-encoding-pragma
14+
- id: requirements-txt-fixer
15+
- id: trailing-whitespace
16+
17+
- repo: https://github.com/pycqa/isort
18+
rev: 5.10.1
19+
hooks:
20+
- id: isort
21+
name: isort (python)
22+
args: ["--profile=black"]
23+
24+
- repo: https://github.com/psf/black
25+
rev: 22.10.0
26+
hooks:
27+
- id: black
28+
language_version: python3.7
29+
30+
- repo: https://github.com/pycqa/flake8
31+
rev: 5.0.4
32+
hooks:
33+
- id: flake8
34+
additional_dependencies: [ flake8-bugbear, flake8-print ]
35+
args: ["--max-line-length=88", "--extend-ignore=E203"]
36+
37+
- repo: https://github.com/nbQA-dev/nbQA
38+
rev: 1.5.3
39+
hooks:
40+
- id: nbqa-isort
41+
additional_dependencies: [ isort==5.10.1 ]
42+
args: [ --nbqa-mutate, --nbqa-dont-skip-bad-cells ]
43+
- id: nbqa-black
44+
additional_dependencies: [ black==22.10.0 ]
45+
args: [ --nbqa-mutate, --nbqa-dont-skip-bad-cells ]
46+
- id: nbqa-flake8
47+
additional_dependencies: [ flake8==5.0.4 ]
48+
args: [ --nbqa-dont-skip-bad-cells, "--extend-ignore=E402,E203" ]
49+
50+
- repo: https://github.com/pycqa/pydocstyle
51+
rev: 6.1.1
52+
hooks:
53+
- id: pydocstyle
54+
args: ["--convention=numpy"]

‎README.md

+7-7
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,19 @@
11
# estimator-evaluation
2-
Evaluation tools for classification and regression
2+
3+
Evaluation tools for classification, regression and clustering using the sktime package.
34

45
Installation
56
------------
67
pip install -r requirements.txt
78

8-
99
Export new requirements
1010
-----------------------
1111
pip freeze > requirements.txt
1212

13-
Sktime install from branch
13+
sktime install from main or branch
1414
--------------------------
15-
- You can use this package with a sktime install that isn't deployed to pypip
15+
You can use this package with a sktime install that isn't deployed to pypip.
16+
1617
- First uninstall current sktime: 'pip uninstall sktime'
17-
- Then install from git using 'pip install git+https://github.com/alan-turing-institute/sktime.git@estimator-evaluation'
18-
- The values after the @ at the end is the branch i.e. another branch would be
19-
'pip install git+https://github.com/alan-turing-institute/sktime.git@another-branch'
18+
- Then install from git using 'pip install git+https://github.com/sktime/sktime.git@main'
19+
- The values after the @ at the end is the branch i.e. another branch would be 'pip install git+https://github.com/sktime/sktime.git@another-branch'

‎sktime_estimator_evaluation/__init__.py ‎estimator_evaluation/__init__.py

File renamed without changes.

‎sktime_estimator_evaluation/debug.py ‎estimator_evaluation/debug.py

File renamed without changes.

‎sktime_estimator_evaluation/evaluation/__init__.py ‎estimator_evaluation/evaluation/__init__.py

+3-3
Original file line numberDiff line numberDiff line change
@@ -22,13 +22,13 @@
2222
'from_metric_summary_to_dataset_format',
2323
]
2424

25-
from sktime_estimator_evaluation.evaluation._result_evaluation import (
25+
from estimator_evaluation.evaluation._result_evaluation import (
2626
evaluate_raw_results,
2727
evaluate_metric_results,
2828
CLUSTER_METRIC_CALLABLES,
2929
CLASSIFICATION_METRIC_CALLABLES
3030
)
31-
from sktime_estimator_evaluation.evaluation._utils import (
31+
from estimator_evaluation.evaluation._utils import (
3232
read_results_from_uea_format,
3333
read_clusterer_result_from_uea_format,
3434
ExperimentResamples,
@@ -46,6 +46,6 @@
4646
from_metric_dataset_format_to_metric_summary,
4747
from_metric_summary_to_dataset_format
4848
)
49-
from sktime_estimator_evaluation.evaluation._bulit_in_evaluation import (
49+
from estimator_evaluation.evaluation._bulit_in_evaluation import (
5050
fetch_classifier_metric
5151
)

‎sktime_estimator_evaluation/evaluation/_bulit_in_evaluation.py ‎estimator_evaluation/evaluation/_bulit_in_evaluation.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
import pandas as pd
55
import numpy as np
66

7-
from sktime_estimator_evaluation.evaluation import (
7+
from estimator_evaluation.evaluation import (
88
evaluate_metric_results,
99
MetricResults,
1010
metric_result_to_summary

‎sktime_estimator_evaluation/evaluation/_result_evaluation.py ‎estimator_evaluation/evaluation/_result_evaluation.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
import platform
44

55
import pandas as pd
6-
from sktime_estimator_evaluation.evaluation._utils import (
6+
from estimator_evaluation.evaluation._utils import (
77
resolve_experiment_paths,
88
Experiment,
99
MetricCallable,

‎sktime_estimator_evaluation/evaluation/_utils.py ‎estimator_evaluation/evaluation/_utils.py

File renamed without changes.

‎estimator_evaluation/evaluation/diagrams/__init__.py

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# -*- coding: utf-8 -*-
2+
"""Critical difference diagram."""
3+
__all__ = ["critical_difference_diagram", "scatter_diagram"]
4+
5+
from estimator_evaluation.evaluation.diagrams._critical_difference_diagram import critical_difference_diagram
6+
from estimator_evaluation.evaluation.diagrams._scatter_diagrams import scatter_diagram

‎sktime_estimator_evaluation/evaluation/diagrams/_critical_difference_diagram.py ‎estimator_evaluation/evaluation/diagrams/_critical_difference_diagram.py

+2-2
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@
1212
import pandas as pd
1313
from scipy.stats import friedmanchisquare, wilcoxon
1414

15-
from sktime_estimator_evaluation.evaluation import MetricResults
16-
from sktime_estimator_evaluation.evaluation.diagrams._utils import metric_result_to_df
15+
from estimator_evaluation.evaluation import MetricResults
16+
from estimator_evaluation.evaluation.diagrams._utils import metric_result_to_df
1717

1818
warnings.filterwarnings(
1919
"ignore"

‎sktime_estimator_evaluation/evaluation/diagrams/_scatter_diagrams.py ‎estimator_evaluation/evaluation/diagrams/_scatter_diagrams.py

+2-2
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,8 @@
66
import pandas as pd
77
import matplotlib.pyplot as plt
88

9-
from sktime_estimator_evaluation.evaluation import MetricResults
10-
from sktime_estimator_evaluation.evaluation.diagrams._utils import metric_result_to_df
9+
from estimator_evaluation.evaluation import MetricResults
10+
from estimator_evaluation.evaluation.diagrams._utils import metric_result_to_df
1111

1212

1313
def scatter_diagram(

‎sktime_estimator_evaluation/evaluation/diagrams/_utils.py ‎estimator_evaluation/evaluation/diagrams/_utils.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
import pandas as pd
55
import warnings
66

7-
from sktime_estimator_evaluation.evaluation import (
7+
from estimator_evaluation.evaluation import (
88
MetricResults,
99
metric_result_to_summary
1010
)

‎sktime_estimator_evaluation/comparison/__init__.py ‎estimator_evaluation/evaluation/test/__init__.py

File renamed without changes.

‎sktime_estimator_evaluation/evaluation/test/diagrams/test_diagrams.py ‎estimator_evaluation/evaluation/test/diagrams/test_diagrams.py

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
1-
from sktime_estimator_evaluation.evaluation import metric_result_to_summary, evaluate_metric_results
2-
from sktime_estimator_evaluation.evaluation.diagrams import scatter_diagram
1+
from estimator_evaluation.evaluation import metric_result_to_summary, evaluate_metric_results
2+
from estimator_evaluation.evaluation.diagrams import scatter_diagram
33
import platform
44

55
def test_scatter_plot():

‎sktime_estimator_evaluation/evaluation/test/test_result_evaluation.py ‎estimator_evaluation/evaluation/test/test_result_evaluation.py

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
from sktime_estimator_evaluation.evaluation._result_evaluation import (
1+
from estimator_evaluation.evaluation._result_evaluation import (
22
evaluate_raw_results,
33
CLUSTER_METRIC_CALLABLES,
44
evaluate_metric_results
@@ -39,7 +39,7 @@ def custom_classification(path: str):
3939
'../../../results/', custom_classification
4040
)
4141

42-
from sktime_estimator_evaluation.evaluation.diagrams._critical_difference_diagram import critical_difference_diagram
42+
from estimator_evaluation.evaluation.diagrams._critical_difference_diagram import critical_difference_diagram
4343

4444
# create_critical_difference_diagram(classification_results)
4545
critical_difference_diagram(clustering_results)

‎sktime_estimator_evaluation/evaluation/test/test_utils.py ‎estimator_evaluation/evaluation/test/test_utils.py

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
1-
from sktime_estimator_evaluation.evaluation._utils import resolve_experiment_paths
2-
from sktime_estimator_evaluation.evaluation import (
1+
from estimator_evaluation.evaluation._utils import resolve_experiment_paths
2+
from estimator_evaluation.evaluation import (
33
fetch_classifier_metric,
44
from_metric_summary_to_dataset_format
55
)

‎sktime_estimator_evaluation/evaluation/test/__init__.py ‎estimator_evaluation/experiments/__init__.py

File renamed without changes.

‎sktime_estimator_evaluation/experiments/classification_examples.py ‎estimator_evaluation/experiments/classification_examples.py

+2-2
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@
55
from sktime.transformations.panel.catch22 import Catch22
66
from sktime.datasets import load_arrow_head
77
from sktime.datasets import load_UCR_UEA_dataset
8-
from sktime_estimator_evaluation.evaluation import fetch_classifier_metric
9-
from sktime_estimator_evaluation.evaluation.diagrams import scatter_diagram, critical_difference_diagram
8+
from estimator_evaluation.evaluation import fetch_classifier_metric
9+
from estimator_evaluation.evaluation.diagrams import scatter_diagram, critical_difference_diagram
1010

1111
import numpy as np
1212

‎sktime_estimator_evaluation/experiments/classification_experiments.py ‎estimator_evaluation/experiments/classification_experiments.py

File renamed without changes.

‎sktime_estimator_evaluation/experiments/clustering_experiments.py ‎estimator_evaluation/experiments/clustering_experiments.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@
2626
from sktime.datasets import load_gunpoint
2727
from sktime.clustering.k_means import TimeSeriesKMeans
2828
from sktime.clustering.k_medoids import TimeSeriesKMedoids
29-
from sktime_estimator_evaluation.experiments.classification_experiments import results_present
29+
from estimator_evaluation.experiments.classification_experiments import results_present
3030

3131

3232

‎sktime_estimator_evaluation/experiments/set_classifier.py ‎estimator_evaluation/experiments/set_classifier.py

File renamed without changes.

‎sktime_estimator_evaluation/experiments/set_clusterer.py ‎estimator_evaluation/experiments/set_clusterer.py

File renamed without changes.

‎sktime_estimator_evaluation/results_by_classifier.py ‎estimator_evaluation/results_by_classifier.py

+3-4
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
import os
22
import numpy as np
33
import csv
4-
from root import ROOT_DIR
54

65
valid_uni_classifiers =[
76
"Arsenal", "BOSS","Catch22","cBOSS","CIF","DrCIF","HC1","HC2","InceptionTime",
@@ -43,7 +42,7 @@
4342
]
4443

4544

46-
def get_single_classifier_results(classifier, root=ROOT_DIR, package="tsml", type="Univariate"):
45+
def get_single_classifier_results(classifier, root="../", package="tsml", type="Univariate"):
4746
"""Load the results for a single classifier on a single resample from local storage.
4847
4948
Parameters
@@ -71,8 +70,8 @@ def get_single_classifier_results(classifier, root=ROOT_DIR, package="tsml", typ
7170
return results
7271

7372

74-
def get_classifier_results(classifiers, datasets, resample=0, root=ROOT_DIR,
75-
package ="tsml"):
73+
def get_classifier_results(classifiers, datasets, resample=0, root="../",
74+
package="tsml"):
7675
"""Collate results for classifiers over problems.
7776
7877
given lists of n datasets and m classifiers, form an n by m array of accuracies,

‎sktime_estimator_evaluation/experiments/__init__.py ‎estimator_evaluation/tests/__init__.py

File renamed without changes.

‎sktime_estimator_evaluation/tests/test_results_by_classifier.py ‎estimator_evaluation/tests/test_results_by_classifier.py

File renamed without changes.

‎sktime_estimator_evaluation/tests/__init__.py ‎estimator_evaluation/utils/__init__.py

File renamed without changes.

‎sktime_estimator_evaluation/utils/format_dataset.py ‎estimator_evaluation/utils/format_dataset.py

File renamed without changes.

‎sktime_estimator_evaluation/utils/read_results.py ‎estimator_evaluation/utils/read_results.py

File renamed without changes.

‎experiments/clustering/averaging/_result_evaluation.py

+4-4
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,14 @@
11
import pandas as pd
22
import os
33

4-
from sktime_estimator_evaluation.evaluation._result_evaluation import (
4+
from estimator_evaluation.evaluation._result_evaluation import (
55
evaluate_raw_results,
66
CLUSTER_METRIC_CALLABLES,
77
evaluate_metric_results
88
)
9-
from sktime_estimator_evaluation.evaluation._utils import metric_result_to_summary
10-
from sktime_estimator_evaluation.evaluation.diagrams import critical_difference_diagram
11-
from sktime_estimator_evaluation.evaluation._utils import read_metric_results, combine_two_summary_df
9+
from estimator_evaluation.evaluation._utils import metric_result_to_summary
10+
from estimator_evaluation.evaluation.diagrams import critical_difference_diagram
11+
from estimator_evaluation.evaluation._utils import read_metric_results, combine_two_summary_df
1212

1313

1414
def load_custom_data():

‎requirements-dev.txt

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
-r requirements.txt
2+
pre-commit
3+
pytest

‎requirements.txt

+1-29
Original file line numberDiff line numberDiff line change
@@ -1,29 +1 @@
1-
attrs==21.4.0
2-
cycler==0.11.0
3-
Deprecated==1.2.13
4-
fonttools==4.33.3
5-
iniconfig==1.1.1
6-
joblib==1.1.0
7-
kiwisolver==1.4.3
8-
llvmlite==0.38.1
9-
matplotlib==3.5.2
10-
numba==0.55.2
11-
numpy==1.21.6
12-
packaging==21.3
13-
pandas==1.4.3
14-
patsy==0.5.2
15-
Pillow==9.1.1
16-
pluggy==1.0.0
17-
py==1.11.0
18-
pyparsing==3.0.9
19-
pytest==7.1.2
20-
python-dateutil==2.8.2
21-
pytz==2022.1
22-
scikit-learn==1.1.1
23-
scipy==1.8.1
24-
six==1.16.0
25-
statsmodels==0.13.2
26-
threadpoolctl==3.1.0
27-
tomli==2.0.1
28-
wrapt==1.14.1
29-
git+https://github.com/alan-turing-institute/sktime.git@main#egg=sktime
1+
sktime

‎root.py

-2
This file was deleted.

0 commit comments

Comments
 (0)