Skip to content

Commit 0a58c11

Browse files
odeimaizsanderegg
odeimaiz
authored and
sanderegg
committed
Fake data extracted from S4L (#378)
* Fixes typos in client after bad merge * webserver.director subsystem: - added a function to retrieve a client provided the application - added notes on a bug * WIP: building coupling between revers_proxy and dicector subsystems * Modifies director API specs. Implementation expected in #363 * cleanup and naming services * WIP: implementing service resolution policy with director sdk * running_services get now returns also the service and its key/version * modified tests according to requirements * modified handlers to work according to requirements * pylint added a main label when more than one docker service is composing a node implemented get_service_details * updated server information * automatically create the resolved oas file * updated generator version * updated director sdk * updated director openapi * updated version check * fixed regex error in client code * updated sample * revert change * formatted director.producer to ready easier * Implemented ServiceResolutionPolicy with director in application_proxy Activated setup_app_proxy in appliction * Minor * Cleanup dev requirements * Cleanup director tests while reviewing them * Fixes duplicated storage in docker-compose * Minor cleanup and adds apihub dependency to storage * Modified director APi specs * WIP: application proxy implementation using new director api * default is empty basepath * generated client for new api with basepath * updated director api server side * Minor cleanup changed APP_SCHEMA by app_schema removed comments so we can pass to xargs in command line * update test for director * updated tests * all tests passing * update to the api * fixed not getting service correctly * WIP: implementing integration tests for reverse proxy * added apihub * start notebook with a base path * version bump * added SIMCORE_NODE_BASEPATH * fixed copy of custom.js to set notebook in single-tab mode * fixed alternative basepath * adding missing veriable and baseurl env * refactoring adding base url using async * added config file moved s4l stuff in s4l_utils * Prepared integration-test for reverse proxy - recreates docker-compose and config using existing files and environs - tests fail * websocket also on alternative basepath * duplicate key * pylint * fix failing test * version bump * empty basepath default * handle error * fix catch * fix default env * final fix * Cleanup: - removed pdb breakpoint - added disabling login option for testing - cleanup tests * Fixed webserver.director handlers * Minor * Using aiohttp client session instead of sdk * Minor * Added makefile to build swarm and tooling * webserver.director found director api endpoint upon setup * application-proxy: - removed reference to director_sdk (using now aiohttp client) - fixed urls * materialDB for LF updated * Fixes responses from director * Fixes on jupyter handlers * application_proxy: service url considers when the app is in swarm or not * tests pass using a sleep to wait the service to start * Added 'fake' data for LF MaterialDB (getItem & getItemList) * s4l converter script added * linting * Disableds proxy if director is disabled * Added 'fake' data for LF Modeler (getItemList) * Minor * Added 'fake' data mapper for Material2Entities * fake data updated * s4l converter extended * fixes eslint * Fixes test-failure in storage * Minor * Adds stdin_open and tty options in docker-compose.devel to enable debugging upon attach * Fixes oas_repo config for docker-dev. repo root does not exists inside the container! * key: label instead of key: value * FakeData moved to frontend * return sorted getItemList for materialDB and modeler * Minor bug in weberver.director subsystem * Front-end now requests interactive services via the reverse proxy * minor * minor * convert APITree into VirtualTree Model * NodeOutputTreeItem added * modeler-neuron fake data added * neuroman default inputs fake data added * qxapp.dev.fake.lf.Data added * minor * All fake data well refactored * New Project Thumbnails * sensors added to LF * sensors added to LF * Fixes reverse_proxy jupyter handler * Fixes webserver.projects handler to retrieve template projects * Minor cleanup * template projects fix * fake data update * minor * this was missing in the patch * clean up * Porjects live in the server * minor * more LF * Modeler has all entities * Neuron simulator fake data added * send nodeKey instead of nodeId for gettings items * send nodeKey instead of nodeId for gettings items (2) * ups * Wokring on Neuron defaults fake data * Nueron defaultIInputs added * more fake data * Mapper default values in and working * Default mapperInputValues get form * test updated * LF sensor can be dropped in Neuron sources * Neuroman list updated * version dump
1 parent b1a92b2 commit 0a58c11

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

64 files changed

+7526
-2154
lines changed

‎Makefile

+1-1
Original file line numberDiff line numberDiff line change
@@ -158,7 +158,7 @@ test:
158158
make run_test
159159
make after_test
160160

161-
PLATFORM_VERSION=3.24
161+
PLATFORM_VERSION=3.27
162162
DOCKER_REGISTRY=masu.speag.com
163163
#DOCKER_REGISTRY=registry.osparc.io
164164

‎services/dy-jupyter/.env-devel

+2-1
Original file line numberDiff line numberDiff line change
@@ -9,5 +9,6 @@ S3_ENDPOINT=minio:9000
99
S3_ACCESS_KEY=12345678
1010
S3_SECRET_KEY=12345678
1111
S3_BUCKET_NAME=simcore
12+
S3_SECURE=0
1213
APIHUB_HOST=apihub
13-
APIHUB_PORT=8043
14+
APIHUB_PORT=8043

‎services/dy-jupyter/docker/boot.sh

+2-3
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ then
99
echo "Creating dummy tables ... using ${USE_CASE_CONFIG_FILE}"
1010
result="$(python scripts/dy_services_helpers/platform_initialiser_csv_files.py ${USE_CASE_CONFIG_FILE} ${INIT_OPTIONS})"
1111
echo "Received result of $result";
12-
IFS=, read -a array <<< "$result";
12+
IFS=, read -a array <<< "$result";
1313
echo "Received result pipeline id of ${array[0]}";
1414
echo "Received result node uuid of ${array[1]}";
1515
# the fake SIMCORE_NODE_UUID is exported to be available to the service
@@ -22,6 +22,5 @@ start-notebook.sh \
2222
--NotebookApp.extra_static_paths="['${SIMCORE_NODE_BASEPATH}/static']" \
2323
--NotebookApp.notebook_dir='/home/jovyan/notebooks' \
2424
--NotebookApp.token=''
25-
#--NotebookApp.tornado_settings="{\"headers\":{\"Content-Security-Policy\":\"frame-ancestors+'self'+http://osparc01.itis.ethz.ch:9081;+report-uri/api/security/csp-report\"}}" \
25+
#--NotebookApp.tornado_settings="{\"headers\":{\"Content-Security-Policy\":\"frame-ancestors+'self'+http://osparc01.itis.ethz.ch:9081;+report-uri/api/security/csp-report\"}}" \
2626
# --NotebookApp.default_url=/notebooks/${NOTEBOOK_URL} #uncomment this to start the notebook right away in that notebook
27-

‎services/web/client/source/class/qxapp/Application.js

-36
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,6 @@ qx.Class.define("qxapp.Application", {
5252
}, this);
5353

5454
this.__restart();
55-
this.__schemaCheck();
5655
},
5756

5857
__restart: function() {
@@ -120,41 +119,6 @@ qx.Class.define("qxapp.Application", {
120119
__disconnectWebSocket: function() {
121120
// open web socket
122121
qxapp.wrappers.WebSocket.getInstance().disconnect();
123-
},
124-
125-
__schemaCheck: function() {
126-
/** a little ajv test */
127-
let nodeCheck = new qx.io.request.Xhr("/resource/qxapp/node-meta-v0.0.1.json");
128-
nodeCheck.addListener("success", e => {
129-
let data = e.getTarget().getResponse();
130-
try {
131-
let ajv = new qxapp.wrappers.Ajv(data);
132-
let map = qxapp.data.Store.getInstance().getFakeServices();
133-
for (let key in map) {
134-
let check = ajv.validate(map[key]);
135-
console.log("services validation result " + key + ":", check);
136-
}
137-
} catch (err) {
138-
console.error(err);
139-
}
140-
});
141-
nodeCheck.send();
142-
let projectCheck = new qx.io.request.Xhr("/resource/qxapp/project-v0.0.1.json");
143-
projectCheck.addListener("success", e => {
144-
let data = e.getTarget().getResponse();
145-
try {
146-
let ajv = new qxapp.wrappers.Ajv(data);
147-
let list = qxapp.data.Store.getInstance().getProjectList();
148-
list.forEach((project, i) => {
149-
let check = ajv.validate(project);
150-
console.log("project validation result " + i + ":", check);
151-
});
152-
} catch (err) {
153-
console.error(err);
154-
}
155-
});
156-
projectCheck.send();
157122
}
158-
159123
}
160124
});

‎services/web/client/source/class/qxapp/component/widget/InputsMapper.js

+37-1
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ qx.Class.define("qxapp.component.widget.InputsMapper", {
2222

2323
let that = this;
2424
tree.setDelegate({
25-
createItem: () => new qxapp.component.widget.inputs.NodeOutputTreeItem(),
25+
createItem: () => new qxapp.component.widget.InputsMapperTreeItem(),
2626
bindItem: (c, item, id) => {
2727
c.bindDefaultProperties(item, id);
2828
// c.bindProperty("key", "key", null, item, id);
@@ -95,6 +95,42 @@ qx.Class.define("qxapp.component.widget.InputsMapper", {
9595
isRoot: true,
9696
children: []
9797
};
98+
if (Object.prototype.hasOwnProperty.call(mapper, "defaultValue")) {
99+
const defValues = mapper["defaultValue"];
100+
for (let i=0; i<defValues.length; i++) {
101+
const defValue = defValues[i];
102+
for (const defValueId in defValue) {
103+
let newBranch = {
104+
key: defValueId,
105+
label: defValueId.replace("-UUID", ""),
106+
nodeKey: nodeModel.getKey(),
107+
portKey: "myPort",
108+
isDir: true,
109+
children: []
110+
};
111+
let newItemBranch = qx.data.marshal.Json.createModel(newBranch, true);
112+
const itemProps = qxapp.data.Store.getInstance().getItem(null, Object.keys(nodeModel.getInputsDefault())[0], defValueId);
113+
if (itemProps) {
114+
let form = new qxapp.component.form.Auto(itemProps, this.getNodeModel());
115+
let propsWidget = new qxapp.component.form.renderer.PropForm(form);
116+
newItemBranch["propsWidget"] = propsWidget;
117+
}
118+
data.children.push(newItemBranch);
119+
const values = defValue[defValueId];
120+
for (let j=0; j<values.length; j++) {
121+
let newLeaf = {
122+
key: values[j],
123+
label: values[j],
124+
nodeKey: nodeModel.getKey(),
125+
portKey: "myPort",
126+
isDir: true
127+
};
128+
let newItemLeaf = qx.data.marshal.Json.createModel(newLeaf, true);
129+
newItemBranch.getChildren().push(newItemLeaf);
130+
}
131+
}
132+
}
133+
}
98134
let model = qx.data.marshal.Json.createModel(data, true);
99135
tree.setModel(model);
100136

‎services/web/client/source/class/qxapp/component/widget/InputsMapperTreeItem.js

+31
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
qx.Class.define("qxapp.component.widget.InputsMapperTreeItem", {
2+
extend: qx.ui.tree.VirtualTreeItem,
3+
4+
construct: function() {
5+
this.base(arguments);
6+
},
7+
8+
properties: {
9+
isDir: {
10+
check: "Boolean",
11+
nullable: false,
12+
init: true
13+
},
14+
15+
isRoot: {
16+
check: "Boolean",
17+
nullable: false,
18+
init: false
19+
},
20+
21+
nodeKey: {
22+
check: "String",
23+
nullable: false
24+
},
25+
26+
portKey: {
27+
check: "String",
28+
nullable: false
29+
}
30+
}
31+
});

‎services/web/client/source/class/qxapp/component/widget/inputs/NodeOutputListIcon.js

+1-1
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ qx.Class.define("qxapp.component.widget.inputs.NodeOutputListIcon", {
3636
}
3737
});
3838

39-
const itemList = qxapp.data.Store.getInstance().getItemList(nodeModel.getNodeId(), portKey);
39+
const itemList = qxapp.data.Store.getInstance().getItemList(nodeModel.getKey(), portKey);
4040
const listModel = qxapp.data.Converters.fromAPIListToVirtualListModel(itemList);
4141
let model = qx.data.marshal.Json.createModel(listModel, true);
4242
list.setModel(model);

‎services/web/client/source/class/qxapp/component/widget/inputs/NodeOutputTree.js

+5-7
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,7 @@ qx.Class.define("qxapp.component.widget.inputs.NodeOutputTree", {
66

77
this.setNodeModel(nodeModel);
88

9-
let tree = this.__tree = new qx.ui.tree.VirtualTree(null, "label", "children").set({
10-
openMode: "none"
11-
});
9+
let tree = this.__tree = new qx.ui.tree.VirtualTree(null, "label", "children");
1210

1311
tree.setDelegate({
1412
createItem: () => new qxapp.component.widget.inputs.NodeOutputTreeItem(),
@@ -18,7 +16,7 @@ qx.Class.define("qxapp.component.widget.inputs.NodeOutputTree", {
1816
},
1917
configureItem: item => {
2018
item.set({
21-
isDir: !portKey.includes("modeler"),
19+
isDir: !portKey.includes("modeler") && !portKey.includes("sensorSettingAPI"),
2220
nodeKey: nodeModel.getKey(),
2321
portKey: portKey,
2422
draggable: true
@@ -32,9 +30,9 @@ qx.Class.define("qxapp.component.widget.inputs.NodeOutputTree", {
3230
}
3331
});
3432

35-
const itemList = qxapp.data.Store.getInstance().getItemList(nodeModel.getNodeId(), portKey);
36-
const showAsDirs = !portKey.includes("modeler");
37-
const children = qxapp.data.Converters.fromAPIListToVirtualTreeModel(itemList, showAsDirs);
33+
const itemList = qxapp.data.Store.getInstance().getItemList(nodeModel.getKey(), portKey);
34+
const showAsDirs = !portKey.includes("modeler") && !portKey.includes("sensorSettingAPI");
35+
const children = qxapp.data.Converters.fromAPITreeToVirtualTreeModel(itemList, showAsDirs);
3836
let data = {
3937
label: port.label,
4038
children: children

‎services/web/client/source/class/qxapp/component/widget/inputs/NodeOutputTreeItem.js

+19
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,25 @@ qx.Class.define("qxapp.component.widget.inputs.NodeOutputTreeItem", {
2626
portKey: {
2727
check: "String",
2828
nullable: false
29+
},
30+
31+
key: {
32+
check: "String",
33+
nullable: false
34+
}
35+
},
36+
37+
members : {
38+
_addWidgets : function() {
39+
// Here's our indentation and tree-lines
40+
this.addSpacer();
41+
this.addOpenButton();
42+
43+
// The standard tree icon follows
44+
this.addIcon();
45+
46+
// The label
47+
this.addLabel();
2948
}
3049
}
3150
});

‎services/web/client/source/class/qxapp/data/Converters.js

+62-4
Original file line numberDiff line numberDiff line change
@@ -24,13 +24,13 @@ qx.Class.define("qxapp.data.Converters", {
2424
return metaData;
2525
},
2626

27-
mergeChildren: function(one, two) {
27+
__mergeFileTreeChildren: function(one, two) {
2828
let newDir = true;
2929
for (let i=0; i<one.length; i++) {
3030
if (one[i].path === two.path) {
3131
newDir = false;
3232
if ("children" in two) {
33-
this.mergeChildren(one[i].children, two.children[0]);
33+
this.__mergeFileTreeChildren(one[i].children, two.children[0]);
3434
}
3535
}
3636
}
@@ -85,7 +85,7 @@ qx.Class.define("qxapp.data.Converters", {
8585
}]
8686
}]
8787
});
88-
this.mergeChildren(children, fileInTree);
88+
this.__mergeFileTreeChildren(children, fileInTree);
8989
}
9090
} else if (file["location_id"] === 1 || file["location_id"] === "1") {
9191
// datcore files
@@ -107,14 +107,72 @@ qx.Class.define("qxapp.data.Converters", {
107107
fileId: file["file_uuid"]
108108
};
109109
parent.children.push(fileInfo);
110-
this.mergeChildren(children, fileInTree);
110+
this.__mergeFileTreeChildren(children, fileInTree);
111111
}
112112
}
113113
}
114114

115115
return children;
116116
},
117117

118+
__mergeAPITreeChildren: function(one, two) {
119+
let newDir = true;
120+
for (let i=0; i<one.length; i++) {
121+
if (one[i].key === two.key) {
122+
newDir = false;
123+
if ("children" in two) {
124+
this.__mergeAPITreeChildren(one[i].children, two.children[0]);
125+
}
126+
}
127+
}
128+
// if (one.length === 0 || "fileId" in two || newDir) {
129+
if (one.length === 0 || newDir) {
130+
one.push(two);
131+
}
132+
},
133+
134+
fromAPITreeToVirtualTreeModel: function(treeItems, showLeavesAsDirs = false) {
135+
let children = [];
136+
for (let i=0; i<treeItems.length; i++) {
137+
const treeItem = treeItems[i];
138+
let splitted = treeItem["label"].split("/");
139+
let newItem = {
140+
"label": splitted[0]
141+
};
142+
if (splitted.length === 1) {
143+
// leaf already
144+
newItem["key"] = treeItem["key"];
145+
if (showLeavesAsDirs) {
146+
newItem["children"] = [];
147+
}
148+
} else {
149+
// branch
150+
newItem["key"] = splitted[0];
151+
newItem["children"] = [];
152+
let parent = newItem;
153+
for (let j=1; j<splitted.length-1; j++) {
154+
let branch = {
155+
label: splitted[j],
156+
key: parent.key +"/"+ splitted[j],
157+
children: []
158+
};
159+
parent.children.push(branch);
160+
parent = branch;
161+
}
162+
let leaf = {
163+
label: splitted[splitted.length-1],
164+
key: parent.key +"/"+ splitted[splitted.length-1]
165+
};
166+
if (showLeavesAsDirs) {
167+
leaf["children"] = [];
168+
}
169+
parent.children.push(leaf);
170+
}
171+
this.__mergeAPITreeChildren(children, newItem);
172+
}
173+
return children;
174+
},
175+
118176
fromAPIListToVirtualTreeModel: function(listItems, showLeavesAsDirs = false) {
119177
let children = [];
120178
for (let i=0; i<listItems.length; i++) {

0 commit comments

Comments
 (0)