Skip to content

EN Data Processing Update #600

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Jul 16, 2020
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,16 @@ In this document, we list all the features currently supported by the Data Proce

### Processing engine

- Currently the only available processing engine is [Apache Spark](http://spark.apache.org/).
- During beta, the only available version of Apache Spark is 2.4.3
- At this moment, OVHcloud Data Processing supports the following engines:

- Spark 2.4.3

- It is not possible to interact with Apache Spark cluster via command line.

### The Apache Spark job in Data Processing beta is limited to:
### The Apache Spark job in Data Processing is limited to:

- Minimum 1 and maximum 30 GB of RAM per Executor
- Minimum 1 and maximum 8 Cores of CPU per Executor
- Minimum 1 and maximum 10 Executors per job
- Minimum 1 and maximum 60 GB of RAM per Executor
- Minimum 1 and maximum 16 Cores of CPU per Executor
- Maximum 24 hours for the duration of each job

### Supported languages for Apache Spark code
Expand All @@ -42,7 +43,7 @@ In this document, we list all the features currently supported by the Data Proce

### OVHcloud vRack

Data Processing can not use or access OVHcloud vRack.
- Data Processing can not use or access OVHcloud vRack.

### Available regions

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,16 @@ In this document, we list all the features currently supported by the Data Proce

### Processing engine

- Currently the only available processing engine is [Apache Spark](http://spark.apache.org/).
- During beta, the only available version of Apache Spark is 2.4.3
- At this moment, OVHcloud Data Processing supports the following engines:

- Spark 2.4.3

- It is not possible to interact with Apache Spark cluster via command line.

### The Apache Spark job in Data Processing beta is limited to:
### The Apache Spark job in Data Processing is limited to:

- Minimum 1 and maximum 30 GB of RAM per Executor
- Minimum 1 and maximum 8 Cores of CPU per Executor
- Minimum 1 and maximum 10 Executors per job
- Minimum 1 and maximum 60 GB of RAM per Executor
- Minimum 1 and maximum 16 Cores of CPU per Executor
- Maximum 24 hours for the duration of each job

### Supported languages for Apache Spark code
Expand All @@ -42,7 +43,7 @@ In this document, we list all the features currently supported by the Data Proce

### OVHcloud vRack

Data Processing can not use or access OVHcloud vRack.
- Data Processing can not use or access OVHcloud vRack.

### Available regions

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,16 @@ In this document, we list all the features currently supported by the Data Proce

### Processing engine

- Currently the only available processing engine is [Apache Spark](http://spark.apache.org/).
- During beta, the only available version of Apache Spark is 2.4.3
- At this moment, OVHcloud Data Processing supports the following engines:

- Spark 2.4.3

- It is not possible to interact with Apache Spark cluster via command line.

### The Apache Spark job in Data Processing beta is limited to:
### The Apache Spark job in Data Processing is limited to:

- Minimum 1 and maximum 30 GB of RAM per Executor
- Minimum 1 and maximum 8 Cores of CPU per Executor
- Minimum 1 and maximum 10 Executors per job
- Minimum 1 and maximum 60 GB of RAM per Executor
- Minimum 1 and maximum 16 Cores of CPU per Executor
- Maximum 24 hours for the duration of each job

### Supported languages for Apache Spark code
Expand All @@ -42,7 +43,7 @@ In this document, we list all the features currently supported by the Data Proce

### OVHcloud vRack

Data Processing can not use or access OVHcloud vRack.
- Data Processing can not use or access OVHcloud vRack.

### Available regions

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,16 @@ In this document, we list all the features currently supported by the Data Proce

### Processing engine

- Currently the only available processing engine is [Apache Spark](http://spark.apache.org/).
- During beta, the only available version of Apache Spark is 2.4.3
- At this moment, OVHcloud Data Processing supports the following engines:

- Spark 2.4.3

- It is not possible to interact with Apache Spark cluster via command line.

### The Apache Spark job in Data Processing beta is limited to:
### The Apache Spark job in Data Processing is limited to:

- Minimum 1 and maximum 30 GB of RAM per Executor
- Minimum 1 and maximum 8 Cores of CPU per Executor
- Minimum 1 and maximum 10 Executors per job
- Minimum 1 and maximum 60 GB of RAM per Executor
- Minimum 1 and maximum 16 Cores of CPU per Executor
- Maximum 24 hours for the duration of each job

### Supported languages for Apache Spark code
Expand All @@ -42,7 +43,7 @@ In this document, we list all the features currently supported by the Data Proce

### OVHcloud vRack

Data Processing can not use or access OVHcloud vRack.
- Data Processing can not use or access OVHcloud vRack.

### Available regions

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,16 @@ In this document, we list all the features currently supported by the Data Proce

### Processing engine

- Currently the only available processing engine is [Apache Spark](http://spark.apache.org/).
- During beta, the only available version of Apache Spark is 2.4.3
- At this moment, OVHcloud Data Processing supports the following engines:

- Spark 2.4.3

- It is not possible to interact with Apache Spark cluster via command line.

### The Apache Spark job in Data Processing beta is limited to:
### The Apache Spark job in Data Processing is limited to:

- Minimum 1 and maximum 30 GB of RAM per Executor
- Minimum 1 and maximum 8 Cores of CPU per Executor
- Minimum 1 and maximum 10 Executors per job
- Minimum 1 and maximum 60 GB of RAM per Executor
- Minimum 1 and maximum 16 Cores of CPU per Executor
- Maximum 24 hours for the duration of each job

### Supported languages for Apache Spark code
Expand All @@ -42,7 +43,7 @@ In this document, we list all the features currently supported by the Data Proce

### OVHcloud vRack

Data Processing can not use or access OVHcloud vRack.
- Data Processing can not use or access OVHcloud vRack.

### Available regions

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,16 @@ In this document, we list all the features currently supported by the Data Proce

### Processing engine

- Currently the only available processing engine is [Apache Spark](http://spark.apache.org/).
- During beta, the only available version of Apache Spark is 2.4.3
- At this moment, OVHcloud Data Processing supports the following engines:

- Spark 2.4.3

- It is not possible to interact with Apache Spark cluster via command line.

### The Apache Spark job in Data Processing beta is limited to:
### The Apache Spark job in Data Processing is limited to:

- Minimum 1 and maximum 30 GB of RAM per Executor
- Minimum 1 and maximum 8 Cores of CPU per Executor
- Minimum 1 and maximum 10 Executors per job
- Minimum 1 and maximum 60 GB of RAM per Executor
- Minimum 1 and maximum 16 Cores of CPU per Executor
- Maximum 24 hours for the duration of each job

### Supported languages for Apache Spark code
Expand All @@ -42,7 +43,7 @@ In this document, we list all the features currently supported by the Data Proce

### OVHcloud vRack

Data Processing can not use or access OVHcloud vRack.
- Data Processing can not use or access OVHcloud vRack.

### Available regions

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -50,20 +50,30 @@ After you submit a job to the Data Processing platform, follow these steps to mo

![Metrics token](images/token.png){.thumbnail}

- Copy the metrics token that is now displayed in the popup.
- Open [https://grafana.com/grafana/dashboards/11844](https://grafana.com/grafana/dashboards/11844) and copy the Grafana ID by clicking the button `Copy ID to Clipboard`{.action}
- Copy and save the metrics token that is now displayed in the popup.
- Open [https://grafana.metrics.ovh.net/login](https://grafana.metrics.ovh.net/login) and login with your OVHcloud account.
- From the main menu, click `Dashboards`{.action} -> `Import`{.action}

- Click on the Grafana logo at the upper left corner of the page and from the main menu select `Data Sources`{.action}
- Click on the `Add data source`{.action} button.
- Fill in the fields like in the pictures with the following data:
- Name: Select a name for your data source (for example `My Data Source 1`).
- Type: `Warp10`
- URL: `https://warp10.gra1.metrics.ovh.net`
- Access: `direct`

![datasource](images/datasource.png){.thumbnail}

- Click on the `Add`{.action} button.
- Click on the `Save & Test`{.action} button.
- Click on the Grafana logo at the upper left corner of the page and from the main menu select `Dashboards`{.action} -> `Import`{.action}
![grafana](images/grafana.png){.thumbnail}

- In the import dashboard page, paste the Grafana ID that you copied before and click on the `Load`{.action} button.
- In the import dashboard page, inside the `Grafana.com Dashboard` textbox write `11844` and click on the `Load`{.action} button. (11844 is the ID of a Grafana template that has been created specifically for Data Processing monitoring)

![grafana import](images/import.png){.thumbnail}

- Write a name for your dashboard
- Paste the metrics token that you copied before.
- For *datasource*, select `My Data Source 1`{.action} from the menu.
- Paste the metrics token that you copied and saved before.
- For *datasource*, select `My Data Source 1`{.action} from the menu (Or any other name that you selected for your data source).
- Click on the `Import`{.action} button to open your dashboard.

![grafana import](images/import2.png){.thumbnail}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,20 +50,30 @@ After you submit a job to the Data Processing platform, follow these steps to mo

![Metrics token](images/token.png){.thumbnail}

- Copy the metrics token that is now displayed in the popup.
- Open [https://grafana.com/grafana/dashboards/11844](https://grafana.com/grafana/dashboards/11844) and copy the Grafana ID by clicking the button `Copy ID to Clipboard`{.action}
- Copy and save the metrics token that is now displayed in the popup.
- Open [https://grafana.metrics.ovh.net/login](https://grafana.metrics.ovh.net/login) and login with your OVHcloud account.
- From the main menu, click `Dashboards`{.action} -> `Import`{.action}

- Click on the Grafana logo at the upper left corner of the page and from the main menu select `Data Sources`{.action}
- Click on the `Add data source`{.action} button.
- Fill in the fields like in the pictures with the following data:
- Name: Select a name for your data source (for example `My Data Source 1`).
- Type: `Warp10`
- URL: `https://warp10.gra1.metrics.ovh.net`
- Access: `direct`

![datasource](images/datasource.png){.thumbnail}

- Click on the `Add`{.action} button.
- Click on the `Save & Test`{.action} button.
- Click on the Grafana logo at the upper left corner of the page and from the main menu select `Dashboards`{.action} -> `Import`{.action}
![grafana](images/grafana.png){.thumbnail}

- In the import dashboard page, paste the Grafana ID that you copied before and click on the `Load`{.action} button.
- In the import dashboard page, inside the `Grafana.com Dashboard` textbox write `11844` and click on the `Load`{.action} button. (11844 is the ID of a Grafana template that has been created specifically for Data Processing monitoring)

![grafana import](images/import.png){.thumbnail}

- Write a name for your dashboard
- Paste the metrics token that you copied before.
- For *datasource*, select `My Data Source 1`{.action} from the menu.
- Paste the metrics token that you copied and saved before.
- For *datasource*, select `My Data Source 1`{.action} from the menu (Or any other name that you selected for your data source).
- Click on the `Import`{.action} button to open your dashboard.

![grafana import](images/import2.png){.thumbnail}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,20 +50,30 @@ After you submit a job to the Data Processing platform, follow these steps to mo

![Metrics token](images/token.png){.thumbnail}

- Copy the metrics token that is now displayed in the popup.
- Open [https://grafana.com/grafana/dashboards/11844](https://grafana.com/grafana/dashboards/11844) and copy the Grafana ID by clicking the button `Copy ID to Clipboard`{.action}
- Copy and save the metrics token that is now displayed in the popup.
- Open [https://grafana.metrics.ovh.net/login](https://grafana.metrics.ovh.net/login) and login with your OVHcloud account.
- From the main menu, click `Dashboards`{.action} -> `Import`{.action}

- Click on the Grafana logo at the upper left corner of the page and from the main menu select `Data Sources`{.action}
- Click on the `Add data source`{.action} button.
- Fill in the fields like in the pictures with the following data:
- Name: Select a name for your data source (for example `My Data Source 1`).
- Type: `Warp10`
- URL: `https://warp10.gra1.metrics.ovh.net`
- Access: `direct`

![datasource](images/datasource.png){.thumbnail}

- Click on the `Add`{.action} button.
- Click on the `Save & Test`{.action} button.
- Click on the Grafana logo at the upper left corner of the page and from the main menu select `Dashboards`{.action} -> `Import`{.action}
![grafana](images/grafana.png){.thumbnail}

- In the import dashboard page, paste the Grafana ID that you copied before and click on the `Load`{.action} button.
- In the import dashboard page, inside the `Grafana.com Dashboard` textbox write `11844` and click on the `Load`{.action} button. (11844 is the ID of a Grafana template that has been created specifically for Data Processing monitoring)

![grafana import](images/import.png){.thumbnail}

- Write a name for your dashboard
- Paste the metrics token that you copied before.
- For *datasource*, select `My Data Source 1`{.action} from the menu.
- Paste the metrics token that you copied and saved before.
- For *datasource*, select `My Data Source 1`{.action} from the menu (Or any other name that you selected for your data source).
- Click on the `Import`{.action} button to open your dashboard.

![grafana import](images/import2.png){.thumbnail}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,20 +50,30 @@ After you submit a job to the Data Processing platform, follow these steps to mo

![Metrics token](images/token.png){.thumbnail}

- Copy the metrics token that is now displayed in the popup.
- Open [https://grafana.com/grafana/dashboards/11844](https://grafana.com/grafana/dashboards/11844) and copy the Grafana ID by clicking the button `Copy ID to Clipboard`{.action}
- Copy and save the metrics token that is now displayed in the popup.
- Open [https://grafana.metrics.ovh.net/login](https://grafana.metrics.ovh.net/login) and login with your OVHcloud account.
- From the main menu, click `Dashboards`{.action} -> `Import`{.action}

- Click on the Grafana logo at the upper left corner of the page and from the main menu select `Data Sources`{.action}
- Click on the `Add data source`{.action} button.
- Fill in the fields like in the pictures with the following data:
- Name: Select a name for your data source (for example `My Data Source 1`).
- Type: `Warp10`
- URL: `https://warp10.gra1.metrics.ovh.net`
- Access: `direct`

![datasource](images/datasource.png){.thumbnail}

- Click on the `Add`{.action} button.
- Click on the `Save & Test`{.action} button.
- Click on the Grafana logo at the upper left corner of the page and from the main menu select `Dashboards`{.action} -> `Import`{.action}
![grafana](images/grafana.png){.thumbnail}

- In the import dashboard page, paste the Grafana ID that you copied before and click on the `Load`{.action} button.
- In the import dashboard page, inside the `Grafana.com Dashboard` textbox write `11844` and click on the `Load`{.action} button. (11844 is the ID of a Grafana template that has been created specifically for Data Processing monitoring)

![grafana import](images/import.png){.thumbnail}

- Write a name for your dashboard
- Paste the metrics token that you copied before.
- For *datasource*, select `My Data Source 1`{.action} from the menu.
- Paste the metrics token that you copied and saved before.
- For *datasource*, select `My Data Source 1`{.action} from the menu (Or any other name that you selected for your data source).
- Click on the `Import`{.action} button to open your dashboard.

![grafana import](images/import2.png){.thumbnail}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,20 +50,30 @@ After you submit a job to the Data Processing platform, follow these steps to mo

![Metrics token](images/token.png){.thumbnail}

- Copy the metrics token that is now displayed in the popup.
- Open [https://grafana.com/grafana/dashboards/11844](https://grafana.com/grafana/dashboards/11844) and copy the Grafana ID by clicking the button `Copy ID to Clipboard`{.action}
- Copy and save the metrics token that is now displayed in the popup.
- Open [https://grafana.metrics.ovh.net/login](https://grafana.metrics.ovh.net/login) and login with your OVHcloud account.
- From the main menu, click `Dashboards`{.action} -> `Import`{.action}

- Click on the Grafana logo at the upper left corner of the page and from the main menu select `Data Sources`{.action}
- Click on the `Add data source`{.action} button.
- Fill in the fields like in the pictures with the following data:
- Name: Select a name for your data source (for example `My Data Source 1`).
- Type: `Warp10`
- URL: `https://warp10.gra1.metrics.ovh.net`
- Access: `direct`

![datasource](images/datasource.png){.thumbnail}

- Click on the `Add`{.action} button.
- Click on the `Save & Test`{.action} button.
- Click on the Grafana logo at the upper left corner of the page and from the main menu select `Dashboards`{.action} -> `Import`{.action}
![grafana](images/grafana.png){.thumbnail}

- In the import dashboard page, paste the Grafana ID that you copied before and click on the `Load`{.action} button.
- In the import dashboard page, inside the `Grafana.com Dashboard` textbox write `11844` and click on the `Load`{.action} button. (11844 is the ID of a Grafana template that has been created specifically for Data Processing monitoring)

![grafana import](images/import.png){.thumbnail}

- Write a name for your dashboard
- Paste the metrics token that you copied before.
- For *datasource*, select `My Data Source 1`{.action} from the menu.
- Paste the metrics token that you copied and saved before.
- For *datasource*, select `My Data Source 1`{.action} from the menu (Or any other name that you selected for your data source).
- Click on the `Import`{.action} button to open your dashboard.

![grafana import](images/import2.png){.thumbnail}
Expand Down
Loading