Add abs vm get #53

mikkergimenez · 2019-10-21T23:17:24Z

Status

Needs Feedback

Description

This adds the ability to request vms from ABS. Since ABS API works a bit differently from the vmpooler API, this has to work a bit differently as well, polling the API before returning the vms. This is a work in progress as the text could be a bit better and the list active vms functionality does not work.

FIXME

Related Issues

Todos

Tests
Documentation

Reviewers

@highb
@briancain

highb · 2019-10-21T23:30:27Z

Gemfile

@@ -7,6 +7,7 @@ gemspec
 gem 'rake', :require => false

 group :test do
+  gem 'faraday', '0.15.4'


You probably want to bump this over in vmfloaty.gemspec, not here

Fixed per slack convo, deleting reference to Faraday in spec headers.

highb · 2019-10-21T23:37:35Z

README.md

@@ -53,7 +53,7 @@ $ floaty --help
 Grabbing a token for authenticated pooler requests:

 ```
-floaty token get --user username --url https://vmpooler.example.net/api/v1
+floaty token get --user username --url https://vmpooler.example.net/api/v1 --api (vmpooler|abs)


Did you actually add this --api flag?

➜ vmfloaty git:(add_abs_vm_get) ✗ be bin/floaty token get --user brandon.high --url https://cinext-abs-prod.delivery.puppetlabs.net --api abs invalid option: --api

No, that was a previous iteration before I realized there was a "service" feature.

mikkergimenez · 2019-10-30T20:59:08Z

FYI: This PR needs some work before it's ready to go go, I'm working on it now.

mikkergimenez

I've tested the code in this PR manually, and it's ready to be tested. This adds the get token, get vms, list vms and delete vms functionality to floaty. There are a few things to note how it works:

ABS requires the username of the person requesting vms in the payload, so I had to add username as a parameter to all the get functions for the various services.
ABS does not return the vm immediately, it adds the request to a queue so the ABS get here polls the endpoint every 10 seconds until the request is fulfilled.
Job ID is generated by the client, and is a 13 digit (millisecond) unix timestamp in order to "guarantee" uniqueness in case two people request around the same time.

mikkergimenez · 2019-10-30T23:25:46Z

lib/vmfloaty/abs.rb

+    response = conn.post "api/v2/request", reqObj.to_json
+
+    i = 0
+    retries = 360


Not sure what to put here, this would retry for an hour, the user can always ctrl-c.

What about a timeout ?

mikkergimenez · 2019-10-30T23:26:40Z

lib/vmfloaty/nonstandard_pooler.rb

@@ -22,7 +22,7 @@ def self.list_active(verbose, url, token)
    status['reserved_hosts'] || []
  end

-  def self.retrieve(verbose, os_type, token, url)


ABS requires posting the username, and the params have to be the same.

mikkergimenez · 2019-11-08T22:05:35Z

There are some endpoints in ABS that don't consistently respond to /api/v2/ they only respond to This makes writing this code challenging. Merging a change on Monday so that ABS has that feature so we don't have to write exceptions here, then after some manual testing, should be good to merge this change.

highb · 2019-11-09T17:06:50Z

I’ll be taking a look on Monday.

…

On Fri, Nov 8, 2019 at 2:27 PM Brian Cain ***@***.***> wrote: Assigned #53 <#53> to @highb <https://github.com/highb>. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#53>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFZQKHBS42JK4YOKVWUF6TQSXRVFANCNFSM4JDHZXEA> .

highb

Tested it out with /api/v2 appended to the URL, we'll work out not making that part of the URL a requirement in another PR.

➜  vmfloaty git:(add_abs_vm_get) ✗ be bin/floaty get centos-7-x86_64 --service abs

Requesting VMs with job_id: 1573510037352.  Will retry for up to an hour.
Waiting 10 seconds to check if ABS request has been filled.  Queue Position: 85... (x1)
- adamant-bride.delivery.puppetlabs.net (centos-7-x86_64)

And getting regular pooler VMs still works:

➜  vmfloaty git:(add_abs_vm_get) be bin/floaty get centos-7-x86_64 --service vm

- deputy-dip.delivery.puppetlabs.net (centos-7-x86_64)

highb · 2019-11-11T22:18:57Z

I think this is good to merge once the Rubocop fixes are in and anyone else who wants to test it out chimes in.

mikkergimenez · 2019-11-27T00:05:01Z

Will mege this in after the holiday.

highb · 2019-12-02T20:17:31Z

lib/vmfloaty/abs.rb

+
+      req_hash['allocated_resources'].each do |vm_name, _i|
+        if hosts.include? vm_name['hostname']
+          if all_job_resources_accounted_for(job['allocated_resources'], hosts)


@mikkergimenez Is this job variable defined somewhere? When I try using delete it seems to blow up here.

➜ vmfloaty git:(add_abs_vm_get) be bin/floaty delete secular-relay.delivery.puppetlabs.net --service abs --trace bundler: failed to load command: bin/floaty (bin/floaty) NameError: undefined local variable or method `job' for ABS:Class /s/vmfloaty/lib/vmfloaty/abs.rb:90:in `block (2 levels) in delete' /s/vmfloaty/lib/vmfloaty/abs.rb:88:in `each' /s/vmfloaty/lib/vmfloaty/abs.rb:88:in `block in delete' /s/vmfloaty/lib/vmfloaty/abs.rb:85:in `each' /s/vmfloaty/lib/vmfloaty/abs.rb:85:in `delete' /s/vmfloaty/lib/vmfloaty/service.rb:115:in `delete' /s/vmfloaty/lib/vmfloaty.rb:229:in `block (2 levels) in run' /Users/highb/.rbenv/versions/2.5.6/lib/ruby/gems/2.5.0/gems/commander-4.4.7/lib/commander/command.rb:182:in `call' /Users/highb/.rbenv/versions/2.5.6/lib/ruby/gems/2.5.0/gems/commander-4.4.7/lib/commander/command.rb:153:in `run' /Users/highb/.rbenv/versions/2.5.6/lib/ruby/gems/2.5.0/gems/commander-4.4.7/lib/commander/runner.rb:446:in `run_active_command' /Users/highb/.rbenv/versions/2.5.6/lib/ruby/gems/2.5.0/gems/commander-4.4.7/lib/commander/runner.rb:68:in `run!' /Users/highb/.rbenv/versions/2.5.6/lib/ruby/gems/2.5.0/gems/commander-4.4.7/lib/commander/delegates.rb:15:in `run!' /s/vmfloaty/lib/vmfloaty.rb:454:in `run' bin/floaty:8:in `<top (required)>'

highb · 2019-12-02T20:19:48Z

lib/vmfloaty/abs.rb

+    conn = Http.get_conn(verbose, url)
+
+    res = conn.get 'summary'
+    JSON.parse(res.body)


Heh, this just returns OK which isn't valid JSON.

➜ vmfloaty git:(add_abs_vm_get) be bin/floaty status --service abs error: 765: unexpected token at 'OK'. Use --trace to view backtrace

I hadn't done status yet, but since it seems relatively easy I just pushed up a change with the status command fixed.

highb reviewed Oct 21, 2019

View reviewed changes

mikkergimenez force-pushed the add_abs_vm_get branch 2 times, most recently from beb3aa3 to 9eb7e49 Compare October 21, 2019 23:57

highb mentioned this pull request Oct 22, 2019

Update Faraday to 0.15, remove unnecessary headers #54

Merged

briancain self-assigned this Oct 22, 2019

briancain added the enhancement label Oct 22, 2019

mikkergimenez force-pushed the add_abs_vm_get branch 2 times, most recently from f45677d to 27a19d6 Compare October 30, 2019 23:24

Adding command to list pools out of ABS

de7d0fd

mikkergimenez force-pushed the add_abs_vm_get branch from 27a19d6 to 4bf7ee9 Compare October 31, 2019 18:43

mikkergimenez added 2 commits October 31, 2019 14:25

Rebasing fixed tests

a77ea84

Adding delete and get active requests

7e27542

mikkergimenez force-pushed the add_abs_vm_get branch from 4bf7ee9 to 7e27542 Compare November 4, 2019 19:42

mikkergimenez commented Nov 4, 2019

View reviewed changes

mikkergimenez force-pushed the add_abs_vm_get branch 2 times, most recently from 77a51ed to 6b2f2e6 Compare November 4, 2019 22:28

briancain assigned highb Nov 8, 2019

mikkergimenez force-pushed the add_abs_vm_get branch from 6b2f2e6 to 78f42a1 Compare November 11, 2019 19:31

highb previously approved these changes Nov 11, 2019

View reviewed changes

mikkergimenez dismissed highb’s stale review via 01b5457 November 15, 2019 21:40

mikkergimenez force-pushed the add_abs_vm_get branch from 78f42a1 to 01b5457 Compare November 15, 2019 21:40

highb reviewed Dec 2, 2019

View reviewed changes

mikkergimenez force-pushed the add_abs_vm_get branch from 01b5457 to e0384a3 Compare December 3, 2019 16:56

mikkergimenez force-pushed the add_abs_vm_get branch from e0384a3 to 4ec084b Compare December 4, 2019 18:23

Adding delete and get active requests

d963e35

mikkergimenez force-pushed the add_abs_vm_get branch from 4ec084b to d963e35 Compare December 4, 2019 19:37

Brandon High added 2 commits December 4, 2019 16:26

Update Rubocop to not use old cop name

9a2cd81

Fix rubocop issues

1e67715

highb approved these changes Dec 5, 2019

View reviewed changes

highb merged commit 6954321 into puppetlabs:master Dec 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add abs vm get #53

Add abs vm get #53

mikkergimenez commented Oct 21, 2019 •

edited

Loading

highb Oct 21, 2019

mikkergimenez Oct 21, 2019

highb Oct 21, 2019

mikkergimenez Oct 21, 2019

mikkergimenez commented Oct 30, 2019

mikkergimenez left a comment •

edited

Loading

mikkergimenez Oct 30, 2019

jcoconnor Nov 13, 2019

mikkergimenez Oct 30, 2019

mikkergimenez commented Nov 8, 2019

highb commented Nov 9, 2019 via email

highb left a comment

highb commented Nov 11, 2019

mikkergimenez commented Nov 27, 2019

highb Dec 2, 2019

highb Dec 2, 2019

mikkergimenez Dec 3, 2019

Add abs vm get #53

Add abs vm get #53

Conversation

mikkergimenez commented Oct 21, 2019 • edited Loading

Status

Description

Related Issues

Todos

Reviewers

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikkergimenez commented Oct 30, 2019

mikkergimenez left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikkergimenez commented Nov 8, 2019

highb commented Nov 9, 2019 via email

highb left a comment

Choose a reason for hiding this comment

highb commented Nov 11, 2019

mikkergimenez commented Nov 27, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikkergimenez commented Oct 21, 2019 •

edited

Loading

mikkergimenez left a comment •

edited

Loading