How to add a header to a request? (using python bindings + webdriver)

[lukeis]

Originally reported on Google Code with ID 2047

Can't find any documentation for how to do this.

I'm writing a new test script using the new Selenium Webdriver code with the python
bindings.

Basic tests work fine, I can start the browser, and control it just fine.

But now I have to add a custom header, and I can't see how. I can see a tantalising
add_custom_request_header() in the top level selenium file, but can't see how that
helps me when all I have is a Firefox webdriver object.

Code looks something like this:

from selenium import webdriver
browser = webdriver.Firefox()
browser.get("/")

How do I get a custom header into that get? 

Reported by rwillmer on 2011-07-13 14:17:32

[lukeis]

Selenium does not provide this capability, as it is not something a user can do. It's
recommended you use a proxy to inject additional headers to the requests generated
by the browser.

Reported by eran.mes on 2011-07-13 14:28:27

• Status changed: Invalid

[lukeis]

By that argument, you should remove the add_cookie() function.

So if I wanted (for example) to test that my site worked well for users with different
language preferences, rather than just tweaking the "Accept-Language" header, I should
set up a separate proxy for each language? And if I wanted to run similar tests for
"Accept-Charset", I'd need to set up a separate proxy for each of those two? Doesn't
sound like this is going to scale...

Reported by rwillmer on 2011-07-13 14:42:57

[lukeis]

There's been extensive discussion about this in issue 141 - in short, things like the
BrowserMob Proxy [1] are very good at this, and make it scale pretty well

1: http://proxy.browsermob.com/

Reported by dawagner on 2011-07-13 15:25:39

• Status changed: Duplicate
• Merged into: WebDriver lacks HTTP response header and status code methods #141

[lukeis]

This is not invalid and has been incorrectly closed/merged.

1) 141 is about response Headers, this is about request headers.
2) It is an wrong to suggest a user cannot change these, they can in several ways,
by changing locale information, arriving from a different refer, installing different
plugins.  However none of these can be handled conviniently for automated web testing.
3) Named key value pairs are ingral part of HTTP/2 request for stream metadata.

Reported by Martin.Spamer on 2013-12-11 09:53:31

[lukeis]

Please reopen - fully agree with #4, setting headers on request is not the same as getting
them from the response, and is integral in a number of complex testing scenarios. We
had to ask a GSOC intern to help at Wikipedia with this.

Reported by [email protected] on 2013-12-16 11:30:47

[lukeis]

The "add_custom_request_header" method is for Selenium RC and requires that you're running
the Selenium Server as a proxy server.  This is basically legacy and will almost certainly
be removed in Selenium 3.  But if you want to use it, here's an example in Ruby for
handling Basic authentication credentials:

http://blog.mogotest.com/2010/06/23/how-to-perform-basic-auth-in-selenium/

If you're looking to add request headers with the Selenium WebDriver protocol, you
should look at BrowserMob Proxy.  Since this sits between the client and the remote
server, it's able to see traffic in both directions.  Indeed, I use this to handle
Basic authentication with WebDriver and it works quite well.  See the README for examples
of adding custom headers:

https://github.com/lightbody/browsermob-proxy

I should note that BrowserMob Proxy allows you to create multiple "virtual" proxies,
so you only need to run one process that can handle hundreds of proxies for you.  So
the multiple proxies per language situation really isn't much more expensive than the
single language case.

Reported by nirvdrum on 2013-12-16 14:12:59

[lukeis]

There are three separate issues with the way this was closed. First, it absolutely is
not a duplicate of issue 141, since it deals with responses, not requests.

Second, the same ludicrous argument discussed to death on issue 141 that the requested
functionality isn't appropriate because it's not something a "user can do" has been
trotted out. There are numerous examples of operations within the WebDriver docs of
things a user can't do directly. One I've used is to find an element on a page via
XPath. Until WebDriver has advanced machine vision, saying it won't do anything a user
can't is a lame excuse at best. There may be a valid reason for not implementing this
functionality within WebDriver, but the official policy isn't one.

Third is the question of whether setting up a previously unknown software package with
its own interface just to do a single operation which is trivial with tools similar
to WebDriver is reasonable. I had little difficulty converting Twill-based tests to
WebDriver except for a couple of things, including the lack of a way to get the HTTP
response code from WebDriver. I may eventually set up a proxy to provide what WebDriver
does not, but the barrier to getting at one integer which Twill has no trouble providing
seems arbitrary and unreasonable to me. If I needed to what the reporter of this issue
needed to do, I'd probably just use Twill.

Reported by JonathanRRogers on 2013-12-17 01:05:37

[lukeis]

The BrowserMob Proxy code was extracted from Selenium so that it could get the attention
it deserves outside the project proper.  I suspect this issue was merged into #141
because the direction of the network traffic doesn't matter; BrowserMob Proxy sits
in the middle and can both inject headers and record traffic.

At this point, if you'd really like to see request headers handled by the driver, you
should comment on the W3 WebDriver specification:

https://dvcs.w3.org/hg/webdriver/raw-file/default/webdriver-spec.html

Several of the driver implementations are outside the control of the project already
and more are slated to.  So there's no feasible way to do what you'd like without a
change to the specification anyway, unless you have a proxy server sitting in between.

Reported by nirvdrum on 2014-01-28 04:26:32

[lukeis]

One vote to open this issue. This should not be merge to #141 (in my opinion).

Reported by harry.soehalim on 2014-07-07 21:27:13

[lukeis]

Seems other drivers supports headers argument, see: http://stackoverflow.com/q/19524069/55075

For example:

    rv = self.app.get('api/v1.0/{0}'.format(ios_sync_timestamp), headers=headers)

Basically adding 2nd argument should do the work.

Reported by kenorb on 2015-05-03 14:19:46

[lukeis]

Similar:  Issue #34
See also: https://github.com/SeleniumHQ/selenium/issues/453

Reported by kenorb on 2015-05-03 19:03:41

[lukeis]

Reported by luke.semerau on 2015-09-17 18:13:15

• Labels added: Restrict-AddIssueComment-Commit