wp_remote_get referrer not being sent correctly

[schrapel]

We use an API that would check the referrer url. In the process of upgrading to 4.6 their API now responds telling us an invalid referrer was provided. If I roll back to 4.5.3 and run wp_remote_get it works without problems.

[swissspidy]

Hey there, thanks for your report!

What kind of arguments are you passing to wp_remote_get()? Some sample code to look at would help greatly.

Looking at the code from 4.5 and 4.6, no referer was ever set by WordPress in 4.5. Now with 4.6, Requests seems to set the referer to the URL being requested:

curl_setopt($this->handle, CURLOPT_URL, $url);
curl_setopt($this->handle, CURLOPT_REFERER, $url);

Citing RFC 2616:

The Referer[sic] request-header field allows the client to specify, for the server's benefit, the address (URI) of the resource from which the Request-URI was obtained

Seems legit in that case.

[schrapel]

Why would the CURLOPT_URL be the same as the CURLOPT_REFERER? If I manually set CURLOPT_REFERER to the URL of the domain I am sending from then I get the response I desire.

The code is below but I obviously removed our API key. I'm going to try get another API key for you that I can share here

<?php
wp_remote_get('http://api.embed.ly/1/feature?feature=oembed&key=***')

[swissspidy]

Why would the CURLOPT_URL be the same as the CURLOPT_REFERER?

I don't know why it was set that way. It has been like this in Requests ever since, see ​https://github.com/rmccue/Requests/commit/78d4f3c117642445cf729a7eaaaa87d8d3897fd6.

There's an ​open issue on GitHub, so maybe @rmccue can chime in.

[schrapel]

I had a look at Guzzle and they don't seem to do this. Hopefully @rmccue can offer his opinion on this

[jeremyfelt]

#37888 was marked as a duplicate.

[differentthemes]

Any news here? Will it be fixed in next updates as a additional argument?

[swissspidy]

@differentthemes As Requests is an external library, this requires some upstream changes first. You can follow ​https://github.com/rmccue/Requests/issues/232 for updates on that.

[differentthemes]

A month has gone and still don't see any changes here :(

[tristangemus]

Wordpress 4.5.4 did not include the referrer at all by default. The referrer should be the root URL of the referring website, site_url should be an adequate solution to this. See patch.

[dd32]

Small note, You can send a Referer header along with your wp_remote_get() which will cause cURL not to send the Referer.
Unfortunately, the Streams handler will send a Referer: header anyway.

This should be fixed upstream at ​https://github.com/rmccue/Requests/issues/232 - inputs and patches would be welcome there.

[differentthemes]

Replying to tristangemus:

Wordpress 4.5.4 did not include the referrer at all by default. The referrer should be the root URL of the referring website, site_url should be an adequate solution to this. See patch.

Thanks! That helps, but only we can't now force all our customers to edit they wp files. This really should be fixed ASAP at least as a new parameter, if not even by default.

[tristangemus]

Replying to differentthemes:

Replying to tristangemus:

Wordpress 4.5.4 did not include the referrer at all by default. The referrer should be the root URL of the referring website, site_url should be an adequate solution to this. See patch.

Thanks! That helps, but only we can't now force all our customers to edit they wp files. This really should be fixed ASAP at least as a new parameter, if not even by default.

Considering this was selected as a solution, it could be part of an upcoming Wordpress release. Hoping to get some support on this or another solution as the referrer being set as the destination URL does not make any sense at all.

Another solution is to not include the referrer and make it an option at all as it is optional in HTTP.

I'd also look into this solution - https://core.trac.wordpress.org/ticket/37820#comment:10

[differentthemes]

Referer header

[differentthemes]

Not quite sure if I get it right, but like this it doesn't work.

Edited: Sorry missed this part

$referer = site_url();

[differentthemes]

I have checked WordPress 4.7 BETA 2, but seems like the changes wan't been made there..

[desrosj]

I am going to close this one out as reported upstream, since action on the GitHub repository is required for this to be merged into WordPress Core.

Please center discussion on ​Requests issue 232.

[nico23]

I am confused about this. I have not digged though the source but looking at the patch it seems the URL is hard-coded in. However, if I got this right, people say we can set the referer with in the $argsheaders? the screenshot above shows code like this and the codex says the arrays should contains a array with header lines. However I have seen a lot of code that uses key-value pairs in the array so my questions are.

1. Can I set the referer with the $args on wp_remote_get/post.
2. If yes, do both of this ways work?
3. Any easy way to test this?

<?php
        $referer = site_url();

        $wp_remote_get_args = array(
                'headers' => array( "Referer: $referer\r\n" );
        );

<?php
        $wp_remote_get_args = array(
                'headers' => array( 'Referer' => site_url() )
        );

        $response = wp_remote_get( $api_url, $wp_remote_get_args );

Edit

Ok I just tested it on ​https://webhook.site awesome site for testing requests. So turns out the code in the Screenshot from @differentthemes above is actually wrong. It end with with header 0 so not sure where this keyless array with newline is coming from. So only my 2nd example will work.

After thinking about this I think its fine if it stays this way or of WP sends no referer at all for privacy reasons as long as you can manually set it if needed. And having it to the same URL as requested seems kind if the same as setting none all all.