A Security Question

To my shame, I don’t have a certificate for my blog yet, but as I was flipping through some referer logs I realized that I don’t understand something about HTTPS.

I was looking into the fact that I sometimes – about 1% of the time – I see non-S HTTP referers from Twitter’s t.co URL shortener, which I assume means that somebody’s getting man-in-the-middled somehow, and there’s not much I can do about it. But then I realized the implications of my not having a cert.

My understanding of how this works, per RFC7231 is that:

A user agent MUST NOT send a Referer header field in an unsecured HTTP request if the referring page was received with a secure protocol.

Per the W3C as well:

Requests from TLS-protected clients to non- potentially trustworthy URLs, on the other hand, will contain no referrer information. A Referer HTTP header will not be sent.

So, if that’s true and I have no certificate on my site, then in theory I should never see any HTTPS entries in my referer logs? Right?

Except: I do. All the time, from every browser vendor, feed reader or type of device, and if my logs are full of this then I bet yours are too.

What am I not understanding here? It’s not possible, there is just no way for me to believe that it’s two thousand and seventeen and I’m the only person who’s ever noticed this. I have to be missing something.

What is it?

FAST UPDATE: My colleagues refer me to this piece of the puzzle I hadn’t been aware of, and Francois Marier’s longer post on the subject. Thanks, everyone! That explains it.

SECOND UPDATE: Well, it turns out it doesn’t completely explain it. Digging into the data and filtering out anything referred via Twitter, Google or Facebook, I’m left with two broad buckets. The first is is almost entirely made of feed readers; it turns out that most and maybe almost all feed aggregators do the wrong thing here. I’m going to have to look into that, because it’s possible I can solve this problem at the root.

The second is one really persistent person using Firefox 15. Who are you, guy? Why don’t you upgrade? Can I help? Email me if I can help.

5 Comments

  1. Posted June 8, 2017 at 10:45 am | Permalink

    You might be missing this piece of the puzzle: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referrer-Policy

    There is also a hidden Firefox preference governing the default, and other browsers might have implemented this policy differently.

  2. Martijn
    Posted June 8, 2017 at 10:51 am | Permalink

    https://serverfault.com/a/520603 ?

  3. Gijs
    Posted June 8, 2017 at 10:51 am | Permalink

    https://moz.com/blog/meta-referrer-tag

    TL;DR: SEO and marketers realized this problem, and so there’s a solution: the tag allows you to opt into sending your referrers to non-https sites.

  4. mhoye
    Posted June 8, 2017 at 11:58 am | Permalink

    That’s it. Thank you all! I’m surprised our tracking protection stuff doesn’t flip those prefs, but I wonder if it should. I’ll file a bug.

  5. Daniel Veditz
    Posted June 9, 2017 at 9:31 am | Permalink

    Francois and I were just talking this week about where we could get away with just sending the origin by default. Sounds like you find the referrer useful though. You and he should talk