Windows Downloads Removed
Windows Downloads Removed2019-12-29 11:22:48
Regrettably, I have no choice but to remove Windows binary downloads from my website.
They are being flagged and blocked in spite of being absolutely clean, on the sole basis that they are uncommonly downloaded. Which by definition is going to keep happening every time I release new software.
I've written more on this subject, which you can read here:
Google's Monopoly is Stifling Free Software
But essentially, I believe I need a Windows EV code signing certificate, and I am unable to get one. The BBB has not responded to repeated attempts requesting to register my business with them, which is a pre-requisite for obtaining the certificate. In fact, I don't even know if EV code signing will be enough, Google does not specify how to prevent this warning in the future, but right now it's the only thing I know to try and resolve this issue.
I will try to come up with something, but for the time being, I am afraid that downloads are only available as source code or Cirrus CI buildbot results. I apologize for the inconvenience, but my hands are tied here.
Permalink • 24 Comments
Peter Kasting2019-12-30 00:07:05I work on the Chrome team. I've sent a message to our safe browsing folks with a link to your article and some requests for clarification and best practices.
justsomedude2019-12-30 08:54:01Wow, this is terrible. Might I suggest putting a link to this post beside the crossed out windows download?
byuu2019-12-30 14:00:25Peter, thank you so much for helping! If it's possible to answer the questions I asked on Medium (here or by e-mail is fine), I would like to update my Medium article with the answers. I really think it will help a lot of software developers out. I've heard from at least ten other people now who have run into similar issues and couldn't find answers. We are basically all afraid of drawing Google's ire here, but without being able to speak to a human we're left in the dark.
justsomedude, I'm going to wait a few days hoping for a response from a Googler. If I don't receive one, I'll create a page on this site to explain the situation in detail.
Everyone else: I'm very sorry, but I received several identical replies so I'm just going to address everything here rather than repeat your questions. I hope nobody minds ^-^; I read everyone's reply and I appreciate the feedback.
For very obvious reasons, I prefer to find a proper, approved solution rather than to try work arounds.
And unfortunately GitHub (or another offsite resource) won't help: those pages can be flagged as well, and then I'm back to where I started. It may also affect my site as well for linking to it, so now it does even more damage.
A Torrent link *could* work, but is a pretty significant barrier to most folks downloading that don't have clients.
SirYodaJedi2019-12-30 17:28:31Removing the binaries altogether seems overkill. This is a standard warning that is easily overwritten by the tech-savvy end-user, and keeps the tech-illiterate end-user from muffing up their computer too much accidentally. The warning goes away in a couple days (unless Google decides that it is legitimately dangerous, in which case Chrome prevents download altogether) and no harm is done.
It isn't actually blocking the download, just collecting analytics to make sure the user meant to download it.
I'm no Google employee; I'm just an end-user of your product that is savvy enough to avoid malware downloads, but not savvy enough to figure out Visual Studio to compile source code.
Kasper2019-12-30 19:25:20Notepad++ had a similar gripe ealier this year.
Frankly code signing certs have turned out to be a massive racket.
Clutz4502019-12-30 20:15:28If you can't link to the binaries can you create or link to a guide on how to take the source and compile it into the binary? I've never had to compile source code before and have no idea what to do. Thanks.
Peter Kasting2019-12-30 21:14:29Hey byuu -- most people are OOO right now due to the holidays, but there's some discussion going on internally with the right folks about whether/how/where to respond. Since I'm an engineer and not a PR person, I don't think I'm at liberty to share the content of those discussions, but I will mention a few things vaguely that I think are public knowledge and/or safe. Hopefully someone will eventually say something more official, since this isn't going to directly answer your questions. Until then, know that I and others internally are doing what we can to try to get you helped here -- I know you're acting in good faith.
* The binaries in question aren't actually blocked. When people using Chrome encounter "uncommon downloads", Chrome shows them a "hey we don't know whether this is safe or not" warning but allows them to go ahead and save anyway. This is different from known malware, where the UI is more like "This is dangerous, Chrome blocked it" and there's no click-through. So even if downloads are "uncommon" users can still get them.
* If you've had a file reviewed, marked safe, and later the warning re-appear for the exact same file, something is clearly wrong on our end; that's a bug that we should investigate directly, and I can probably put you in contact with the right people to escalate to if you have specific details we can look at.
* Repeatedly falsely flagging legit software as questionable causes people to lose confidence in Chrome's warnings and click through them, which is bad for user safety. So we certainly have a vested interest in learning over time what software is trustworthy, and not just blindly flagging every new .exe as bad. This is another way of saying: AFAICT your interests and Google's align here, so when there seems to be conflict, it might be a case where Hanlon's Razor is applicable.
* That said, Google also has some incentive to be vague here. Imagine that the rule was "every signed .exe was automatically marked safe", and we said that directly. Then malware authors could simply sign their executables and bypass the warning. It's very difficult to build a system that does a good job of distinguishing safe and unsafe software, in the presence of malicious actors, AND when the evaluation criteria are known. Historically, Google has generally been vague about things like how search ranking works, for similar reasons. This hinders the chances of you getting a clear response about "do things X, Y, and Z and your binaries will be marked safe", which is frustrating for all concerned.
* As you note, you're not the first person to publicly express concern over this. People being confused about what to do and mistrusting Google is bad for our public perception, so again, I think we have a vested interest in making sure developers aren't harmed/making it at least somewhat clear what the right thing to do is.
* In the absence of knowing how safety is being evaluated, I'd be hesitant to speculate on/spend money on "fixes" that may or may not help. For example, the Medium piece says "...a consequence of this is directly harming free and open source software developers from being able to release their software without paying expensive certificate authority rent-seeking fees," which implies that getting a cert will make a difference. But later on in the piece you make clear that it's not obvious whether getting a cert would make a difference, and https://support.google.com/webmasters/answer/3258249 says "...an unsigned binary is not a reason for flagging your binary as unwanted software...", which suggests that it likely wouldn't make a difference. Nonetheless, the summary ends up resulting in interpretations like https://twitter.com/perpetualmaniac/status/1211394527405072384 .
Grizzly2019-12-30 21:19:09Hello byuu, Its sad that you had to take the windows dl down. Have you thought for at least the time being to make a Mega.nz link for the time being? You can make a free account and do it from there. Just trying to help.
Now does this also affect the lib retro bsnes core as well. I ask because you can run retro arch on windows, so does it affect the binaries on there also?
Hope you can get this fix soon.
byuu2019-12-31 04:19:55Thank you Peter. I removed the links because I was outright told to by Search Console. A good first step would be to change the wording in Search Console. Right now it states: "Google has detected harmful content on some of your site's pages. We recommend that you remove it as soon as possible."
Given that I have friends who have had their entire sites blocked for hosting safe binaries (eg Kawa), and heard of GitHub pages being blocked as well (eg notdan), I'm a little hesitant to simply ignore the warning and hope that it will go away. A public statement or even reply to me that it's safe for me to continue hosting my binaries would be sufficient for me to trust that and repost them. Until then I'd like to play it safe.
So I can't definitively state that Google flagged the same file twice, because Search Console doesn't actually tell you which file it flagged. But I can say that higan v107 is the only new binary I have posted recently. I also had binaries up for bsnes and beat (the latter is a binary patcher like Xdelta), so maybe it flagged one of those next and the timing was just very unfortunate.
I understand your reasoning for not providing exact answers. It seems that my site being up for 14 years doesn't factor into any sort of trust of my domain. Could I at least ask if signing my code (with or without an EV certificate) would begin to associate trust with my binaries going forward, and would that built-up trust reset every two years when I need a new signing certificate? (the CN should be the same of course.) Or does Safe Browsing ignore the certificate entirely? That could save me a lot of money to have an answer on.
The full passage from your link states: "We recommend that you sign your code. While an unsigned binary is not a reason for flagging your binary as unwanted software, we recommend programs have a valid and verified code signature issued by a code-signing authority that presents verifiable publisher information."
Stating that not having a signature won't necessarily flag software as unwanted doesn't clarify if having one will help prevent it. Given that "We recommend that you sign your code" is bolded for emphasis, I don't feel the Medium article was necessarily deceptive, but I will make a good faith effort to make corrections to my article and post those corrections everywhere I can find. If code signing is truly irrelevant, I feel it shouldn't be in the guidelines section at all. Google is effectively advising developers to buy these certificates which will either require doxing ourselves (personal certs) or spending exhorbitant sums of money (business EV certs.)
Nonetheless, I do sincerely apologize if my bringing light of this caused any spread of misinformation, that is the last thing I want. But in my defense, I would like to add that if I could have spoken to a real person about this, I wouldn't have needed to address this publicly. I do agree that our interests are aligned, at least on this matter, and I'd be most happy if we can both come up with something that benefits everyone from this.
Thank you again for your continued help with this matter!
byuu2019-12-31 04:27:01SirYodaJedi2019 and Clutz450, please be patient a few more days. It seems like bad timing to have happened around the holidays. Once I'm sure on how to move forward, I'll take steps to remedy the situation.
Grizzly, offsite linking doesn't alleviate the underlying risk. This won't affect RetroArch as I do not distribute libretro binaries of bsnes myself. Their project handles that.
Hias2019-12-31 08:19:12I don't think this is a good idea. Windows people who do not want to compile their binaries for themselves will be looking for unofficial win32 binary sources, e.g. on forums. That way your users could end up with malicous code the binary provider included.
Could you not simple create a win32 download page (extra subfolder-path), with does not get indexed with Google (robots.txt or similiar)?
This way Google does not see the binary links and can not complain about them...?
Thanks and regards
byuu2019-12-31 09:02:27Hias, it will crawl noindexed pages anyway, and blocking it would be considered cloaking and that risks a much bigger penalty.
It's important to me to fix this the right way that Google approves of.
0072019-12-31 13:24:28Im starting to really dislike google.
Joe2020-01-01 01:34:55Google is trash byuu. I hope you find a solution that is satisfactory to you. If anyone needs the windows binary or any others of higan or bsnes I have archived all of your work (that I have been able to find) for the past seven years.
Wowfunhappy2020-01-02 15:54:56Byuu, would you consider putting up hash's of the exe's in place of the download links until this gets sorted?
I don't personally need it, but it seems like a good idea to keep others safe. This situation is invariably going to drive people off site.
Peter Kasting2020-01-02 22:30:27byuu: Regarding whether it's safe to continue hosting binaries, please see the "Uncommon Downloads" section of https://support.google.com/webmasters/answer/9044101 , which states this:
"...Find and confirm whether your downloads conform to the download guidelines. If any of the downloads violate these guidelines, remove them. If they do not violate these guidelines, you do not need to remove them, simply request a review, as described next. ..."
So I think the official public statement would be that you don't need to remove downloads that don't violate the download guidelines ( https://support.google.com/webmasters/answer/3258249#guidelines ).
Regarding whether signing helps, note that code signing is required for some OSes to not give users warning dialogs on running, and this guideline is an advisory one inside a section about how to make sure your users don't see any warnings. It's also the _only_ advisory guideline; everything else is normative. Sorry, that's as non-vague as I can be :(
Finally, just because it's been radio silence from me -- the discussion is still ongoing on this end. I'm hopeful something more definitive will happen, eventually. It takes a frustratingly long time...
byuu2020-01-03 02:45:32Peter, thanks again for the status update. I understand it will take some time. I agree with you and I really hope something good comes of this.
As stated before, the review process didn't work in my case. I also really don't want to have to request a review every time I release a new build. My intention with the higan v107 series was to post new builds every few days and collect feedback (via byuu.org/higan/feedback) -- unfortunately those plans have been kind of nixed here =/
For the time being, I've set up offsite mirrors for obtaining my binary releases. Once this is all resolved, hopefully I can move back to hosting the binaries on my site.
Long-term, hopefully I can figure out something with code signing, but I'm not willing to obtain a personal developer certificate in my legal name, so it might prove difficult.
Wowfunhappy, there are hashes in my latest news post already. It looks like it's going to be a while longer, so I'll revamp my bsnes and higan download sections here soon.
Peter Kasting2020-01-03 10:28:28As far as I know, we're not aware of things being re-flagged after they've been reviewed, since the "uncommon downloads" warning is supposed to only be for cases where there has not actually been a review. I would strongly suspect something like "one binary was flagged, then a different binary was flagged" instead. Like I said before, in any case where you have strong reason to believe something's gone wrong, try and collect details and I'll try to forward it to someone who can investigate thoroughly.
Regarding requesting reviews on every build release, note that "these warnings are lifted automatically" ( https://support.google.com/webmasters/answer/9044101 ) and that (as previously mentioned) the files aren't being blocked. Your site should not be blocked either; if it is (if users are getting some kind of interstitial trying to visit your site in this case), please let me know, as that would mean something went wrong. All users should be seeing for an "uncommon download" is an "are you sure" sort of message in the actual Chrome download shelf on downloading the file in question; and eventually people won't see that as the system figures out that the binaries are safe. Requesting reviews should, hopefully, merely speed up the process.
byuu2020-01-03 13:58:16Peter, I don't know how else it would happen, I only posted one new binary in the past month. Maybe Google suddenly found bsnes or beat after the fact and flagged that, I'm not sure. Your team has my permission to look at my domain and see what happened if you like; it doesn't actually tell me which page or file it flagged so I really don't know.
I'm still very uneasy getting warnings in Search Console every time I post a new binary, but if I have an inside contact at Google in case it escalates to a site warning, I'm willing to rehost my binaries here again. Would you be able to use my contact page (byuu.org/contact) and drop me your contact info? I also had a separate question if you have the time, but if not I'll leave it for just this one issue ^-^;
Thanks again for your help with this!
trig2020-01-03 20:48:08I don't understand. Is your web host refusing to host the files?
Not everyone uses a browser that scans downloads / checks signatures. I don't understand why Google is so involved in the process of downloading a file.
I am not a Chrome user. I don't have Google software. Is there any way I can download??
byuu2020-01-05 04:33:15trig, 80% of web browsers use Safe Browsing out of the box. That's far too big of a user group to simply say "well turn off this overly aggressive feature in your browser."
Google is involved in everything: that's the power of being a monopoly. You should read up on what they're doing to publishers on the web with AMP.
You can download my software from https://byuu.itch.io instead now.
William2020-01-22 23:50:14If you were to change hosting providers, i.e. to Linode, would that help out at all? I've had bad experiences with the company that hosts your website's server.
I run a website much less popular than yours, and have never had issues with Google Safe Browsing (knock on wood).
byuu2020-01-23 07:59:22William, it wouldn't be related to the hosting. Google flagged my binaries on itch.io as well.
The main reason I don't use something like Linode is that I require a VPS that offers FreeBSD via KVM hosting, which limits my options significantly.
Sterophonick2020-01-25 21:23:10Damn. That really sucks.