Falling Out of Love With EPH 19

In 2015, I was pretty clearly on the EPH train. I sure did vote for its passage. And now it’s 2016, and I’m tweeting out stuff like this:

So yeah, I am not surprised that got some confused reactions.

To be perfectly frank, this year I voted with the minority against ratification of EPH. I’m not going to claim that the passage of EPH is going to Destroy Life As We Know It And The Cats And Dogs Will Play Shuffleboard Together. I am not upset that it passed. But I’m no longer convinced that it’s the great solution I thought it was in 2015, because I’ve been doing a lot of listening and thinking.

First of all, we had some data to look at with how EPH would have processed this year and previous years.

  • It’s become very obvious that EPH isn’t the silver bullet that will slay all slating that I think a lot of us convinced ourselves it was. For the most part, it would have only made a 1-2 nominee difference in this year’s categories, and a weaker difference in 2015, which seems very counterintuitive considering the slating was much stronger last year. (This year there were actually more nominating ballots than final voting ballots, which is weird.) So yes, it’s better to have one or two “genuine” nominees in a category than none at all. But having categories that are still composed mostly of shit is really not a workable long-term solution.
  • I’m extremely concerned how EPH will affect the dramatic presentation categories, which were not included in the report on Sunday just because they were such an unholy mess to figure out. But one point we can’t ignore is that for those categories, there’s so much less on offer that people are likely to cluster around certain works in ways that will act as a natural slate—and considering that this is all about trying to stop the deliberate manipulation by the shitbag faction of the fandom, I don’t think it’s fair that the organic growth of something that would appear slate-like to the un-nuanced eye of an algorithm be treated in the same way.
  • Looking at the previous year’s data, EPH would have eliminated one of the Hugo winners from 2014. In the meeting, Dave mentioned this as something of a surprising aberration. And if it’s a thing that only happens once, good. But I find myself very concerned that similar unintended consequences could happen. The rule should really be first do no harm, here.

After talking to a lot of people and listening, I’ve also got some concerns about how EPH as a system will affect voter confidence in the process.

  • It’s still damn complicated. Yes, I’m aware that you can boil an explanation of EPH down to a single powerpoint slide. And if you grasp it well enough to explain it from that slide, good for you. I know I can’t, and I rarely have a powerpoint at hand when this comes up in conversation. I don’t think this is terribly good when it comes to trying to promote participation in the process. (Seriously, it’s hard enough to explain how the ranking works for the final voting.)
  • And considering all the points above? If I’m concerned and I’m among the group of high information voters when it comes to the mechanics, how is someone who hasn’t gotten to go through the all the presentations and question times and the like going to feel?
  • EPH is effectively a black box. It doesn’t really allow for simple hand verification. That doesn’t help efforts at transparency, which is the number one thing that encourages people to have any kind of faith in the system. And let’s not forget the conspiracy-mongering, just-repeat-something-until-it’s-true shitbag that made all this necessary. It’s not that I give a crap one about actual comic book villain Theodore Beale’s good opinion, but the less transparent something is, the easier it is to make up things about it.

I’ve become a convert to the Three Stage Voting proposal. We get told again and again that the practice has always been to leave it to the voters and have faith in them. And we’ve seen over the last two years that the voters as a whole have definite opinions about people who try to manipulate them, the system, and their beloved award. So I’m willing to have faith that the voters can decide in a semifinal if they think someone doesn’t have any business being one the final ballot with a purely up or down vote.

That said, I think we’re going to be fine. EPH has been amended so that it can be easily suspended for a year by the business meeting if there’s something going really haywire with it. We’ll see what 2017 brings.

By the way, Cheryl Morgan has written a good post that covers this and other WSFS stuff. You should read it.

19 thoughts on “Falling Out of Love With EPH

  1. Reply shaunduke Aug 23,2016 20:26

    The problem isn’t the “voting” stage. It’s the nomination stage. The voters have shown again and again that they won’t put up with this kind of thing, but that doesn’t stop people from pushing out a lot of folks who otherwise would be on the ballot. Unless the Three Stage will solve that, I don’t see a solution to this problem that doesn’t involve some very radical decisions.

  2. Reply Gary C Aug 24,2016 15:50

    The biggest problem with EPH is that there are too many ways to sum up what it doesn’t do; namely, to produce a shortlist that is acceptable to the larger body of Hugo voters, and that represents a sampling the best that SFF has to offer in a given year. At this point I’m not even convinced it will do anything to dampen the effect of slate voting for next year’s awards. I agree that 3SV is a clearer and more effective solution, and I think the vote threshold for rejecting a work is significant enough to prevent abuse – but it is a system that can still be gamed. The griefers can simply produce a slate made entirely of “hostages” and still effectively control the nominations. They have shown that they are determined enough to do this. I can see them slating the likes of Neal Stephenson and Alastair Reynolds and Lois McMaster Bujold and so on, forcing these individuals to stress over whether to accept the nomination they may or may not have otherwise come upon fairly, and making it nearly impossible for enough voters to reject them in the second stage. Yes we will be spared having to no award Vox Day and a bunch of Castalia House detritus on the final ballot, but they will still get to decide who the nominees are. I know the intention behind 3SV is to act as a deterrent to bad behavior in the first place, but as long as they are determined to disrupt the process I believe they will continue to do so.

    • Reply Rachael Aug 24,2016 17:01

      Yeah, unfortunately we can only control the process on this end. We can’t compel period to grow the fuck up.

  3. Reply homunq Aug 26,2016 07:36

    Thanks for a thoughtful post. I think it deserves a response.

    I’m the EPH originator. Please don’t use my name here; I’m still clinging to a shred of pseudonymity for this nym.

    On the charge of “too complex” and “unverifiable”, there really is no defense. It’s just a matter of whether you think the advantages of EPH (slate resistance, diversity, reduced strategic incentives/ increased fairness) are worth these downsides.

    But on your concerns around the BDP categories, and especially BDP:Long, I think you have it wrong. And, frankly, I think that this is partly Dave’s fault. At the meeting, I didn’t want to highlight this disagreement, because frankly I’m grateful for all he’s done, and deeply cognizant that he has been acting in good faith and with the highest ideals of impartiality. But he got this one wrong.

    BDP: long is an unusual category, in that there are relatively few strong candidates. This makes it less likely, not more, that EPH will change results in a year without explicit slating. Say that there are 4 dominant candidates plus bunch of also-rans. The typical ballot will still not nominate all 4 of the strong ones; tastes still differ, and fans, like anyone else, enjoy making a pet of the occasional contrarian opinion. So each ballot will nominate 1-3 of the strong, and maybe an also-ran or two. That’s a lot more overlap than a typical category; but it’s not more “slate-like”. You can still see that a ballot votes for work A, and remain mostly in the dark in terms of guessing which other works it nominates.

    The upshot, in EPH, will be that the gap between “points” and “nominations” will be a bit higher for BDP:long, but that the rank ordering of “points” will remain largely the same as that of “nominations”. And in this kind of situation, EPH does not change the results from a system based solely on “nominations”.

    Yes, it’s true, there will be some slight correlation between two ballots, along the dimension of “mainstream” versus “alternative”. And “mainstream” will act as effectively a slate, so EPH could have an effect there, contrary to what I said in the previous paragraph. But I think that effect would actually be healthy; to get rid of the weakest “mainstream” option, as most people who voted for that would have other finalists they like, and replace it with the strongest “alternative” one. In an alternate version of this year, with EPH and without slates, that might mean “Age of Ultron” would drop off; it’s pretty much consensus by now that it’s not the strongest of the MCU films, and most people who nominated it probably actually preferred at least one of The Martian, The Force Awakens, or Mad Max.

    What about BDP:Short? Honestly, I didn’t include that in my paper with Bruce because data cleaning was a nightmare. I did a lot of work cleaning up the 2015 ballots, and I think I got it right, but I still couldn’t be fully confident that I’d made exactly the same calls as the admins. What do you do when people write things like “any Korra episode” in BDP:Short? Since my first priority was not to undermine the legitimacy of the admin decisions, I punted. But… Dave was the admin, and he had the cleaned-up data. I didn’t want to fight with him about leaving this category off of his analysis, but I actually do think he made the wrong choice, even though he thought that I agreed with him.

    Obviously, in BDP:Short, each show is a natural slate. But most individual-show fandoms are coherent enough that there’s clearly one episode that stands out. In that case, minor fans of that show can afford not to act like a slate. They don’t have to say “I’d better vote for the 5 best episodes because if I leave one off that might be the one the other voters like the best”, so they can just vote for the 1 stand-out and then use the rest of their ballot for other things they like. So in this case, EPH is just encouraging the kind of voting behavior that I think most of us would agree is healthy anyway.

    ….

    One other point you bring up is that you find it surprising that EPH worked better this year, when the slate was weaker. To me, once I’d seen what it did last year, that’s not surprising at all. The most a slate can possibly get is all the nominees; as it gets weaker, it loses the first and second and even perhaps third nominee under EPH one by one, before it loses any under the prior system (bloc approval voting, in technical terms). So, that’s not at all counterintuitive to me.

    As for 3SV: as I argued in the business meeting and in the EPH+ FAQ, I think that 3SV without some form of EPH is a recipe for disaster, inviting slaters to simply take over the longlist.

    Once again, thanks for your thoughtful post. I’m quibbling, but it’s clear to me that you’ve considered the issue with good faith, wisdom, and intelligence. I hope I’ve convinced you to rethink certain aspects, but you are still free to come to whatever overall conclusion you see fit. Cheers.

    • Reply Rachael Aug 26,2016 10:28

      Thank you for the excellent response. I’m going to take some time and think about it.

  4. Reply greghullender Aug 26,2016 10:24

    3SV is great–I voted for it–but it should be remembered that 3SV will not work without EPH.

    The reason for that is that a slate of 300 can simply divide itself into 3 blocks of 100 and dominate the entire top-15 list. If you doubt that, look at the detailed voting breakdowns for the last few Hugo Awards and notice how few non-slate works had more than 100 nominations. (2016 was exceptional in that there was huge participation in nominations, but I doubt that can be sustained.)

    On the other hand, EPH really comes into its strength with a list of 15. No realistic slate could ever take 10 out of 15 slots–even if you allowed unlimited nominations.

    • Reply Rachael Aug 26,2016 10:29

      Also a very good pointto think on.

    • Reply Chris Gerrib Aug 26,2016 11:31

      @Greg – well, no, not really. The slate divides into three and each third then competes with the 100 or so non-slate votes.

      It’s easy to say “divide into thirds, here’s your list” but in practice harder to pull off. I mean, Presidential elections (the election of 1800) went wrong trying to do that. We see that slate discipline is not that strong anyway.

      • Reply Rachael Aug 26,2016 11:32

        Thanks for putting that into words, Chris. That’s what I was thinking about.

      • Reply greghullender Aug 26,2016 12:57

        I’d rather not depend on all future slates having the same weaknesses that current ones have. And dividing in three is very simple; just tell people to use the last non-zero digit of their phone numbers or some equivalent trick.

        The point is, it’s a serious vulnerability, and simply hoping that no one ever takes advantage of it is a poor strategy.

  5. Reply lauowolf Aug 26,2016 11:21

    “(This year there were actually more nominating ballots than final voting ballots, which is weird.)”
    That was the drop off of less-hardcore slaters, unwilling to pay out for another year with no real payoff.
    They nominated on last year’s membership but couldn’t vote on that dime.

    As for EPH, in addition to a lot of “what he said,” I think the thing is, we still need it and the rest of the fixes to get a relatively honest ballot.
    Thing is, people don’t seem to me to want to uniformly punish anything some idiot tries to slate, so 3SV isn’t likely to do much about hostages or kingmaking.
    A sufficient number of the voting membership is not likely to downvote anything but outright griefers on a regular basis.
    Gaiman’s on the long list?
    That doesn’t look inappropriate, or, what, Slow Bullets?
    It’s just not what people want to do.
    So it’s appropriate to have something that nibbles away at their slate-driven boost.
    Does this give a completely organic ballot?
    Maybe not, but it should improve things.

    • Reply Rachael Aug 26,2016 11:31

      I know some people uniformly punished for slating. And others said “we know you’re trying to manipulate us to vote against something that probably would have been nominated anyway, so no.” Which was apparently the thinking that won out this year. That’s actually why I’m very willing to go with the “trust in the voters” metric here. The voters as a whole have shown that they really don’t appreciate being manipulated either way and know how they want to deal with it.

      • Reply greghullender Aug 26,2016 12:59

        I agree 100% that “trust the voters” is the right strategy. But you have to give them a chance.

      • Reply lauowolf Aug 26,2016 14:01

        I totally trust the voters, just not the slaters.
        And I don’t like the excess impact on the ballot that 3SV working alone would leave in place for kingmaking by slaters, diluting the voice of all other voters.
        The thing is, 3SV is all-or-nothing.
        It totally works for booting awful stuff, and thus closes the door to insulting nominations.
        If enough voters boot want to boot a thing (Beale), it’s gone.
        It can’t be implemented too soon for me.

        But the intermediate cases – slate-boosted, perhaps unimpressive? – 3SV doesn’t touch it, or its standing on the eventual ballot, unless voters are more hardline than they seemed to be this year.
        Voters will not lock-step vote to eliminate hostages.
        Nor should they, necessarily.
        There’s the issue that puppies are fans too, and to the extent that some of their nominations reflect actual preferences, they need to be included in some way in the final making of the ballot.
        And that’s a difficult balancing act, that 3SV can’t handle.
        Anything under the qualifying number of voters seeking to eliminate a work under 3SV does nothing.
        But if an equivalent number of nominators totally ignored the work, EPH may undermine its standing enough to take it out.
        It gives a chance for a needed reweighting of questionable, but not irredeemably bad, nominees.

        I just want a votable ballot, and a packet that at the least offers stuff some real person actually thought was good, even if not necessarily stuff I’d choose, rather than crap that is supposed to explode my poor head.

    • Reply greghullender Aug 26,2016 14:38

      I think the best hope is that between EPH+, 5 of 6, and 3SV, the rewards of slating will be so few that the slaters will go off and find better targets. They already seem to be focusing on the Dragon Awards. The old Hugo voting system was extremely vulnerable to abuse and always depended on an honor system. We can never go back to that.

  6. Reply Dave Wallace Aug 28,2016 00:20

    Rachel – I think what you are describing might be better described as “End of the EPH Honeymoon Period,” at least for me. I.e., that state in which we begin to realize that our beloved has flaws that we didn’t notice or minimized at first, and (hopefully) start the work of building a relationship with the real person, warts and all, rather than the idealized image we had previously built up of them.

    I had certainly shared your hope last year that EPH was going to essentially solve our problems with the slate, confining them to a small number of nominations roughly proportionate to their numbers until they either got bored and went away, or started nominating works with more crossover appeal, while the rest of us got to see a ballot with mostly non-slate works. I was disappointed when I first heard about the results on recent data that suggested it was only regaining an average of a bit over one non-slate work per category, while still leaving the slate in control of the majority of the ballot.

    But that’s not nothing. It still means more genuinely contested categories, with fewer No Awards given out, and fewer categories where the only non-slate entry wins by default. So I’m still supporting EPH (and EPH+) for now, among other measures.

    People now have high hopes for 3SV, which I think is likely to also result in “it helps, but not as much as we hoped.” (Part of that is that we don’t yet know just how hard it will really be to get enough voters to participate to get something offensive off the ballot.)

    So at this point, I’ve concluded that a single “silver bullet” is unlikely, and that instead what we will probably see is an incremental mixture of different measures that address various parts of the problem, until we get to an acceptable final ballot in most years. I think both EPH and 3SV have a role to play in that grand experiment. It looks like EPH+ can reclaim some additional non-slate nominations, if we judge its side-effects acceptable. And there are some additional ideas that people have floated that may help further. There may come a time when the BM concludes that EPH isn’t worth the cost, but if we can keep getting reports that show how EPH compares with the old system each year, I think we will get to a consensus about the value it provides within a few years.

  7. Reply Karl-Johan Norén Aug 28,2016 15:37

    I’ve written down my own thoughts and detailed (as in from raw ballots to the finalist list) on EPH:

    http://kjn.livejournal.com/65023.html

    I hope it can help you and others to better grok EPH; it really works on a fundamentally different principle than FPTP.

    Thanks for your thoughts and your reporting from the Worldcon, and I hope I can see you in Helsinki!

  8. Pingback: Pixel Scroll 9/9/16 Pixel Trek: The Search For Scrolls | File 770

Leave a Reply to greghullenderCancel reply