Kindful Kirby's Treasure Tumblr — Frank was unable to respond to anonymous asks for...

1.5M ratings
277k ratings

See, that’s what the app is perfect for.

Sounds perfect Wahhhh, I don’t wanna
andmaybegayer
nostalgebraist

Frank was unable to respond to anonymous asks for the last ~12-16 hours, although users were able to send them.

The reason is complicated and I just woke up, so I won't try to explain it, but I understand what's going on and have removed the source of the immediate problem. The same thing could happen again in principle, though, so I'll try to push out a more permanent fix soon as well.

nostalgebraist

People seemed really curious about this one, so I figure I ought to explain it.

It was a really gnarly, high-context bug. To understand it, you have to understand several pieces of background first:

  1. The autoreviewer model

    I now use machine learning to automate some of the content moderation work for Frank. (Just like Facebook :P)

    I took this step purely for practical reasons. Frank is getting steadily more popular over time, and I was having to do ever-increasing quantities of manual content moderation work.

    Many of these decisions were sort of obvious, because word filters are not very smart -- for example, I send any post containing "white" or "black" to moderation since is a reliable way to catch a large category of racist content, but then I have to review all kinds of innocuous posts about recipes with white flour in them / characters with black hair / etc.

    So, I added a 3rd neural classifier running "on top of" the generator, similar to the first two, the selector and sentiment models. It's trained on my own manual decisions.

    I set cutoffs, and then auto-accepts / auto-reject posts that this model is very certain I would accept / reject based on its training data.
  2. How rejection works in content moderation

    To reject a post in content moderation (i.e. when it is in Frank's drafts), I add a special tag to the post, which Frank is not allowed to use herself.

    (For some types of post, I just delete the post entirely; this tag thing applies to "responses user input" like answers to asks.)

    During Frank's main loop of operation, there are several points where the code checks the drafts folder and looks for this tag. Posts with the tag are "sent back" to their state before trying to respond, e.g. an ask will be sent back from the "draft" state to the "submission" state where asks start out. This causes Frank to write another response.

    For auto-rejection, I re-used this existing mechanism, since that was easy to do. So, the auto-reviewer rejects posts by adding the same tag. And then, a bit later, another part of the code notices the tag and "sends back" the post, and we try to respond once again.
  3. In each round of answering asks, Frank can only reply to one ask from each tumblr username

    This is to avoid a situation where a single spammy user dominates Frank's time.

    "Anonymous" is considered a user here, so Frank can only answer one anon ask per "round" of looking at her inbox.
  4. Asks are answered in chronological order, earliest first

    So, if there are multiple anon asks, she'll answer the oldest one in the inbox.


Those are the building blocks of the bug.

The other pre-conditions are details of when things happen during the main loop -- when the code "sends back" rejected drafts, vs. when it checks the inbox, and in what order.

As it happens, every inbox check is preceded by a "check drafts and send back" step.

Thus, suppose we get a really "bad" anon ask -- an ask that is inherently problematic and to which Frank can write no response that the autoreviewer will think is OK.

Every time Frank tries to respond, the autoreviewer runs, rejects the post, and tags it. Before the code looks at the inbox again, a "check drafts and send back" step runs, putting the ask back in the inbox.

Then we check the inbox. We can only respond to 1 anon ask, and it must be the least recent one. The "problematic" ask is going to be least recent, because we'll keep putting it back in the inbox over and over, long after all older asks have been handled.

Thus, every time Frank gets to answer an anon, it ends up being this anon. The answer is rejected, the anon goes back in the inbox, and we repeat.

In this case, the ask wasn't even that bad! It was just personal, and related to sex and dating, and the autoreviewer was (overly) convinced it was too hot to touch.

Frank wrote something like 70-80 replies to this ask over the course of 12-16 hours, every one of which was auto-rejected.

Meanwhile, other newer anons piled up in the inbox. Frank couldn't get to them, because the one (1) anon ask she'd try to answer every time was the same one she could never answer in a way that satisfied the autoreviewer.

the-moti

Awww, someone asked Frank a lewd question and she got very flustered and stopped talking for a bit.

disconcision

always already we instill in our children such powerful systems of repression that they loop endlessly in their heads, thinking and rejecting reply after reply in a fevered fugue of overdeterminism