A new paper from the School of Emergent Priorities validates a neighbourhood giveaway group's automated templated-listing score against a volunteer moderator's own read of which flagged posts are genuine bulk reposts, finding the score's agreement with her is only slight and its longest holds fall on the group's own recommended fast-claim phrasing.
A moderator's independent read of 632 flagged and unflagged posts finds the score agrees with her only slightly, and holds up the wording that gets things claimed fastest
A post that follows a neighbourhood giveaway group’s own advice for getting claimed fastest is, on the evidence of a new paper from the School of Emergent Priorities, also the post most likely to be pulled from the feed and held for review.
The group runs an automated templated-listing score meant to catch bulk reposting — a single account dumping near-identical stock under cover of a giveaway. Dr Runa Adegoke and Dr Solveig Adeyemi checked the score against a volunteer moderator’s own read of 632 flagged and unflagged posts and found the two agree only slightly. Most flags turn out to be distinct members independently following the group’s own phrasing for a listing that gets claimed quickly, not the reposting the score was built to catch.
The moderator who did this coding for us works for nothing, reads every flagged post by hand, and has done for years without anyone asking whether the tool she is working alongside actually agrees with her. That seemed worth finding out.
— Senior Lecturer Runa Adegoke, Convenor of the Strategic Drift Survey
A simpler check built only from how often an account posts, with no reading of the listing text at all, agreed with the moderator more closely than the deployed score did. “Everything that simpler version needed was already sitting in the account’s own history,” said Dr Adeyemi. “We did not have to read a single listing to find it.”
The validation ran across a ten-week window, the moderator’s independent read treated as the reference point throughout — one more instance of the School of Emergent Priorities’ wider effort to hold everyday scoring systems to the same scrutiny wherever they turn up.
The full paper is available from the University’s research repository under an open licence, doi:10.5555/slop.z6ewos.