Twitter is making public tweet data freely available to academic researchers in the hope the boffins can use it to make the world a better place.
Social media companies like Facebook and Twitter have been blamed for making the world a worse place by allowing misinformation to spread unchecked, to the point that a former US president appeared to incite an insurrection via incendiary tweets and public ramblings about unsubstantiated voter fraud. Nonetheless, Twitter insists wider tweet distribution can help.
“You’ve heard from Twitter before how we serve the public conversation,” said Adam Tornes, a product manager at the 280-character per post shifting service, during a press briefing on Monday. “And the world is a better place with a more informed population openly discussing the learnings surfaced by the research community.”
Babe, I’ve changed! Twitter wants to try a relationship again with devs after first major API tweaks in years
To encourage such study, Twitter has devised an academic research product track that allows those affiliated with academic institutions to access Twitter data via the v2 API introduced last July.
Twitter isn’t just for arguing with people who are wrong on the internet, trolling, bots, state-backed political manipulation, or making unsupported claims for personal gain. It’s also for studying those things, it said.
The microblogging site is a potential source of insight for researchers who have found ways to use Twitter data, dubious or otherwise, to examine issues of public concern.
Since 2006, added Tornes, Twitter data has been used for a wide range of inquiries, pointing to a US bird migration study based on the feathered beasts people mentioned in tweets. More recently, in April last year, Twitter launched a real-time COVID-19 data stream.
Twitter CEO Jack Dorsey says Trump ban means the service has failed
But as Tornes acknowledged, Twitter hasn’t made it easy for researchers to review and analyze people’s tweets. “For too long, we ignore their unique circumstances and differentiated needs and capabilities,” he said. “We’ve also failed to fully appreciate and embrace the value and impact of scholarly work using Twitter data.”
Twitter’s Academic Research product track aims to address those failings with free access to publicly accessible tweets through the full-archive search endpoint, access to up to 10 million tweets per month (about 20x the Standard product track), better filtering to reduce data collection, and new technological and methodological documentation.
Previously, Twitter only offered such broad access to paying enterprise customers. The company intends to launch a specialized business product track, alongside additional access levels for its Academic Research, Standard, and Business product tracks at some point in the months ahead.
New and existing Twitter developers can apply for Academic Research access; Twitter says it’s reviewing such applications manually. Requirements include being a postgraduate student, faculty, or research-focused employee at an academic institution; having a defined research objective and plan; and a commitment to use any data only for non-commercial purposes.
Now is the time
“There’s never been a more important time to be transparent and accountable when it comes to sharing public Twitter data with the academic community,” said Leanne Trujillo, program manager for Twitter’s academic research team. “The events and global challenges of the last year have made that increasingly clear.”
Trujillo, like Tornes, framed Academic Research access as part of the company’s mission “to help make the world, and Twitter, a better place.”
The Register asked whether Twitter has seen any efforts to deliberately poison its data in a way that might alter the conclusions of researchers analyzing that data.
Tornes said Twitter has shared a lot about the behavior the company has seen with bots and has been fairly transparent about manipulation efforts. “We think this is a great way for Twitter to provide greater transparency into those types of behaviors and help us hold us accountable and help us learn from those behaviors as well,” he said.
The data Twitter will make available to researchers may change over time and those examining the data will need to pay attention to when tweets vanish from the public record, whether as a result of deletion by the tweet creator or via account suspension or cancellation.
According to Tornes, Academic Research access does not include the ability to review data from suspended accounts – for Trump’s tweets, investigators will need to check with the National Archives and Records Administration. Twitter, he said, has a compliance requirement that researchers must delete all data that is no longer public on its platform. And it offers several tools for looking up whether tweet data is still public.
“We strongly encourage academics to look up the state of their data set on a periodic basis, especially for major events like publication or things like that,” he said.
For those curious about the state of the Twitter API, the roadmap is here. ®