A new article published on the front page of otn/java, by Yogesh Tewari and Rajesh Kawad, of Infosys Limited Labs in Bangalore, India, titled “Real-Time Topic Modeling of Microblogs,” explores “the challenge of real-time extraction of topics from a continuous stream of incoming microblogs or tweets that are particular to an application” that they created. From a simple tweet text, the application is designed to accurately suggest relevant topics discussed in the tweet, and provide real-time timelines of topics generated from the tweet streams.

They explain that this is no simple tasks since a tweet, “considered as a text corpus, contains only 140 characters and second, given their brevity, tweets may not provide useful information and may contain different forms of text such as ‘smileys’ and short-form URLs. Finally, tweets are generated in real time.”

Yogesh and Rajesh apply LDA (latent Dirichlet allocation) to topic model tweets and make use of the Machine Learning for Language Toolkit (MALLET) API as the implementation for LDA – all performed in a Java environment. The LDA implementation is in turn encapsulated within the MALLET API, which here functions as a command line–based Java tool.

As they state: “Our targets are the actual Java classes that perform the LDA logic whose methods we invoke with required input in real-time. Storm is our choice of a free and open source distributed real time computation engine implemented in Java and running in a distributed mode. Storm is highly scalable and easily capable of handling incoming tweet streams. We use Twitter4J to stream tweets, which require valid Twitter authentication. So our task is to design a topology that will consume tweet streams and output a timeline of topics.”
Check out the article here.