We present langid. py, an off-the-shelf language identification tool. We discuss the design
and implementation of langid. py, and provide an empirical comparison on 5 longdocument …

Language identification (" LI") is the problem of determining the natural language that a
document or part thereof is written in. Automatic LI has been extensively researched for over …

In recent years, online social networking has revolutionized interpersonal communication.
The newer research on language analysis in social media has been increasingly focusing …

Word-level language detection is necessary for analyzing code-switched text, where
multiple languages could be mixed within a sentence. Existing models are restricted to code …

We present an evaluation of “off-theshelf” language identification systems as applied to
microblog messages from Twitter. A key challenge is the lack of an adequate corpus of …

Social media services such as Twitter offer an immense volume of real-world linguistic data.
We explore the use of Twitter to obtain authentic user-generated text in low-resource …

In recent years, sentiment analysis has been highly used on social media datasets to get
conclusive information, opinions of users about different topics, such as politics, events, and …

Hidden properties of social media users, such as their ethnicity, gender, and location, are
often reflected in their observed attributes, such as their first and last names. Furthermore …

Language identification is an important first step in many IR and NLP applications. Most
publicly available language identification datasets, however, are compiled under the …

Working from a dataset of 118 billion messages running from the start of 2009 to the end of
2019, we identify and explore the relative daily use of over 150 languages on Twitter. We …