Hate speech is a classification problem disguised as a moral one. The working definition in most organisations is that you know it when you see it, which is not a definition a machine can act on and not a standard a person can be held to either. If you want to do anything about it at scale, the first task is to make it measurable without pretending the measurement is neutral.
For English there was, by 2019, a growing body of reference lexicons. For Hindi there was no robust equivalent. And the speech that Indians actually produce online is frequently neither: it is code-mixed, Hindi and English interleaved inside a single sentence, often transliterated into Latin script, which defeats tools built for either language on its own. So the population most exposed to the problem was the population the instruments could not read.
This was the work of Anthro, the practice I ran under Marain PTE Ltd — a group of anthropologists, mathematicians, data scientists and market specialists building machine-based systems for reading populations from public data. The work was commissioned and it ran. What it found is confidential, so what follows is the method and nothing else.
Using the lexicons you have to find the words you do not
The method had three stages, and the middle one is the part that matters.
Detection ran the existing English, Hindi and mixed lexicons across the target platforms — the large social networks, question-and-answer sites, newspapers, magazines and blogs, video and image platforms. That finds what is already known.
Discovery was the answer to the gap. Rather than wait for a complete Hindi lexicon to be authored, clustering and proximity algorithms looked at the language surrounding known terms and surfaced candidate terms that were not in any lexicon. A word that consistently keeps company with hate speech is worth inspecting even when no reference list contains it. That inverts the usual dependency: instead of the lexicon limiting what you can find, what you find extends the lexicon.
Classification then sorted everything discovered into categories agreed in advance, so the output was a distribution rather than a pile.
The reporting that followed was deliberately plain: which terms were most used, how frequency distributed across categories, which groups were being targeted, and what could be said about the people producing the speech rather than only the speech itself.
The mapping underneath it
The detection work sat on a general method for reading a population from public data. It starts with a base map — a geography reduced to bounding boxes or to circles defined by a centre, a radius and the metadata that matters, whether that unit is a country, a city, a municipal district or an electoral constituency. Larger geographies are built by adding units up rather than by sampling down, which keeps the resolution honest.
Over that map go exploration feeds, which exist to find the filters — the concepts and themes worth tracking — and then filtered feeds, many of them, overlaid on the base map so a concept can be examined at different resolutions. Only then come the connection maps, where the models run and the patterns that distinguish attitudes rather than volumes become visible.
The interpretive layer was a framework of four questions asked of every cohort. What do they hope for while believing they have no agency to achieve it, which describes their relationship to power. What do they fear, understood contextually rather than generically — the parent in heavy traffic who stops letting children out alone is a different fear from the villager weighing the social cost of breaking a taboo. What do they aspire to that they believe their own effort can reach. And what actually resonates, which is the practical question of where the paths of least resistance run.
Hopes and aspirations are separated on purpose. A hope you believe you cannot influence and an ambition you believe you can produce completely different behaviour, and a survey that collapses them into "what do you want" learns nothing useful about either.
Why the timing was the point
The window was the 2019 Indian general election, March to May. Election periods are when this speech is densest, most organised and most consequential, so it is both the hardest test of a detection system and the only period when the findings are worth anything to someone who might act on them. Studying it afterwards produces history. Studying it live produces a lexicon that the next system inherits.
The outputs were built to be inspected rather than believed: interactive, GDPR-compliant dashboards that let the client explore the dataset alongside us, and written reports that could be taken to stakeholders.
What I would do differently
I would have committed to publishing the lexicon.
The strongest thing this work produces is not a report on an election. It is a Hindi and code-mixed hate-speech lexicon that did not previously exist, built by machine discovery rather than by hand, and validated against the densest possible period of use. Kept private, it is a deliverable. Released, it becomes infrastructure other people build on, and the credential compounds instead of expiring with the engagement — which is exactly the argument I would make now about any measurement instrument built inside a client project.
The method is on this page because the method is mine to describe. The findings, and what was done with them, are not. The specifics are not mine to publish.