I hate to complain . . . No one is without difficulties, whether in high or low life, and every person knows best where their own shoe pinches. – Attributed to Abigail Adams.
In financial inclusion circles there is a recent vogue for analyzing social media data to identify consumer protection abuses. As lower income people around the world increasingly use WhatsApp, Twitter, and the like to express themselves, researchers are applying tools to capture their sentiments and package them to inform regulators and financial service providers.
More broadly, calls to listen to the voice of the customer are finally on the upswing, and complaints are seen as a key source of that voice. Complaints – whether expressed on social media or through formal grievance redress channels – are transparent windows into what consumers are thinking and experiencing.
I am heartily in favor of learning from complaints. When converted to data using tools that range from simple counting to natural language analysis, the cacophony of consumer voices can be made to yield insights for providers and regulators about problems that need fixing. When complaints data are made public, in an appropriate format, they warn consumers about abuses to watch out for and providers to avoid. Good stuff.
Turning complaining voices into useful complaints data can be tricky. It is not enough to simply quantify complaints, because although they are genuine, they contain serious biases.
However, turning complaining voices into useful complaints data can be tricky. It is not enough to simply quantify complaints, because although they are genuine, they contain serious biases. Here are three sources of distortion that we observed over the course of the Smart Campaign’s research project “Client Voice,” and related research.
1. People complain about some problems but not others.
With face-to-face services, people often complain about personal treatment – staff who disrespect them or collections officers who humiliate them publicly. With digital services, many complaints arise about lack of response to inquiries or inability to connect. In the psychology of complaining, there is something about insulting treatment that moves people to respond, as does the immediate frustration of being unable to complete a task. On the other hand, far fewer complaints surface about high interest rates and fees, even though these may cause greater harm than a momentary snub or failure to connect. Complaints about harassment from collections officers may in effect be blaming the messenger, deflecting attention from the underlying problem of a predatory loan. More broadly, complaints tend to reflect ex post consumer problems around the experience of use. Poor treatment does matter – greatly, but it is important to remember that consumer protection abuses include ex ante problems in the design and terms of the product itself.
2. Prevalence is hard to determine.
In converting social media or grievance narratives into analyzable data, complaints are categorized and counted in a process that reduces them to key words or problem types. The resulting counts require context before judgments can be made about whether a problem is isolated – in which case it should be resolved by the financial institution – or widespread – signaling a possible need for policy change. A raw number is difficult to interpret. During the coronavirus pandemic, the consumer protection agency of Peru announced that it had received about 20,000 complaints through a new digital complaints window. Was that a lot? Or a few? Compared to what? Such numbers take on meaning only when tracked across time, geographic areas, or companies.
A raw number is difficult to interpret. During the coronavirus pandemic, the consumer protection agency of Peru announced that it had received about 20,000 complaints through a new digital complaints window. Was that a lot? Or a few?
The need to determine prevalence has implications for the call to publish complaints data. During the Obama administration, the U.S. Consumer Financial Protection Bureau published a complaints database on its website, laudably available to both researchers and individuals. One could see the number of complaints of a given type about a specific bank. But there was no context, such as scaling the number of complaints by the size of the institution. Citi, Bank of America, and Wells Fargo had many complaints. Was this because their service was poor, or did it merely reflect their large scale? A researcher could correct for institution size, but individuals – and importantly, journalists – are unlikely to be equipped to make such adjustments.
3. Complaints can become sensationalized.
When complaints have an emotional aspect, as many do, they grow like fish stories, stoked by the craving for salacious gossip. A few egregious examples can morph into widespread opinions about the treatment given by an entire class of institutions. This is an important challenge in determining prevalence, and it calls for analysts to triangulate with a variety of methods and data sources to get a true picture. In our research we used focus groups and in-depth interviews to hear about the kinds of abuses customers experienced, turning up many shocking incidents. But only through quantitative surveys could we estimate the frequency of such abuses. Often, the most colorful stories turned out to be apocryphal, or nearly so.
Social media analytics are a breakthrough new tool, but may incorporate distortions, because social media tend to amplify signals with highly emotional content. Particularly harrowing complaints may go viral, and social network analytics may pick them up as more prevalent than they are. The social media accounts of digital lenders in Asia distributing photoshopped pornographic images of customers are a case in point. No one knows how often this happens, but everyone has heard about it.
Other tools may have less tendency to distort, but analysts must still consider whether bias influences the data they see. Even a simple form of complaint— a rating site—is susceptible to emotion. Because people often complain when they are angry, they are more likely to slam a company with a one-star negative rating, instead of two or three star ratings denoting moderate problems. Data derived from formal grievance redressal processes may carry fewer such distortions.
Sensational complaints must be taken seriously, because outrage narratives can become political, with major consequences.
Even if they point to relatively rare experiences, sensational complaints must be taken seriously, because outrage narratives can become political, with major consequences. The best-known example of this is the case of farmer suicides in Andhra Pradesh, India that shook the microfinance sector in 2010. The suicide accounts propelled a genuine over-lending problem into the political sphere, precluding a more sober, regulatory approach that might have corrected the problem with less damage to the sector and its customers. Similarly, in Cambodia in 2019, an investigative journalism report about land seizures arising from microfinance debt caused a public sensation, even though only 28 cases were cited. This isn’t just a developing world or financial sector problem—FAI Managing Director Tim Ogden wrote a book about how a broken complaints system led to unnecessary panic in the U.S. about Toyota vehicles.
One of the best ways to respond to these challenges may be to make it easier and more appealing for customers to lodge complaints directly, yielding more complaints data with fewer biases. Customers may be less inclined to vent publicly when their complaints have already been attended to. Peru’s new digital complaints window, mentioned above, is a good example of an effort to broaden access to complaints channels. Such mechanisms need to be available first at the provider level and then at the regulator level, with industry associations as a third possible avenue. They need to be funded well enough so that the provider or regulator can actually respond to the complaints received.
Complaints data are a critical source of information for protecting consumers, but they require care in interpretation and response. The voice of the consumer should be heard, and policy makers and providers must listen intelligently.