in case I will review your work:

Billboard Magazine has published an analysis of Top 100 charting song lyrics since 1958, when the modern chart began. The result? Songs about love top the list (65.8%), then songs “about” dancing–as in, “The Hustle” (7.1%), self-empowerment songs are next (6.9%), followed by anti-love songs (6.6%), a category labeled “surf city” which includes Toto’s “Africa” and “Wild Wild West” (4.1%), war (2.1%), funny novelty songs (1.4%) and then an “other” category that is 60 songs large.

Since I review about two dozen papers a year that employ the same methodology, and come to the same conclusions as Gary Trust, I thought this was an opportune moment to help future paper authors learn my preferences and biases. I trust other culture/quant scholars–or, social scientists in general–would have the same sort of concerns, so I think about this as a kind of community intervention with me as the voice. I hope this will change my life. Here we go:

1. Present us with a puzzle. Just reporting counts of things in categories doesn’t make for interesting reading. Why should I care about the lyrical content of the Top 100? If, at the end of your analysis you write something like, “When it comes to smash songs, for half a century and counting, simply, we’re obsessed with obsession”, you’ve got several massive problems.

  • First, this is boring and an extraordinarily trite observation about American culture. And even if I spin this as a critique of the pop charts, you’re still not telling me anything that hasn’t been said a million times before. Americans love cheezy pop music that focuses on love and dancing, and almost never addresses stratification or politics. Wah, wa. It just makes me think you’re kind of an idiot for expecting differently.
  • Second, you’re not able to draw this conclusion (“we’re obsessed with obsession”) from your data. In the preparation to write this sentence, your mind twisted the generic concept “love” into the pathological condition of “obsession.” Your love might be unhealthy, but you’ve provided no evidence to support the claim that the love described in these songs has an “obsessive” character.
  • Third, you’re overgeneralizing. “We” don’t all listen to pop music, and even if “we” did, the pop charts are a totally inappropriate index of our state of mind.

2. Describe your method. In this case, we are told a great deal of information (for a short article) about the count of songs, and time range of the data, and their relative numbers within these categories, but I have no freaking idea how the input (song lyrics) produced the output (8 categories, and an “other”). Relatedly:

  • How did you decide what the categories would be? Did you count all the nouns in the songs first, then identify synonymous concepts and keep reducing complexity in this fashion until you arrived at a total of 6 groups? Did you look for some of these themes on purpose, and others emerged as you noticed patterns in the data? Did you do this by hand? Use a data analysis tool of some kind? What about just “word count” in Word? Did you have anyone else look at the data, and see if their categories matched yours?
  • How did you sort  songs into categories? If a song primarily focused on romance, but it took place at the beach (e.g., the Beach Boys’ “Surfer Girl”), did you put this into the “surf city” category or the “love” category? Was the decision a function of the emphasis in the song (some internal count of word mentions?)? Was it more impressionistic? Did the sound of the song influence your decisions, so that love songs that had more beats per minute were eligible for both the “love” and “dancing” categories?

3. Make sure your categories are non-redundant and have face validity. In this case, I simply can’t make any sense out of the “Surf City” category (which doesn’t actually seem to be about surfing at all, but rather songs that mention places?). Love and anti-love seem like the same conceptual terrain to me, particularly because most love songs I know express a little skepticism about whether love is actually some kind of trick or drug that’s going to wear off, leaving the victim alone and in pain. “Dancing” is a total cop-out, and seems heavily weighted toward a particular moment in the pop charts when inventing dances to accompany songs was part of the larger branding process within the music industry. That you’ve collapsed those songs (e.g., “The Mashed Potato,” “Land of a 1000 Dances”) in with dance songs (e.g., the Madonna catalogue) is more ammunition for my complaint about your methods.

I’m sure this is better-than-average reporting for the Billboard folks, and it is one of the few times I’ve seen them publish something that purports to be social science (he writes: “(And, I could finally put my psychology minor to use).”). But in the effort to appeal to an audience, this article both fails to meet the lowest criteria for social science research and, perhaps worse, it fails to be information.

One final note: Don’t ever write a paper that I might review that focuses on the following puzzles:

  • “Why doesn’t the audience for X music realize the artists aren’t good musicians / aren’t authentic / aren’t political enough? It is clear as day, to me.”
  • “Why isn’t pop music more political, like I’d like it to be?”
  • “How come rap artists are so preoccupied with money and sex?”
  • “Is rap sexist?”

These are almost irretrievably stupid questions, roundly criticized by every legitimate scholar (and most non-scholar, music experts), and So. Played. Out.





Filed under Uncategorized

8 responses to “in case I will review your work:

  1. p.s., to the future paper authors. i’m the other person who reviews all these papers and Jenn is completely right.

  2. “Top 100 charting song lyrics since 1958” — you can listen to them all here, five seconds each (I think it begins a bit before 1958).

  3. Noah

    Aren’t most of these basic skills that should be taught to most grad students taking a quant methods class at some point? Operationalizing variables and designing good research questions are two skills that every quant needs to be successful.

  4. Jenn Lena

    Being taught, learning and applying are three stages of a process. In this case, I think there’s huge attrition from the ranks at each stage.

  5. noah,
    i think they cover research design in 212AB (research methods) in our department. regardless, i’ve long been of the opinion that getting the standard error right is only one piece of quantitative research and you wouldn’t know from the way we teach that data cleaning and theory operationalization are equally important.
    with the specific kinds of work that jenn is critiquing, i think some of this is a selection effect. a lot of people get interested in sociology because they think it’s an area that will allow them to explore their political and artistic interests (eg, is rap sexist) rather than an area that is more about questions like focused identity, endogenous growth processes, structural brokerage, etc. this misunderstanding is completely understandable given the vast gulf between what we teach undergrads and what appears in our top journals.

  6. Jenn Lena

    Gabriel is exactly right. There is a chasm between the kinds of content taught to undergraduates (which draws them into the prospective graduate student pool) and the kinds of projects professional sociologists are typically willing to supervise, review positively, and respond to in their own research. “Is rap sexist” is not among the latter group, but absolutely in the former.
    To illustrate the point, while I’m extraordinarily flattered to have one of my articles about rap music included in the second edition of “That’s the Joint!: A Hip-Hop Studies Reader” (the most comprehensive and well-curated of the readers on this topic), the article selected is the only one I’ve written that is (1) qualitative and (2) in the mode of “cultural studies” and rather less “sociological.” Undergraduate readers of the book will rightly imagine I am interested in such methods and approaches as a rule, when I am only interested as an exception to the rule.
    I hasten to add that this is only one of many contributing factors. I am of the opinion that there are many unqualified students admitted to graduate programs in sociology, and compounding this, there is social promotion within programs.

  7. Nicole

    Why on earth are you reviewing two dozen papers a year?

  8. Jenn Lena

    Jesus told me to.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s