pedagogical experiment

A senior sociologist of note is struggling to invent a great new assessment instrument and turned to me for advice. “If it existed, you’d be using it by now” was essentially my response. But the truth is that i’m also always looking for a new way to gauge student learning, and to capture all the skills they might have. I’m particularly keen to balance the distance between how they have been conditioned to excel (standardized tests, memorization, boring essays), and what kind of things they need to do and know “out there” (information for cocktail conversations, stuff that will help them to evaluate evidence, facts that help guide civic engagement). And there’s the need to establish synergies with other classes in the university and department, and (in my case now) to prepare them for independent research they will do as seniors (we have a thesis requirement). So for my fall seminar, I’ve doubled down on “prep for senior thesis” and emphasized the need to synthesize information accurately and efficiently, and to be thoughtful about what you’d need to do to understand questions that arise from doing so. In case it helps you, here’s how I’m going to do that:

Practicing sociologists establish competency within a literature (research designed to answer or understand the same problem or puzzle) and then design original research projects to contribute to this literature. Our goal is to become practicing sociologists.

The obligation of reading comprehension and discussion preparation. Each week’s reading focuses on a single problem or puzzle. You will establish competency by completing the reading, and bringing a memo to class that summarizes your understanding of that problem. It will also contain a list of questions you have, the answers to which you think will help you to clarify, or solidify, that understanding. Our discussion will focus on answering each others’ questions.

Just to interject quickly, and for those wondering: yes, I did write my syllabus as a series of obligations. I was raised Catholic. Sue me. (No, please don’t.)

  • I will spot-check these memos. If you come to class with coherent and reasonably accurate notes on all of the assigned readings, and discussion questions that are topical and reasonable, you will receive credit for that memo. If your memos are incomplete or incoherent, you will not receive credit for them.
  • Your memos do not need to be written as prose. They can be lists, for example. They can be written on notecards. They must be on paper, however.
  • If you like, you can start your memo by using the “reading guide” I have included in this syllabus. However, the goal of our discussion is not be become preoccupied with individual elements of specific arguments, but rather to discuss the research questions that unite the week’s readings, and the contributions of each piece to an answer. I am not spot-checking to see if you have answered all of the questions on the reading guide; I am checking to see that you’ve identified a question/hypothesis/explanation that unites all of the readings, and how their approaches to it are similar and different from one another’s.
  • “What is the author’s research question?” is not a good question. That’s asking us to write your memo for you. But you might need help addressing some of the main components of the memo, and in that case, you’re better off starting your question by stating what you think is correct, and asking for more information, detail, or critique. E.g., “The author’s research question is something having to do with the relationship between social structures like religion, and individual behaviors, like treating things as sacred, but the connection between the two is too obvious for me to think it is actually a question that needs sociological research. What am I missing?”

In the second hour of each session, we will work to collectively produce one or several research project/s based on the lacuna we identify in class.[1] This is a brainstorming exercise. We will discuss various research sites or data sources, various ways of conceptualizing our research question and what it might add to the literature, and the ease or difficulty of completing data collection and analysis. We will also discuss the potential results each design might produce, and the degree to which these results will address our research question.

The obligation of science. For the “midterm,” you will select two weeks from the first half of term, and develop a research project proposal for both.

  • Each project proposal should be more than 4 pages in length, but no more than 8 (that’s 8 to 16 pages total, double-spaced, one inch margins, 12 point type).
  • Each should include a clear statement of one research question, an explanation of how it contributes to the literature (n.b. here is where you will incorporate your weekly memos, in addition to research you do independently on the topic), a description of the data and method you will use [including comments on access, bias, and resources (including staff and methods/software training) you might need], a list of hypotheses, expected results, and a statement of their potential generalizability.
  • You are not under any obligation to write about the project that appears to you to have generated the most consensus in discussion. However, if you do not address reasonable objections raised in class, you will be penalized.

For the final exam, you will submit two additional research reports, of similar length, drawn from the second half of the course. You may also submit rewritten versions of your midterm submissions (with the original copy that includes my comments), and can earn “back” as much as half the points you lost.

You may not use the same research question for any two reports.

  • You may use the same data source for as many as two (but not more than 2) reports. [E.g., you can gather observational data on the same corner, or use the Census twice.]
  • You will not receive rewrite grades if you do not submit the midterm copy with my comments at the time you submit your final exam. Your original midterm grade score will remain unchanged.

[1] “Second hour” is an approximation. As this is an experiment, we’ll make a decision on-the-fly about how much time each component of class will take.


As you can see, this is a small seminar–right now there are 6 students enrolled and they all look to be seniors and/or students I’ve taught in another course. I know this isn’t a practical design for most of you, but I am interested to see if it helps me to generate a better discussion than in previous seminars.



Filed under Uncategorized

7 responses to “pedagogical experiment

  1. I had been using haruspicy to assess student learning but our facilities staff is just too busy after finals week to haul away the carcasses and steam clean my carpets. So anyway, that’s why I went back to grading.

  2. Jenn Lena

    You’ll have to re-write the course description, so those pre-med students can’t get bio credit anymore.

  3. JLessard

    With your colleagues across the university, identify the 10-15 attributes of an effective practicing sociologist.

    Determine the level of importance of each attribute in relation to the others. Suggest a simple scale – one through five but with definitions for each number (e.g. a five means that this attribute is “critical”, a one means this attribute is “nice to have but not career breaking”).

    Develop a similarly simple but specific scale that can be used to rate student proficiency across each attribute (e.g. a five means “fully proficient”, a one means “novice”). To the extent you need / want to, you can get very specific around the definition of proficiency for each attribute.

    At the beginning of each semester, have students rate how important they think each attribute is using the same scale that you and your colleagues used. Then have students rate how proficient they think they are across each attribute. Professors should also rate each student’s level of proficiency.

    The gaps between the attributes that you think are important and the attributes that they think are important help guide the student’s understanding of the terrain.

    The gaps between how proficient you think the student is and how proficient the student thinks they are across each attribute helps establish the learning base line.

    The gaps between importance and proficiency identify improvement opportunities.

    Repeat at the end of the semester to measure progress.

  4. Jenn Lena

    @JLessard (hey! buddy!): Setting standard criteria is a better method in educational contexts that are linked tightly to specific and routinized professions. If all my students were training to become auto mechanics, for example, such criteria could be easily set and achievement easily documented. But the goal of a liberal arts education is not, as you know, to track students into routinized professions but rather to equip them to enter their work lives with a great range of skills that they can orient and refine once they enter work environments, and we hope they can get jobs that are relatively open and allow for entrepreneurship. In cases such as these, universities and employers trust in us (experts in varied fields) to balance the inevitable contradictions between broad and specialized criteria. Our syllabi reflect this, and this is one reason why I try out different assessment methods, and why I know none of them is ideal for all students.

  5. JLessard

    @Jenn. Hello! Hope that you are enjoying NYC.

    I understand your point (no assessment method is ideal in industry either) but believe you may be thinking too narrowly about what those attributes could be. If your goal is for students to become practicing sociologists, then there must be certain traits / attributes exhibited by professionals, no? Or better yet, and to your point above, a set of cross-disciplinary attributes that together define what we would expect of an engaged and intelligent member of society.

    Again, you all are the teachers, but could not attributes be similar to:

    – Creating researchable questions that would add to the literature
    – Identifying the data required to answer a researchable question
    – Translating findings into a comprehensible report
    – Identify and analyzing logical fallacies
    – Creating memorable and impactful visualizations of research findings

    Best of luck!

  6. Jenn Lena

    @JLessard: Succinctly, yes. I have defined this course’s objectives in those terms (“becoming a practicing sociologist”), and have set almost exactly the learning goals you itemize above (they’re just on the part of the syllabus I didn’t cut and paste into the post). I mistakenly thought you were urging toward a model that should be applied to ALL sociology courses. And my objection to that was simply that not all courses or all instructors set the goal of making majors into practicing sociologists.

    • JLessard

      “They must be on paper, however.” Nice.

      Re, your above. The more I thought of the idea the more it evolved into something that could be applied across courses, disciplines, and years. Of course, getting all users of the tool – professors and students – to agree on what a “5” or a “3” actually means on the importance and (especially) proficiency scales would be crucial. Thus the need for precise and understandable definitions. A longitudinal assessment of student performance could be way more powerful and insightful than a simple (and increasingly meaningless) letter grade / snapshot in time.

      Do you read the blog “Marginal Revolution”? If not, you may find it interesting. It’s an economics blog primarily written by Tyler Cowan, a professor at George Mason. He actually writes a fair amount on global music, so there is likely value for you there regardless (he travels so has primary interactions with different forms).

      Anyway, he uses what I think is a great question on some of his finals:

      “Write your own questions. Write your own answers. Harder questions and better answers get more points.”

      If a professor had rolled that out at Colgate I think my head would have exploded.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s