Tuesday, October 28, 2014

#worthassigning: Matt Reed on the tensions caused by loss of local faculty governance

Matt Reed, who writes the Confessions of a Community College Dean blog at Inside Higher Ed, has a post up that traces the shifts in decision making on policies and curriculum in higher education, a shift that's veering away from faculty governance/local control.
The post is at https://www.insidehighered.com/blogs/confessions-community-college-dean/bossypants-conundrum

Here's an excerpt:
In many ways, higher education’s mode of production is still artisanal. Each professor sets her own standards for grading, selects her own materials, and to a significant extent reigns supreme in the classroom.  The apprentice-journeyman-master structure of grad school makes some sense in the context of an artisanal model. The artisanal model has its own dogma, in which academic freedom and shared governance are supposed to ensure that the artisans are substantially left alone.  As with any working dogma, it has its own internal contradictions -- shared governance can work against the autonomy of dissenters, for example, which is why dysfunctional department meetings are endemic to the industry -- but it has held up for long enough that some people think it’s natural.

Over time, the economic limitations of the artisanal model led to a wave of unionization based on the industrial model.  Unionization had clear benefits, although it introduced a whole new set of tensions.  For example, the artisanal ideal that every professor is a special snowflake sits uneasily alongside payscales determined solely by seniority.  “Master” status relies on being somehow special; collective bargaining relies on solidarity.  And the boundaries between curricular decisions, which are subject to shared governance, and economic decisions, which are subject to collective bargaining, aren’t always clear.  Is program elimination curricular or economic?  (The correct answer is “yes.”)  Still, to the extent that the unionization drive reinforced the artisanal ideal of faculty being substantially left alone, most of the contradictions could be contained.

Now a new logic is emerging, and it’s bringing new tensions.  State governments, often following initiatives from national foundations, are starting to look more intentionally at community and state colleges as branches of state workforce development systems.  In so doing, they’re working to shift the locus of decision-making from the campus, where shared governance remains the preferred method of decision-making, to the state.
I recommend the link for the full context. 

Why is this a piece worth assigning?

The changes Reed describes, even if you don't agree with his description of them or if what he says isn't true for you locally, are broadly true that the analysis is useful. The piece will be useful especially for graduate students to read, as they begin their careers and need to think strategically about where they want to be in ten and twenty years. It's a hard thing to do, think ahead about where a career in learning and teaching will go, but as tenure and full-time positions are already being reduced, it's also the new reality. 
The skill sets that most current faculty needed and used to get jobs still apply, but onto that, adjuncts and graduate students and newly hired assistant professors, need to develop other skills, and will need to be ready to adapt to conditions that most of their graduate school mentors cannot imagine. Newly minted or early career faculty in traditional programs will need to do the traditional things -- publish, present at conferences, serve on committees -- but they'll also need to be ready to shift gears if their programs are cut, or colleges are reorganized, or simply because opportunities for full time work that do not emerge in traditional colleges might be found in nontraditional programs.

What's also true is that new models of teaching are emerging, ones that are clearly not artisinal. Competency based education programs (see shar.es/10oyYn for new programs announced at public colleges)  -- where entire degrees and certificates are offered -- often do not follow an artisanal model at all. Faculty roles are unbundled -- curriculum and course design is separated out from leading discussion, giving feedback to students on their work, tutoring, and mentoring. In that world, what will professional development look like?
Will the raw numbers of traditional faculty roles in traditional colleges and campuses remain steady, either as adjuncts or full-time tenured artisans,  and will competency based programs, or badging programs, or other alternative educational ecosystems  simply emerge side by side, creating an alternate higher ed universe? Or will competing models merge, will institutions become hybrid, and faculty roles hybrid?

If faculty lead hybrid lives -- doing some artisanal course sections on brick and mortar campus where they design the curriculum and buttress their choices with academic freedom, but also serve on a curriculum review/design board for a competency based program, are hired to design a course for continuing education, or teach as learning coach in an online degree program -- what does that mean for their professional identity, how their work is valued, how promotions and pay increases are determined?

Sunday, October 26, 2014

Why ETS Might Not Want to Let Les Perelman In

Update on October 27, 2014.

In a discussion on WPA-L, Les Perelman responded to this post with two e-mails that shed more light on the nature of the proposal he submitted and his research goals. I've added those below.


At  Is MIT researcher being censored by Educational Testing Service?, Valerie Strauss has a post by Les Perelman, of MIT, detailing what he describes as a censorship condition imposed by Educational Testing Services (ETS) on research Les proposed to them.  ETS would only give Les access to E-Rater, their technology for automatically assessing writing, if he agreed to have his findings reviewed and to make corrections to errors ETS might find. If Les opted not to take their corrections, he could publish his findings but could not mention ETS or its products by name. The post includes a response by ETS asserting that the practice, which they do not dispute, is not censorship.

So my post here is not about whether ETS conditions are censorship, but something else. Why automated testing companies might choose to make it difficult for Les to have access to their products, and how other researchers might opt to test the claims of these products.

Automated Writing Scoring Companies Do Not Trust Les Perelman


In his post, Les cites the model of consumer watch dogs -- "All I want to do is what organizations like Consumers Union and the Underwriters Laboratory do all the time: determine 1) if an advertised product meets its claims and 2) whether or not it is defective."

Les observes that ETS is selective in applying the research policy under discussion, "Over the next few months, I discovered that the provisions ETS had told me were common practice were not consistently applied. Around the same time, another researcher had applied to use Criterion and had no problem gaining access."

In addition to ETS, Les also asked Pearson for access to their automated essay scoring technology, he writes:
Pearson Educational Technologies wouldn't even reply to my request to test their WriteToLearn® software, and Peter Foltz, a Pearson Vice President, was quoted in the 2012 New York Times article as justifying Pearson’s refusal to give me access to their product because “He wants to show why it doesn't work.”
I'm not surprised that ETS and Pearson (LightSide Labs is a welcome exception on this front) will not give Les easy access to their technologies. Les's goal is in fact what Foltz says it is it is: to prove that, quoting from Les's post,  "computer generated nonsense could receive high scores from Automated Essay Scoring (AES) computers," or put another way, that the software can be tricked into giving high scores for bad writing.

Les's goal, in their view, is not to test their products with the kind of prose most writers who use their products will write. So from their point of view, the review he proposes isn't of the kind Consumer Reports does -- using the products under the conditions they're designed for -- but is instead to show that their software can be tricked with a program that students will not be using. And if that is in fact their view, then I'm not surprised access isn't forthcoming.

I am surprised that ETS didn't just say -- if that is their reasoning -- as much in their rebuttal. From the point of view of these companies, my guess is that the see Les as provocateur, and not as open-minded curious researcher. Les's work is invaluable, and I love reading it. But I can see why these companies might not want to give him access at this stage. It may be cowardly on their part, but it also has a logic to it: very few us make things easy for those we think are out to get us.

So Can, Automated Essay Technology Be Reviewed Openly and Fairly?


But back to testing this technology, which does need to be studied and tested. Les evokes the work of the Consumer Union, publisher of Consumer Reports, and Underwriters Laboratory, a product safety project, saying his research is akin to theirs.

But Consumer Reports, doesn't ask for permission or access to products; it goes out and buys the stuff they will test. That specifically frees them from the kind of entanglements Les's proposal instigated.

So a Consumer Reports model that looked at ETS would not be the kind of project Les has proposed -- using software specifically designed to fool the technology, but instead using the technology under the kinds of conditions and writing that ETS claims to have designed Criterion for.

A research project could be done this way: access to Criterion is purchased for students, and the professors who will be using it would go through the required Criterion training.  The teachers would teach with Criterion as part of the course mix, students using it to do the work of the course,, with the writing emerging for submission to the software under course conditions. The study can include a comparison of feedback from Word on the same draft submitted to Criterion, as Les proposed. But the basis of the study would be using the product the way it was designed to be used.

And ETS doesn't have to be told at all that the product is being studied in a class test of this kind.

I know when I did a back of the envelope analysis of grammar checkers using a single student essay -- and I accessed a variation of e-rater/Criterion by paying for a Turnitin.com WriteCheck (http://en.writecheck.com/) account, Word, Grammarly (another platform I checked) and E-Rater as expressed in WriteCheck all got some things wrong. See the following two links for a summary of that:

http://nccei12carbone.blogspot.com/2012/10/an-experiment-with-grammar-checkers.html
http://nccei12carbone.blogspot.com/2012/10/sampling-of-grammar-checker-errors.html

But even that quick study doesn't get at what a class test would reveal about how the technology affects students and teachers, how they need to adapt or how the classroom feedback ecology and workloads are shifted.

The important question about automated writing assessment technologies is less about how accurate they are compared on one another, and more about how their presence in classrooms and in the hands of novice writers may hurt or help the teaching and learning of writing. And to know that, it's important to study the technology under the conditions where teaching and learning happen.

Responses from Les Perelman

First e-mail
Nick,

I think you are being too kind to ETS and the other companies., Lightside excepted (we both have considerable respect for Elijah).  My purpose with BABEL and my other experiments is to demonstrate that overall AES does not work.   Various testing companies make absurd claims in volumes like the recent one edited by Burnstein and Shermis (2013).  MT Schmidt in that volume states “IntelliMetric is theoretically grounded in a cognitive model often referred to as a “brain-based” or “mind-based” model of information processing and understanding..”  I an trying to refute that claim and others.

Secondary to my claim that AES does not work are several other claims. First, students could game AES machines like E-rater, that already grades high stakes tests like the GRE simply by memorizing word lists and peppering them throughout their paper with no regard for making meaning.  I have done that already with E-rater GRE online test.  There is a second human reader for machine graded tests like the TOEFL and the GRE, but given the scoring conditions, I would be interested to see if human readers would catch such strategies.  E-rater gives a score that is a continuous variable (e.g. 4.6), while  humans are limited to integers.  The two scores go to another human reader only if the difference is greater than 1.5 point on a 6-point scale.  If the second human reader’s score is between the scores, the three scores are averaged.  Otherwise the outlier score is thrown out.  ETS has a research report on E-rater but does not present the crucial statistic of when there are outliers, what percentage of the time is the E-rate score the outlier.  Given that Pearson wants to use their AES scoring engine as the second reader for the PARCC Common Core tests, these questions are extremely relevant.

Moreover, the specific study I was proposing was going to use student papers from the ASAP study to observe how well e-rater compares to MS Word.  It would have been similar to the excellent study comparing instructors to e-rater in the current issue of Assessing Writing.  In a phone conference with ETS, I even offered to show then the results afterwards before dissemination and discuss it with them.  The only condition I would not accept was censorship.

As for the Consumer Reports model, I mentioned buying Criterion and using it in a class as you suggested during the conference call with ETS, and was told that the Terms-of-Use agreement with ETS for classroom use prohibits published research without ETS’s permission.

I believe that there may be a few legitimate uses for AES (and Elijah may well find them).  However, I also am convinced by argument from people like Noam Chomsky and other cognitive scientists that our knowledge of semantics is way too insufficient for  most uses of computers to evaluate writing.

Les
Les Perelman, Ph.D
Excerpt from a second e-mail from Less, written after I offered to post the first here:
I did not mention the source of the papers in the OpEd, although I did in the proposal, because most people reading the Washington Post would not know what the ASAP study was and it would take too much verbal real estate to explain. 
I did say, however, "I submitted a detailed proposal to compare the accuracy of Criterion to that of the Microsoft® Word Spelling and Grammar tool. I would conduct the study with a colleague from MIT who has a Ph.D. in linguistics from MIT and who worked with Noam Chomsky and Morris Halle, the founders of modern linguistics.” 
In my conference with them I told them I would agree to use Criterion for no other purpose.  That was not good enough for them.



Wednesday, October 22, 2014

#worthassigning: Mark Bernstein on Notetaking

Some of you will remember, or perhaps still use, Storyspace, software that EastGate released for creating hypertext fiction. Storyspace via Michael Joyce's _Afternoon, A Story_, were described in a 1992 NY Times Book Review piece called "The End of Books" by Robert Coover (http://www.nytimes.com/books/98/09/27/specials/coover-end.html).

Bernstein's current project, which grew out of academics trying to use Storyspace for note taking and information organizing instead of post-modern fiction, is Tinderbox.  From chapter one of his book, The Tinderbox Way, which is both a user manual and a meditation on the value of systemic note taking, Bernstein describes the software this way:
Tinderbox is designed to help you write things down, find them, think about them, and share them. Tinderbox is an assistant. Its meant to help, to facilitate. Its not a methodology or a code. Its a way to write things down, link them up, and share them. Its a chisel, guided by your hand and your intelligence.
I pulled the quote above from Sources and Methods #5: Mark Bernstein, a fascinating podcast interview with Bernstein conducted by Alex Strick van Linschoten and Matt Trevithick. The interview runs about an hour, but below the podcast recording, you'll see a time line, indicating at what minutes in the discussion different topics arose. The full interview is worth a listen for context and continuity.

Here's an excerpt from that timeline to give you both a sense of the conversation, and how the timeline helps you to find key areas to return to:
8:54 - Idea: Discovering the structure information should take is the essence of what research is about.
13:14 - Idea: Agile software development has come into force more recently, rather than structuring all the rules first. Writing the software and then revising the software - where most of what you do is editing, rather than designing and debugging - has been extremely fruitful, and has gone in just 10-15 years gone from outlying heresy to the dominant paradigm of software development today. [Note: Bernstein goes on to describe how Agile software resembles writing prose.]
15:08 - Idea: When you’re writing, you’re talking to yourself, or rather to the page. When you write, you are meeting minds on the screen, and in fact one of those minds is a manifestation of ourselves.
20:38 - Idea: People don’t like to think about their process of writing. We have this essentially romantic conception of idea generation writing, that it’s essentially inspiration, and it should come to you in a flash, and that it’s mystical, and that it’s based in someway on your innate goodness, and therefore people don’t spend much time thinking about how to improve because you can’t improve on your own innate goodness.
There's a lot in the discussion that maps on to teaching writing, teaching research, teaching thinking, and faculty development for those professors who want to help students get better at writing, research, and thinking. 

The interview can be assigned in time points for students, or one might scroll to to a point and play a snippet as a way to launch a discussion. For students especially, this discussion focuses on the role of noting, of seeing and recording, and in the act of doing so, of thinking, organizing, and find order. 

In a way, it's about slowing down, of taking the time to start a system that will serve a learner as a writer, and over time, as they change as writers, learn more, know more, and will find it more and more useful to be able to go back into their reading and writing history to recall, reorganize, and rethink, note taking as a key element for revision.



Friday, October 17, 2014

#worthassigning -- 3 essays on rape and death threats against women in the age of #gamergate

Online violence against women scares and worries me. As it morphs from virtual threat, which is bad enough and still violent even if not overtly physical, into offline threats that drive women from their homes, offices, and families and into hiding, the damage and danger has become palpable enough to make news.

Writing in her Washington Post Blog, Act Four, Alyssa Rosenberg sums up the three high profile cases:
[Anita] Sarkeesian has had to leave her home because someone who threatened her claimed to have her address and that of her parents. Brianna Wu, who co-founded the video gaming company Giant Spacekat, also moved to avoid threats made in response not even to sustained criticism of video games, but to jokes she made about the Gamergate campaign. “Depression Quest” developer Zoe Quinn went into hiding after a vengeful ex-boyfriend published a long account of her alleged infidelities that seemed to imply she chose her partners for professional advancement.
These women's stories are in the news now, coming shortly after a spring and summer that brought much needed attention to campus sexual assaults and the lack of protections and justice most of its victims endure. It seems to me, then, a look at the issue of virtual assault having devastating consequences in physical world, can also increase understanding and shed light on campus sexual assault.

To that end, here are three pieces on virtual assault that I recommend and would assign. Note, these are frank discussions and long pieces. But they're compelling.

A Rape in Cyberspace


I'd start with Julian Dibbell's "A Rape in Cyberspace," available online at his http://www.juliandibbell.com/articles/a-rape-in-cyberspace/

A version Dibbell's essay first appeared in the Village Voice in 1993. That's right, twenty years ago. Dibbell's piece focuses on how a violent virtual rape in LambdaMOO, a text based virtual world where rooms, characters, spaces and actions are all described in words that allowed players to enter, create avatars, and interact, shocked those LambdaMOO members into realizing they were a community. The article describes how the victim of the rape, though virtual, suffered physically -- fear, anxiety, tears. While the community recognized the rape as an assault, they realized there was nothing they could do to punish the perpetrator in real life, no way to bring charges, to get an arrest. And so they develop rules to try to deal with future actions virtually, by setting up guidelines for expelling offenders from the Moo.

I'd start with this piece for a few reason. It's one of the first, perhaps the first, documented case study of cyber rape and its affects. Unlike the underbelly of the contemporary accounts of virtual sexual assault under consideration in the next two pieces, the rapist didn't have recourse to comrades, drum up justification that blamed the woman he assaulted, or wage the assault in multiple sites and social networks. The rape happened in a relatively small and closed community, a community that became more formally formed in response to the rape, and because it was small, was able to adopt more quickly rules and policies to better protect members.

Why the Trolls Will Always Win


First published by Kathy Sierra at her own blog as "Trouble at the Koolaid Point," Wired Magazine republished her original post verbatim just days later with the title "Why the Trolls Will Always Win," and that's the version I link to here: http://www.wired.com/2014/10/trolls-will-always-win/. I'm opting for Wired because Sierra indicates that she might at some point take her original post down.

But off the bat, a first question for students to consider -- how does the different title give by Wired shape their reading?

Sierra's piece marks the tenth year anniversary of her first online threat, of which, Sierra writes, "I thought it was a one-off, then. Just one angry guy. . . .  But looking back, it was the canary in the coal mine…". What's changed? Sierra's work picks up from the one angry guy, the kind of person the LambdaMOO community faced, to a world where social networking -- Twitter, especially in her personal experience -- amplifies and spreads the assault against a lone woman exponentially, as trolls amplify, get picked up, facts are ignored, lies are believed. She explains:
I now believe the most dangerous time for a woman with online visibility is the point at which others are seen to be listening, “following”, “liking”, “favoriting”, retweeting. In other words, the point at which her readers have (in the troll’s mind) “drunk the Koolaid”. Apparently, that just can’t be allowed.
[. . .]
But the Koolaid-Point-driven attacks are usually started by (speculating, educated guess here, not an actual psychologist, etc) sociopaths. They’re doing it out of pure malice, “for the lulz.” And those doing it for the lulz are masters at manipulating public perception. Master trolls can build an online army out of the well-intended, by appealing to The Cause (more on that later). The very best/worst trolls can even make the non-sociopaths believe “for the lulz” is itself a noble cause.
Sierra describes first what her experience tells her about the logic and motivation of trollers and then why and when she stepped back from online life -- how she was harrassed out, and why, with this post, she is stepping back in.

The Future Of The Culture Wars Is Here, And It's Gamergate


Written by Kyle Wagner in Deadspin, this piece (http://goo.gl/5yWvnL), published Tuesday, October 14, begins with the news that Brianna Wu fled her Boston home over the weekend, "after an online stalker vowed to rape and kill her."

Wagner does three very useful things.

  1. He explains the origin of #gamergate, a movement that claims to be about holding the press who cover the gaming industry to journalistic ethics, but that began with a deranged ex-lovers lies and rants about Wu Zoe Quinn, including the false claim that her romance with a journalist lead to a game she wrote being praised.
  2. He critiques the coverage of #gamergate in the traditional press, The New York Times and the like, noting that their instance on even handedness creates what press critics have called false equivalence, a logical fallacy. Such coverage lends undo legitimacy to those who committing the assaults.
  3. This is key, Wagner describes the mindset of the trollers and assaulters, why they feel aggrieved enough to do this, and he likens it the kind of motivations that drive certain segments of the Tea Party and other reactionary groups: change they don't like is coming. For some in the Tea Party, its the recognition, embodied in our first black President, that the nation's demographics are changing and that whites will become a minority.  For those in gamergate, its a reaction against the fact that more and more women play games, and that new games are emerging designed to appeal to those players. 

What Can One Learn From These?


Taken together these three essays move both historically and technologically. Dibbell's piece sets an early example, when the Web was young and new and a smaller place; Sierra's first person account gives voice to a victim of assault and violence in her own terms, introducing and explaining terms and techniques that show the role of social networks in turning the derangement of a lone actor into a a deranged and even more dangerous mob; and Wagner's work takes a step back, offering scathing attack on Wu's assailants and their motivations, and then tying it to a larger cultural trend.

Combined, these are long reads, but as I said, compelling. Each is written in Web vernacular, each is frank, each has its own truths to convey. I like them because they are one-sided, advocate for women and for a civil online social communities, even as the work by Sierra and Wagner show how difficult that goal is. I also think they bring forth the physical and psychological cost of assault, how the relentless nature of these kinds of attacks, have consequences in the offline world. This is not a painless crime, not just words.

And finally, they pieces shed light on the anger and entitlement mindsets of the attackers. So those insights, empathy for the attacked and anger and hate of the attackers, can provide some grounding for a discussion of campus sexual violence.