Dogsourcing

On April 1st, the Amazon team published findings from a groundbreaking experiment:

"A couple of weeks ago Martin and I arrived at the office at about the same time. He was trying to get Rufus to settle down. Rufus wasn't cooperating and Martin told me that Rufus was a bit bored. Jokingly I suggested that we could probably train Rufus to do some Mechanical Turk HITS."

If your company is interested in pursuing a dogsourcing initiative, some useful resources can be found on Wikpedia:

• Evaluating the intelligence of a dog: "It is likely that dogs do not have the ability to premeditate an action to solve a problem." Systems like Mechanical Turk can help remedy this shortfall, by prompting the canine to action.
• Jobs performed by dogs, including the most impressive...
  
• Assistive Dogs: No joke here. Assistive dogs have to employ constant decision making to keep their disabled owners out of harm's way.
  

We're Exporting Jobs to... Milwaukee?

Outsourcing in high-tech companies has become a familiar enough cliché - it's regular fodder for network TV news, and even Homer Simpson's had his 2 cents. So when Jason Calcanis of Mahalo published a list of 17 tips for start-ups and added outsourcing as #18 ;) it was no surprise. But the destination was.


From Calcanis' blog:

Outsource to middle America: There are tons of brilliant people living between San Francisco, Los Angeles, and New York who don't live in a $4,000 one bedroom apartment and pay $8 to dry clean a shirt--hire them!

The myth that outsourcing = offshoring is something the media loves to indulge in, but the fact is that anyone visiting an outsource portal (Elance, Odesk, Mturk) will find thousands of Americans competing successfully for contract and task work.

Anyone in the US with time to spare and skills to apply can be a part of this, and they don't have to settle for third-world wages to do it. In fact, the very idea of hourly wages starts to get hazy when people do things like Mechanical Turk tasks in their spare time just to unwind.

Intro to Amazon's Mechanical Turk

Mechanical Turk is a crowdsourcing network set-up by Amazon. The service gives you access to a network of freelancers, who log-in to an online system and can pick and choose tasks to work on. The key distinction: unlike Elance and other outsource portals, Mechanical Turk is aimed at piecework, not entire projects.

There's an e-commerce set-up on both ends. Tasks are uploaded by "requesters" - companies and individuals that define the work to be performed. Requesters deposit funds and set pricing for the tasks, while workers who perform tasks get funds transferred to their accounts. These are conventional Amazon.com accounts, so they can be linked to bank accounts or used for purchases on Amazon.

Real world experience

That's the long-story-short version - what's it like to actually use Mechanical Turk? I've assembled some of my experiences below, as well as links to tools and info I've stumbled upon along the way.

What tasks can be performed by Mechanical Turk workers?

MTurk is ideal for any task that involves the manipulation of small amount of information: data entry, analysis of written or photographic materials, research, and transscription of audio recordings are common tasks.

So far, I've assigned work in two areas: writing content, and doing web research. As a web marketer, I'm always looking for ways to generate useful content for websites, and I've been impressed by the high quality of writing that I get from Turk workers.

• Related Post:
  Mechanical Turk Workers Mostly US-Based

Typically, I'll send out sets of 50-500 tasks. A set of several hundred tasks typically gets done in a day or two, although this will vary depending on the complexity of the task, the time of day they're posted, etc. Some requesters launch thousands of tasks at a time, which I've seen workers churn through with a week.

Why use Mechanical Turk?

As a solo consultant, MT gives me access to a very flexible, always-on labor pool. I live in Cambridge, Massachusetts and have access to college students in a wide range of disciplines - they're smart, in plentiful supply, and generally inexpensive. But MT is an easier resource, a more reliable resource, and (at least for now) a much cheaper resource.

What are the advantages, versus employing a traditional subcontractor?

From the requester (employer) standpoint, it has several advantages: not having to scout for talent, file paperwork, keep track of invoices or timesheets. Mechanical Turk automates HR, and allows you to structure freelance work in a way that isn't otherwise possible.

The key to using MT profitably is to recognize what types of projects can be diced into the taskwork that MT can digest easily. MT is best for tasks that are easy to define, and can be clearly expressed in a text message without any back-and-forth clarification. It takes a bit of upfront work to define the tasks, and prepare instructions that can't be misinterpreted - this is crucial to ensuring that the results are accurate, and returned to you in a usable format.

One aspect of the system that helps improve results is the ability to pre-screen workers using a test. If they pass the test, they receive a "qualification" which then allows them to perform tasks available only to them. I have a copywriting qualification set-up for my tasks, and other companies have created dozens more, focusing on areas like language translation or technical expertise.

Finally, it's incredibly scalable. If you have thousands of tasks, so much the better - the upfront cost of preparation work can be spread across those tasks. And workers tend to swarm on larger groups of tasks, realizing that they can settle-in for a few hours and make larger amounts of money.

What are the wages like?

Amazon sets a very low floor of 1 cent per task, so it's left to the free market to decide what rates are fair. At the moment, there are only a few companies participating on a regular basis, so there's a tendency to price tasks very, very cheaply: they know that someone, somewhere will eventually pick-up the assignment.

Wages have become a contentious matter among workers, especially now that the system has been operating for over a year. The overall sentiment expressed on MT forums and blogs is that pricing is too low, and the most active workers seem to be expressing the most dissatisfaction. I think effective wages have to go up to retain workers of the caliber that I've dealt with.

• Related Post:
  Beer Money: Mechanical Turk on Campus

Some companies are trying to remedy this by assigning bonuses or awarding prizes to the best workers, but this creates its own inequities. It also creates a gold-rush mentality, as workers swarm-in to perform tasks that are linked to a bonus.

Eventually more companies will enter the system, and competition for workers' attention will drive prices to a more equitable level. But that depends on how well Amazon will succeed in attracting companies with large workflows to perform. If free market activity can't float a higher wage, and workers begin to drift off, then a more hands-on policy may be necessary.

What tools are available to access the system?

To access the system via any method, you first must sign-up for an Amazon account at Mturk.com.

• The web interface at Mturk.com is pretty rudimentary, and is useful only for a few initial experiments.

• True power users will want to take advantage of the API - developer resources can be found at the Amazon Web Services site.

• Intermediate users may want to check out HIT Builder, a web-based interface for piping tasks into the system. It's a bit kludgy at times, but allows you to do bulk uploading of assignments without touching code.

Further Updates

This article will be continuously updated, with all updates posted to this location (so feel free to bookmark or del.icio.us it.)

Mechanical Turk Workers Mostly US-Based

I've got some results to share from a questionaire deployed on Mechanical Turk to gather stats and feedback from the Turk workforce.

Here's a taste of the info gathered from the first 100 responses: a breakdown of country of origin. As suspected, Amazon's U.S.-centric payment system has resulted in a mostly-U.S. workforce, but a few workers from elsewhere are participating.

I also see this same geographic spread in the web stats for this blog:

• 79% United States
  
• 4% Canada
  
• 2% UK
  
• 2% Israel
  
• 2% France
  
• 1% each from Cyprus, Finland, Germany,
  India, Italy, Malta, Netherlands, Romania,
  Singapore, Thailand, and (yes!) Turkey.

I'll leave you with one comment from Finland-based worker that sums-up the payment issue nicely:

The idea is very intriguing, but unfortunately U.S. businesses aren't taking advantage of it and the service isn't open to bold people elsewhere.

Influencing an Election with Crowdsourcing

An article in the New York Times highlighted an interesting new political tactic: campaign resources are being used to promote certain newspaper articles online, such that they appeared at the top of a Google search for a candidate's name.

This makes for a powerful propaganda tool. For example, a Democrat could promote a negative article on her Republican challenger, such that it appeared #1 in Google for that person's name. Anyone searching for info on that Republican would get an eyeful of criticism, giving them something new to think about. The fact that a newspaper article (a supposedly objective source) is used as the target makes its impact even more compelling.

This is a great example of crowdsourcing - not just because many people were involved, but because many people's websites were being used for a focused task.

Miserable Failure

You might recall the story a couple years back, where Bush's White House profile became the #1 result for the search "miserable failure." (Yes - it's still there.) This is an example of Google bombing: a mischievous version of a marketing practice called search engine optimization (SEO).

SEO is a technique used by companies to get more visibility in search engine listings. One interesting aspect of SEO has to do with links between websites.

Search engines add-up links from other websites as a form of reputation scoring: CNN.com has a "better" score than NPR.org - not because it has better news, but because more websites link to CNN articles than NPR articles. Link scoring is Google's search sauce, and it's become standard practice for all search engines.

Link scoring is crowdsourcing at its best, because the crowd (the universe of websites) is largely unaware that their actions are being analyzed and used to serve some secondary purpose.

Gaming the System

Google bombing (and SEO in general) introduces a new twist: the idea that this "crowd" can be contacted, requested to link to a specific website in a specific way, and create an intentional outcome.

All of this brought to mind a discussion I recently had with one of the Amazon reps working with Mechanical Turk. They have a provision in their terms of service that prevents companies from using their workforce for promotional work. So you can't hire an army of Turk workers to build links from other websites.

I thought this seemed unduly restrictive, because I'm a search engine marketer and saw Turk's potential as a useful tool in my work. The rep explained that Amazon was trying to avoid having someone come-in and use their workforce as part of some "gaming" effort.

Given the New York Times article, I'm inclined to agree with the wisdom of that logic... if somewhat reluctantly :)

Feedback from the Welcome Mat

Last week I set-up a Welcome Mat HIT group on Mechanical Turk, to announce the Paylancers blog and gather feedback from workers. I received some interesting comments about both the site and Mturk in general.

"Very interesting. . . mainly because I didn't even know that this stuff (i.e., Mechanical Turk, "crowdsourcing") even existed before today."

"I didn't realize that Mturk was still not drawing big companies. It's a shame that companies don't realize how much research they are missing out on."

Wages are an evergreen issue...

"I do HITs mostly outta boredom and for a few extra bucks a week, but at this point the only people making "big bucks," i.e. more than beer money, are those able to place high in the contest answering the research questions. As such, those HITs have become ridiculous and the goal seems now to see how many you can snag, not how many great answers you can really give."

"I find it amazing that anyone can find things to comment on... that don't involve how we, the workers, get screwed money-wise. ;) I think there have only been maybe one or two days where I've hit minimum wage on here."

"The blog is nice, and I like the link to the message board. I think, though, that the author of the blog ought to take complaints about low pay rates for HITs more seriously than he does."

I think I've been taking it seriously, but perhaps because I'm a requester I don't take it personally enough. Payout is Mturk's biggest credibility issue with workers. And for outside observers too, who look at Mturk as some bizarre social experiment that will never catch on commercially.

Another reader was curious...

"...why you like blogspot, instead of your own hosted instance of Wordpress or another solution?"

Simple - I wanted to hit the ground running and focus on writing. I have friends that have installed Wordpress, and it takes them a while to get comfortable with the settings and template design. I'd rather invest those first 50 hours learning, using, and writing about Mturk - and setting-up Blogger is as easy as rolling out of bed.

Overall, the feedback was very encouraging: a lot of workers were not aware of the Turking sites that have sprung-up, and were happy to see that people were discussing the issues they've come across.

HIT Groups as Welcome Mats

As an experiment, I posted a "My 2 Cents" HIT group on Mechanical Turk to announce this blog to Mturk workers. In the space of a few minutes, I've already gotten some very interesting feedback - including some comments that suggest the "Read More" links on every post aren't visible enough. So it's doing double-duty as a promotional tool and a focus group.

New PHP Tools for Mechanical Turk

Santa Cruz Tech has just released a first draft of their PHP code for Mechanical Turk. This will help take some of the burden off developers trying to code applications that tap into Mturk.

Santa Cruz joins DPA, which has had its HIT-Builder tool out for several months. The two tool sets are very different: HIT-Builder is a full-blown web interface for Mturk, whereas Santa Cruz's work only covers the underlying plumbing.

But PHP is an ideal choice for underpinning a web interface, so one would imagine that their work (released as open source) will eventually power some user-friendly tools.

Beer Money: Mechanical Turk on Campus

A constant issue in the Mechanical Turk community is the rate of pay: given the low prices of HITs, and the time it takes to execute them, many people have complained that - even at breakneck speed - they're having a hard time hitting minimum wage.

Others take a more casual attitude, saying that Mturk is an online pastime like blogging, chatting, or surfing - only this one pays you to do it.

Perhaps it boils down to personality types. Or maybe it's a matter of finding the right audience. In the early days of Mturk, the inevitable trend seemed to be that India would dominate the worker population, with their killer combo of English skills, internet access, and low wage demands.

This hasn't happened (yet) mostly because Amazon's payment system is very low-tech and US-centric. If PayPal becomes an option, then it'll be a very different ball game.

But for now, given the current set-up, is there an ideal workforce for Mechanical Turk? College students seem like a no-brainer: they are smart, low on cash, and have high-speed connections on many campuses. But let's look more closely at the factors, because they do a lot to highlight the benefits of working at Mturk.

• Many on-campus college students don't have cars, so job opportunities are limited. Mturk's telecommute not only makes work more reachable, but also saves the cost of driving for students that do have cars (a huge factor these days.)
  
• Mturk is a casual take-it or leave-it workplace. I can log-in for 15 minutes or an entire Saturday afternoon - how many jobs offer that? College students have their studies and their social lives to think about, so flexibility is a huge draw.
  
• Also, think about the upfront cost: all the steps a person must take to find even the simplest part-time position. Plus the commute, plus the workplace friction, etc. For a college student looking for a couple hundred bucks a month, Mechanical Turk is compelling because there's no barrier to entry.

In short, the appeal of Mechanical Turk is not that it's big money, or even fair money, but that it is literally easy money.
I'm sure a lot of other audiences share some of those factors, and would find Turk equally appealing.

Cottage In, Cottage Out

Amazon probably hopes that Mechanical Turk will evolve to the point where large companies will jump in, dumping 6- and 7-figure HIT groups into the mix. But for now, it's a cottage industry on both ends of the transaction. Should Amazon do more to foster small-scale Turking? And (more critically for Amazon) are they stuck with it?

Corporate IT departments are strange creatures. As individuals, IT pros love to experiment, learn about new technologies, and use whatever resources are available to get things done quickly.

But group them together in an air-conditioned cube farm, and the collective instinct that results will steer them away trying something new. Your typical IT back office is very much like your father's Oldsmobile - full of sensible choices, industry standards, and minimized risks.

Why? Because they are financially accountable for every mistake they make. They are conservative in this environment because it makes sense to do so.

Big companies are not biting

So, here we have this new web service from Amazon, one which provides a highly-scalable human labor pool that is perfect for large corporations with huge projects and deep pockets. Undoubtedly, this is the cheese cake that Amazon is after, because when you're dealing with 1/2 cent commissions on HITs, you need millions of them make a profit.

But corporate customers are not biting yet - judging from what I see on the HIT lists, and the reactions I've gotten from my own clients.

Why? I think it's due to a couple factors.

Security

Almost any company that possesses millions of pieces of information is absolutely paranoid about their competitors getting access to it. Turk script writers know that fetching this info and reassembling it would be a cinch. Corporate IT knows it, too.

Predictable capacity

Second, large volumes of work require large volumes of throughput, or else your 4 million assignments will take about 4,000 days to complete. Right now, Amazon isn't too keen on releasing workforce volume data, or even a tachometer to see how fast HITs are getting done. Without this data, a corporate manager would be very uneasy about committing a large project to Mturk.

Yes, they could try a small project first - a pilot program to demonstrate the value of the process. But corporations like solutions tailored to their needs, and right now Mturk customization is entirely self-service - download the API docs and off you go. Not very inviting.

Behind door #2...

Fortunately, small businesses do not have these concerns. And thanks to systems like DPA's HIT Builder you don't even have to be technical to use the system, even for complex HITs.

So Amazon should make a genuine effort to cater to the 1- to 1000- HIT requester. Who knows, the repeat business could mount up quickly ;)