Jeremiah Owyang posted today on a Round Table he attended with the folks of Factiva and some others (bloggers, corporate program managers, PR consultants and social media practitioners) to discuss what should be measured in the Social Web. They came up with a nice list which they voted against and ranked according to the most votes.

Here’s the list:

1. Participation & Engagement
2. Influential Ideas: Memes and their intensity over time
3. Relevance
4. Sentiment/Tone/Opinion/Favorability/Emotion
5. Content
6. Relationships & Connections
7. Analytics & Activity
8. Community Activity or Call to Action
9. Reach
10. Conversational Index/Engagement Tied With Demographic/Who

Because Jeremiah did not provide definitions, I’m going to take a jump out into the blue and start playing word association…Participation & Engagement sounds like something to be measured while the Conversational Index sounds like a KPI that could be used to affect that measure – for blogs. As defined by Stowe Boyd the Conversational Index is:

[# of posts] / ([# of comments] + [# of trackbacks])

An index score that is less than one indicates an engaging blog while a score of 1 or more indicates an unengaging blog.

While I agree with the principal of this metric, it seems to me that this score is too easily gamed – and maybe not even on purpose.

In blogging, one of the best practices I’ve heard is to interlink between your posts (Jeremiah does this a lot) which is very helpful to the readers but since platforms like wordpress will treat that intralink as a trackback, the total # of trackbacks will be polluted with your own linking and therefore susceptible to gaming.

Next, when we’re looking at comments, this is a conversation where (hopefully) the author is responding to the comments from her readers. Now, this could be gamed if rather than responding to comments normally, the author starts to break her comments into smaller bits in order to drive a better CI score. On the flip side, bloggers that tend to write shorter posts more often as opposed to bloggers that write longer posts less frequently will be penalized simply because of their writing style.

So what to do? How can this excellent kernel of an idea be taken to something that might be acceptable industry-wide as a standard? I don’t have the perfect answer but here are some thoughts:

1. Intra-blog links need to be excluded from the trackback count, shouldn’t be that hard to do from a technology POV
2. Normalization: Because the CI is at the mercy of the writing style used, a way would have to be found to normalize the score across a spectrum of writing styles from very short/very frequent to very long/very infrequent
3. Author comments – I really don’t have any sort of clue on this one which either makes me incredibly daft or makes it very dangerous to the integrity of the CI score. My WAG is that some sort of algorithm would be needed to determine if the author’s commenting style is a deliberate attempt to positively impact the index.

I hope I don’t sound like I am beating up on the Conversation Index because I really like the idea, so let’s move on…

Memes and their ‘intensity’ over time seems to me to be very much like Reach or at least a special kind of reach. Lacking a definition of intensity, I guess that it means something like how quickly it spreads and how far it spreads and then the question is; how does it pace over time? Isn’t this like Reach?

Relevance: Relevant to What? To Whom? Either this is very atomic in nature (in which case measurement is going to be meaningless) or it means measuring to how many people something is relevant to – wait isn’t that Reach?

Sentiment/Tone/Opinion/Favorability/Emotion – Looks Like A Duck, Sounds Like A Duck so it must be Voice of Customer data. The one difference here being that it doesn’t necessarily require a survey or contact us, rather it can be directly scored (although somewhat subjectively) from blog comments, trackbacked-posts, etc.

Content – Umm… not enough to go on here

Relationships & Connections: Again I ask ‘isn’t this Reach?’ Maybe it’s reach in the context of a LinkedIn but it still looks like Reach.

Reach: ‘Nuff Said

Conversation Index/Engagement – See Engagement/Participation

Demographic/Who: Who is always important whether its from a demographic, technographic or behavioral standpoint. The CMO or advertiser or product manager will always want/need to know who is viewing/engaging with content.

Of course, that’s just a critique of the list that the round table came up with and not necessarily what Jeremiah would want to know from some one in the business of measurement.

But that will have to wait, this post is already making me twitchy from it’s length so I’ll have to do a follow up post on what I think should be measured in the social web – don’t let me forget.

Update

We’ve been having a pretty lively discussion about engagement over at Eric T. Peterson’s Blog, his posts on the topic are:

How Do You Calculate Engagement? Part 1

Frank Faubert Writes In

How Do You Calculate Engagement? Part 2

How Do You Calculate Engagement? Part 3 

Additionally, Eric has started a Web 2.0 Working Group over on Google Groups discussing methods, etc. for measuring Web 2.0

This entry was posted on Thursday, January 4th, 2007 at 7:41 am and is filed under web 2.0, Web Analytics, web marketing. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.