Jeremiah Owyang posted today on a Round Table he attended with the folks of Factiva and some others (bloggers, corporate program managers, PR consultants and social media practitioners) to discuss what should be measured in the Social Web. They came up with a nice list which they voted against and ranked according to the most votes.

Here’s the list:

1. Participation & Engagement
2. Influential Ideas: Memes and their intensity over time
3. Relevance
4. Sentiment/Tone/Opinion/Favorability/Emotion
5. Content
6. Relationships & Connections
7. Analytics & Activity
8. Community Activity or Call to Action
9. Reach
10. Conversational Index/Engagement Tied With Demographic/Who

Because Jeremiah did not provide definitions, I’m going to take a jump out into the blue and start playing word association…Participation & Engagement sounds like something to be measured while the Conversational Index sounds like a KPI that could be used to affect that measure - for blogs. As defined by Stowe Boyd the Conversational Index is:

[# of posts] / ([# of comments] + [# of trackbacks])

An index score that is less than one indicates an engaging blog while a score of 1 or more indicates an unengaging blog.

While I agree with the principal of this metric, it seems to me that this score is too easily gamed - and maybe not even on purpose.

In blogging, one of the best practices I’ve heard is to interlink between your posts (Jeremiah does this a lot) which is very helpful to the readers but since platforms like wordpress will treat that intralink as a trackback, the total # of trackbacks will be polluted with your own linking and therefore susceptible to gaming.

Next, when we’re looking at comments, this is a conversation where (hopefully) the author is responding to the comments from her readers. Now, this could be gamed if rather than responding to comments normally, the author starts to break her comments into smaller bits in order to drive a better CI score. On the flip side, bloggers that tend to write shorter posts more often as opposed to bloggers that write longer posts less frequently will be penalized simply because of their writing style.

So what to do? How can this excellent kernel of an idea be taken to something that might be acceptable industry-wide as a standard? I don’t have the perfect answer but here are some thoughts:

1. Intra-blog links need to be excluded from the trackback count, shouldn’t be that hard to do from a technology POV
2. Normalization: Because the CI is at the mercy of the writing style used, a way would have to be found to normalize the score across a spectrum of writing styles from very short/very frequent to very long/very infrequent
3. Author comments - I really don’t have any sort of clue on this one which either makes me incredibly daft or makes it very dangerous to the integrity of the CI score. My WAG is that some sort of algorithm would be needed to determine if the author’s commenting style is a deliberate attempt to positively impact the index.

I hope I don’t sound like I am beating up on the Conversation Index because I really like the idea, so let’s move on…

Memes and their ‘intensity’ over time seems to me to be very much like Reach or at least a special kind of reach. Lacking a definition of intensity, I guess that it means something like how quickly it spreads and how far it spreads and then the question is; how does it pace over time? Isn’t this like Reach?

Relevance: Relevant to What? To Whom? Either this is very atomic in nature (in which case measurement is going to be meaningless) or it means measuring to how many people something is relevant to - wait isn’t that Reach?

Sentiment/Tone/Opinion/Favorability/Emotion - Looks Like A Duck, Sounds Like A Duck so it must be Voice of Customer data. The one difference here being that it doesn’t necessarily require a survey or contact us, rather it can be directly scored (although somewhat subjectively) from blog comments, trackbacked-posts, etc.

Content - Umm… not enough to go on here

Relationships & Connections: Again I ask ‘isn’t this Reach?’ Maybe it’s reach in the context of a LinkedIn but it still looks like Reach.

Reach: ‘Nuff Said

Conversation Index/Engagement - See Engagement/Participation

Demographic/Who: Who is always important whether its from a demographic, technographic or behavioral standpoint. The CMO or advertiser or product manager will always want/need to know who is viewing/engaging with content.

Of course, that’s just a critique of the list that the round table came up with and not necessarily what Jeremiah would want to know from some one in the business of measurement.

But that will have to wait, this post is already making me twitchy from it’s length so I’ll have to do a follow up post on what I think should be measured in the social web - don’t let me forget.

Update

We’ve been having a pretty lively discussion about engagement over at Eric T. Peterson’s Blog, his posts on the topic are:

How Do You Calculate Engagement? Part 1

Frank Faubert Writes In

How Do You Calculate Engagement? Part 2

How Do You Calculate Engagement? Part 3 

Additionally, Eric has started a Web 2.0 Working Group over on Google Groups discussing methods, etc. for measuring Web 2.0

Yep, that’s short-hand for Eric Talks Engagement. Eric gets one step closer to brass tacks by defining the activities that occur on his site, or with his content that are relevant and engaged - for him.

Here’s his list of activities, grouped into moderate engagement and high engagement sets (as defined by Eric):

Moderate-Value Activities

• Read my weblog
• Read about the Web Analytics Business Process
• Research web analytics jobs
• Add a link to my link database
• Read comments about my books
• Give me an email address
• Host a Web Analytics Wednesday
• Join the Web 2.0 Measurement Working Group

High-Value Activities

• Consider buying one or more of my books
• Buy one or more of my books
• Read about any of my books
• Read about my Key Performance Indicator Worksheets
• Download a sample copy of one of my books
• Email me directly
• Submit a comment to my weblog
• Go to Amazon.com to check out my books

Now, score one for Eric for actually thinking through a framework for measuring engagement, and putting it into practice, but it’s late and I’m feeling nit-picky.
Because Engagement has been talked about so much lately in terms of Web 2.0 and ‘Social Media‘ I’m putting Eric’s listed activities through that lense. Of the 16 activities, only 5 measure social media engagement:

• Read my weblog
• Add a link to my link database
• Host a Web Analytics Wednesday
• Join the Web 2.0 Measurement Working Group
• Submit a comment to my weblog

7 seem to be about commerce - dealing with buying Eric’s books

• Consider buying one or more of my books
• Buy one or more of my books
• Read about any of my books
• Read about my Key Performance Indicator Worksheets
• Download a sample copy of one of my books
• Go to Amazon.com to check out my books
• Read comments about my books

Two of them I would equate to traditional CRM efforts

• Give me an email address
• Email me directly

The last two I’m having a hard time categorizing, but my guess is they relate directly to the Eric T. Peterson brand

• Research Web Analytics Jobs
• Read about the Web Analytics Business Process

So, here’s the breakdown:

• Social Media Activities: 31.25%
• Commerce Activities: 43.75%
• CRM Activities: 12.5%
• Other (Brand) Activities: 12.5%

Ok, so Eric’s in a bit of a hybrid situation, which ‘traditional’ (web 1.0?) companies will also find themselves in where social media is just one of many things going on.

But the blogger in me says ‘Come On! Only five of the activities are related to the social aspect of the site AND only one of those five is of high engagement value?’

Eric you gotta get out of your traditional corporate shell! (Disclaimer: I’m a traditional corporate dog too)

Where are the subscriptions to your RSS feed and the associated click-backs?
Where are the track backs?
Where are the buzz-rankings (e.g. post/discussions picked up in other places)?
Let me put this another way, is engagement a practical measure for non-social web activities?

It seems like forever ago that Eric and I talked about engagement and how I expressed some frustration over Scoble’s post about the need for Engagement metrics. Haven’t we web analysts been talking about this like FOREVER (well maybe 18 months or so anyway).

At the time, Eric told me he was measuring engagement on his site and that he planned a blog post on the subject. So I held off because Eric had some real-world experience, and better tools for testing it out. So here we are, more than a month afterwards and that very busy man finally had a chance to get out his thoughts on his experimenting - yeah! Something to look at and noodle over.

Anyway, it’s time to get to work, so more thought on this will have to wait until later. In the meantime, go read Eric’s post and see what you think.

Hey, did anyone notice that the Web Analytics Association Research Committee recently published the first two in a series of podcasts on Best Practices? In these two podcasts, Wendi Malley interviews Avinash Kaushik from Intuit on the topics of measuring Blogs and RIAs (Rich Internet Applications such as Ajax).

These are in-depth interviews (running over a half hour each) with lots of great gems from Avinash on topics from simple ‘how-tos’ to strategic approaches for measuring these new platforms.

(I’d better go look at my search analytics to see what I should be blogging about)

Here are the links to each podcast - ENJOY!

Web Analytics Association: Best Practices - Avinash Kaushik on Blogs & RSS

Web Analytics Association: Best Practices - Avinash Kaushik on Rich Internet Applications

Disclaimer: I am on the Web Analytics Association Board of Directors

Tech Crunch is reporting a rumor that the founders of iRows have signed with Google and iRows will be shut down. (source is here - in hebrew)

If true, it’s a big bummer. iRows along with Edit Grid are my favorite online spreadsheet tools with the best breadth of tools. Certainly, they are currently a lot better than Google Spreadsheets, which still has no graphing capabilities, unless you count the repeat function hack.
There’s no word from the guys at iRows
Hopefully it is all smoke, but if true, I’ll miss iRows - it was my favorite

Edit: It’s True, read the announcement at the irows blog.