The Digg Beta Recommendation Engine: A Review

Today Digg released it’s Beta Version of “The Digg Recommendation Engine”.  This Beta Release will be presented to registered Digg users first, based on a random sampling of logged-in users.

The Recommendation Engine is a new way of discovering new content on Digg, .it uses  your past digging activity to identify what Digg calls “Diggers Like You”.  According to Anton Kast, lead scientist at Digg:

“When you Digg a story, you tell the Recommendation Engine two things: that you recommend the story to other users and, less obviously, that the users who Dugg the story before you are good at finding content. The Recommendation Engine keeps track of users who Dugg particular stories before you did, and it recommends you the stories they Dugg. The more content you Digg, the smarter the Recommendation Engine becomes.”

The Recommendation Engine utilizes the Digger’s past history over the last 30 days and it formulates it’s recommendations based upon your activity.  Anton says:

“When it’s time to calculate your Recommendations, the Engine draws from this pool of matched Diggers. For each matched Digger, it computes a correlation coefficient between you and them. It then picks a cutoff for this correlation coefficient, and the Diggers who make the cut are called ‘Diggers Like You.’ “

The engine determine how many stories that other Diggers have in common and divides that number by the total number of stories you or they Dugg. The ratio is a correlation coefficient, sometimes called a “Jaccard Coefficient”.

Once the engine has determined your ‘Diggers Like You’, your recommendations concist of stories that your ‘Diggers Like You’ have alredy Dugg, minus the stories you already Dugg or Buried.  Other steps, like the Diversity Rules and the Promotability Constraint, add into the equation.

What are Promotable Stories?

Promotable stories are stories which are recent enough to be eligible for the Digg homepage, but haven’t appeared there yet.

What is Diversity?

Digg wants you to be diverse, they don’t want all your stories on the same topic or Dugg by the same people. The Beta Version of this Digg Recommendation Engine takes this into account.  To insure your Recommendations are diverse, the Engine imposes limits that keep things from getting too focused. It makes sure that no one ‘Digger Like You’ determines too many of your stories. The Engine also limits the influence of any single one of your Diggs.
How did the Upcoming Page change?

I won’t give screenshots of the entire page, mostly because much of it is either unchanged or irrelevant.  I will show the most important things.  When you go to the Digg homepage and click on the Upcomng Page tab, you see the following notice:

This notice can be closed, or hidden, and the main purpose is to show you that something new was added.  It only provides a brief description of the changes.

Next thing you will see is along the right side of the page, it is the Recommendation Engine.  In this section you will see how many items you have Dugg in the past 30 days, it will give you some stupid comment about “Digging On” and it asks you how are your Recommendations.

There is also a “Learn More” button that you can click. That will take you to a page by Kevin Rose that explains the Beta Version and if you follow through it will take you to Anton Kast’s Whitepaper blog.  I won’t show screenshots of those.

The next section is something most of us will pay attention to the most, it’s the “Diggers Like You” section. This section tells you how many Diggers Like You it found and it shows you the compatibility ranking and the name of the Diggers.  The screenshot below shows the section:

I clicked on the Blue Bar in the Compatibility column, it took me to a page showing this information:

Notice that it shows how many Diggs the Digger has, how many you have, and the difference between the two.  It also shows your Compatibility Score. You have the option of adding the Digger as a Friend or removing the Recommendation.  The screen also shows what submittal the Digger made which is compatible to you.

Summation of the Digg Recommendation Engine

I have a problem with the Digg Recommendation Engine. I see it as is a way for Digg to force Diggers into a thing they call “Diversity”.  In short they want every Digger to post on a variety of topics, instead of leaving it up to the Digger as to which topics they wish to submit or be a part.

Don’t get me wrong… I do not oppose Diversity, but I do feel that it’s up to the individual members whether they want to participate in a wide variety of topics or still stick to one or two topics.  It seems that Digg is going out of it’s way to penalize people for doing not becoming a part of some “diversity” scheme.

As a person who ran his own net and several message forums on the net over the past 15 years, I find this extremely uncomfortable.  The members of any board or group ought to have the freedom to post and reply to whatever topics they wish on the forum, but most of all they should not be penalized for only posting to one or two topics.  To try to force them to do otherwise is completely against the reasons for any Social Group in existence.

The one thing I really like about the Beta Engine is that it gives me one more way to search for Friends and any help is a help.  I can look at what Digg recommends and ignore it or accept it as I will.