"whatcha talking 'bout?" version 4!

Ok, this is pretty much the final definitive version of analysing the latest hot topics from library/librarian blogs…

hotstuff
http://161.112.232.18/hotstuff.php

The page is updated approximately every 15 minutes and uses the following methodology…

1) posts older than 48 hours are analysed and the frequency by which every unique word appears is calculated

2) posts from the last 48 hours are analysed in the same way and the word frequency is compared to the older posts

3) when a word has become noticeably more frequently used in the last 48 hours, it’ll appear in the word cloud — the bigger the increase in frequency, the larger the size in the cloud

4) if a word appears in multiple new blog posts then the shading is darker

5) if a word only appears lots of times, but only in a small number of new posts, then the shading will be lighter

So, in the last 24 hours, the usage of “2007″ has increased substantially.  In new posts the word has a frequency of 28%, but only 7% in older posts.

Several bloggers have picked up on the sale of ProQuest — if other bloggers talk about it today, then it will grow in size and be shaded darker.

The usage of the word “disallow” has also increased, but it only appears in a single blog post (by the Baby Boomer Librarian) and is therefore shaded lightly.

Unlike the previous versions, this one doesn’t require a stop word list — words like “library” and “the” tend to have a high frequency of usage in both old and new posts, so the relative difference in usage is usually too small for the words to appear in the cloud.

The other cool thing is that this version uses the “network effect” — the more posts it has to work with, the better the cloud becomes as delivering the latest hot topics.  For example, Stephen Abram‘s RSS feed is currently delivering posts from the last 3 days and he usually ends them with “Stephen”, so he’s currently making a strong appearance in the cloud.  However, over time, the number of older archived posts with the word will increase which means he’ll no longer (relatively speaking) be a hot topic in the cloud …although not in real life, of course!