Crowdsourcing request: Help us create a list of blogs for v. 2.0

Update: Thanks to everyone for your help! We’ve finished updating the database. Look here for news on our launch on Saturday, January 15.

In just under three weeks, we’ll be unveiling the beta version of the next generation of this site.

The new site will work very differently from this one; it is a custom-created database that collects information from hundreds—and ultimately thousands of blogs. Users will easily be able to select just the topics they want, instead of seeing posts based on what network they are on. We want the beta site to be usable from day one, but to do that, we need some help.

I’ve created a Google Docs Spreadsheet for this purpose. Anyone can access the spreadsheet and make modifications. What we need are the name, URL, RSS address, and topic of each blog. What we have, in most cases, is just the URL. If everyone pitches in and visits 10 to 20 blogs, then we should be able to generate this information in a matter of days, if not hours.

Most of the blogs are listed on the Master Blog List (the first tab at the bottom of the spreadsheet). To start helping, just fill in the information in the space provided. If you figure out an automated way of doing this, you can reserve a block of blogs by typing your name in the designated column; then no one will duplicate your efforts.

The reason we need humans to do this is that we want the blogs to be classified by topic. We’ve generated a list of topics (on the last tab in the spreadsheet). When you visit a blog, figure out what topic from our list best describes the blog, and enter it in the space provided (most web browsers will display a drop-down menu to make this easy for you).

The other tabs are for blog networks that are a little more difficult to suss out; either there was no easy way for us to find a list of blogs, or there are non-science blogs mixed in with science blogs. So, we’ve given specific directions for what to do in each case.

FAQs

• My Blog Isn’t Listed!
  Don’t worry! Either we’ve already got all the info we need (in the case of some blog networks) or you’re an independent blogger and you’ll be able to register your blog when the site launches. If you don’t think you’re in either of those camps, let us know in the comments below
• None of the official topics apply to this blog
  Just pick the closest match. You can get more specific in the secondary topic
• I don’t agree with your list of topics
  We had to start somewhere. The list will be easily modifiable in the future.
• One of the listed blogs is not scientific
  Explain your objection in the Notes section on the spreadsheet
• Someone has reserved a block of blogs for hours
  You can use File –> See revision history to see how recently an update was made. If it’s been more than an hour, feel free to delete their name, substitute yours, and work on that entry
• There’s no drop-down menu of topics
  Try using a different browser. I’ve tested it on Safari and Firefox, but I can confirm it doesn’t work on Chrome for Mac.
• What’s in it for me?
  Our eternal gratitude? Plus, if we see you at a conference, we’ll buy you a beer

Thanks again. Let us know if you have any other questions in the comments.

Here’s another link to the Google Docs Spreadsheet

Building a better network: Identifying trends/posts of interest

When you build a network of blogging networks, the problem quickly escalates from “how do I collect as much data as possible?” to “how do I manage all this data?”

Take a look at the Science Blogging Aggregated home page. There’s lots of great stuff there — too much for the typical reader to handle. Even if you visit several times a day, the information rushes by too quickly to discern any trends, and it’s hard to know which posts are really well thought out and which are just one-off posts that hardly merit your attention at all.

We talked yesterday about one way of sorting through the data — tags. However, this method alone probably won’t satisfy all users. A person might be interested in all posts tagged “psychology,” but they might just want to see the highlights of what’s going on in other fields, and tagging won’t help them identify the most interesting, thoughtful posts.

We see at least four possible ways of sifting through the posts to find the most interesting ones.

1. Crowd-sourced ranking. Users rate or recommend posts they like, so others can sort by rating or number of recommendations to find the posts they want to see. An advantage is that there is no central authority telling readers what to like. A disadvantage is that blogs that are already very popular are perhaps most likely to be recommended, so this system might not help users identify up-and-coming blogs that are very high quality.

2. Self-promotion. Bloggers could promote a small number of their posts, indicating these are their best work (one per week? one per month?). This overcomes the “up-and-comer” problem, but a blogger whose work is mediocre could exploit the system by promoting posts that aren’t very interesting or useful to others.

3. Active curation. Editors could be chosen for each field (physics, biology, etc.) and actively promote one or two posts each day. That way readers would know that an expert has read all the posts on a topic and selected the most interesting or relevant. Advantages are that editors may be able to identify trends that more automated systems don’t catch, and that editors may be less swayed by the most popular blogs. Disadvantages include possible bias of editors, and variable editor quality. It would also require coming up with a system for selecting editors. Would a central person be in charge of that, or would we need to create some sort of a system for nominating/voting for editors?

4. Social networking. We could create a truly social network where users are only shown the “likes” of their friends. However, this requires a significant programming effort, and people are reluctant to join new social networks when they already participate actively in one or more networks. I think we might be better off using the social features of other networks, rather than building our own. If we could make it really easy for people to post their “likes” to Twitter and Facebook, then we could leverage those networks to perform the social function.

There is, of course, no reason that we shouldn’t do all of these things over the long run. But we have limited resources. Which of these approaches is most useful? Are there any other approaches that would work better? Do you have any specific suggestions for how to implement any of these ideas? Let us know in the comments.

Building a better aggregator: Goals, Tagging

The ScienceBlogging site you see now was always intended to be a temporary solution. What we really need is a site that not only aggregates blog posts, but also allows users to classify them, search them, highlight their favorites, point their friends to them, and do many other things we haven’t even imagined yet.

Behind the scenes, Bora, Anton, Jessica, Mark, and I have been discussing how to do that, but we realized that limiting the discussion to just ourselves is depriving us of a valuable resource: The people who’ll be using and contributing to the new site.

So, over the next few days, I’ll be offering some thoughts about how to proceed and inviting your comments. Our plan is to have at least a partially functional, working prototype of the new site by the ScienceOnline conference in January 2011. Let’s get that started right now by discussing the goals for the site.

Goals
Here are the goals we came up with for the site:

• To be a central site where scientists, media, other experts, and laypeople see what scientific topics are being discussed on blogs, in real time
• To be a resource for locating past discussions
• To promote science blogging and other online discussion of science
• To promote scientific accuracy and avoid pseudoscience and crackpottery
• To be encyclopedic and inclusive
• To be searchable and filterable
• To have a system (or multiple systems) for highlighting discussions and posts that are especially topical / high quality
• To have a means of removing or hiding posts that are not scientific (e.g. vacation photos, political rants unrelated to science, etc.)
• To be multilingual
• To be open source / open access

Should anything be added, changed, or removed?

Tags
One of the first considerations will be how to keep track of all this information, and a huge key to that will be classifying it. That’s why we think it will be essential to have a unified tagging system in place. If bloggers don’t select their primary tags from a central list, then it will be difficult for users to find posts on the topics that interest them. On the other hand, if bloggers must visit our site to choose primary categories, then usage will suffer. We can allow bloggers to set default tags for their posts using their registration page, but there should be some way to override those settings for individual posts, still using our list of preferred tags.

Could we create a WordPress plugin for this? Or adapt an existing plugin? What about other blogging platforms? What about templates that don’t support tags? One possibility is using a bookmarklet, which would be platform neutral but not ideal. Any other ideas on how to implement a tagging system?

That’s just the first bit — there’s a lot more to discuss, but we thought this would be a good way to get the conversation started. So please, let us know what you think in the comments.

The Open Laboratory – what, how and why

The Open Laboratory is the annual anthology of the best writing on science blogs. Yes, this is an actual, physical book, printed on paper.

The aim of the book is twofold: first, to showcase the quality of science blogging to the audience that does not read blogs and perhaps has a negative opinion of blogs due to the anti-blog propaganda in the mainstream media, and second, to build and strengthen the science blogging community.

The idea for the compilation came from a discussion between Anton Zuiker and a representative of the online book publisher Lulu.com. They were trying to find a fun and useful way for the company to sponsor the first ScienceOnline conference (then called Triangle Science Blogging Conference). As it was late December 2006 there were only about four weeks left until the conference, so they thought there was not sufficient time to collect and publish such a book and have it ready in time for the meeting.

But I thought it could be done if the project was completely crowdsourced. I posted a call for submissions on my blog and e-mailed hundreds of science bloggers asking them to recommend either their own or other people’s best posts which they promptly did. I then asked several science blogging friends to help me read and evaluate all the entries. This narrowed the field from 218 submissions down to 62. Out of those 62 finalists, I picked 50 essays, making sure that different areas of science, as well as different formats and styles, were represented in the final version. I contacted the authors and, with huge help from Anton Zuiker on the technical side of things, put the book together and had it published just in time for the first Conference. You can buy the first edition here.

The book was an instant success – both among the bloggers and in reviews published in several media outlets and journals (including in Nature). It became obvious that this had to become an annual project. But it was also obvious that this project is too big for one person to handle alone.

Thus, for the second anthology, I asked Reed Cartwright to act as the 2007 guest editor. The number of entries doubled, so his help in setting up the technology for submission, judging and sorting the entries was invaluable. His technical skills also made the book look much better. Thus, the second book was born. You can buy it here.

In 2008, guest editor Jennifer Rohn brought her editorial skills (as well as skills in persuading several other people to help) to produce an even more professionally edited and prettier book – you can find it here.

For the 2009 edition, guest editor SciCurious refined the judging method further and the result was, again, a great improvement. You can see it for yourself here.

The work on the 2010 book is in progress. The guest editor is Jason Goldman. The Submission form is here and the instructions for submitting are here. You can buy all four annual collections here and you can read Prefaces and Introductions to older editions here.

I post the full updated listing of all the submissions every Monday morning. This serves as a reminder for bloggers to submit their (and other people’s) posts, and to some extent prevents duplicate entries. But most importantly, it presents a growing listing of some of the most exciting work on science blogs. This is a weekly post where bloggers can discover each other and discover blogs they were not previously aware of. Thus it is also a promotion for all the bloggers involved.

The complete transparency of the process and the community involvement in the entire project are the biggest strength of it. Everyone in the science blogging world feels a little bit of pride in it and a little bit of ownership in it. Competition is tough, but everyone is very sportman-like when the final winners are announced in late December or early January, everyone congratulates the winners and everyone helps promote the book to their friends and families. Thus the project serves both as a glue for the community and as a means for the community to promote itself to the people outside of it, including people who are not online at all. Thus both the science and the world of blogging gain new readers from the project.

Some thoughts about science blog aggregation

(Adapted from two posts on Word Munger)

After the summer’s “PepsiGate” affair and the subsequent departure of 20 or so bloggers from ScienceBlogs, I suggested that if the departing bloggers want to continue to have the kind of influence they used to have at ScienceBlogs, they need to do something more than just start or restart their old, independent blogs. They need to form a new network — perhaps built around different principles, but a network nonetheless. They might choose to have a central site based on RSS feeds or some other aggregation system, but there needs to be a systematic way to connect their conversations. Otherwise, most readers will tune out. It’s simply too much work for most readers to follow a diverse set of disconnected blogs. Social networking sites like Twitter can bring important individual posts to light, but are less effective at sharing the extended conversations that go on between blogs.

Sure, there are some other burgeoning science blog networks, but none seem to be prepared to assume the ScienceBlogs mantle (which ScienceBlogs itself hasn’t actually yet ceded). There are also some lists of all the bloggers who’ve left ScienceBlogs, but they don’t capture all the other science bloggers who were never a part of ScienceBlogs, or the many excellent bloggers who chose to stay.

To me, the obvious next step would be to find some way of collecting all these disparate voices in one place. Sure, ResearchBlogging does some of that, but it only captures posts specifically about peer-reviewed research, which is probably less than ten percent of what scientists and science communicators actually blog about.

One idea that shows promise, at least as a stopgap, is to use an existing social network to do the task. There’s already discussion over at Friendfeed about doing just that. The advantages of such a system is that Friendfeed already has tools in place to help people “like” and “dislike” posts, discuss them, and so on.

To see how this might work, I created a FriendFeed group for Anthropology, based on blogs registered with ResearchBlogging.org. You can check it out here. But this isn’t all Anthropology blogs, or even all Anthro blogs registered with ResearchBlogging — I cheated a bit because my default report of regisered blogs doesn’t include RSS addresses. I only used blogs from Blogger and Wordpress since their RSS URLs are easily reproduced based on the blog URL. And there are other problems. Many blogs cover multiple topics. How would you decide how which list(s) to put them on? What if someone started posting pseudoscience, or moved their blog? Who would be in charge of monitoring the list to make sure it remains useful? And how many people would actually register with FriendFeed just to follow blogs? The beauty of a site like ScienceBlogs is it stands on its own — you go there to read blogs about science. Someone who’s only interested in science (and not social networking) is less likely to hang around a site like FriendFeed just to read science blogs. I’m unconvinced that a set of feeds could have the same influence as a dedicated science blog aggregator.

In the wake of these thoughts, Bora, Anton and I came up with something we think is at least a little better. This site is sort of an aggregator of aggregators. We’re letting others do the work of collecting blogs into bundles; we’re just sharing all those bundles. If other bundles are promising, we’ll add them to the aggregator here, with a minimum of fuss. It’s not ideal — I think the ideal aggregator would have more active curators, and a way to sort through all the posts by topic — but it’s certainly a good start. Let us know what you think.