This site is made possible by member support. ❤️
Big thanks to Arcustech for hosting the site and offering amazing tech support.
When you buy through links on kottke.org, I may earn an affiliate commission. Thanks for supporting the site!
kottke.org. home of fine hypertext products since 1998.
Steven Johnson responds to (blasts? slams?) the endangered joy of serendipity piece I just linked to, arguing that the web is a much better serendipity engine than the library. (BTW, I think Steven is part machine himself…after posting that link, I took out the trash and ducked out to get something at the bodega around the corner and when I got back, there’s a message from him in my inbox with a link to his rant. Jesus.)
William McKeen on the “endangered joys of serendipity”. “Do people browse anymore? We have become such a directed people. We can target what we want, thanks to the Internet. It’s efficient, but dull.”
pb fills us in on how he finds lost URLs. In addition to the techniques he lists, I use the search function on my newsreader, which content also gets indexed with Spotlight so that works as well.
Google can be used for finding scientific papers that are more popular (and influential?) than their number of citations would otherwise indicate. “The technique might also emerge as a more useful measure of scientific impact than merely the number of citations alone.”
kottke.org: #1 Google search result for “nude paddleball players”. (thx, jonah)
John Battelle’s book, The Search, is not available on Google Book Search because his publisher, Penguin, is suing Google over Book Search. On Penguin’s decision, Battelle says, “I totally disagree with it” and “It’s very irritating to me”. (via jb)
Business 2.0 imagines Google’s future: as The Media, as The Internet, its death, and as God.
In 2002, Dave Winer of Scripting News and Martin Nisenholtz of the New York Times made a Long Bet about the authority of weblogs versus that of NY Times in Google:
In a Google search of five keywords or phrases representing the top five news stories of 2007, weblogs will rank higher than the New York Times’ Web site.
I decided to see how well each side is doing by checking the results for the top news stories of 2005. Eight news stories were selected and an appropriate Google keyword search was chosen for each one of them. I went through the search results for each keyword and noted the positions of the top results from 1) “traditional” media, 2) citizen media, 3) blogs, and 4) nytimes.com. Finally, the scores were tallied and an “actual” winner (blogs vs. nytimes.com) and an “in-spirit” winner (any traditional media source vs. any citizen media source) were calculated. (For more on the methodology, definitions, and caveats, read the methodology section below.)
So how did the NY Times fare against blogs? Not very well. For eight top news stories of 2005, blogs were listed in Google search results before the Times six times, the Times only twice. The in-spirit winner was traditional media by a 6-2 score over citizen media. Here the specific results:
1) Hurricane Katrina hits New Orleans.
Search term: “hurricane katrina”
3. Top citizen media result (Wikipedia)
13. Top media result (CNN)
56. Top NY Times mention (NY Times).
61. Top blog result (Kaye’s Hurricane Blog)
Winner (in spirit): Citizen media
Winner (actual): NY Times
2) Big changes in the US Supreme Court (Rhenquist dies, O’Conner retires, Roberts appointed Chief Justice, Harriet Miers rejected).
Search term: “harriet miers”
4. Top media result (Washington Post)
5. Top citizen media result (Wikipedia)
8. Top NY Times mention (NY Times)
11. Top blog result (TalkLeft)
Winner (in spirit): Media
Winner (actual): NY Times
3) Terrorists bomb London, killing 52.
Search term: “london bombing”
1. Top media result (CNN)
2. Top citizen media result (Wikipedia)
21. Top blog result Schneier on Security
No NY Times article appears in the first 100 results.
Winner (in spirit): Media
Winner (actual): Blogs
4) First elections in Iraq after Saddam.
Search term: “iraq election”
1. Top media result (BBC News)
6. Top blog result (Iraq elections newswire)
6. Top citizen media result (Iraq elections newswire)
14. Top NY Times mention (NY Times)
Winner (in spirit): Media
Winner (actual): Blogs
5) Terri Schiavo legal fight and death.
Search term: “terri schiavo”
2. Top blog result (Abstract Appeal)
2. Top citizen media result (Abstract Appeal)
4. Top media result (CNN)
65. Top NY Times mention (NY Times)
Winner (in spirit): Citizen media
Winner (actual): Blogs
6) Pope John Paul II dies and Cardinal Joseph Ratzinger appointed Pope Benedict XVI.
Search term: “pope john paul ii death”
1. Top media result (CNN)
3. Top citizen media result (Wikipedia)
58. Top blog result (The Pope Blog: Pope Benedict XVI)
No NY Times article appears in the first 100 results.
Winner (in spirit): Media
Winner (actual): Blogs
7) The Israeli withdrawal from the Gaza Strip.
Search term: “gaza withdrawal”
1. Top media result (Worldpress.org)
31. Top blog result (Simply Appalling)
31. Top citizen media result (Simply Appalling)
No NY Times article appears in the first 100 results.
Winner (in spirit): Media
Winner (actual): Blogs
8) The investigation into the Valerie Plame affair, Judith Miller, Scooter Libby indicted, etc..
Search term: “scooter libby indicted”:
1. Top media result (CNN)
15. Top blog result (Seven Generational Ruminations)
15. Top citizen media result (Seven Generational Ruminations)
43. Top NY Times mention (NY Times)
Winner (in spirit): Media
Winner (actual): Blogs
And just for fun here’s a search for “judith miller jail” (not included in the final tally):
1. Top media result (Washington Post)
3. Top blog result (Gawker)
3. Top citizen media result (Gawker)
No NY Times article appears in the first 100 results (even though there are several matching articles on the Times site).
In covering the jailing of their own reporter, the Times lagged in the Google results behind such informational juggernauts as Drinking Liberally, GOP Vixen, and Feral Scholar.
Winner (in spirit): Media
Winner (actual): Blogs
Here’s the overall results, excluding the Judith Miller search:
Overall winner (in spirit): Media (beating citizen media 6-2).
Overall winner (actual): Blogs (beating the NY Times 6-2).
Some observations:
The eight news stories were culled from various sources (Lexis-Nexis, Wikipedia, NY Times) and narrowed down to the top stories that would have been prominently covered in both the NY Times and blogs.
The keyword phrase for each of the eight stories was selected by the trial and error discovery of the shortest possible phrase that yielded targeted search results about the subject in question. In some cases, the keyword phrase chosen only returned results for a part of a larger news story. For instance, the phrase “pope john paul” was not specific enough to get targeted results, so “pope john paul ii death” was used, but that didn’t give results about the larger story of his death, the conclave to select a new pope, and the selection of Cardinal Joseph Ratzinger as Pope Benedict XVI. In the case of “katrina”, that single keyword was enough to produce hundreds of targeted search results for both Hurricane Katrina and its aftermath. Keyword phrases were not tinkered with to promote or demote particular types of search results (i.e. those for blogs or nytimes.com); they were only adjusted for the relevence of overall results.
The searches were all done on January 27, 2006 with Google’s main search engine, not their news specific search.
Since the spirit of the bet deals with the influence of traditional media versus that of citizen-produced media, I tracked the top traditional media (labeled just “media” above) results and the top citizen media results in addition to blog and nytimes.com results. For the purposes of this exercise, relevent results were those that linked to pages that an interested reader would use as a source of information about a news story. For citizen media, this meant pages on Wikipedia, Flickr (in some cases), weblogs, message boards, wikis, etc. were fair game. For traditional media, this meant articles, special news packages, photo essays, videos, etc.
In differentiating between “media” & citizen media and also between relevent and non-relevent results, in only one instance did this matter. Harriet Miers’s Blog!!!, a fictional satire written as if the author were Harriet Miers, was the third result for this keyword phrase, but since the blog was not a informational resource, I excluded it. In all other cases, it was pretty clear-cut.
Chris Anderson has one of the best descriptions I’ve read of collective knowledge systems like Google, Wikipedia, and blogs: they’re probabilistic systems “which sacrifice perfection at the microscale for optimization at the macroscale”.
Update: I fucked up on this post and you should reread it if you’ve read it before. After reading this post by Niall Kennedy, I checked and found that I have mentioned or linked to the site for Freakonomics 5 times (1 2 3 4 5), not 13. The other 8 times, I either linked to a post on the Freakonomics blog that was unrelated to the book, had the entry tagged with “freakonomics” (tags are not yet exposed on my site and can’t be crawled by search engines), or I used the word “Freakonomists”, not “Freakonomics”. Bottom line: the NY Times listing is still incorrect, Google and Yahoo picked up all the posts where I actually mentioned “Freakonomics” in the text of the post but missed the 2 links to freakonomics.com, Google Blog Search got 2/3 (& missed the 2 links), Technorati got 1/3 (& missed the 2 links), and IceRocket, Yahoo Blog Search, BlogPulse, & Bloglines whiffed entirely. Steven Levitt would be very disappointed in my statistical fact-checking skills right now. :(
I wish Niall had emailed me about this instead of posting it on his site, but I guess that’s how weblogs work, airing dirty laundry instead of trying to get it clean. Fair enough…I’ve publicly complained about the company he works for (Technorati) instead of emailing someone at the company about my concerns, so maybe he had a right to hit back. Perhaps a little juvenile on both our parts, I’d say. (Oh, and I turned off the MT search thing that Niall used to check my work. I’m not upset he used it, but I’m irritated that it seems to be on by default in MT…I never intended for that search interface to be public.)
———
The NY Times recently released their list of the most blogged about books of 2005. Their methodology in compiling the list:
This list links to a selection of Web posts that discuss some of the books most frequently mentioned by bloggers in 2005. The books were selected by conducting an automated survey of 5,000 of the most-trafficked blogs.
Unsurprisingly, the top spot on the list went to Freakonomics. I remembered mentioning the book several times on my site (including this interview with author Steven Levitt around the release of the book), so I checked out the citations they had listed for it. According to the Times, Freakonomics was cited by 125 blogs, but not once by kottke.org, a site that by any measure is one of the most-visited blogs out there.[1] A quick search in my installation of Movable Type yielded 13 5 mentions of the book on kottke.org in the last 9 months. I had also mentioned Blink, Harry Potter, Getting Things Done, Collapse, The Wisdom of Crowds, The Singularity is Near, and State of Fear, all of which appear in the top 20 of the Times’ list and none of which are cited by the Times as having been mentioned on kottke.org in 2005.
I chalked this up to a simple error of omission, but then I started checking around some more. Google’s main index returned only three distinct mentions of Freakonomics on kottke.org. Google Blog Search returned two results. Yahoo: 3 results (0 results on Yahoo’s blog search). Technorati only found one result (I’m not surprised). Many of the blog search services don’t even let you search by site, so IceRocket, BlogPulse, and Bloglines were of no help. (See above for corrections.) I don’t know where the Times got their book statistics from, but it was probably from one of these sites (or a similar service).
Granted this is just one weblog[2], which I only checked into because I’m the author, but it’s not like kottke.org is hard to find or crawl. The markup is pretty good [3], fairly semantic, and hasn’t changed too much for the past two years. The subject in question is not off-topic…I post about books all the time. And it’s one of the more visible weblogs out there…lots of links in to the front page and specific posts and a Google PR of 8. So, my point here is not “how dare the Times ignore my popular and important site!!!” but is that the continuing overall suckiness of searching blogs is kind of amazing and embarrassing given the seemingly monumental resources being applied to the task. It’s forgivable that the Times would not have it exactly right (especially if they’re doing the crawling themselves), but when companies like Technorati and Google are setting themselves up as authorities on how large the blogosphere is, what books and movies people are reading/watching, and what the hot topics online are but can’t properly catalogue the most obvious information out there, you’ve got to wonder a) how good their data really is, and b) if what they are telling us is actually true.
[1] Full disclosure: I am the author of kottke.org.
[2] This is an important point…these observations are obviously a starting point for more research about this. But this one hole is pretty gaping and fits well with what I’ve observed over the past several months trying to find information on blogs using search engines.
[3] I say only pretty good because it’s not validating right now because of entity and illegal character errors, which I obviously need to wrestle with MT to correct at some point. But the underlying markup is solid.
Google search for “i don’t read kottke” versus a search for “i don’t read boing boing”. Nottke** wins, 39 to 37! Sit on it, Cory!
** Nottke = not Kottke, coinage by John Gruber.
Amazon/Alexa is opening up their index, letting people access the raw data, processing power, and even the crawlers. What a huge idea. (via bb)
AIGA Voice has an interview with Peter Morville about his new book, Ambient Findability. A question from the interview that everyone responsible for a web site should be asking themselves (emphasis mine): “Can [people] find your content, products and services despite your website?” Love that.
I got an email this morning from a kottke.org reader, Meghann Marco. She’s an author and struggling to get her book out into the hands of people who might be interested in reading it. To that end, she asked her publisher, Simon & Schuster, to put her book up on Google Print so it could be found, and they refused. Now they’re suing Google over Google Print, claiming copyright infringement. Meghann is not too happy with this development:
Kinda sucks for me, because not that many people know about my book and this might help them find out about it. I fail to see what the harm is in Google indexing a book and helping people find it. Anyone can read my book for free by going to the library anyway.
In case you guys haven’t noticed, books don’t have marketing like TV and Movies do. There are no commercials for books, this website isn’t produced by my publisher. Books are driven by word of mouth. A book that doesn’t get good word of mouth will fail and go out of print.
Personally, I hope that won’t happen to my book, but there is a chance that it will. I think the majority of authors would benefit from something like Google Print.
She has also sent a letter of support to Google which includes this great anecdote:
Someone asked me recently, “Meghann, how can you say you don’t mind people reading parts of your book for free? What if someone xeroxed your book and was handing it out for free on street corners?”
I replied, “Well, it seems to be working for Jesus.”
And here’s an excerpt of the email that Meghann sent me (edited very slightly):
I’m a book author. My publisher is suing Google Print and that bothers me. I’d asked for my book to be included, because gosh it’s so hard to get people to read a book.
Getting people to read a book is like putting a cat in a box. Especially for someone like me, who was an intern when she got her book deal. It’s not like I have money for groceries, let alone a publicist.
I feel like I’m yelling and no one is listening. Being an author can really suck sometimes. For all I know speaking up is going to get me blacklisted and no one will ever want to publish another one of my books again. I hope not though.
[My book is] called ‘Field Guide to the Apocalypse’ It’s very funny and doesn’t suck. I worked really hard on it. It would be nice if people read it before it went out of print.
As Tim O’Reilly, Eric Schmidt, and Google have argued, I think these lawsuits against Google are a stupid (and legally untenable) move on the part of the publishing industry. I know a fair number of kottke.org readers have published books…what’s your take on the situation? Does Google Print (as well as Amazon “Search Inside the Book” feature) hurt or help you as an author? Do you want your publishing company suing Google on your behalf?
Google finally launches a blog search service. The default search is by relevance, which I’m not sure is correct, and it’s pretty bare bones so far, but I’m sure that many other people will be saying so long, Technorati. Also available in Blogger flavor. (via waxy)
Use the Technorati Accelerator to “search on any URL and get the same response you would have to wait thirty seconds for on their site”. Zing!
In reaction to some ads of questionable value being placed on some of O’Reilly’s sites (response from Tim O’Reilly), Greg Yardley has written a thoughtful piece on selling PageRank called I am not responsible for making Google better:
Google, Yahoo, Microsoft and the other big search engine companies aren’t public utilities - they’re money-making, for-profit enterprises. It’s time to stop thinking of search engines as a common resource to be nurtured, and start thinking of them as just another business to compete with or cooperate with as best suits your individual needs.
I love the idea that after more than 10 years of serious corporate interest in the Web that it’s still up to all of us and our individual decisions. The search engines in particular are based on our collective action; they watch and record the trails left as we scatter the Web with our thoughts, commerce, conversations, and connections.
Me? I tend to think I need Google to be as good a search engine as it can be and if I can help in some small way, I’m going to. As corny as it sounds, I tend to think of the sites I frequent as my neighborhood. If the barista at Starbucks is sick for a day, I’m not going to jump behind the counter and start making lattes, but if there’s a bit of litter on the stoop of the restaurant on the corner, I might stop to pick it up. Or if I see some punk slipping a candy bar into his pocket at the deli, I may alert the owner because, well, why should I be paying for that guy’s free candy bar every time I stop in for a soda?
Sure those small actions help those particular businesses, but they also benefit the neighborhood as a whole and, more importantly, the neighborhood residents. If I were the owner of a business like O’Reilly Media, I’d be concerned about making Google or Yahoo less useful because that would make it harder for my employees and customers to find what they’re looking for (including, perhaps, O’Reilly products and services). As Greg said, the Web is still largely what we make of it, so why not make it a good Web?
Before we get going, here are some alternate titles for this post, just to give you an idea of what I’m trying to get at before I actually, you know, get at it:
Now that your hyperbole meter has pegged a few times, hopefully the rest of this will seem tame in comparison. (And apologies for the length…I got rolling and, oops, 2500 words. But many of them are small so…)
Way back in October 2004, this idea of how the Web as a platform might play out popped into my head, and I’ve been trying to motivate myself into writing it down ever since. Two recent events, Yahoo’s purchase of Konfabulator and Google’s release of a new beta version of Google Desktop have finally spurred me into action. But back to October. At the Web 2.0 conference, Stewart pulled me aside and said something like, “I think I know what Google is doing with Google Browser.” From a subsequent post on his site:
I’ve had this post about Adam Bosworth, Alchemy and the Google browser sitting around for months now and it is driving me crazy, because I want all the credit for guessing this before it happens. So, for the record, if Google is making a browser, and if it is going to be successful, it will be because there is a sophisticated local caching framework included, and Google will provide the reference apps (replying to emails on Gmail or posting messages to Google groups while on the plane).
At the time, Adam Bosworth had been recently hired by Google for purposes unknown. In a blog post several months before he was hired, Bosworth mused about a “new browser”:
In this entry, I’m going to discuss how I imagine a mobilized or web services browser handles changes and service requests when it isn’t connected. This is really where the peddle hits the metal. If you just read data and never ever alter it or invoke related services (such as approving an expense report or booking a restaurant) then perhaps you might not need a new browser. Perhaps just caching pages offline would be sufficient if one added some metadata about what to cache. Jean Paoli has pointed out to me that this would be even more likely if rather than authoring your site using HTML, you authored it as XML “pages” laid out by the included XSLT stylesheets used to render it because then you could even use the browser to sort/filter the information offline. A very long time ago when I was still at Microsoft (1997) we built such a demo using XSLT and tricky use of Javascript to let the user do local client side sorting and filtering. But if you start actually trying to update trip reports, approve requests, reserve rooms, buy stocks, and so on, then you have Forms of some sort, running offline, at least some of the time, and code has to handle the inputs to the “Forms” and you have to think through how they are handled.
A couple weeks later, Google introduced the first iteration of their Desktop Search. To me, the least interesting thing about GDS was the search mechanism. Google finally had an application that installed on the desktop and, even better, it was a little Web server that could insert data from your local machine into pages you were browsing on google.com. That was a new experience: using a plain old Web browser to run applications locally and on the Web at the same time.
So this is my best guess as to how an “operating system” based on the Web (which I will refer to as “WebOS”) will work. There are three main parts to the system:
That’s it. Aside from the browser and the Web server, applications will be written for the WebOS and won’t be specific to Windows, OS X, or Linux. This is also completely feasible, I think, for organizations like Google, Yahoo, Apple, Microsoft, or the Mozilla Foundation to make happen (more on this below).
Compared to “standalone” Web apps and desktop apps, applications developed for this hypothetical platform have some powerful advantages. Because they run in a Web browser, these applications are cross platform (assuming that whoever develops such a system develops the local Web server part of it for Windows, OS X, Linux, your mobile phone, etc.), just like Web apps such as Gmail, Basecamp, and Salesforce.com. You don’t need to be on a specific machine with a specific OS…you just need a browser + local Web server to access your favorite data and apps.
For application developers, the main advantage is that instead of writing two or more programs for multiple platforms (one for the Web, one for Windows, etc.), they can write one app that will run on any machine with the WebOS using the same code base. Bloglines and NetNewsWire both do about the same thing and have radically different codebases (Bloglines uses HTML/JavaScript + some sort of backend programming/scripting language while NNW is a Cocoa app only for OS X), but a version of Bloglines developed for the above platform could utilize a single codebase.
You also get the advantages of locally run applications. You can use them when you’re not connected to the Internet. There could be an icon in the Dock that fires up Gmail in your favorite browser. For applications using larger files like images, video, and audio, those files could be stored and manipulated locally instead of waiting for transfer over the Internet.
There are also disadvantages to WebOS applications, not the least of which[1] is that HTTP+JavaScript+XHTML+CSS+Flash is not as robust in providing functionality and user interaction as true desktop applications written in Cocoa or Visual Basic. But as Paul Graham points out, Web applications may be good enough[2]:
One thing that might deter you from writing Web-based applications is the lameness of Web pages as a UI. That is a problem, I admit. There were a few things we would have really liked to add to HTML and HTTP. What matters, though, is that Web pages are just good enough.
Web pages weren’t designed to be a UI for applications, but they’re just good enough. And for a significant number of users, software that you can use from any browser will be enough of a win in itself to outweigh any awkwardness in the UI. Maybe you can’t write the best-looking spreadsheet using HTML, but you can write a spreadsheet that several people can use simultaneously from different locations without special client software, or that can incorporate live data feeds, or that can page you when certain conditions are triggered. More importantly, you can write new kinds of applications that don’t even have names yet.
And how about these new kinds of applications? Here’s how I would envision a few apps working on the WebOS:
I’m looking at the rest of the most commonly used apps on my Powerbook and there’s not too many of them that absolutely need to be standalone desktop applications. Text editor, IM[3], Word, Excel, FTP, iCal, address book…I could imagine versions of these running in a browser.
So who’s going to build these WebOS applications? Hopefully anyone with XHTML/JavaScript/CSS skills, but that depends on how open the platform is. And that depends on whose platform it is. Right now, there are five organizations who are or could be moving in this direction:
So yeah, that’s the idea of the WebOS (as I see it developing) in a gigantic nutshell. The reality of it will probably be a lot messier and take a lot longer than most would like. If someone ends up doing it, it will probably not be as open as it could be and there will likely be competing Web platforms just as there are now competing search engines, portals, widget applications (Konfabulator, Dashboard, Google Desktop Sidebar), etc., but hopefully not. There’s lots more to discuss, but I’m going to stop here before this post gets even more ridiculously long. My thanks if you even made this far.
[1] Actually, the biggest potential problems with all this are the massive security concerns (a Web browser that has access to data on your local hard drive?!!!??) and managing user expectations (desktop/web app hybrids will likely be very confusing for a lot of users). Significant worries to be sure, but I believe the advantages will motivate the folks developing the platform and the applications to work through these concerns.
[2] For more discussion of Web applications, check out Adam Rifkin’s post on Weblications.
[3] Rumor has it that Google is releasing an IM client soon (more here). I’ll be pretty surprised if it’s not significantly Web-based. As Hotmail proved for email, there’s no reason that IM has to happen in a desktop app (although the alerting is problematic).
[4] Maybe Google thinks they can’t compete with Apple’s current offerings (Spotlight, Dashboard, Safari, iPhoto) on their own platform, but that’s not a good way of thinking about it. Support as many people as you can on as many different architectures as you can, that’s the advantage of a Web-based OS. Microsoft certainly hasn’t thought of Apple as a serious competitor in the OS space for a long time…until Web applications started coming of age recently, Microsoft’s sole competitor has been Microsoft.
Google introduces a new (beta) version of Google Desktop featuring Sidebar, their answer to Dashboard and Konfabulator. Here’s more on Google’s move from the Times, which also includes speculation on the possible release of an IM client this week.
Odd size comparison of Yahoo and Google indices. I think their assumption (that a “series of random searches to both search engines should return more than twice as many results from Yahoo! than Google”) is pretty flawed. The number of returned results could vary because of the sites’ different optimizations for dictionary words, for searches with small result sets, and differences in how their search algorithms include or exclude relevant results. Put it this way: if I’m looking for a frying pan in my apartment, I’m gonna refine my search to the kitchen and not worry about the rest of the house, no matter how large it is. (via /.)
David Filo and Jerry Yang are organizing the entire WWW into a hierarchical category system. They’ve named their site “Yahoo”.
As We May Think by Vannevar Bush. This influential essay that introduces Bush’s Memex concept was published 60 years ago this month.
Nokia.com comes up first in a Google search for “motorola mobile phones”. I suspect it’s because Motorola’s site isn’t optimized for Google (lots of Flash, little text) and a difference in usage: it’s “cell phones” in the US versus “mobile phones” in Europe (where Nokia is from).
Google now offering customized home page, with weather, stocks, movies, etc..
Stay Connected