Connect The Dots: A Human-Based TSP Solver

Today was a good day because I printed my thesis for examination! The integration of non-contiguous colour pages was a total pain in the ass (x3 copies of the dissertation!). Not long now before it gets sent out and assessed (torn apart?) by my peers. Crazy times...

I managed to find some time to get stuck into the design for the human-based TSP solver project (my rails project is called solver for now). The first iteration of the project will be involve a gimmick/"single player game" user experience. The motivation for the user will be to maximise a user ranking score whilst being initially drawn to the gimmick of the application. Follow-up iterations will consider some of the other proposed user experiences (such as real-time competition) as well as those not yet considered (thought up).

The so-called game involves unlimited repetitions of "connecting the dots" on small (5-10 nodes) user problems. Each user problem represents a sub-portion of a broader TSP, and the straight line through the nodes (or sub-tour) will contribute towards a holistic "collectively intelligent" solution. The specifics of the ranking equation have not been tied down as of yet, although will likely involve an aggregation of contributions across problem instances in the context of the globally optimal solution (magnitude/quality trade-off with regard to effort).

We are having a research meeting next week and I elected to present. I'm going to discuss this problem and tender suggestions for contribution aggregation schemes as well as presentation methods from my research group (complex intelligent systems lab).

The data model (schema) is simple enough. Tables for problems, problem parts, problem instances, user contributions and users. The accumulation of contributions is compressed at this time. My brain is full of nice batch update approaches using message queues and I'm finding it hard to suppress my inner architect in order to "just hack". Ideally, I'd like to store distinct contributions, although the compression approach is the way forward for now.

The screen mock-ups (pen and paper) are rudimentary although block out the core functionality. The principle screen is the presentation of the sub-problem toward capturing user contributions. The current problem instance is summarised as well as the scope of holistic contributions received for the instance thus far (users, contributions, scores, human cycles? etc.). Contributions may be captured from drive-by users (anonymous) or by those users logged in (like reddit and karma). Additional principle screens include the holistic summary of the problem instance, the user leaderboard, past problem instances with outcomes, and user details that also summaries their contributions to date (all public and RESTful).

I got my hands dirty, scaffolding much of the application, seeding the database, and reading up on rails and the new 2.0 features in particular. Firstly, most tutorials out there on rails suck, too much hello-world CRUD, not enough departure into complexity. An overview of basic rails gotchas from Shash7 was useful, as was a two-part tutorial on rails 2.0 features called Rolling with rails 2.0 (part1 and part2). I took some time and came to grips with the naming conventions so that everything plays nicely together. I climbed into ActiveRecord, fixed some plural problems, came to terms with associations and foreign keys in the model (good solid example), and investigated best practices regarding seeding the database.

I'm enjoying rails at the moment, it reminds me a lot of hacking early releases struts although with a lot more scripts and hard coded conventions. "Learning while doing" is fine while primitively architecting, although I expect there to be a crunch when I need to actually put some functionality in the application (expect lots of ruby based rants in the future).

I intend to invest a lot more time into the the user experience and psychology/presentation concerns of the subproblems after the first iteration. This last comment feels like one of those things for which time is never alloted, lucky I captured the thoughts and comments in previous posts (accountability!?). I may push the application out to the web (hosting?) late next week to start getting feedback from a private user pool.

Considering User Experience in Crowdsourced Problem Solving

I received a lot of positive email feedback regarding yesterdays idea of using collective intelligence to address NP-Complete problems. Thanks a lot!

Firstly the problem and solution are unashamedly inspired by Luis von Ahn's research. Specifically, his notions of Games with a Purpose (pdf) and the games and services he has released along these lines: reCAPTCHA, The ESP Game, Peekaboom, and Phetch. Click them, try them, love them. Awesome ideas! This is not a big secret, as I have spammed my interest in this research many times on this blog.

What I clearly lacked from the project proposal was any consideration of user experience. This was reflected in much of my feedback (cheers): Why would the user come back? What is in it for the user? How does it help the user? How are the puzzles compelling? What motivates the user to solve your problem in a piecewise manner? So on.. This area needs work, superseding all technical work, for now.

Some user experience models that come to mind are:

• Gimmick: This was the original notion. I was caught up in the emergent effect and considered user participation based on excitement over the gimmick, and at a stretch as a single player game. In my mind I relate the gimmick model with interesting AJAX sites (one hit wonders), specifically collaborative drawing sites like The Broth, beautyandchaos, draw ball, and may others.
  
• Educational: The site could be geared towards education, demonstrating how local problem solving contributes to holistic problem solving, emergence in human-based algorithms, or specifically education regarding the TSP itself.
  
• Single Player Game: Beyond the gimmick of the project, I thought the accumulation of scores for a user may turn the "quality" or number of contributions into a competition. A hazard with this approach is that scoring could bias the resultant holistic solutions. For example, rewarding local Nearest Neighbor solutions would merely approximate the NN solution at aggregation time. A leader board based approach would be compelling enough for me to play (I think), as long as it looked cool (back to gimmick).
  
• Two Player Game: This is a more interesting approach where users are paired randomly and compete with each other with random sub-graphs. The goal may be to match solutions, compete based on time (exploit reflexive structure identification), compete against the same objective (minimising) or compete with conflicting objectives (one minimise and one maximise).
  

All interfaces could be provided and iterated to find the most compelling user experience (at least during early testing). It is clear to me that contributions must capture the powerful human capacity for pattern recognition in the identification of underlying structure. Specifically, this must be discriminated from automated nearest neighbour solutions which can be automated. I think of the human contributions as an innate mixture of heuristic and acquired knowledge (if you are familiar with ACO) that when aggregated over many local sub-problems provides an approximately optimal solution. This is the primary question to be answered by the application.

Some additional insights (thanks Craig and Cam):

• Consider an integration with Google maps, solving real TSP's is more compelling than arbitrary points on the screen. A brilliant suggestion I overlooked on the first pass.
• Consider a facebook application if for the size of the userbase if nothing else. I had a similar thought (blog widget). A winner, as long as compelling variations of the game + meaningful aggregates can be defined.
• Consider the use of the sub-tours/sub-graphs as reCAPTCHA's. I think in their native form, this may not be as straight forward because there must be a single solution that is easy for the user (NN? Globally optimal?) and which is hard for the computer to locate (not the first, but defiantly the second).
  

This last point got me thinking about transformations of the problem. My first inclination was to show random sub-problems with random rotations to capture invariant structures, although there are more powerful polynomial time transformations of the problem into related problems that could be exploited. You could also visually display local information in many different ways (terrain, etc). I need to think about this, specifically with regard to visual transformations which can exploit human pattern recognition useful to the global problem.

I didn't get into any tech today other than some brief playing with the tools and the database. I invested available time today on mocking out the data model (on paper), and thinking about user experience and collaborative games (time well invested). I'm staying with the user for now.

Crowdsourcing NP-Complete Problems

I'm shifting gears and have decided to start pushing ideas out as web sites rather than technical prototypes. The reason for the gear change, is because I want users and I want to try broader experiments. I've got some ideas regarding drive-by (casual) contributions and patterns of collective intelligence I want to try out that require lots of varied user behaviour. Specifically, I intend on building a webapp that captures casual contributions made by users towards playing a game or solving a simple puzzle, the aggregation of which will contribute toward solving an NP-complete problem.

My intention is to capture my progress, thoughts, and problems on this blog while realising this modest project. After deciding to commit to the idea, I spent today mocking some screens, coding a proof of concept prototype for the Travelling Salesman Problem (TSP), and setting up a development environment.

The Application
The early vision for the site focuses on the user contributions as a game or puzzle, and the aggregation of all contributions received to date as a holistic solution. User contributions may be made anonymously, although signing-up permits the capture of user statistics, puzzle scores, and a leader board. The minimal interface focuses on the puzzle solving aspect, although provides some visualisation of the state of the holistic solution such as number of contributions, users, and holistic scoring. The holistic solution may be visualised, along with a graph of community performance over time. Problems would rotate after converge (lack of improvement) or a time limit, and a past-solutions repository will be accessible with solution visualisation. Tomorrow I want to find some free software to capture screen mock-ups or just hack out the templates.

The Problem
I chose the TSP first because I have worked with the general problem a lot in the past. Specifically with regard to a multiple user interaction model across a P2P network (see IIDLE), and with regard to a stardardised problem solving framework (see my OAT project). Simple symmetric Euclidean TSP's can be addressed by the aggregation of multiple sub-graph solutions. The sub-problems or "puzzle" to be addressed by users will be to connect a set of points to form a sub-tour (connect the dots). My assumption is that the human pattern recognition will be at least as good as a Greedy Nearest Neighbour (NN) solution, and hopefully with tweaking of the interface and aggregation function, better than NN. My technical prototype verified that the greedy aggregation of multiple sub-tours demonstrates that a NN-like solution can be achieved easily as long as enough contributions are received. I need to formalise notions of "enough" and "convergence" for problem rotation.

The Technology
I have decided to build the application using Ruby on Rails, based on my previous assessment of the technology landscape. I have played around with rails before in late 2006 on my mac, although that was some time ago, and I'm developing this site on a PC (for now). I hit the rails site, got running with InstantRails and RadRails using Aptana. I then did a read a quick refresher with the ONLamp articles "rolling with rails revised" (parts one and two). The tools seem nice and work together well. At this stage it feels like the old PHP and Tomcat/Struts days.

I only had the notion for the project at lunch time, so it's all still fresh and new. I expect to bash out some page and database templates tomorrow and think through the user experience. I'm going to try and put-off getting into the technical aspects of problem definition/integration/visualisation for as long as I can because that's the fun stuff and I want to get on top of the mundane (user-centric) things first.

Funnelling Cognitive Surplus With and Without Your Permission

A popular post at the moment is Clay Shirky's written version of his talk at the Web2.0 conference titled: Gin, Television, and Social Surplus. The thesis of the post is along the lines of television as a system for addressing the cognitive surplus from average peoples scheduled spare time. Before TV, during the industrial revolution it was gin (drink your self stupid), and Shirky relates such wasted human cycles to the thought-effort required to create a Wikipedia sized project. This point is used to push the notion that there is currently a shift in media from passive consumers (TV and newspapers) to active consumers, producers. and sharers.

These notions hook in to the classical architecture of participation (O'Reilly 2003, 2004, discussion 2004) where systems are created to allow users to be produces and consumers and more importantly define what and how they produce and consume within the systems loosely defined constraints. Shirky highlights that there is a growing expectation for media to be deployed into this model, and promotes the idealised notion of donating a few percent of annual human cycles online.

Does tagging a popular blog post in del.icio.us have a higher or lower cognitive load than discussing an episode of Lost at work? Are the aggregate effects of those tags more valuable to society than the aggregated impact of such conversations? Different domains, so naturally the methods (computational, social) vary for aggregation and information extraction.

Something I have noticed personally (myself and friends) is that there are plenty more ways to soak up you cognitive surplus online, even in architectures of participation. So-called tracking the technology market itself and reading popular tech news and commentary are still reasonably passive activities. You may contribute clicks, comments, and maybe opinions (like this post), which I presume sometimes is better than watching a TV show and chatting about it with others. My point is that on the spectrum of contributions, these are really low, and even in aggregate are relatively low quality (feed into page rank or something similar). People that want to produce, will produce and some of those things will be as generally useful as contributions to Wikipedia articles. Those that want to passively consume (lurk) will continue to do so. The shift is that automated computation may allow organisations to exploit such passive gestures for their and maybe our (as users) utility.

This last point highlights that although TV may be dissipating cognitive or attention surplus, it may not be dissipating contribution surplus. Regarding this discussion of surplus, I would like to see something done with all the computational surplus. Here, right now, while you are reading this post, what is your CPU doing? 99% consumed with an idle process? Awesome.

Thoughts like that inspire me to bash out a distributed in-browser data processing system (firefox plug-in? AJAX?) that consumes these cycles toward helping the user. A massive probabilistic model of a users interests would be useful, even if it's crap most of the time or in most domains.

Regarding architectures of participation, a related notion at the time the term popularised was Dan Bricklin's napster inspired three models of contribution: Expert, Mechanical, and Volunteer, from his 2001 post The Cornucopia of the Commons: How to get volunteer labor. This was rephrased by O'Reilly as Expert, Volunteer, and Shared (for example). Beyond public by default in the shared model, one may consider the broader notion of the exploitation of contributions through selfish user action (as was discussed in that post). Instead of direct public effects like public photo or link streams, the system solves hard problems, the products of which are publicly consumable (yes, the digitized books example again).

Such efficient and automated exploitation of passive and active contributions may herald a new age of participation (whether users choose to participate or not), although more importantly will likely demarcate the next generation of successful corporations like the factories and media conglomerates before them.

Hierarchical Temporal Memory and Thoughts on Conversations

Yesterday I listened to an ITConversations podcast by Jeff Hawkins at ETech last year titled Why Can't a Computer Be More Like a Brain? I have been following Jeff, his HTM technology, and his new company Numenta since they released their development environment early last year. Anyway, listening to Jeff's marketing spiel again got me thinking about a broader application of abstracted brain models to the web.

In his book titled On Intelligence, Hawkins outlined his theory for human intelligence called the Memory Prediction Framework. The approach is focused a functional abstraction of the neocortex (part of the brain that does lots of useful stuff). The premise is that the hierarchical architecture for this region of the brain relates to the interesting information processing that it does. The model focuses on the architecture, the spatiotemporal abstraction (feature selection) up the hierarchy, and the memory-based prediction afforded by such acquired abstractions.

Numenta's Hierarchical Temporal Memory is a machine learning realisation of the theory. It's kind of like an Artificial Neural Network (neurons, layers, and propagation) although uses a varied inspiration, nomenclature, and training algorithm (Bayesian belief networks). They provide a non-commercial platform called NuPIC to play around and try out examples. Alternatively, there is an open source realisation based on the Hawkins book called Neocortex. As a tech, HTM is an interesting and popularised variation on the theme (ANN broadly conceived), although I suspect hardly a breakthrough in either machine learning or neuroscience.

The specific train of thoughts I had involved relating the decentralised information processing of the brain to the web. For example:

• Content Perspective: Web content are the neurons connected implicitly (pointers) and implcitly (contextually or semantically) to other content. Users provide the active information processing and learning algorithms for the otherwise passive and unintelligent system.
  
• Users Perspective: Users are the neurons connected to each other through their local interfaces such as popular sites and web applications. Each has wide variations to their connections and therefore reach to other users. Content manifests artifacts of communication (links) and therefore the information (memory).
  

Not particularly insightful, but the common theme resonates with notions of web collective intelligence.

A final thought is in relating the hierarchical nature of information processing to the transformations in user interaction space (geography, application, social, interests, profile clusters, etc). Each involves feature selection as a means for defining and raising the level of abstraction. I believe there is a pattern to those successful/intelligent web applications that exploit easy (localised) connections across multiple user interaction spaces simultaneously (for example social and profile cluster or geography and application). The interesting question is can you emerge (learn) such levels of user interaction, or do they have to be engineered (system) or contrived (user)?

Patterning Interestingness: Artefact, Context, and Relevance

Yesterday, I posted about my document interestingness side project and highlighted some related research and software libraries. The domain was constrained to academic research, addressing the specific question: how interesting is a given paper in the context of my research? I received a fair bit of personal correspondence, so I thought I would capture and integrate some of the ideas and related abstractions.

Firstly, it is important to highlight that the post didn't give anything of value away. As we know, ideas are worth nothing except for their inspirational quality, and only success is worth ripping off. The approach was a crude version of a 30+ year old method, and I can only assume that specific application has been addressed in broader academic software (perhaps bibliographic management software). Success with this approach will be in its effective execution to a specific (real) user pain point, with domain-specific special source (technique customisation).

Anyway, I had some interest regarding the academic application, some interesting elaborations for news and feed filtering, and some suggestions for related domains like job applications. The following provides a summary of motivating questions (problems):

• Academic:
• How interesting is a given paper?
• Should I read this paper?
  
• From the papers referenced, what are the relative relevance of each?
• Who may be most interested in my paper?
  
• How does my paper related to my field? (in what ways...)
• What fields are most likely interested in my paper?
• How can I optimize my position (for review/acceptance/examination)?
• Who will receive my work favourably?
  
• How relevant is my work for submission to a given journal/conference?
• Will they be interested in my paper?
  
• Blogs/Feeds:
• How interesting is a post in the context of what I read?
• Should I read this post?
• How interesting is this feed in the context of the feeds I read?
• Will I read posts from this feed?
• What is the relevance of outbound links?
• Should I click a link?
  
• Resume:
• How interesting is a given applicant?
• Should we interview this applicant?
• How interesting is a given job?
• Should I apply for a given job?
  

I have phrased may of the suggestions in terms of questions regarding interestingness/relevance as well as domain specific decisions. The problems cover a range of motifs including search, filtering, and recommendation which may be abstracted into a general pattern. This general pattern involves three core concerns:

1. An artefact to assess which may be as small as a query or as large as a set of document.
2. A context to assess against which may be as small as a document or as large as the Internet.
3. A the relevance between the two that translates into interestingness for the user (some kind of investment, such as time to read).
   

What was interesting for me from this experience was the flip in concerns. I was focused with the personalisation of the process: the assessment of foreign artefacts against a user model. The suggestions were free of that constraint, highlighting concerns such as the modelling of others toward addressing a potentially unrelated user concern. For example, in the academic setting: the modelling of the interests of high-reputation individuals or organisations toward a users economic payoff (attention, acceptance, etc.).

This trend may be extrapolated. For example, the equivalent in the blogsphere would be the modeling of the readers of your blog based on their own blogs (and related content) and suggesting articles of yours for them to read. A crazy but intriguing notion of pushing per-user recommendation onto content producers: show me (not my group) your best goods. Alternatively, you use such information to campaign other bloggers into becoming readers/subscribers/advocates of your own work product.

The abstraction has a clear relationship with internet search (query as the artefact against the internet corpus) as well as context-sensitive advertisement (mapping posts to an ad corpus or the other way around). In the latter case and many of the cases listed above, classical problems of keyword spamming are not a concern, as there is no payoff to do it. That does not mean the system cannot be gamed, it means the goal posts are moved. For example, if a system was devised based on the premise of the automated relation of your content with other people's content (read posts related to what you write about), you may be able to achieve a broader readership (through recommendations) by having generic content (broader but shallow keyword frequency). Similarly, in the academic domain, a conference may promote a general interest in their corpus to promote higher submission/attendance numbers.

Finally, this experience has confirmed to me that short development (hack) cycles followed by broader disclosure is a powerful force. The old me would have been satisfied with progress captured in a private word document, and moved on to the next self-delimited cycle or idea. Cheers for the email and IM!

Side Project: Automating Document Interestingness

For the last week and a half I have been hacking together a simple "document interestingness" application (while not making corrections my PhD dissertation). Basically, the user outlines a set of sources (directories of PDF's, web pages, blogs, etc), the application builds a probabilistic model of the users interests, which in turn is used to automatically assess the interestingness of artifacts. The application is currently restricted to an academic domain addressing the specific question: is this paper interesting with regard to my research? Primitive, sure, but it's just a proof of concept.

My approach was rudimentary, using context-insensitive whitespace separated symbols (words) extraction and modeling of extracted text. Interestingness involved a series of calculations involving word sets and some vector algebra with weightings. I tested it on myself (naturally) using my own technical report archive. Models were serialised to disk and I created some visualisations (tag clouds in HTML and corpus graphs in JUNG). Relevance measures were as expected across my test datasets (papers from my field, related fields and unrelated fields). It's a minimal solution with all command line/API user interactions with some Swing for visualisations (yes it's written in Java).

My methodology while working on the project was to get something working as quickly as possible (just hack), and I explicitly avoided research into my topic (information retrieval research). NLP is not my area, so I was surprised today when I finally did do a little research to discover that my first pass at the problem was quite reasonable.

The basic modeling approach may be referred to as a Term Count Model (word frequencies) and the general document interaction measures fit nicely into the so-called Vector Space Model and related tf-idf weighting and similar. Basically the approach involves modeling the word frequencies in a vector space and using simple algebra (distances, magnitudes, angles) to assess relevance of indexed documents.

Digging a little deeper reveals a discussion of such classical document retrieval approaches in the context of search and SOE on an interesting site called Information Retrieval Intelligence. The sites provides a detailed look at the term count model, the vector space model, a consideration of word densities in such models, and much more. I also came across a good paper on term weighting titled Term Weighting Approaches in Automatic Text Retrieval (1987). Finally, digging deeper still reveals Lucene, a Java library that does pretty much all of these things in what seems like a best practices manner (Apache products are typically solid).

Although my hacked application and it's rudimentary indexing and interestingness measures is minimal, it is functional. I'm not so sure I would have something functional if I had taken a deep research approach first. Specifically, I doubt a simple VSM approach with word weightings would have satisfied me and I doubt I would have considered a suitable base application (academic research paper relevance). Anyway, the next step for this side project will be to consider whether there is a market (web/desktop) for my chosen problem (or variations there of), or whether I should graft a different problem into my codebase (post filtering application?).

reCAPTCHA Makes Me Want To Work On Machine Vision Problems

One of the problems I perceived in my field (computational intelligence) was how the algorithms we played with related to real world problems. The majority of the "optimisation", "pattern recognition", and "classification" problems were sanitised versions of so-called real world problems, with known optimal solutions. Such problems are exactly what you need in the investigation and demonstration of experimental algorithms and systems, although what I felt was missing were motivating real world examples beyond lofty and loose relations (likely an artifact of my narrow perspective of the space).

At the moment, many sites you visit that require you to sign up provide a CAPTCHA, and more interestingly, they use the reCAPTCHA service. I have a high-frequency of signing up to services (I like to try things out) and each time I complete a CAPTCHA challenge, I feel like dropping whatever project I have on the go and getting stuck into computer vision research.

Some background. A CAPTCHA is a test to make sure you are a human and typically involves reading, interpreting, and typing visually messed up words. reCAPTCHA is a project that exploits such human computation toward solving a computer vision problem (complete digitisation of books) that is too difficult for todays (off the shelf?) optical character recognition softwares. A reCAPTCHA challenge involves the human-based transcription of two words, one that is known to the system and one that is not, and the human is not told which is which. The system-known word is used for the test, whereas the results for the second word are aggregated into a probabilistic model that eventually solves the second word (to a desired level of confidence).

The reCAPTCHA project provides widgets to embed the service in your website (in only 4 lines of code), lots of API's and plug-in's, and provides a variation for protecting email addresses listed on websites from automated harvesters called mailhide. The use of the service appears (qualitatively to me) pervasive among web applications, and is even used by Facebook and their massive user base. Harnessing that amount of human computation is inspiring (I want that!).

Frankly, I've never approach any computer vision problems other than toy datasets for demonstration purposes, although the process of automating the apparently effective "human approach" to the problem makes my mouth water with anticipation. I would inductive models layered from sub-symbolic to symbolic representations of the domain. Lots of training, lots of tweaking, lots of fun. What holds me back is the knowledge that the devil is in the detail. Serious math-heads have hit this problem for >30 years (78?), and as such I'm sure there are no quick fixes (remember, it's hard) and I'm sure that is a lifetime of methods (dogma) to consider (lots of book time). Nevertheless, this OCR problem is in everyones face, advertising the need for effective computational intelligence algorithms. The more AI-Hard problems that are out there in the open like this one, the more likely substantial (interesting) work is completed toward addressing it.

The human computation + blind trials + probabilistic modeling approach toward problem solving is a pattern promoted by Luis von Ahn whom I've posted about before. The pattern is cool, and is being followed in academic circles beyond von Ahn's work, for example both popularized and elaborated (distributed human computation). I really like the discrete contributions made by users as a service (security, game, etc.) that feeds into a broader model used for a different purpose (digitize books, image search corpus, etc.). It is a different way of thinking about collective intelligence, where the emergent service is disjoint from the primary discrete contributions, and the parts of the model are exposed although slyly represented. For example you do not see "come help us digitize books" or "label images" (boring), instead "solve this so you can do what you came to the site to do" (alternate intent).

I relate it to the explicit construction of your social graph on Facebook, firstly satisfying your own interest, and secondary satisfying the sites (once they figure out how to exploit it). Same deal with del.icio.us and flickr (user->site or discrete->emergent). Interestingly, AideRSS uses a reverse of this pattern where the user gets their selfish service from the aggregation of other's contributions, requiring the contributions before the aggregate service (emergent->user). The same can be said for Google's search.

Beyond the collective intelligence patterns, I want to do problem solving the von Ahn way. I have been racking by brain as to how to represent an optimisation or classification problem in such a way to exploit discrete human computation (nothing substantial so far). An obvious path is to exploit human pattern recognition and approximation capabilities (almost reflexive they are soon fast/good), although it is clear that the difficulty does not lie in the selection of a problem, but rather it's perception. For example citizen science and crowdsourced a football team are nice gimmicks (direct perception of emergent effects), but the ESP game is a popular game and reCAPTCHA is a popular security service (indirect perception/interest). Interactive EC is a good start, but one must think broader regarding inductive methods, and narrower regarding application. Any thoughts?

Intelligent Content Filtering with AideRSS and Some Extensions

I was searching for various intelligent filtering web applications and I came across the AideRSS service that has the tag line: "read what matters". I played around with it today and have come to the conclusion that I both like the model of the approach (makes sense) and the specific service itself (makes me feel warm inside).

The filtering model is constrained to a user-defined feed, in which each post is assessed based on it's "user engagement". Specifically, this is a social assessment mechanism they call PostRank that assesses each post in terms of the web-based contributions of others related to the post (so-called conversations), such as: comments, bookmarks, diggs, page rank, and so on. The tractability of the model is in its restriction to a single feed, and therefore the feed-relative scoring and normalisation that assigns each post a PostRank between 1-10 (I believe).

An early proposal of the per-feed model is described on Ilya Grigorik's (the project lead) blog titled: PostRank RSS Filtering. This model grew out his early assessment of the state of RSS filtering problems and solutions in a post titled: Reinventing RSS Readers. I like the simplicity of the model because it solves a different problem to the state of reader applications, specifically: what is interesting in a user specified feed (filtering), rather than: what is interesting in the blogsphere related to what I read (recommendation). The chosen model feels like an abstraction of the simpler problem of displaying popular or interesting posts on a blog (mentioned in the comments of the above post), although the use of broader social indicators is clever.

The core service is the application of the "user engagement" filter to popular high-traffic feeds (Slashdot, RWW, TechCrunch, Reddit). This is done by simply subscribing to a filtered version of a specified feed at the desired interest-level. Variations on this theme include the application of the filtering approach to aggregations of RSS feeds (filter the stream), and the integration into GoogleReader via a Greasemonkey-based plug-in.

Publishers (bloggers) can use the service to assess their content (see the state of NeverReadPassively), display a listing of interesting posts in a widget (I've included one on the right panel of this blog), and offer feeds of their work at various interesting levels (top, best, great, good, all). Finally, for developers the site provides an API, which some applications have exploited such as Lijit and integration into NewsGator RSS tools.

The site is built on ruby (back-end) and rails (front-end) and uses AWS (at least EC2). The service came out of private beta mid 2007 and since then has received support in the tech-press including: a review on Slashdot, a review on RWW, hype regarding the GoogleReader plug-in, and hype regarding funding. There are many approaches to RSS filtering and applications that claim to possess similar features (fav.or.it and thoof for example), although the standout features of this service in my eyes are as follows:

• It is free
• It does not require a user context (no lock-in!)
  
• It is simple
• It does what it claims to do (qualitatively well!)
  

I personally love the widget and as such some extensions I would propose include: (1) the ability to customise the existing "top posts" widget (colours and such, like amazon and Google allow), (2) the ability to embed the time-line graph as a widget, and (3) the ability to include a PostRank on each post in the same way the Google Toolbar gives a PageRank for sites and Technorati's use of per-post "authority".

Blog and Post-based widgets are good for publishers and would help promote this cool service virally. Regarding widget customisability, the present state of the widget does not display properly on my blog (cannot read the PostRank scores), so I had to right align the thing (ugly but readable).

The social-basis for the ranking algorithm means that there may be a delay between post time and bubble up time. Rather than a negative, I like this feature because like social news, old topics can resurge and break through to the front page (filtering levels). In this way, the chosen model allows each feed to be treated as it's own isolated social news site, presumably reassessed daily by the AideRSS systems. It would be interesting to see for example how the reddit front page compared to a filtered version of the raw-story submission scheme (whether reddit conversations correlate with broader conversations). Anyway, I think I may have a tinker with the API.

Specialised Human Information Processing: The Case of Scoble and the FeedReader

I can't remember what I was researching, but I came across a post on Tim Ferriss' blog (fourhourworkweek) with an interview with Scoble last May titled How Scoble Reads 622 RSS Feeds Each Morning. What I found interesting was the way Scoble described the cognition involved. It seemed obvious and familiar.

I cannot remember the specifics, although generally he described the process in terms of iterative scans of an item to extract information of interest. A variety of different filters were used such as title key words, image in the content, and author reputation. The iterations were used as discriminators for interesting pieces, where attention and subsequent knowledge extraction and acquisition are treated as a commodity (naturally, 622 feeds is a lot). The first pass is raw pattern recognition: does the piece look interesting? Importantly, discrimination and acquisition are intertwined, such that as the granularity of the discrimination process progresses so does the granularity of extracted and acquired information, such that if a piece is truly interesting it is extracted and acquired in its entirely (I presume).

I'm not sure about the level of recall available after such a process, although Scoble does point out the multiple objective outcomes. Specifically, in tracking memes (for example technology he is interested in), tracking the personal details of high-reputation individuals (CEO's), and ultimately tracking broader industry trends. Scoble points out that the application of the acquired information is seeming ad hoc and passive, remaining latent until called upon in a specific circumstance (How's the kid? What do you think of X?). Active application occurs at the time of acquisition, where he in turn produces a feed of human-discriminated pieces of interest.

There is nothing special about the Scoble case other than my use as it as a catalyst for considering the general case. I proceed with a similar algorithm daily and have done for many years, therefore I'm sure many perform such a ritual. Further, I use a specialised version of the case when intently researching a technical report or paper, applying levels of discriminators toward productively educating myself on a space and achieving my goal (bi-objective at a minimum). In this latter case, the algorithm provides a ruthless top-down information acquisition process which I have come to rely on.

The process as captured in systems occurs along similar lines, using various indicators (signals) and indicators in aggregate until sufficient information is acquired to make a decision. What I have not seen, is an effective equivalent for the human-first pass "does it look interesting?". This is something humans are very good at although poorly understand, specifically the process of holistic estimation/approximation toward snap decision making.

How would you automate such a first-pass to the process? I presume a vision-based templating system, accumulated over time from experience reading interesting and non-interesting articles (inductive). I also presume that it exists in scales between context specific and non-specific decision making (specific-feed to all/any feed). How is it different from the aggregate effect of per-indicator recrimination? I'm not sure, but my gut tells me that it is (scientific method, bah). It is approximate, learned (not instinct), and typically wrong, but right enough that it is given support (you trust your own judgment). In this case, thinking over Scoble's comments makes me think of things like visual composition (hence the templating approach).

I think abstracting and modeling this process would be fun. The layering of complexity reminds me of Brook's subsumption architecture, and the emergent complexity would be exciting (if it did something useful).

Seed Funding?

Two posts related to seed funding caught my attention this week, so I thought I would capture something on the topic. I've played the roles of student, software engineer, hacker, and research scientist, and across all of those roles money has never entered the equation (funding from: self, company, clients, and government). With thoughts of scaling up my own projects, I have to start think about more substantial sources of cash other than savings.

Seed funding is about getting an idea off the ground. It's about market research and product development toward producing (understanding) something that has value or is realised as a dead end. The cash comes from you, friends, and family. If at the end of seed funding the product has value, one seeks additional rounds of funding from angel investors, or venture capital. The amount of cash thrown around for seed funding is in the order of $10K-$100K, and is obviously high risk. The level of risk means that anyone with business experience giving you money will seek a high rate of return, and bringing in outsiders likely requires careful consideration of value, monetisation, and exit strategy (business plan). Important, but not as important as building something people want to use.

Anyway, the first post I came across was on RWW titled 13 Seed Funding Options For Entrepreneurs. This was interesting because it forces you to think about the varied ways for acquiring seed funding. This is important because each choice is a trade-off, and blindly making a decision such as "self funded" or "bootstraped from consulting" without being informed is madness. More importantly, there are a lot of people out there (likely ex-entrepreneurs) that want to be involved in high-risk startup projects (even if it is merely advisory)!

The second post fits in well with the first titled How do we set the valuation for a seed round? Importantly, if one considers firstly that outsides want to be involved in a monetary sense, then the first step down the path of angel seed investment is a consideration of the results of the process. Although high-level, this post from VentureHacks sets up the appropriate nomenclature and focuses on the business concerns: what you give up and what you get in return.

I'm sure there is a wealth of such information out there, although I found these posts useful simply for their no-frills style. The average Google search is full of business spam and advertisement-bias. If one is considering the money problem, an excellent staring point is Graham's in depth How to Fund a Startup.

Attention Profiling and APML

I was chatting about lifestreams and thick profiles with a friend (Matt) and he pointed out the Attention Profiling Mark-up Language (APML). An attention profile makes sense in an attention economy where the way a user spends their time is treated as a scarce commodity to be optimised and exploited. An attention profile captures and describes this commodity for a user, which may be exploited to enhance their web experience (such as a service), or by others (such as retail or advertisement).

APML is an approach toward standardising the description of a users attention profile toward promoting the idea that those who own such collected information (the so-called silos such as Google, Amazon, Ebay, etc, as well as the little guys) make it available to users. The effort is driven by Australian company Faraday Media, that have a broader interest in data portability and providing services related to such information. They describe themselves as "epicentre of the attention ecosystem". The core of the standard appears to be motivated by empowering users by making such information accessible and portable given its inherent value. The standard appears to neglect that this value is exactly what the companies that possess it are designed to capture and exploit. Giving it up may not be desirable for many cases. As with Google scanning books pissing off publishers, this will likely make the owners of the collected attention information uncomfortable (disruptive is good right?).

At this early stage, the standard is quite general, describing concepts, sources, free form data, and applications, with dates of access and scalar user ratings. There are some interesting characters involved in the working group (Digg, Ma.gnolia, popurls, etc), and a growing list of supporting applications most notably, the Faraday Media based particls and engagd. Two other early adopter applications that look interesting are dandelife that converts the lifestream into a personal bibliography, and cluztr for social news (click stream as the attention profile).

Particls is interesting in that it is a desktop application that provides an intelligent (attention mediated) information alert system, learning about the user from the contents of their hard disk (see coverage least year on RWW, TC1, 2). The idea is cool, although appears too technical for the average user case. They need to be bought or license the technology to someone to build a killer user-friendly webapp. Faraday Media are obviously tech guys, because their engagd product is a broader solution to the same problem, providing tools and a service to convert a users lifestream into an APML for use elsewhere (such as in a mashup). If the format takes off, this will be a standard practice across lifestream applications (their vision I suspect).

Faraday seemed to have kicked all of this off in 2006, although APML and their applications seeped into the sphere last year with broader coverage. For example: expectations for the use of APML in personalised advertising by Ross Dawson, a generic overview of the state of APML, Mashable highlighting privacy concerns, the potential for integration in GoogleReader, and another generic overview.

Frankly, user's do not want access to their attention profile, they want the services that do. I see the same arguments as are used with OpenID, in the end, it is better for users and businesses. What is missing are good idea's for using this information! Exploitation of an attention profile into an existing social-based service (social news, etc) is a gimmick at best. Social sites are built around aggregate decisions, not personalised for a user based on a deep profile. Modeling a person based on behaviour is hard and expensive, although collecting attention information is relatively easy. This is an important distinction because all APML provides is a standard way for describing such attention information in aggregate. This is an excellent next-step for the the lifestream applications, although still requires innovators to make use of it.

Frankly, attention information will be used to solve completely different problems to the use of such information in aggregate. First-pass cases naturally include filtering, recommendation, and advertisements all of which are currently performed based on a users group rather than the actual user (an easier problem). I believe that as with the shift to social-powered websites, we will see new paradigms for thinking about personal-powered websites.

Considering Twine and Semantic Computation

Reading my feeds, I came across a review of Twine from an AI perspective titled: Why I Migrated Over to Twine. I had read about it before, and paid attention to the post on O'Reilly Radar, although I thought I would have a deeper look and think about it.

The product appears to be a wiki like knowledge management system requiring the user to load in their content explicitly. The expectation appears to be that future content creation (documents, media, etc) occur from within or in conjunction with the application. It also seems to provide all the social network features one would expect for relating a user to additional content, people, and interest groups.

Twine is a semantic application, in that they claim to use semantic understanding to learn about the users interests. The product of the learning is exploited through recommendations, connections, and ultimately productivity and or user experience. The system processes text for resources like peoples, places, things, and uses classical (W3C) semantic technologies such as RDF (subject, predicate, object) for resource description, and OWL to assign resources properties. Interestingly they seemingly provide REST-like accessibility of resources, and mined Wikipedia to seed their knowledge base. One thing that I am interested in are the technical ways in which the semantic data from extracted resources is maintained (does a RDBMS suffice?).

The site has seen a lot of press, among which was it's review and relative comparison with the state of semantic web applications late last year on RWW.

To me, the slow take-off of the semantic web feels a lot like the slow take-off of DVD. It was hyped and talked about for what seemed like years, creeping into shops and homes, until before you knew it, DVD was all any body wanted or could acquire. The semantic web feels as though it is doing the same thing, only it is strange because this is not the way information technology operates. Attrition of technology and standards is ruthless, and one would think that slow adoption would result in a similar state of affairs as that of SOAP (adoption constrained to enterprise in favor of lean alternatives).

Informed users are screaming for solutions to the overload of information, and informed computation (based on the meaning of information) is considered to be a direct solution. It is hard to think out effective uses for semantic computation beyond recommendation and discovery, at least off the cuff. I think the Twine application appears interesting, for example I would love to have a play (when they reach me on the list of beta users), although the lock in of the application for personal knowledge management would be a concern for me. It begs the question of whether centralisation of user content (as was also the case with lifestreaming) is an effective path to exploiting semantic computation. I think this would only be the case if the semantic-driven features offered outweigh the required churn.

Grasping the Lifestream

I recently came across the lifestreaming meme. I am not sure where it originated (who really cares anyway), but it effectively describes the aggregation of your contributions to the internet and those of people you care about (we are all makers now after all).

There is a blog dedicated to the subject, which was started in early 2007 appropriately called Lifestream. RWW provided a brief overview in Jan this year titled Lifestreaming: a ReadWriteWeb Primer, which highlighted a diverse set of products. This review was followed up at the end of Feb with a broader treatment titled 35 Ways to Stream Your Life. The focus of the review was on sites that aggregate your contributions and the contributions of your friends. Broader reviews are excellent for considering the general problem, the general solution, and various positions in the solution space toward improving the adaptive fit of the solution in its niche.

The notion is that such "lifestreaming" applications address the problem that new contributions you are interested in are made in a decentralised and concurrent manner. The general solution involves requiring the user to actively define the decentralised areas of interest in the web (that have remote access) and monitor them for changes of interest which are integrated and presented. Naturally, there are many tweaks on the approach, such as the cool idea of the application actively seeking areas of interest for you in lifestrea.ms, and performing search biased the contributions of yourself and your network in the proposed Delver (called Social Search). Google killer, hardly.

I gave the much discussed Friendfeed a crack. I can imagine the application addressing the proposed pain point if and when I have enough friends producing enough content I care about. Not yet though. My visits are low frequency across "the social sites", and direct communication and conventional syndication work just fine at the moment. This may highlight firstly the highly clustered nature of the paint point (the valley, popular people), and given the growth in the social application space, it suggests the likely flattening in the distribution in the future.

The first notion I had was that when such aggregation is useful, it will rapidly create a situation of "social network information overload" like my mail inbox and feed reader right now. I see the same thing with twitter and even social news. The only way to wade through the stream is with third party added value (meta meta meta...). The same solution will likely apply to the lifestream case, unless lessons are learned and integrated into the product.

A second more interesting point is the notion that such sites are successful if and when they trap traffic from the accessible "sites of interest". This was pointed out last month in a post titled The Conversation Has Left the Blogosphere, where it was highlighted that traffic is stolen because the conversations are promoted on the aggregate not the individual artifact. Importantly, this conversational property is the basis of news aggregation sites as well, so it is reasonable to assume the model is viable for the lifestream pain point.

Finally, it is clear that the real value for these applications is the exploitation of the information about the individual across multiple sources, and the usage of such information in aggregate. Search and Ads are obvious, but the influence may exploited out the bounds of user interaction with the Internet (get ambitious kids!).

Startups in Oz

Something I have been thinking a lot about is the extent of "startup culture" in my home country and city (Melbourne in Australia incidentally). I don't expect anything like the (perceived) ferociousness of the valley, but I do expect innovation and tech, only because I've seen it around in small doses (well the tech guys anyway).

A quick first pass highlights the RRW review of the Top Web Apps in Australia written by Vishal Sharma in late 2006, and The Top 60 Web 2.0 Applications in Australia written by Ross Dawson in early 2007. The top web applications from both lists are familiar, although provide a good grounding for the uninitiated. Vishal also runs a startup portal with the tag line: Discovering the next generation of startups, innovation & tech trends in Australia. Most interestingly he coordinated a recent Australian Startups Carnival 2008, providing a broad and in-depth coverage of the state of home grown webapps (which also got techcrunched).

Others have asked the question about the visibility of OZ startups, and were bombarded with an assortment of links including: an Australian startups wiki, Sydney BarCamp for meeting up (just passed in early April), and a weigh in by one of the Atlassian guys. A little further digging revealed an interview with Ross Hill, and a Melbourne based entrepreneur group called the hive which looks interesting.

I'm sure there are a lot more Australia/Startup information sources out there, but this small array of pointers provide a good jumpstart for baseline comparisons and starting points for networking.

Anything is Better than Nothing or Random

I read a set of posts recently which threaded together in my thoughts as motivating a startup. Firstly to the posts, which are as follows:

• The Young Man's Business Model: Contrasting naive and informed problem solving, specifically in the context of development work, with interesting anecdotes.
  
• On Leaving Google: Comments from an engineering regarding the good times had at Google, and an overview of his personal reasons for leaving to do start-ups.
  
• Rails Is A Ghetto: A massive rant about the personal and resultant technical problems with the core ruby on rails team.
  
• Creatively Speaking Interview: (reddit co-founder Alexis Ohanian) Commenting on the founding of the reddit social news web site.
  
• The end of Moswer: A personal account of a startup idea for the mobile web that was pushed for a year and ultimately failed to gain traction, and ran out of money.
  

I took away a lot from this set, although I'll try and articulate the highlights.

The attitude of the hacking-startup culture promotes user-centric results over everything, which is exactly what you want addressing considering hard problems with the web. The reason is obvious and stamped into the title of this post: some result, no matter how crashy, buggy, or slopping is generally better than no result, or the first pass which is likely a random result.

This was reinforced for me by the young businessmans model approach which was basically a re-phrasing of 'quick an dirty', the naivety of which is contrasted with the informed old man approach. The author advocates considering both approaches before just hacking because he has "been there and done that". I would give the same retrospective advice to those considering starting a PhD (do x! do y! don't do it!), when the point of the exercise is the apprenticeship nature of the process.

Both the rails rant and the reddit interview provided insight into the crashy, buggy, and generally crappy nature of the early released versions of rails (crashing a lot, massive memory leaks) and reddit (sleeping with the laptop). These cases are important because both products are successful in their respective fields (web frameworks, social news), although were hacked into existence from a generally naive and uninformed position (of varying degrees, naturally). This is an extreme of worse is better, where nothing is worse and crappy product is workable. This, to me, is the core of the Graham 'just hack' methodology.

The leaving Google story links into the perspective as the alternative. You can join one of the big guys and be trained to do things properly, to rapidly become the "old man" in the way you think and execute projects. The work will be solid and interesting, although as pointed out by the author, your passion and responsibility are bounded by someone else's conservative (for the shareholders) vision. This links into the Mowser case where although the founder is broke, in debt, and the company a "failure", he is still proud and willing the to give startup thing another crack.

Ultimately, these cases solidified my perspective that the continuous development (agile) of a perpetually wrong (beta) product is the way I want to take on hard problems. Using passion and hard work, from a naive and generally uninformed perspective that ultimately result in promoting risky behaviour, e.g. things committed businesses simply cannot or will not do.

Grasping REST

I have been thinking about the old days, where it was all raw technology and trends. An important trend I have picked up on is the formalisation of the notion of opening up the pieces of an application, called REST. The Wikipedia article is pretty comprehensive, although I found the combination of Rest in Plain English from the RestWiki in conjunction with some examples more tangible.

For me, the contrast of REST with RPC is meaningful (it was what we used to do right?). I like the perspective of thinking about web services as a top-down approach with pre-designed contracts mediating interaction. REST seems to promote a simplified bottom-up perspective, allowing complexity to be defined as aggregates when and where it is needed. Further, the notion of 'application pieces' (resources) as nouns which can be presented and manipulated with verbs provides a nice pattern compared to dragging procedural calls out into the open. Allowing users to define the use of a set of loosely coupled primitives comes into line with neo-web notions of mashups and allowing users to locate and refine the value of your data (classic web2.0), and I presume that's the point.

It is interesting how business is now driving this institution to formalise the approach, specifically with regard to the need to integrate it into SOA. So-called Web Oriented Architecture provides an effective tool for the enterprise to internalise the notions, although I'm not so sure that they are applicable. Business is where you want heavy lifting with well defined contracts like web services, lightweight interaction and 'application as the platform' doesn't make sense unless you're opening it all up to the wild (or you're massive!). The point is, the enterprise market is seeking alternatives to the over-complicated state of SOA. It appears that commentators are suggesting business heed the lessons of "being open" and "simplification" and are not suggesting a disregard web services completely. Perhaps most important is the observation that it's not "REST or nothing", and that partial (weak and implicit) adoption has shown to be good enough even on the open wild web. The explosion of mashups happened (is still happening) using scraping and RPC-based API's without explicit REST, micro formats, or automated tools.

The obvious point is that acknowledgment of the approach and exploitation where appropriate must become a best practice for architecting new web-based systems. The Ruby on Rails community have identified this and have made REST a foundational principle for the platform going forward. This is good for the approach as the RoR community of early adopters will quickly anneal the how's and when's, and progress is being made. I seem to remember a similar situation with the adoption of AOP and the early adopters annealing the applicability in appservers and frameworks.

Highlighting the Peaks in the Web Development Landscape

I have been out of the loop for a while. In fact more recently while writing my dissertation I have been totally consumed, oblivious to the goings on in technology. The effect is that there are embarrassing gaps in my knowledge, specifically with regard to the web technology development landscape. In an effort to consolidate and internalize, this post highlights some of the interesting peaks of the landscape with regard to the web, with an implicit negation of my now outdated background from 4 years ago (LAMP/JEE). This post was created from surfing popular and social technology news. Amendments and corrections are welcome!

Programming Languages: Popular programming languages have changed little. The general TIOBE programming community index highlights the continued pervasive usage of Java, C, VB, the importance of Perl and Python and the growth of Ruby. O'Reilly book sales from 2006 provide insight into the interest in languages, highlighting the decrease in the pervasive approaches, and growth of JS and ruby. C# seems to be the .NET language of choice and interest, and I'm sure ASP.NET is dominating MS shops. Surveying programming social news is somewhat misleading, biasing perspectives on esoteric computer science languages like Lisp, Scheme, Eiffel, Erlang, Scala, and Haskel. I have to point out that I do get a kick out of learning new ways for thinking about programming when time permits. Safe bets would be Java, .NET, Python, and Ruby.

Development Methodology: Agile software engineering methodologies were the rage when I was working last (lots of XP), and it seems the trend has continued. Agile web development seems to be the rage these days (naturally), promoted by successful startups, evangelists, and productivity promoting frameworks. To me, notions of perpetual beta appear to be the application of XP practices where the users are the "client".

Web Frameworks: To show my vintage, the web frameworks of choice when I was last useful where things like Struts, JSF, Spring, and all manner of containers, code generators, and persistence frameworks. By far ruby on rails is the rising star of todays web frameworks, and Java integration will play an important role in enterprise. Django appears to own the python perspective, and old PHP players are still there, and MVC still seems to be the way to do things. Regarding presentation, the popularity of AJAX has not only boosted interest in JS, but also resulted an exposition of presentation frameworks. Some highlights include Dojo, YUI, and GWT, the latter of which I like a lot given the Java->JS complication approach capturing best practices. Some additional lightweight frameworks which may be useful to grasp include OpenID, JSON, REST, and I'm sure many others I have missed and forgotten.

Platforms: In addition to the classical model of building a site and getting hosting, there appear to be many ecosystems in which to participate these days. There are communities for mashups that exploit API's of big sites, including automated tools, and ecosystems centered around large service sites (like twitter for example). There are communities for creating widgets and gadgets for customizable home pages and blogs. Social sites like facebook provide application development platforms that are fostering vast ecosystems given the size of the committed user base involved. Beyond the browser, there are large communities focused on bring the web to mobile devices such as Android, as well as developing content for Second Life.

Tools: The Eclipse platform is stilling going strong for Java, and provides an effective entry point on other popular languages used on the web (Ruby, Python, PHP, etc). Beyond Eclipse, the classical IDE platforms support the popular interpreted languages, in addition to language-specific IDE's providing a lot of choice (for example: ruby and python IDE roundups). Firefox provides a powerful array of presentation development tools, including the popular Greasemonkey and FireBug.

Operations: Operations haven't changed that much, same old databases and operating systems. Regarding Linux, Ubuntu and Gentoo have risen up, although the classics still engender trust. Regarding hosting, the Amazon Web Services, specifically EC2 and and S3 provide a compelling solution, and Google's App Engine will be game changing with regard to barrier to entry for simple sites.

Finally, it seems the popular themes at the moment are the geo web, the social web, and the rich web.ns >>

Twitter, Micro-blogging, and Notions of a Public Idea Funnel

A post on the problems with twitter caught my attention today. I signed up, had a play, and came to the conclusion that the author may have missed the point. Regardless of the partial validity of her claims (hard to explain, what's it for), the platform clearly fills the useful void of persisted and syndicated micro-blogging. The success of the Twitter approach to the problem is grounded in the founders focus on those two features, the recognition of the importance of lightweight authoring mechanisms (140 characters from IM, SMS, and the web if you have to), and their opening up of the whole thing up with API's.

The Twitter in Plain English video totally nailed the platform in my eyes for the average user, and Middlebrook's Big Juicy Twitter Guide nailed the power user case.

The baseline usage of "what are you doing" is banal, but so what. The hierarchy of each person's geographically disjoint clique defines the relative utility of tweeting for the average user. More interesting is the consumption (following) behaviour, promoting the elevation for those commentators that are effective in the medium (not necessarily writers, bloggers, or journalists!). Participation for the average user makes sense in the same way as blogs and before that personal webpages make sense.

Twitter in the broader sense (at the moment) is all about breaking news. Specifically the identification of an event that may or may not have been identified by other media (speed), and the coverage of an event from multiple perspectives including those participating (accessibility). This useful application of twitter is widely acknowledged, for example in identification of natural disasters and in coverage of technology conferences. This application of the platform provides the first viable pathway of the approach toward wider adoption, and will/is promoting the development of effective aggregation tools to extract and exploit this value.

TC and RWW have a lot of coverage of Twitter. I found RWW's comment on the importance of the open API interesting (10 times more traffic than the website) and the founders recognition and exploitation of its power in allowing the community to find and refine the value of the product. This reminded me of id Software doing the same thing with the Quake computer game (which I experienced first hand in my youth), and the patterning of this successful approach thought the 'web as a platform' crowd. Twitter seems to provide yet another case study of a humble (smart) project, that provides core features and acknowledges both that they may or may not know the best way to use and exploit the product, and that the users defiantly do and will. Whether this was by design or convenience, doesn't matter in the face of wide adoption.

What I don't agree with are ideas of such a service as a platform for serious discourse, or the new tech water cooler. It is clear that the early adopters are annealing the usage of product, helping to define utility beyond the average user case (for example). I don't really get the feedback's of the service yet, I need to play with it some more.

One thought that did occur to me was that of taking selfish usage to the extreme (average user). Something I do a lot of is take notes. I take them down on paper, word documents, google notebook, etc, and eventually I formalise them as technical reports, blog posts, projects, whatever. A useful 'single user centric' application of micro-blogging like Tumblr or Twitter would be for the capturing of semi-formal thought snippets for later application in the highlighted ways. Unlike private automatic-aggregated capture, the micro-blog format would promote better single-user aggregation tools, and permit discrete reuse beyond the single user. The first notions that come to mind are the use of a tag cloud and post classification for improved single-user thought aggregation, per-user/per-day clustering of feeds/posts for promoting discovery of related people, thoughts, and materials, and finally keyword search for good old fashioned broader discovery and potential comment.

This notion of a 'public thought funnel' is an application of micro-blogging I would immedatly find useful for project, research, and blogging related activities. I have only considered note taking, which does not fit into the broader notion of micro-blogging solutions, as such Twitter is a good starting point. Unfortunately the high-frequency of the tweets and selfish nature of what is posted do not provide a natural fit with twitetiquette, and the aggregation tools require third party mashing (surmountable). The power here is the social factor for discovering materials related to and ultimately refining you perspective on your present interest. For example, the amazon approach would be something along the lines of "you may be interested in..." and "people who had thoughts on ... on also notions on ...".