Tags: mashups

Pipe Dreams: The life and times of Yahoo Pipes

Yahoo PIpes was ahead of its time. Here’s a nostalgic retrospective by Glenn Fleishman.

I think it was Jason who once told me that if you want to make someone’s life a misery, teach them about typography. After that they’ll be doomed to notice all the terrible type choices and kerning out there in the world. They won’t be able to unsee it. It’s like trying to unsee the arrow in the FedEx logo.

I think that Stewart Brand’s pace layers model is a similar kind of mind virus, albeit milder. Once you’ve been exposed to it, you start seeing in it in all kinds of systems.

Each layer is functionally different from the others and operates somewhat independently, but each layer influences and responds to the layers closest to it in a way that makes the whole system resilient.

Last month I sent out an edition of the Clearleft newsletter that was all about pace layers. I gathered together examples of people who have been infected with the pace-layer mindworm who were applying the same layered thinking to other areas:

• Rich applied pace layers to career paths,
• Mark applied pace layers to the design process, and
• Jorge Arango applied pace layers to reading.

My own little mash-up is applying pace layers to the World Wide Web. Tom even brought it to life as an animation.

See the Pen Web Layers Of Pace by Tom (@webrocker) on CodePen.

Recently I had another flare-up of the pace-layer pattern-matching infection.

I was talking to some visiting Austrian students on the weekend about design principles. I explained my mild obsession with design principles stemming from the fact that they sit between “purpose” (or values) and “patterns” (the actual outputs):

Purpose » Principles » Patterns

Your purpose is “why?”

That then influences your principles, “how?”

Those principles inform your patterns, “what?”

Hey, wait a minute! If you put that list in reverse order it looks an awful lot like the pace-layers model with the slowest moving layer at the bottom and the fastest moving layer at the top. Perhaps there’s even room for an additional layer when patterns go into production:

• Production
• Patterns
• Principles
• Purpose

Your purpose should rarely—if ever—change. Your principles can change, but not too frequently. Your patterns need to change quite often. And what you’re actually putting out into production should be constantly updated.

As you travel from the most abstract layer—“purpose”—to the most concrete layer—“production”—the pace of change increases.

I can’t tell if I’m onto something here or if I’m just being apopheniac. Again.

On Monday, I linked to Tom’s latest video. It uses a clever trick whereby the title of the video is updated to match the number of views the video has had. But there’s a lot more to the video than that. Stick around and you’ll be treated to a meditation on the changing nature of APIs, from a shared open lake to a closed commercial drybed.

It reminds me of (other) Tom’s post from a couple of year’s ago called Pouring one out for the Boxmakers, wherein he talks about Twitter’s crackdown on fun bots:

Web 2.0 really, truly, is over. The public APIs, feeds to be consumed in a platform of your choice, services that had value beyond their own walls, mashups that merged content and services into new things… have all been replaced with heavyweight websites to ensure a consistent, single experience, no out-of-context content, and maximising the views of advertising. That’s it: back to single-serving websites for single-serving use cases.

A shame. A thing I had always loved about the internet was its juxtapositions, the way it supported so many use-cases all at once. At its heart, a fundamental one: it was a medium which you could both read and write to. From that flow others: it’s not only work and play that coexisted on it, but the real and the fictional; the useful and the useless; the human and the machine.

Both Toms echo the sentiment in Anil’s The Web We Lost, written back in 2012:

Five years ago, if you wanted to show content from one site or app on your own site or app, you could use a simple, documented format to do so, without requiring a business-development deal or contractual agreement between the sites. Thus, user experiences weren’t subject to the vagaries of the political battles between different companies, but instead were consistently based on the extensible architecture of the web itself.

I know, I know. We’re a bunch of old men shouting at The Cloud. But really, Anil is right:

This isn’t our web today. We’ve lost key features that we used to rely on, and worse, we’ve abandoned core values that used to be fundamental to the web world. To the credit of today’s social networks, they’ve brought in hundreds of millions of new participants to these networks, and they’ve certainly made a small number of people rich.

But they haven’t shown the web itself the respect and care it deserves, as a medium which has enabled them to succeed. And they’ve now narrowed the possibilites of the web for an entire generation of users who don’t realize how much more innovative and meaningful their experience could be.

In his video, Tom mentions Yahoo Pipes as an example of a service that has been shut down for commercial and idealogical reasons. In many ways, it was the epitome of what Anil was talking about—a sort of meta-API that allowed you to connect different services together. Kinda like IFTTT but with a visual interface that made it as empowering as something like the Scratch programming language.

There are services today that provide some of that functionality, but they’re more developer-focused. Trys pointed me to Pipedream, which looks good but you need to know how to write Node.js code and import npm packages. I’m sure it’s great if you’re into serverless Jamstack lambda thingamybobs but I don’t think it’s going to unlock the potential for non-coders to create cool stuff.

On the more visual pipes-esque Scratchy side, Cassie pointed me to Cables:

Cables is a tool for creating beautiful interactive content.

It isn’t about making mashups, but it does look something that non-coders could potentially use to make something that looks cool. It reminds me a bit of Bret Victor and his classic talk on Inventing On Principle—always worth revisting!

This Video Has {{ viewcount }} Views - YouTube

Tom’s videos are so good! Did you see his excellent in-depth piece on copyright?

This one is all about APIs and the golden age of Web 2.0 when we were free to create mashups.

It pairs nicely with a piece by another Tom from a couple of years back on the joy of Twitterbots.

Linked Data at the Guardian | Open Platform | guardian.co.uk

A great write-up of the latest additions to the Guardian's Open Platform API including a lukewarm assessment of Semantic Web technologies like RDF.

Everything Is a Remix

It's well worth paying attention to this site, the accompaniment to the four-part series of videos entitled "Everything is a Remix."

Amazon® AWS HMAC signed request using PHP

Since Amazon decided to require signed requests for its API, I'm going to have to use this code to keep Huffduffer and The Session working. Grrrr... cool APIs don't change.

Flickr Services: Flickr API: flickr.places.placesForUser

This new Flickr API method makes it really easy to get a list of visited places for a Flickr user.

Show Us a Better Way

Tell the UK government what you'd build with public information and they could help fund your idea. Time to put your hacking hat on.

Scripting Enabled

Christian is using the prize money he won at Mashed to put on an event in London in September devoted to "ethical hacking": creating mashups to make social networks more accessible.

Build Last.fm: Extend your Last.fm experience

Last.fm have gathered together the best apps built on their API and put them all in one handy browsable spot.

Dopplr Blog » Announcing the first release of Dopplr’s API

Dopplr can has API.

A couple of months ago I wrote:

Jessica speculated a while back about reverse Google Maps. Suppose that when you entered an address, instead of just showing you the top-down view of that point on the planet, you also got to see how the sky would look from that point. Enter a postcode; view the corresponding starmap.

It isn’t in Google Maps yet but it is in Google Earth. The newest version features a button labeled “Switch between Sky and Earth”. This new Sky feature allows you to navigate photographs of space taken from the Palomar observatory and the Hubble telescope. It’s just one more example of what you can do with geodata.

Location information is the basis for a lot of the mashups out there—of which, Overplot remains my favourite. The possibilities in mashing up geodata with timestamps are almost limitless.

Getting datetime information is relatively easy. Every file created on a computer has a timestamp. Almost everything published on the Web is also timestamped: that’s the basis of lifestreams.

I look forward to the day when geostamps are as ubiquitous as timestamps. If every image, every blog post, every video, every sound file had a longitude and latitude as well as a date and time… I can’t even begin to imagine the possibilities that would open up.

I’m not the only one thinking about this. Responding to the question, what parts of the Web need to be improved or fixed in order for the Web of today to evolve into the Web of the future?, Jeff Veen writes:

I wish every device that was capable of talking to the network could send its geolocation. I’d like this to be fundamental—let’s send longitude and latitude in the HTTP header of every request. Let’s make it as ubiquitous and accessible as the time stamp, user agent, and referring URL.

This doesn’t necessarily mean that every electronic device needs to be geo-aware. As long as devices can communicate easily, you may ever only need one location-aware device. Suppose my phone has GPS or some other way of pinpointing location. As long as that device can communicate with my computer, perhaps using Bluetooth, then my computer can know my location: a very short string of two numbers. Once my computer has that data, my location can be broadcast and a whole ecosystem of services can be enhanced. Sites built around travel or events are the obvious winners but I can imagine huge benefits for music sites, photo sharing or any kind of social networking site that boils down to real-world activity.

The technology isn’t quite ubiquitous enough yet and there are privacy concerns (though the granularity of geodata negates a lot of the worst fears) but I hope that as the usefulness of geodata becomes clearer, location enhanced services can really begin to bloom.

Ever since I hacked up my little life stream experiment and wrote about it, it’s been very gratifying to see how people have taken the idea and run with it. Emily Chang has written about the resources she came across when she was putting her life stream together. Sam Sethi has been talking about life streams as a rich vein of attention data (which reminded me of John Allsopp’s thoughts on why blogging as we know it is over).

Of course this idea of mashing up time-stamped (micro)content—usually through RSS—isn’t anything new. Tom Armitage touched on this during his presentation at Reboot in Copenhagen last year:

Whenever I publish anything with a date attached, there’s a framework for ongoing narrative. The item published is our narrative, but the date gives it ongoingness. It takes time for the pattern to emerge; initially, throwing data at that black box, it seems random. For instance: I upload photos to Flickr at arbitrary intervals. I go silent on my blog without explanation. It may seem, in the short-term, like a blip, but in the long-term, it’s an important part of my story. My blog is full of delicious bookmarks right now because I’ve been busy at work, and writing this talk. That’ll be reflected in the longer game, when I write my post-Reboot blog entry, and suddenly the pattern becomes clear.

If you haven’t yet done so, I strongly urge you to read the rest of Telling Stories — What Homer, Dickens, and Comic Books can teach your (social) software. It’s quite brilliant and discusses many issues that are even more relevant today with the rise of OpenID and the clamour for portable social networks.

Jeff Croft has been pioneering the life stream idea for quite a while now, originally calling it a tumblelog. His implementation uses APIs rather than plain ol’ RSS. He’s right in thinking that APIs are a more robust solution for long-term archiving but I think of my life stream as being a fleeting snapshot of current activity.

As Jeff points out:

The result is that most people’s lifestream looks great for the first several days back, but then get all sparse at the bottom, where only one or two sources are still providing information.

CSS to the rescue. I’ve updated my life stream to give vibrant colours to newer entries and faded, eventually illegible colours to older, less relevant content. It’s kind of like Shaun’s recent experiments with age and colour.

I love APIs but when something as simple as RSS does the job, I’ll go for the simple solution every time (hence my love of microformats). In fact, I see RSS as being a kind of low-level short-term API or, as Rob Purdie put it, the vaseline of Web 2.0.

The ubiquity of RSS is what makes Yahoo Pipes possible. Now anybody can make a life stream by plugging in some RSS feeds into a pipe. Here’s one I made earlier. When I tried to do this a few days ago, I couldn’t get it to sort by date properly: it was sorting the pubDate field alphabetically—that seems to be fixed now.

Using Yahoo Pipes isn’t quite as straightforward as it could be. It still feels kind of techy and intimidating for non-geeks. This is the same problem that Ning used to have. Its services were ostensibly being provided so that non-techy people could start mashing stuff up but the presentation was impenetrably techy. That’s all changed now.

Ning has completely rebranded as a social network builder. Personally, I think this is a brilliant move. After just a few seconds on the front page, it’s absolutely clear what you can do. By providing example sites, they make the point even clearer. You can still make all the same stuff that you always could on Ning—videos, photos, blogs—but now it’s all wrapped up as part of a clearer goal: creating your own social network site.

When Yahoo Pipes launched, it looked like it might be competing directly with Ning. Now that’s not the case. The two services have diverged and are concentrating on different tasks for different audiences.

I’ll be keeping an eye on Ning to see how it deals with the issue of portable social networks. I’ll be watching Yahoo Pipes as a tool for creating life streams.

I’ve been playing around with Twitter, a neat little service from the people who brought you Odeo. You send it little text updates via SMS, the website, or Jabber. It’s intended as a piece of social software, but I think it has potential for more selfish uses.

Every time I ping Twitter, the message is time stamped. Every time I post a link to Del.icio.us, that’s time stamped. Every time I upload a picture to Flickr, a time stamp of when the picture was taken is also sent. Whenever I listen to a song on iTunes, the track information is sent to Last.fm with a time stamp. And of course whenever I blog, be it here, at the DOM Scripting blog or Principia Gastronomica, each entry has a permalink and a time stamp.

Just about every time somebody publishes something on the Web, it gets time stamped. Wouldn’t it be nice to pull in all these disparate bits of time stamped information and build up a timeline of online activity?

The technology is already in place. Most of the services I mention above have APIs. In this case, a fully-blown API isn’t even necessary. Each service already offers an easily parsable XML file of activity ordered by time: RSS.

At the recent Take Back The Web event here in Brighton, Rob Purdie talked about RSS being the vaseline that’s greasing the wheels of Web 2.0. He makes a good point.

Over the course of any particular day, I could be updating five or six RSS feeds, depending on how much I’m blogging, how many links I’m posting, or how much music I’m listening to. I’d like to take those individual feeds and mush ‘em all up together.

There are a couple of services out there for mashing up RSS. FeedBurner is probably the most well known, but you are limited to a pre-set choice of RSS feeds that you can mix in. RSS Mix offers a more open-ended splicing service but it seems a bit confused when it comes to date ordering. There’s some other service I was playing around with last week but for the life of me, I can’t remember the name of it. All I remember is that it had an extremely annoying interface full of gratuitous Ajax.

I’ve mocked up my own little life stream, tracking my Twitter, Flickr, Del.icio.us, Last.fm, and blog posts. It’s a quick’n’dirty script that isn’t doing any caching. The important thing is that it’s keeping the context of the permalinks (song, link, photo, or blog post) and displaying them ordered by date and time. What I’d really like to do is display the same information in a more time-based interface: a calendar, or timeline.

Annoyingly, the Last.fm feed of recently listened to tracks disappears if you don’t listen to anything for a while. Grrr…

Update: Here’s the PHP source code.

YUI’s Matt Sweeney and Hands-on YUI Workshops at Yahoo! Developer Day/Hack Day September 29-30 in Sunnyvale » Yahoo! User Interface Blog

Yahoo is opening up Hack Day to the masses. If you're anywhere near Sunnyvale on September 29th, this should be fun.

In all the many mashups out there, Google Maps is probably the most used API (version 2 is out now).

One of the latest in the long line of map mixes is Galker Stalker. It takes user-submitted celebrity sightings and displays them on a map of Manhattan.

Has Nick Denton gone too far this time? George Clooney certainly thinks so. Of course, for a site like Gawker, any publicity is good publicity. Jessica and Jesse are just so excited that George Clooney has noticed their existence.

From Web page to Web platform

CNET's News.com explains why web services are so cool.