This is my first blog post.
For the past several months, my teammates (ed, jonathan, daniel raffel, kevin) and I have been working on a project called Pipes. It is now live. Since Pipes is all about feeds, I thought that it would only be appropriate if I finally started a blog so that I could produce one too.
I am very excited to see Pipes’ vision resonating with some of the most visionary people on the web. I’d like to think that we pushed some boundarys and I am happy to see both the concept and the execution of Pipes being commented on.
Early on in the development of Pipes we recognized it’s large scope beyond simple processing of feeds. For example, would it not be cool if Pipes could be just as easily be applied to REST web services to allow truely generalized mashups?
Stay tuned for more…
Well, I’ll be your first comment then. And the first to congratulate you publicly on what is a “stunning” “milestone”. I couldn’t be prouder to be working a cube over from y’all. This is Yahoo!
very nice work Pasha. congrats
Congratulations Pasha!
Congratulations! It’s beautiful
Pasha! What can I say. I’m so happy for you. I’ve been hoping this would come out since the first time I saw it all those months ago.
Please take in the accolades. You of all people deserve the attention. You’ve been evangelizing and hacking innovation at Yahoo! long before it was encouraged and supported.
Keep fighting and keep innovating. The internet is better because of your determination and brilliance.
-beach
Awesome work on Pipes. You’ve taken a hard problem and put a great interface on top of it. I’m very impressed.
Have you seen Dapper? http://www.dapper.net – we have a feature that’s not entirely different called Linked Dapps which lets a user visually link together any number of sources on the Internet (including those that do not have feeds or web services). Would love to hear your input on Dapper.
Again, great work!
Haven’t seen it yet, but the concept is exactly what I’ve been looking for….almost as exciting as the appearance of the first browser to make it easier for people to connect to and locate info on the Internet. Can’t wait to try Pipes out….
Excellent! Now how about exposing an API for programmatic creation of new Pipes?
Congrats, great work!
“would it not be cool if Pipes could be just as easily be applied to REST web services to allow truely generalized mashups” – to mix & splice you’ll need a generalized data model, time you started reading Planet RDF!
This is great, Pasha. Inspiring.
One question is whether Y! intends always to be the processing center for pipe operations. Could the individual processing blocks somehow be distributed as work units in a P2P fashion, so that Y! need only be a pipes directory service/coordination hub?
It seems reasonable, for example, that a site like Techcrunch might be willing to help out in the filtering/transformation of its own feed, because it would give them valuable insight into how their readership is using their content; remixing it. At the same time, it’s a win for Yahoo! because you needn’t host all the horsepower needed to do the rote grinding.
Even better might be if some of the processing could be offloaded to clients e.g. via a desktop agent or browser plug-in. If I’m reading a pipe on my desktop (as opposed to e.g. My Y! or Bloglines), my PC may as well help out with some of the heavy lifting that otherwise Pipes has to do.
Hey Pipes is great…but I can think of something that would help it out a lot. One of the strengths is the content analysis…but the output is an element and not text. For those of us who want to built dynamic url’s with the content analysis string, it would be a great help to alllow this.
Hi Jake,
We are aware of the limitations of our current modules/data types. We are working on introducing more generalized data types and have individual module/functions produce them instead of embedding the results in the items as we do today.
Danny: you are correct
Evan: Pipes already relies on a fair bit of processing power from all the web services that it calls while executing. We are planning on supporting external processing modules in the future. In general, I expect Pipes to be part of an eco-system of data sources, processors and consumers.
Hello Pasha,
Nice work, these Pipes seem very promising! If you would consider my comments I will find useful to add some generators on top of the opérators. It would make things easier, let say, to create dynamic dates and use such basic generator as building blocks for URL etc.
I suggest this since there are not so much RSS based webservice taht provide date, time, etc.
Hi Denis: agreed.
The User Input modules provide a way to this. For example, you can use a Date Input module and set the default value to a relative value such as “2 hours ago”. The output date object can be fed to the date formatter and URL builder to construct dynamic URLs which you can feed to the Fetch module.
This approach has the side-effect that your Pipe will have a date input in its UI which you may not want. We are thinking of adding constructors for all the primitive types to solve this problem.
Great
Thanks for the explanation I didn’t noticed this !
This is great! How long did it take you to build the system?
- Sean
sorry sean: yahoo policy does not allow me to comment on that.
Pasha jan this is awesome. You guys have done a fantastic job
One input I would give would be to make an additional module to pipes that allows us to query with SQL. Alternativly enabling stored procedure like language inside one of the pipes would be ideal too since that would open up the development door tremendously. Looking forward to building many pipes 
Nice Pasha! (and whateva…it’s way over my head!) actually i was just trying to find an email for you just to say hi! what are you up to …(though i can see you’ve been very busy) and if you remembered me from DC.
-Hazel
Pipes is very impressive I first tried it a couple of weeks ago. The IPhone interface looks great.
very interesting, but I don’t agree with you
Idetrorce
plz. give me informations on generators. i want to market in bangladesh
thanks
Hello webmaster
I would like to share with you a link to your site
write me here preonrelt@mail.ru