An Open Source Success Story: A History of Plucker
Plucker started out in the late 90's as a way to get webpages "plucked" onto a PalmPilot handheld device, to replace the proprietary AvantGo software that only ran on Windows at the time.
Since then, Plucker has evolved quite a bit into the advanced application it is today. You can find Plucker in every major Linux distribution, running on thousands of machines daily.
Plucker went from a single application that was 7k in footprint size, to a full-blown viewer and e-book reader with tools in every language and for many different operating systems.
The first release of Plucker was quite rough (but it worked!). It could only deal with textual content (ignoring any images, and without color), was based on lynx and an awk and sed parser, and syncronized directly to the Palm handheld. This meant you had to have your Palm in the cradle to gather and syncronize content.
Below you can see what it looked like. No images, limited error handling for bad HTML. The lynx browser was used to fetch the HTML, dumped to text, and parsed it with sed and awk parsers. There was no scrollbar, and you had to use the scroll buttons or tap your stylus and move the page up and down to navigate up and down the content. There were no menus, options, or preferences. What you see here is what you got, and it worked, and worked well. Heck this was 1998, we didn't need that much.
Version 0.02 was a bit more robust, but still offered the same basic features as 0.01. It also ran on older Palm devices, such as the Pilot5000 and Pilot Professional. Still no images, no menus, no preferences, and no scrollbar, but again, it worked, and was very fast at rendering text.
Look... IMAGES! v0.03 of Plucker now contained the first pass at image support, and the first version of the perl conduit to sync the Plucker data to the Palm device.
There were some minor glitches with the text formatting, but those would be fixed.. in time. Also has the first pass at a basic menu structure for navigation. lynx was replaced by wget in 0.03, visited links were implemented, and some basic tap-navigation was implemented (up and down). The source was also put under revision control in this version, and has been there ever since (3 years and running)
Better image support, initial cut at scrollbar for navigation, first pass at preferences, the first Deutsch translation, and OS/2 support.
The text rendering and layout in 0.5 was much improved over previous releases. There was also the addition of a new Python parser in this release that replaced the awk and sed scripts from previous releases.
Also new was a set of new configuration options and the first introduction of zlib compression of the data. You can also see that the graphics rendering in 0.5 was improved quite a bit. Images that were links were no longer underlined as they were in 0.04.
Italic fonts, horizontal rules, and the introduction of multiple databases also made their appearance in this release.
0.6 brought us the internal Find function in Plucker, which was quite useful. The first Czechoslovakian translation was done, 2-bit greyscale images, movable scrollbar, Database Manager, and a percentage accellerator to move quickly through a document.
Fullscreen mode, new HTML elements supported (text alignment), support for mailto tags, and various other small additions.
Support for beamed Plucker content, Flash support, and the first instance of the global configuration file for the Python parser.
Large image support (pan around), more custom preferences (screen depth), an Italian translation of the viewer, and a few bug fixes.
A small update to v0.9 that adds the ability to copy URLs to the clipboard, and a global configuration file for Unix and Linux (~/.pluckerrc). Also support for ZLib-compressed databases.
Quite a few new things in this version, including bookmarks, more custom preferences, hardware tap navigation, a more robust Python parser, and more.
Custom font selection, hardware button navigation, a new French translation of the viewer, more toolbar commands, new Python parser options, and inital support for beaming Plucker documents from within the document manager, including those from compact flash.
Three new translations, Faroese, French, and Japanese, more database manager options to show additional information on each document, beam, delete, and rename.
Interview with Mark Lillywhite, original author of Plucker
David A. DesrosiersThursday October 10th 2002 - 21:47:15
Who really was the original creator of Plucker? A gentleman named Mark Lillywhite from Australia. I had the chance to interview Mark (online) to see just what gave him the idea for the project. Below is the full interview.
David A. Desrosiers: When did you first get the idea for Plucker?
Mark Lillywhite: Well ... I moved to Canada to work on some medical software and they gave me this nice new laptop which ran Windows NT. I decided to get a Palm Pilot and it came bundled with AvantGo, and every morning during the 30 minute ride on the subway/bus I would read news etc that AvantGo had collected. When I finally got sick of NT on my laptop (as you do) I installed Linux, but of course AG didn't work with linux. I always wanted to do some coding for the Palm and I saw this as an opportunity... so that's how it started.
David A. Desrosiers: What year was that?
Mark Lillywhite: It would have been '98 I think. It's hard to remember now.
David A. Desrosiers: Have you ever wanted to go back to AvantGo? (I had to ask)
Mark Lillywhite: No, never haha!
Mark Lillywhite: Sadly at the moment my pilot is broken so I can't use any of these packages.
Editor: I've sent Mark a spare TRGPro since this interview, so he could continue to use Plucker, a project he started for all of us.
David A. Desrosiers: Did you ever imagine that Plucker would take off as it has done?
Mark Lillywhite: No, I had no idea. After I released it I wondered if anyone would actually care.
Mark Lillywhite: Really it's a personal project gone horribly out of control
David A. Desrosiers: (stops tape recorder) I actually used to hit your website and poke around at your webcam pics, and found your Plucker page a few years back.
David A. Desrosiers: That's how I found the project myself.
Mark Lillywhite: It used to beep every time someone looked.
David A. Desrosiers: The core team has grown from 1 person, yourself, to almost a dozen people, and now Plucker is used in major distributions by thousands of people every single day. Do you have any wishes that Plucker should have done 'Feature X' or 'Option Y' since you started the project?
Mark Lillywhite: No not really. I mean the Plucker I released - the one I was happy with, 0.03 - had basically everything in it that I wanted. Now I see all these e-books and so forth, I think they're really, really great ideas -- but I never imagined them myself.
Mark Lillywhite: I mean that's the thing with free software right, that's why I released it.
David A. Desrosiers: "Release Early, Release Often", right.
David A. Desrosiers: People are using parts of it in their own free software projects now.
Mark Lillywhite: Right on, but of course you have to release something useful too, otherwise people won't pick it up.
David A. Desrosiers: ..and people are using it in major institutions (the medical-interest topics on the lists lately)
David A. Desrosiers: Hey, version 0.01 was pretty rough ..and it was also only 7k in footprint size.
Mark Lillywhite: Cool, you know I didn't know that it was in distributions.
Mark Lillywhite: I also didn't know about medical apps using it, which is odd 'cos I used to work in the medical industry.
Mark Lillywhite: 0.01 was very rough.. I couldn't get the linker to work properly so I had to put it all in one ginormous file, I hated that.
David A. Desrosiers: If you could change anything, what would the top three things be?
Mark Lillywhite: Probably I would have liked to release a parser that I was proud of!! I think everyone probably thinks I'm a crap programmer (but maybe I am).
David A. Desrosiers: The awk parser wasn't so bad... it was just cryptic as heck.
Mark Lillywhite: I would have liked to have more time to spend on Plucker, I am a bit of an outsider now and am only really tolerated 'cos I started it all.
Editor: We do more than tolerate Mark, we rely on him to keep the mailing lists running, and for other things related to Plucker. He's a bit more pivotal than he admits.
David A. Desrosiers: Personally, I wish I had more time to work on the perl parser..
David A. Desrosiers: The same could be said of Eric Raymond and fetchmail, or RMS and emacs though.
Mark Lillywhite: And I wish my pilot hadn't died, or that I could justify the cost of a new one, but I'm waiting for the MP3/cellphone/pilot magic combo.
David A. Desrosiers: There are now OS/2, Windows, Macintosh ports which work, as well as many Unixes. Now people are talking about browser plugins and web-based sync services for Plucker. What do you think of all these ideas?
Mark Lillywhite: Oh I think it's great! Really, I love the fact that I started all that stuff you know?
David A. Desrosiers: Actually... .if you want, I can send you a Palm... I have way too many units. Gratis. What Palm did you have that died?
Mark Lillywhite: I mean I don't take credit for anything that Plucker looks like now, I just love the idea that my code bootstrapped it!
David A. Desrosiers: What's next on the Plucker horizon, for you that is?
Mark Lillywhite: Well for me, just to get it working again would be nice. I would really like to grab some time and just look at the code and get some of these ebooks.
David A. Desrosiers: Well, I'll just have to send you some hardware, so you can get it running again.
Mark Lillywhite: One of the most exciting things I've seen lately is that guy who's working on the forms support.
Mark Lillywhite: The thing I find exciting is that these people have picked it up and said, Ok, we need this, let's do it.
Mark Lillywhite: Someone is getting paid to work on this, I think that's so cool, I've indirectly employed someone!
David A. Desrosiers: We have developers in Sweden, Germany, California, Texas, and Australia... as well as from dozens of other places on the globe, and Plucker is now in major linux distributions' standard installations. Other tools are beginning to support Plucker's format as an output format (Sitescooper) and now some tools are using the Plucker format in their own code (CSpotRun's new update includes full Plucker support).
Mark Lillywhite: You said earlier that 0.01 was a bit rough and it was. Mike once said to me that if I hadn't released 0.03, where I managed to get the linker to work (by upgrading gcc) then he would never have bothered to suss out the structure of the thing.
Mark Lillywhite: So in a way, I almost released "too" early because the 1st release was so hard to read, 'cos I couldn't split it up.
David A. Desrosiers: It's getting larger now, and almost time to begin using segments, and "snap-ins" to support new features like DOC support, etc.
Mark Lillywhite: Yeah, I think it's important to keep in mind that one of the huge advantages of Plucker over a.g. was that itactually loads and renders much more quickly because of the pre-parsing and a small binary footprint.
David A. Desrosiers: 74k is the current size, and aside from any online functions (which we don't use), it still surpasses AvantGo in features and is still 8x smaller.
Editor: The code has since grown, with the addition of VFS support and other features, to a current size of 101k
Mark Lillywhite: I would hate to see pure HTML stuck onto Plucker (though I imagine a tag-encoded XHTML would be feasable). I think parsing on the pilot is silly.
Mark Lillywhite: ..at least, trying to parse such a stupid format as HTML
David A. Desrosiers: I am still of the belief that the configuration files should be XML, and there should be some sort of storage format for the content which is in some way XML'ized, but that's not going to be possible until we begin having sites support Plucker directly. Having Plucker handle RDF/RSS though, should be somewhat trivial to add.
Mark Lillywhite: Right, yeah. I'm a huge fan of XML, I'm actually writing XML in my other xchat window.
Mark Lillywhite: The use of pre-encoded XML tags would make the file size very small while maintaining the advantages of XML. I think you can have both small size and XML, just not unicoded-XML.
Mark Lillywhite: It's just tokenizing, token #1 represents <xhtml>, token #2 prepresents <p>, etc. etc. then the "XML" is not really much different from Plucker
Mark Lillywhite: In fact you could make an XML format that represents the actual plucker database I suspect.
Editor: The conversation diverged from here into a discussion of XML, but I'll save that for another time..
This website is 100% compliant with XHTML 1.0 Strict and uses valid CSS.
Only very minimal use of tables allows us to achieve this result.
Plucker runs on Microsoft Windows, Linux, Mac OS X, Palm handheld devices, Windows Mobile devices and handheld devices powered by Linux.
Plucker is a Free Software project. We do this entirely in our spare time, to provide a great piece of software for you.
None of the Plucker developers get paid for their work on Plucker.