Raining on the OpenClip Parade

Thursday, 21 August 2008

The OpenClip project, which debuted this week, describes itself as “a non-profit, open-source, community-effort project, which promotes a framework for the iPhone that allows users to copy/paste between participating applications.”

The obvious shortcoming, compared to a hypothetical system-wide clipboard from Apple, is that apps which don’t explicitly support the OpenClip scheme don’t work with it — including all of Apple’s apps, like Mail and Safari. That’s to be expected. But it’s worse than that.

The OpenClip framework, by Zac White, is a very clever implementation of a fundamentally unwise idea. White’s description of the project on his own weblog is fairly open regarding its inherent problems. The OpenClip.org web site, however, is hosted and written not by White but by Proximi, the developers behind MagicPad. Their developer FAQ states:

How does it work?

OpenClip utilizes a shared space on the iPhone. Applications that use the OpenClip framework can access this common area to write to and read from, allowing copy/paste between participating apps.

That struck me as curious, as I wasn’t aware of any inter-application “shared space” on the iPhone. White’s own description of how it works and the OpenClip source code itself show that such a description is disingenuous. The “How does it work?” section of their regular (i.e. non-developer) FAQ is more technically accurate:

OpenClip utilizes an application’s ability to read into other application’s Documents directory. Applications that use the OpenClip framework can access this read only area to read pastes from other applications and then OpenClip can offer the newest data to the current application.

The fundamental problem the OpenClip project faces is that of data interchange. If you copy something in app A, and wish to paste it in app B, the clipboard data needs to exist somewhere where app A can write (when you copy), and where B can read (when you paste). On Mac OS X, the system provides this to Cocoa apps via the NSPasteboard class and associated APIs. Individual applications don’t have to worry about the details of how and where clipboard data is stored; it’s an implementation detail completely managed by the system.

But even ignoring Mac OS X’s standard system-wide clipboard, apps on the Mac face no challenges when it comes to exchanging data with other apps via the file system. On the Mac, all apps can read and write wherever they want within your entire home folder. So if Mac apps A and B wish to share data with each other via a custom file format, they can both agree to do so via a shared file, in, say, the user’s Documents folder.

iPhone apps can’t do that. Or, more specifically, third-party iPhone apps written with the official iPhone SDK can’t do that; Apple’s own iPhone apps can do whatever they want.

iPhone Sandboxing

The idea with sandboxing is that each app executes in its own space, with limited resources and with no ability to alter or modify anything outside its own sandbox. The downside is that some of the things Mac apps can do but which iPhone apps cannot are potentially very useful (and/or very cool). The upside is that those same things are potentially dangerous (both in terms of security and in terms of stability). It’s a trade-off.

Here’s how sandboxing works in iPhone OS 2.0. Given its shared roots with Mac OS X, the iPhone OS unsurprisingly has a very familiar file system layout. The system’s standard apps reside in a top-level folder named /Applications/, just like on Mac OS X. Apps that you install via the App Store don’t go there, however. Instead, there is a separate Applications folder for these applications. In iPhone OS 2.0, that folder is at /private/var/mobile/Applications/, but it doesn’t really matter exactly where it is. (/private/var/mobile/ is more or less the iPhone equivalent of your home folder.)

Every time you install an application, a new sandbox is created within that Applications folder. The sandbox is a folder named with a UUID, for example, “68813987-A628-493F-90E2-A6ABCD922A89”. The application itself is installed inside the sandbox folder, along with its own directories for writing data. So, if you install two iPhone apps from the App Store, named, say, “Foo” and “Bar”, they’ll be installed in two separate sandboxes that look something like this at the file system level:

/private/var/mobile/Applications/04A74595-4DE8-4026-8459-63B2D153D13C/
    Documents/
    Foo.app/
    Library/
    tmp/

/private/var/mobile/Applications/77C9A482-F5F8-4284-9E16-C629763B9162/
    Bar.app/
    Documents/
    Library/
    tmp/

Each app gets its own Documents folder, its own Library folder, and even its own temporary scratch space (“tmp”). Each application can only write to the file system within its own sandbox directory. This isn’t just a guideline from Apple — it is enforced by the OS. Any attempt by an app to write to the file system outside its sandbox will fail.¹

However, in iPhone OS 2.0, an app can read from anywhere in the file system. This serves as the basis for how OpenClip works.

How OpenClip Works

As an API, White’s OCPasteboard class is a clone of Cocoa’s NSPasteboard. For every NSwhatever call in the standard Cocoa NSPasteboard class hiearchy, OpenClip offers a corresponding OCwhatever. The advantage to this design is twofold: first, it’s exactly what Cocoa developers are accustomed to on the Mac; second, if Apple eventually ports NSPasteboard to the iPhone OS, it’s likely to have a very similar API.

When an app using OpenClip copies data,² it writes the data into two files within its own Documents folder:

Documents/
    OpenClip/
        OCGeneralPboard.data
        OCGeneralPboard.metadata
Foo.app/
Library/
tmp/

When an app using OpenClip pastes, the OpenClip framework peeks into the Documents folder of every other app sandbox in the file system, looking for the most recently-modified OpenClip data files. So if you have three apps, A, B, and C, and you copy something in A, then copy something else in B, then do a paste in C, C will paste the data copied from app B, because it was created more recently.

The Problems

On his weblog, White writes:

How it works is relatively simple and doesn’t break the SDK agreement. OpenClip works by looking into the Documents folder of other applications to get their pastes. Applications are allowed to write all they want to their own Documents directory (for copy), so no foul there. Applications are also allowed to read outside their sandbox into the Documents directories for other apps (for paste), so no foul there.

This is not accurate. It’s more like “slipping through a temporary loophole” than “no foul”. In the Security section of chapter 4 of Apple’s iPhone OS Programming Guide, the sandbox is described by Apple as follows (boldface emphasis added):

For security reasons, iPhone OS restricts an application and its preferences and data to a unique location in the file system. This location is part of the security feature known as the application’s “sandbox.” The sandbox is a set of fine-grained controls limiting access to files, preferences, network resources, hardware, and so on. In iPhone OS, an application and its data reside in a secure location that no other application can access.

Not simply that no other application can write to, but which no other application can access. That this restriction is not yet enforced at a technical level (such as is the case with an app attempting to write outside its own sandbox) does not mean it’s permitted.

And, indeed, in the 2.1b4 release of the iPhone OS, it is enforced. The OpenClip demo apps, which work as advertised on iPhone OS 2.0.2, do not work in the current 2.1 beta, because apps are no longer able to read or even see other apps’ sandboxes.³ To be clear, this change is clearly not in response to OpenClip; Apple began seeding the 2.1 betas with these tightened sandbox restrictions before OpenClip debuted, and the iPhone OS Programming Guide has stated all along that apps can’t “access” the contents of other sandboxes.

There is no “shared space” for iPhone apps to exchange data. (One workaround I’ve seen bandied about is to use the system-wide Address Book database as a storage location for shared clipboard data. Needless to say, such an implementation would not qualify as an intended use of the AddressBook framework.) Wishing it were otherwise doesn’t make it so.

The intent of OpenClip is fine. That there’s been so much coverage regarding OpenClip in the past 24 hours shows just how much demand there is for inter-application copy-and-paste. But developers would be foolish to adopt a framework that only works today because of a loophole in iPhone OS 2.0 that is already closed in iPhone OS 2.1. ★

Title Case Update

Wednesday, 13 August 2008

Back in May, I published a Perl script I wrote for properly title-casing text. Its main trick was in being smart about “small words” that should not be capitalized even in title case.

I wrote:

The source code itself is, uh, rather convoluted, to say the least. It’s one of those pieces of code that started small and simple, and grew ugly over time as edge cases were worked around one at a time. I’ve been using this script for years, but have put off publishing it on the grounds that it looks like the sort of punctuation-riddled code that gives Perl phobics the heebie-jeebies.

A slew of clever people took my script and ported it to other languages — JavaScript, Python, Ruby, PHP, Objective-C, even Nu. Crackerjack work, one and all. But two stand out, at least for my own needs.

David Gouch wrote his own version in JavaScript; his script handles all the edge conditions my script did, and also some others. He also published a list of test cases and the results of my original script, John Resig’s bug-for-bug port to JavaScript of my script, and Gouch’s superior solution. This is the best JavaScript solution I’ve seen; I’m using it in the bookmarklets I use for posting Linked List items.

Aristotle Pagaltzis was kind enough to take my Perl script and refactored it in a straightforward and logical way, fixing some of the edge conditions along the way. He took Gouch’s test cases and turned them into a proper testing suite. The result is better than my original script in every way, and his test suite can be used to improve any implementation.

My thanks to everyone who pitched in on these scripts, and to David Gouch and Aristotle Pagaltzis especially. ★

Memoranda

Monday, 11 August 2008

Apple and Microsoft, as ever, offer a study in contrasts. Take, for example, two recent company-wide memos from CEOs Steve Jobs and Steve Ballmer. Jobs’s, leaked last week, regarded the botched launch of MobileMe. Ballmer’s, from two weeks ago, outlined Microsoft’s strategic goals for the next year.

The difference in their leadership styles is evident simply from studying the differences in their writing styles. Jobs’s memo is brief, humble, and focused. (That’s not to say Jobs is humble, only that the memo is.) Ballmer’s is long, full of bluster, and more or less describes Microsoft as being in competition with every other company in the entire software industry.

Let’s be clear, Microsoft is making a boatload of money under Ballmer’s leadership: $15.8 billion in revenue, $4.3 billion in net profit for the just-ended quarter. Apple, in the same quarter, reported $7.4 billion in revenue and $1 billion in net profit, and Google reported $5.4 billion in revenue and $1.2 billion in net profit — which means Microsoft had more revenue and nearly twice the net profit of Apple and Google combined. And of course a strategic outline for the entire year is going to be a longer, less focused memo than one that’s focused just on MobileMe.

So take my criticism of Ballmer and his style with a few billion dollars worth of salt. But, still.

There are some similarities between Ballmer and Jobs. For one, they both sign their memos, simply, “Steve”. For another, they’re both non-engineers leading engineering companies. Engineers, in general, crave facts and detest bullshit. My sense is that by and large, engineers at Apple are often frustrated by Jobs’s (relative) lack of technical acumen, but in terms of overall leadership and company strategy, they believe what he says.

Ballmer, however, has the demeanor of a successful car salesman. He’s so full of bluster that he comes across as being either delusional or full of shit.

Example. Ballmer, in his memo, on Apple:

Apple: In the competition between PCs and Macs, we outsell Apple 30-to-1. But there is no doubt that Apple is thriving. Why? Because they are good at providing an experience that is narrow but complete, while our commitment to choice often comes with some compromises to the end-to-end experience. Today, we’re changing the way we work with hardware vendors to ensure that we can provide complete experiences with absolutely no compromises. We’ll do the same with phones — providing choice as we work to create great end-to-end experiences.

Engineering and design are all about trade-offs. With Windows — and before it, DOS — Microsoft has made trade-offs in the interests of ubiquity. Do what it takes to get it everywhere. Apple’s trade-offs for the Mac have been in the interests of the cohesiveness and quality of the overall experience. Apple’s recent gains in computer market share have been a huge deal for Apple — but they’re a drop in the bucket for Microsoft. Microsoft’s still-growing profits show that Windows doesn’t necessarily lose as the Mac wins. What Ballmer is arguing is that Microsoft plans to somehow have its cake and eat it too — “absolutely no compromises” is not possible. Everything involves compromises.

Likewise with Ballmer on Google:

Google: We continue to compete with Google on two fronts — in the enterprise, where we lead; and in search, where we trail. In search, our technology has come a long way in a very short time and it’s an area where we’ll continue to invest to be a market leader. Why? Because search is the key to unlocking the enormous market opportunities in advertising, and it is an area that is ripe for innovation. In the coming years, we’ll make progress against Google in search first by upping the ante in R&D through organic innovation and strategic acquisitions. Second, we will out-innovate Google in key areas — we’re already seeing this in our maps and news search. Third, we are going to reinvent the search category through user experience and business model innovation. We’ll introduce new approaches that move beyond a white page with 10 blue links to provide customers with a customized view of their world. This is a long-term battle for our company — and it’s one we’ll continue to fight with persistence and tenacity.

The inherent bravado in Ballmer’s statement-as-fact that Microsoft “will out-innovate Google in key areas” sets off alarm bells. That’s a goal, not a fact. And his mockery of Google’s search as “a white page with 10 blue links” indicates that he has no idea why Google has been so successful. I’d wager that if anyone is ever going to gain on Google in search, it will be by presenting even more focused results — less clutter, fewer distractions, more emphasis on making the results easily scanned. The old Microsoft could recognize good ideas and copy them; now Microsoft can’t even recognize genius.

The strategic bottom line is that Microsoft, under Ballmer, feels compelled to compete everywhere — that they must directly confront any company achieving any significant success, no matter how far afield that success is from the areas where Microsoft is already winning or doing well.

Jobs’s memo, on the other hand, was an acknowledgement that Apple had tried to do too much at once:

It was a mistake to launch MobileMe at the same time as iPhone 3G, iPhone 2.0 software and the App Store.  We all had more than enough to do, and MobileMe could have been delayed without consequence.

And as for what the company is going to do:

The MobileMe launch clearly demonstrates that we have more to learn about Internet services.  And learn we will.  The vision of MobileMe is both exciting and ambitious, and we will press on to make it a service we are all proud of by the end of this year.

Not a guarantee that Apple will somehow magically do everything better. Simply a promise to learn and to press on. Credible and humble.¹ And, more importantly, realistic. Apple employees may not always — or even often — agree with Jobs, but they do believe him. Apple tends to do and achieve exactly what Jobs says they will. (His declaration in January 2007 that Apple would be selling 10 million iPhone per year by 2008, for example.)

Ballmer’s promises, in contrast, defy belief, at least regarding where Microsoft stands against Apple in terms of “end-to-end experience” and against Google in terms of search and online advertising. He’s either ignorant or lying — neither of which is inspiring to the rank-and-file engineers.

I’m reminded of this great line from Tim O’Reilly back in 2002 regarding Bill Gates:

Microsoft gets a lot of heat for not leaving enough on the table for others. My mother, who’s English, and quite a character, once said of Bill Gates, “He sounds like someone who would come to your house for dinner and say, ‘Thank you. I think I’ll have all the mashed potatoes.’” This isn’t quite fair, but it gets the point across, at least about some of Microsoft’s behavior.

Gates may be gone, but the attitude is infused in the company’s culture. The difference between the ’90s and the ’00s, though, is that today there are far too many potatoes on the table for any single company, no matter how large, to eat.

Microsoft ought to be more worried about doing well in their core competencies — OS licensing and developer platforms — than in expanding into unrelated new areas. Ballmer is keen on pointing out that Microsoft sells 30 Windows licenses for every Mac that Apple sells, but Windows Mobile, which has been on the market for eight years, doesn’t even outsell the year-old iPhone by 2-1. And Apple is gaining fast; it seems possible that by 2010 Apple could be selling more iPhones than all Windows Mobile handsets combined, and it arguably already has more third-party developer interest.

Ballmer, I think, needs a little more Developers Developers Developers and a little less Advertisers Advertisers Advertisers. ★