Feb 28 2017 update

Apple has responded quickly to address this issue. Their Developer ID page, which I believe is brand new, specifically addresses provisioning profiles and their relationship to the Developer ID certificate. Here’s what they say (emphasis added):

For apps that utilize advanced capabilities with a Developer ID provisioning profile
Gatekeeper will evaluate the validity of your Developer ID certificate when your application is installed and will evaluate the validity of your Developer ID provisioning profile at every app launch. As long as your Developer ID certificate was valid when you compiled your app, then users can download and run your app, even after the expiration date of the certificate. However, if your Developer ID provisioning profile expires, the app will no longer launch.

That section addresses the crashes seen in PDFpenPro and 1Password: It is now documented that an expired provisioning profile will prevent your app from launching. That’s not necessarily good news…but the good news is that this will, going forward, be a much rarer event:

To simplify the management of your Developer ID apps and to ensure an uninterrupted experience for your users, Developer ID provisioning profiles generated after February 22, 2017 are valid for 18 years from the creation date, regardless of the expiration date of your Developer ID certificate.

So any app that uses a provisioning profile created after February 22nd of this year will not crash due to an expired provisioning profile—even if the developer does nothing and lets their Developer ID certificate expire—until February 22, 2035. That’s effectively forever in the world of a macOS app (it’s longer than macOS/OS X itself has existed, in fact.)

Thanks, Apple, for the quick response! We’re leaving the original article posted as a non-techie overview of the Developer ID system; keep reading if that’s of interest to you.

---

Recently, some well-known Mac apps, including 1Password, PDFpenPro, and Soulver, had a big problem: They all failed to launch. Nothing had changed with these apps (i.e. no updates had been released), and yet they simply stopped working.

So what happened? All three of these apps (and probably some others we haven’t heard from yet) contained an expired code signing certificate. That expired certificate prevented the apps from launching, though no developer would have expected that, based on Apple’s own documentation. And an expired code signing certificate can’t just be renewed to extend its expiration date (like you would a driver’s license); it needs to be replaced with a new non-expired certificate, which requires distributing an update to the app.

Follow me now, if you wish, for a somewhat deep dive into the world of code signing, as I attempt to explain—from a consumer’s perspective yet with a developer’s hat on—what is code signing, why these apps broke, why the breakage wasn’t expected, and other related questions and answers.

Update: AgileBits has a very detailed blog post that covers this issue in even more depth—well worth the reading time.

Code signing is a digital signature for apps, proving that an app was created by the developer who signed the app, and it allows the operating system to check whether a given app’s code has changed since it was created. Here’s how Apple summarizes it for developers:

Code signing is a macOS security technology that you use to certify that an app was created by you. Once an app is signed, the system can detect any change to the app—whether the change is introduced accidentally or by malicious code.

You participate in code signing as a developer when you obtain a signing identity and apply your signature to apps that you ship. A certificate authority (often Apple) vouches for your signing identity.

Signing apps is a reasonable way to add security to a system without overburdening the end user with authorization dialogs—like the infamous User Account Control (UAC) in Windows Vista, which popped up “Allow this?” dialogs seemingly every 10 seconds.

On macOS, code signing’s visible front-end is Gatekeeper, as seen in the Security & Privacy System Preferences panel:

macOS ships with Gatekeeper set to only allow apps from the Mac App Store or from identified developers, so it’s a big advantage for a developer to register with Apple and sign their apps: Any unsigned apps require additional authorization work by the end user before they can be run. Would you want to do that extra work when presented with a scary dialog like this one?

This is why almost all mainstream apps are signed, as well as small little one-off apps from single coders: Nobody wants their app to generate the above dialog. (The app in my screenshot, Disk Inventory X, was last updated in 2005—long before signed apps came to be a thing. There’s nothing malicious about it; it was just a good example for the screenshot.)

Here’s how Apple explains the safety aspect for users on the Gatekeeper page:

For apps that are downloaded from places other than the Mac App Store, developers can get a unique Developer ID from Apple and use it to digitally sign their apps. The Developer ID allows Gatekeeper to block apps created by malware developers and verify that apps haven’t been tampered with since they were signed. If an app was developed by an unknown developer—one with no Developer ID—or tampered with, Gatekeeper can block the app from being installed.

There are two key security advantages that come from signed apps. The first, as noted above, is that the system can confirm that the app you’re trying to run hasn’t had its code changed since it was signed. So if a piece of malware did manage to modify the binary for a particular signed app, that app wouldn’t launch because its signature would be found to be invalid.

The second big advantage is that Apple can revoke a certificate if a developer were to do something truly malicious. Once revoked, any apps that were signed with that certificate will stop working. Here’s how Apple explains it to developers…

Code signing your app allows the operating system to identify who signed your app and to verify that your app hasn’t been modified since you signed it. Your app’s executable code is protected by its signature because the signature becomes invalid if any of the executable code in the app bundle changes. Note that resources such as images and nib files aren’t signed; therefore, a change to these files doesn’t invalidate the signature.

Code signing is a good thing from the user’s perspective; from a developer’s perspective, it adds to the complexity of the build-and-release process, but the benefits outweigh the added complexity.

One element of this complexity is that the certificates do expire, and an expired certificate must be replaced with a new non-expired certificate, which requires distributing an update to your app. But until recently, this hasn’t been a time-critical item, and an expired certificate wasn’t a big deal.

Historically, the answer to that question was “No, it’s not.” In fact, Apple still explicitly tells developers that an expired code signing certificate won’t affect existing apps (added emphasis mine):

If your certificate expires, users can still download, install, and run versions of your Mac applications that were signed with this certificate. However, you will need a new certificate to sign updates and new applications. If your certificate has been revoked, users will no longer be able to install applications that have been signed with this certificate.

This is why developers haven’t viewed an expired certificate as a critical issue: It shouldn’t have any effect on already-released apps. To reiterate the point, Apple states so again on another page, even calling it out with a highlight:

The message can’t be any clearer: There’s nothing inherently bad about an expired signing certificate, at least relative to already-released apps.

In fact, you probably have a number of apps on your Mac—or your iPhone—with expired certificates. For example, an older app from a developer who has since gone out of business. Apps from such developers will run just fine, even though they contain an expired certificate. This is a good thing for users, because you don’t need a developer to still be in business to use their app.

Once the certificate data is embedded in the app, the fact that it’s expired doesn’t remove the safety provided by the signature: If a piece of malware modifies an app with an expired signing certificate, the code signing check will still fail (and the app won’t launch), so the user is protected.

Apple can also revoke an expired certificate (in the case of malware) and the app will fail to launch, so the user is protected.

Neither of these safety valves require the certificate to be current, only that the app contains a valid signature, created at the time it was built.

From a developer’s perspective, it’s also good that expired certificates don’t cause apps to fail to run. For example, we distribute old versions of our apps, for users on older versions of macOS/OS X. At some point, the certificates in those apps will expire. If we had to create new builds for all 50+ versions of the apps we distribute, that would be a nightmare. And if we didn’t do that, we’d have a lot of upset users of older versions, complaining that their apps no longer run.

Yes, they should. But as noted above, Apple explicitly tells developers—in at least two places—that they only need these certificates to build new apps or update existing apps. So if you have a certificate that’s set to expire, but you don’t have an urgent need to update an app or create a new app, it’s supposed to be a non-event.

It’s a bit of speculation, but it’s possible (probable, even) that the affected apps’ developers actually had new, non-expired certificates in hand. But those certificates would only be used to create an update to the app, and they’d been told that an expired certificate in an existing app wasn’t anything to worry about.

Another important fact is that simply getting a new non-expired certificate doesn’t solve the problem: The certificate data is built into the app itself, not stored externally somewhere where a newer version can be retrieved. To get their new certificates into their apps, these developers would have had to push an update to users—and unless they had a minor update ready to go, the sole purpose of the update would be to replace the expired certificate.

Most developers dislike releasing numerous updates, especially updates with only one change (assuming it’s not a critical bug fix), and again, Apple tells developers that an expired certificate isn’t a big deal. A normal business approach to this situation would be…

1. Set up the new certificates, so they’re ready for the next update.
2. When the time comes for the next update, the build will contain the new non-expired certificate data.

But because the expired certificates broke these apps, the developers had to release an update just to refresh the certificate data. That’s not good for developers or users.

Given Apple’s position that an expired certificate should have no impact on an app’s ability to run, there’s no doubt that Smile and AgileBits and others were very surprised by reports of their apps failing to launch. So really, why did these apps fail?

The common thread in all three of these apps is that they included support for using iCloud for sync or storage. Access to iCloud was originally restricted to App Store apps, but that changed with the release of Yosemite: Now any registered developer can include iCloud support in their apps.

To do so, a developer needs to flip a switch in Xcode to add iCloud support:

Flipping the toggle from Off to On creates something called a provisioning profile. This profile says “It’s OK for this app to use iCloud, even though it’s not in the App Store.” This was best explained by Smile’s Greg Scown to Adam Engst in this tidBITS article:

What’s new with PDFpen 8 is that, in addition to being code signed, it has a provisioning profile, which is essentially a permission slip from Apple that’s checked against an online database in order to allow the app to perform certain actions, called entitlements. For PDFpen, the entitlement that’s being granted is the capability to access iCloud despite being sold directly, rather than through the Mac App Store, a feature that wasn’t possible until about a year ago.

The only apps that crashed due to expired certificates (so far, at any rate) were those that used this iCloud provisioning profile outside of the Mac App Store. What happened with PDFpenPro and the other apps is that the expired signing certificate rendered the iCloud provisioning profile invalid. The invalid provisioning profile, in turn, prevented the app from ever launching.

According to everything Apple has told developers, this should not happen. To the best of my knowledge, developers were never given any warnings about expired certificates and iCloud provisioning profiles. If they had been told, it’s a fair certainty that they all would have updated their apps with new certificates before the expiration date. Of course, this wouldn’t help any user who didn’t update their copy of the app before the certificate’s expiration date.

What made the situation even worse is that the provisioning profiles are checked by the system itself, before the app even runs. As a result, the app can’t tell the user something is wrong, because the app didn’t actually launch. Again, from tidBITS:

An app called taskgated-helper determines this even before PDFpen’s code runs, so there’s no way for Smile to detect the error condition and present an error to the user.

So the app can’t tell the user what’s going on, and even worse, the OS doesn’t tell the user what’s going on: The app just seemingly dies in an instant. That’s bad for the user, and bad for the developer, because they’re blamed for something they didn’t even know was occurring (because their app never loads, and no crash reports are generated). If the OS is going to kill the app, the OS should tell the user why it did so, so the user has some understanding about the problem.

First, from our perspective: As we currently don’t use iCloud (or other) provisioning profiles, our behavior won’t change: An expired certificate is not reason alone to update an app. And as discussed above, doing so for all old apps with (eventually to be) expired certificates would be a nightmare.

For developers who do use iCloud (or other) provisioning profiles, it appears that unless Apple fixes something, they will have to pay very close attention to their certificate expiration dates, and make sure they push out app updates (and hope all their customers update) before the certificates expire.

As a user, if you’re really interested in this stuff for some reason, there’s an app called RB App Checker Lite that can take a look inside any application and see its certificates (and a bunch of other data).

The output is definitely way up on the geek scale; here’s what it shows for Flying Meat’s Acorn image editor:

Here I’ve clicked the arrow next to “The signature contains 3 certificates” in order to view the Flying Meat code signing certificate. As you can see, it appears their certificate was recently replaced, given the nearly-five-years-from-now expiration date.

---

Code signing is a complex topic; hopefully this article made the subject a bit clearer, and you gained an understanding of how it works, and for how seemingly-irrelevant expired certificates brought down a number of Mac apps.

This entry was posted on Thursday, February 23rd, 2017 at 12:16 pm by Rob Griffiths, and is filed under Behind the Scenes, Developer. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.