func addOne(_ x: Int) -> Int {
fatalError("Haha! No Int for you!")
}
That’s legal Swift. I don’t think many would find that surprising. Of course that’s legal, right? But it is surprising. addOne
claims to be a function that accepts an Int and returns an Int. It does accept an Int, but…it doesn’t return an Int.
“Don’t be silly, Rob. It crashes. It can’t return an Int if it crashes.”
Well, yeah. But it promised to return an Int. It didn’t promise to “return an Int or crash,” did it? The whole point of strong types was that the compiler would enforce our promises, and the compiler doesn’t bat an eye at this code.
“The compiler can’t possibly know everything that might crash.”
I’m not ready to concede that, but even so, the compiler clearly can see that this function doesn’t return an Int. There’s no return intValue
anywhere. This should prick our ears a bit. Something is strange here. Is it just compiler magic, or is there something deeper?
Hint: There’s something deeper.
Let’s take a step back and think about another thing that should surprise us.
let printSquid: () -> Void = { print("🦑") }
So, printSquid
is a closure that takes no parameters and returns…what?
You might be tempted to say “nothing.” But that’s not what the code says. It says it returns Void. What’s Void?
public typealias Void = ()
Void is a real thing. It’s the type of the empty tuple. It’s not “nothing.” In Swift, the type ()
and the value ()
happen to have the same spelling. When we mean ()
as a return type, we traditionally write it using the typealias Void. But () -> ()
is exactly the same thing as () -> Void
. Why “Void?” Because Swift has ObjC roots, and ObjC is C-like, and in C there are void
functions that literally return nothing.
In Swift, every function returns something. There are functions that return ()
, and there’s some syntactic sugar that inserts -> Void
and return ()
for you in some cases. But you’re free to include them if you like (it’s not particularly idiomatic Swift, but it’s legal).
func f() -> Void {}
func g() { return () }
And functions like print
that seem to have no return value, really do return Void:
let printResult: Void = print("🦑") // 🦑
print(printResult) // ()
If we don’t add the : Void
here, the compiler will emit a warning because you usually don’t want Void values, but they’re totally legal.
Each type has a set of values that are part of that type. Some types have an enormous number of values. Int has around 18 quintillion values. String has even more. And some types have very few values. Most enums have just a handful of possible values. Bool has just 2. A lot of the power of strong types is that we can create small types; types that can only hold exactly the values that are legal for our program. That’s why we prefer enums to String and Int unless we mean “arbitrary text” or “an arbitrary number,” and we prefer structs to Dictionary unless we mean “an arbitrary mapping of keys to values.” Smaller, more constrained types help the compiler help us.
So how small can a type get?
Void has just one value. So that’s pretty small. In most languages that have this type, it’s called Unit. The one-value type.
Can they get smaller?
It’s time to get back to the original example:
func addOne(_ x: Int) -> Int {
fatalError("Haha! No Int for you!")
}
Looking at fatalError
, it feels a lot like print
. The syntax is similar. But we can’t just throw print
into places fatalError
goes:
func addOne(_ x: Int) -> Int {
print("Haha! No Int for you!") // Cannot convert return expression of type '()' to return type 'Int'
^^^^^
}
How does the compiler know we’re allowed to use fatalError
here, but not print
? Maybe the compiler just knows that fatalError
is special. That’s what was done prior to Swift 3:
@noreturn public func fatalError(...)
^^^^^^^^^
The @noreturn
attribute told the compiler that this function doesn’t return, and then the compiler had logic to handle that case and let us skip returning the Int we promised. But…bleh. I hate that solution. It opens up weird corner cases. For example, what happens if we add @noreturn
to a function that we claim does return something, or something that throws:
@noreturn func addOne(_ x: Int) -> Int { ... }
@noreturn func runForever() throws { ... }
The first one is probably an error, and maybe the compiler should forbid it. But what about the second one? Is throwing a “return?” At the implementation level, it actually is. But should this attribute allow it? There’s not a really obvious answer. Is it possible for a function to require a @noreturn
parameter? How does this impact function overloading?
In Swift 3 they got rid of the attribute hack, and solved the problem with a type: Never. The signature of fatalError
is now:
public func fatalError(...) -> Never
^^^^^
So what’s Never? Is it some new compiler trick? Nope. It’s just a type, an enum with no cases:
public enum Never {}
How many Never values are there? Well, none. You can’t construct one. That has all kinds of interesting implications.
func f() -> Never { ... } // f never returns
func g(_: Never) { ... } // g can never be called
struct S { // S can never be constructed
let nope: Never
...
}
enum E {
case ok(Int) // E.ok can be constructed
case nope(Never) // E.nope can never be constructed
}
// But also interesting:
struct G<Element> {}
let ok = G<Never>() // This is fine. Never can be a phantom type.
Another interesting implication is that [Never]
is an empty array.
Never is the smallest possible type. We call it an “uninhabited type.” There’s nothing special about the name “Never.” You can create your own no-case enum, and it’ll work the same.
// Our own custom uninhabited type
enum NeverReturn {}
func neverReturn() -> NeverReturn {
// We could call fatalError() here, since generating any uninhabited type
// is sufficient to create any other. But we can also use an infinite loop.
// The compiler can prove this will never return. Never isn't just for crashing!
while true {}
}
func addOne(_ x: Int) -> Int {
neverReturn() // It's fine not to return Int, because the compiler knows this doesn't return
}
// While Never can be used to create a NeverReturn, they're not the same type
let never: Never = neverReturn() // Cannot convert value of type 'NeverReturn' to specified type 'Never'
let neverEver: NeverReturn = fatalError() // Cannot convert value of type 'Never' to specified type 'NeverReturn'
While it’s possible to create your own uninhabited types, I don’t really recommend it. The Swift team considered having different types for things like “exit” vs “abort” and intentionally chose not to. One uninhabited type is probably plenty. But it’s nice that it’s not some magical name.
In type theory, an uninhabited type is often called a bottom type, and written as ⊥. A bottom type is a subtype of every other type. So Never would be an Int and a String and a UIViewController and every other type. The opposite is the top type (⊤), the supertype of every other type. In Swift, that’s Any.
But in Swift, Never isn’t actually a bottom type. If it were, you could write this, and you can’t:
let x: Int = fatalError() // Cannot convert value of type 'Never' to specified type 'Int'
It’s easy to fix this with a little extra syntax if you need it:
let x: Int = { fatalError() }()
So Never acts like a bottom type when it’s being returned from a function, but not when it’s being passed to a function or assigned to a variable. It would be more consistent for Never to be a true bottom type, and for it to conform to every non-static protocol (i.e. protocols without static or init requirements). Whether that’s worth the weird corner cases it might create, I’m not sure. But maybe.
Never is my favorite type in stdlib. It’s been my favorite type since it was introduced in Swift 3, and the Combine framework has completely justified my love of it by applying it to generics.
Publishers generate a series of values or a typed failure:
public protocol Publisher {
associatedtype Output
associatedtype Failure : Error
func receive<S>(subscriber: S) where S : Subscriber, Self.Failure == S.Failure, Self.Output == S.Input
}
But what should Failure be if the Publisher never generates errors? Never, of course. No magic needed. It just works. What if it only can generate a Failure? Well, then Output is Never (see IgnoreOutput). When there’s a proper type, the special cases disappear, and it just works.
So Never has gone from a little-known type solving a little-known problem around fatalError
to something Swift developers will probably use every day without even thinking about it. And that makes me very happy.
final class APIClient {
static let shared = APIClient()
let baseURL = URL(string: "https://www.example.com")!
let transport: Transport
init(transport: Transport = URLSession.shared) { self.transport = transport }
// Fetch any Fetchable type given an ID, and return it asynchronously
func fetch<Model: Fetchable>(_ id: Model.ID,
completion: @escaping (Result<Model, Error>) -> Void)
{
// Construct the URLRequest
let url = baseURL
.appendingPathComponent(Model.apiBase)
.appendingPathComponent("\(id)")
let urlRequest = URLRequest(url: url)
// Send it to the transport
transport.send(request: urlRequest) { data in
let result = Result { try JSONDecoder().decode(Model.self, from: data.get()) }
completion(result)
}
}
}
This fetch
method is great for getting a model by ID, but I have other things I want to do. For example, I’d like to periodically POST to /keepalive and return if there’s an error. That’s really similar, but kind of different.
// GET /<model>/<id> -> Model func fetch<Model: Fetchable>( _ id: Model.ID, completion: @escaping (Result<Model, Error>) -> Void) { let urlRequest = URLRequest(url: baseURL .appendingPathComponent(Model.apiBase) .appendingPathComponent("\(id)") ) transport.fetch(request: urlRequest) { data in completion(Result { let decoder = JSONDecoder() return try decoder.decode( Model.self, from: data.get()) }) } }
// POST /keepalive -> Error? func keepAlive( completion: @escaping (Error?) -> Void) { var urlRequest = URLRequest(url: baseURL .appendingPathComponent("keepalive") ) urlRequest.httpMethod = "POST" transport.send(request: urlRequest) { switch $0 { case .success: completion(nil) case .failure(let error): completion(error) } } }
Both basically follow this pattern of build an URL request, pass it to transport, and then deal with the result. I know it’s just one line that exactly duplicates, but the structure is still really similar, and it feels we could pull this apart. The problem is that fetch
is doing too much.
So maybe we pull out the part that changes and call it Request. But what should Request be? So often, I see people jump to a PAT (protocol with associated type) like this:
// This is a bad idea
protocol Request {
var urlRequest: URLRequest { get }
associatedtype Response
var completion: (Result<Response, Error>) -> Void { get }
}
So what’s the question we ask whenever we make a PAT? Would I ever want an array of these? I think we would definitely want an array of requests. A list of pending requests. Chaining requests together. Requests that should be retried. We definitely want an array of requests. This is a great example where someone might come along as say, if only we had generalized existentials then everything would be wonderful. No. That wouldn’t fix anything. The problem is this treats a PAT like a generic, which isn’t the right way to think about it.
Generics and PATs are very different things that solve very different problems. Generics are type-parameterization. That means that the types are being passed as parameters to the function. They’re passed at compile time, but they’re still passed by the caller. When you say Array<Int>
, you, the caller, get to decide what kinds of elements Array holds. In Array<Int>(repeating: 0, count: 10)
, Int is just as much a parameter as 0 and 10. It’s just a different kind of parameter.
PATs aren’t like that. Their associated types are not parameters passed by the caller. They’re hooks provided by the implementor of the conforming type (or whoever wrote the conforming extension). When you conform a type to a PAT, you have to provide a mapping of stuff that algorithms need to stuff this type has. Collection requires an Index type in order to implement subscripts (among other things). Set says “here’s my Set.Index type that Collection algorithms should use when you need an Index type.” Array says “please use Int as my Index for those algorithms.” As the consumer of Set or Array, you can’t change those choices. You can’t say “I want an Array indexed by Character.” That’s not up to you. It’s not a type parameter.
The point of a PAT is to allow algorithms to use the type. If you’re thinking about storage (like putting things in an Array) rather than algorithms, you probably do not want a PAT.
Rather than focusing first on how to construct a Request, let’s focus on how we’d like to use one. I wish something would just know all the stuff I needed to send to the transport….
class APIClient {
func send(_ request: Request) {
transport.send(request: request.urlRequest,
completion: request.completion)
}
}
This is a kind of “wish driven development.” We “wish” there were some type that could handle the URLRequest and completion handler for us, we pretend it exists, write the code that uses it, and then make it a reality. And the reality couldn’t be simpler:
struct Request {
let urlRequest: URLRequest
let completion: (Result<Data, Error>) -> Void
}
OK, that’s simple, but that’s still not quite what we want. There’s no model information in there. I want to create Requests that know about model types, like this:
client.send(Request.fetching(id: User.ID(1), completion: { print($0)} ))
So I want to put User.ID into a system and get User back out in the completion handler, but the system (Request) only understands Data. That means we’re making a type eraser. We’re hiding a type (User) inside Request. How? With one of the simplest type erasers you can have: a generic function or closure. Basically, we just take fetch
and wrap it into a closure. Here’s fetch
:
class APIClient {
func fetch<Model: Fetchable>(_ id: Model.ID,
completion: @escaping (Result<Model, Error>) -> Void)
{
// Construct the URLRequest
let url = baseURL
.appendingPathComponent(Model.apiBase)
.appendingPathComponent("\(id)")
let urlRequest = URLRequest(url: url)
// Send it to the transport
transport.send(request: urlRequest) { data in
let result = Result { try JSONDecoder().decode(Model.self, from: data.get()) }
completion(result)
}
}
}
And here’s fetching
:
extension Request {
static var baseURL: URL { URL(string: "https://www.example.com")! }
// GET /<model>/<id> -> Model
static func fetching<Model: Fetchable>(id: Model.ID,
completion: @escaping (Result<Model, Error>) -> Void) -> Request
{
// Construct the URLRequest
let url = baseURL
.appendingPathComponent(Model.apiBase)
.appendingPathComponent("\(id)")
let urlRequest = URLRequest(url: url)
return self.init(urlRequest: urlRequest) { // Here's the closure that hides (erases) Model
data in
completion(Result {
let decoder = JSONDecoder()
return try decoder.decode(Model.self, from: data.get())
})
}
}
}
fetching
is a generic method, but it returns a non-generic Request struct. This kind of generic->non-generic conversion is an incredibly powerful way to simplify your system and keep generics from spiraling out of control.
You may ask “why a static fetching
method rather than creating an init(fetching:completion)
extension” For this one, init
would probably be fine, but as you think about other kinds of Requests, especially ones with no parameters, it would get messy. For example, it’s hard to build a nice init
for /keepalive. (This isn’t a deep design point; it’s just a stylistic choice. You might prefer init(keepAliveWithCompletion:)
, and that’s up to you.)
In any case, this is how I’d build the /keepalive handler:
extension Request {
// POST /keepalive -> Error?
static func keepAlive(completion: @escaping (Error?) -> Void) -> Request
{
var urlRequest = URLRequest(url: baseURL
.appendingPathComponent("keepalive")
)
urlRequest.httpMethod = "POST"
return self.init(urlRequest: urlRequest) {
switch $0 {
case .success: completion(nil)
case .failure(let error): completion(error)
}
}
}
}
This is the end of my discussion of this little network stack (though not the end of my discussion of generics). It’s not designed to be a “real” network stack. I don’t expect anyone to use this directly as described. I build stacks based on these principles all the time, but I’ve never had one look exactly like this. They’re each quite specialized to their particular API, and the particular needs of the app. The goal here wasn’t to create a general purpose library to solve all networking problems. The goal was to show how you would extract generic code tailored to a problem. Your API is probably different, and you’ll probably build your solution in a different way. Don’t feel you have to use a Transport and an APIClient and a Request. (Though maybe you should use Transport… :D)
If you want to build a general purpose library around this, I suggest you first build small, custom libraries around several APIs that are very different from each other. Then look for the abstractions. Abstracting too soon, before you really understand the problem, is the most common cause of generic headaches.
There is no such thing as “as generic as possible.” Generic code is abstraction, and abstraction is choices. Choosing to make things flexible in one direction often makes it harder to be flexible in another. I have a long list of code bases where we needed more flexibility, and the first step was to rip out all the “flexible” code that made the wrong assumptions about what kind of flexibility would be needed and was never actually used.
Write concrete code first. Then work out the generics.
If you stick to that, it’ll probably work out ok.
]]>The current models are User and Document:
struct User: Codable, Hashable {
let id: Int
let name: String
}
struct Document: Codable, Hashable {
let id: Int
let title: String
}
But now the server API is changing. Document IDs will be Strings, not Ints. (True story.) But really, IDs never really were Ints. I mean, IDs aren’t numbers. What would it mean to add two IDs together? Or divide them? How can I pretend that an ID is a kind of number if most number-like operations would be nonsense? The current design allows me to pass document IDs when I mean user IDs. It even lets me pass random integers when I mean an ID. That can’t be right. IDs are their own thing. They want a type.
As usual, I’ll start very concretely with User and see if anything generic develops. The first step is to lift the ID into its own type.
struct User: Codable, Hashable {
struct ID: Codable, Hashable {
let value: Int
}
let id: ID
let name: String
}
So now creating a User looks like this:
let user = User(id: User.ID(value: 1), name: "Alice")
That’s ok, but I don’t like the value:
label. It violates one of the principles of the API Design Guidelines:
In initializers that perform value preserving type conversions, omit the first argument label, e.g.
Int64(someUInt32)
.
To comply, I should I add another initializer.
struct User: Codable, Hashable { struct ID: Codable, Hashable { let value: Int init(_ value: Int) { self.value = value } } let id: ID let name: String } let user = User(id: User.ID(1), name: "Alice")
Much better. Document will be almost exactly the same.
struct Document: Codable, Hashable { struct ID: Codable, Hashable { let value: String init(_ value: String) { self.value = value } } let id: ID let title: String }
It’s not a lot of code, but anytime I’m tempted to cut and paste, it’s time to wonder if there’s generic code hiding in there. After all, most of the model types in this system will probably have an ID.
When I see code duplication, I often reach first for a protocol so I can extract a generic algorithm. That’s something protocols are very good at. In the case of ID, there are two duplicated concepts: identifiers conform to Codable and Hashable, and identifiers have a “no label” initializer.
It’s important to focus on the duplication of concepts, not keystrokes. DRY doesn’t mean “never type the same letters twice.” The point is to extract things that will vary together. I don’t want to capture “types that include the characters : Codable, Hashable
and init(_...
.” I want to capture “things that behave as identifiers.” So I’m going to capture that concept as Identifier:
protocol Identifier: Codable, Hashable {
associatedtype Value: Codable, Hashable
var value: Value { get }
init(value: Value)
}
extension Identifier {
init(_ value: Value) { self.init(value: value) }
}
With that, User.ID is simplified to:
struct User: Codable, Hashable {
struct ID: Identifier { let value: Int }
let id: ID
let name: String
}
To use it, APIClient.fetch
needs to accept an ID type rather than an Int:
func fetch<Model: Fetchable>(_ model: Model.Type, id: Model.ID,
completion: @escaping (Result<Model, Error>) -> Void)
And of course Fetchable needs to add an ID type:
protocol Fetchable: Decodable {
associatedtype ID: Identifier
static var apiBase: String { get } // The part of the URL for this fetchable type
}
Wait a minute… There’s nothing “of course” about that last change. Fetchable used to be a simple protocol. Now it’s a PAT (protocol with associated type). That’s a big change in Swift. Whenever you find yourself typing associatedtype
, you need to stop for a moment and think “would I ever want to put this in an Array?” Once you put an associated type on a protocol in Swift today, it’s no longer a “thing.” It’s only a constraint that can be used for extensions and generic functions. It can’t put put in a variable, or be passed to a function, or in any other way be treated as a value.
Yes, someday generalized existentials will improve this in some cases. But before you pine for those days, or reach for a type-eraser, it’s time to think harder about the protocol.
I want to roll back to the Identifier protocol and ask that question, “would I ever want to put an Identifier in an Array?” I’ve used this protocol in production projects for a long time now, and the answer so far has been no. It just hasn’t come up. As I wrote this article, I tried to invent use cases that needed an Array of Identifiers, and each time the example kind of fell apart. I was always forcing it. But it’s worth walking through the thought process anyway.
If I try to create an Array of Identifiers today, it spits out that infamous error:
let ids: [Identifier] = [User.ID(1), Document.ID("password")]
// Protocol 'Identifier' can only be used as a generic constraint because it has Self or associated type requirements
And this it the point where you cry out “generalized existential!” But that wouldn’t actually change anything. Let’s just imagine that we have a generalized existential or I’ve written an AnyIdentifier type-eraser. Eventually I’m going to wind up with some loop over ids
:
for id in ids {
// ??? the only property is .value, which is an unknown type ???
}
I call this the “what now?” problem. The only thing I can do with id
is get its value, because that’s the only thing in the protocol. But each ID can have a different value type. So what can I do with it? Even with the fabled generalized existential, the type of .value
would have to be Any. What else could it be? I can’t call fetch
with that. I don’t even know the Model type.
“I don’t even know the Model type.” As I said, I’ve used this protocol in several projects and I’ve never needed a list of Identifiers, but as soon as I started writing this article, I realized how weird it is that an Identifier doesn’t know what type it identifies. Originally I was going to just rewrite this article to ignore it, but these kinds of…mistakes?…are important to explore. I hesitate to call it a mistake, because it’s never mattered in any shipping software I’ve worked on. If a type is solving your problems, it’s not wrong. But maybe it could be better.
Before I make it better, I want to show how to solve a “what now?” problem without changing Identifier. I know that sounds a little strange, but sometimes you inherit types that you can’t easily change, and it’s good to have lots of tools in your belt that don’t require rewriting half your project every time something is less than ideal. So let me walk through an example where you think you want to use an Array of Identifiers, but don’t.
Let’s say that once an hour I want to refresh all the model objects by re-fetching them. So I build a list of Identifiers to refresh, and get the “can only be used as a generic constraint” error, and now have to decide what to do. The answer is to look again at what I really want. I don’t want a list of Identifiers. I want a list of refresh requests. A refresh request is a future action, and a future action is closure. I typically like to wrap that closure into a type. Maybe something specialized to this problem like:
struct RefreshRequest {
// The delayed action to perform.
let perform: () -> Void
init<Model: Fetchable>(id: Model.ID,
with client: APIClient = APIClient.shared,
updateHandler: @escaping (Model) -> Void, // On success
errorHandler: @escaping (Model.ID, Error) -> Void = logError) // On failure, with a default
{
// Smash together updateHandler and errorHandler into a single () -> Void.
perform = {
client.fetch(Model.self, id: id) {
switch $0 {
case .success(let model): updateHandler(model)
case .failure(let error): errorHandler(id, error)
}
}
}
}
// Just a helper so errorHandler can have a default value
static func logError<ID: Identifier>(id: ID, error: Error) {
print("Failure fetching \(id): \(error)")
}
}
let requests = [
RefreshRequest(id: userID, updateHandler: { users[$0.id] = $0 }),
RefreshRequest(id: documentID, updateHandler: { documents[$0.id] = $0 }),
]
The point of all of this isn’t this specific data structure. It’s that () -> Void
is an incredibly powerful and flexible type, and you can construct it from all kinds of other functions. It’s another case of “common currency.” If you want a delayed action, that’s just a function. A lot of complicated generic code comes from trying to keep track of all the parameters to a generic function you intend to call later. You don’t need to keep track of parameters (and their types) if all you need is the function itself. This is the heart of type-erasure rather than focusing on type-erasers. It’s hiding types I don’t care about any more, like Model. Note in this example how two generic closures that rely on Model (updateHandler
and errorHandler
) are combined into a single () -> Void
, non-generic closure that relies on nothing. This is very common technique, and it’ll come up again in this series.
There are more improvements I could make here. The basic closure { someModel[$0.id] = $0 }
is going to be duplicated a lot and I could fix that. But I’m going to leave it for now and focus on a better identifier.
What I really want is the model type to know its ID type, and the ID type to know its model type. If you remember the APIClient.fetch
method, it takes both a type and an identifier:
func fetch<Model: Fetchable>(_ model: Model.Type, id: Model.ID, completion: @escaping (Result<Model, Error>) -> Void)
This creates awkward repetition in the API:
client.fetch(User.self, id: User.ID(1), completion: { print($0)} )
I could add an extra “Model” associated type to the Identifier protocol, but it gets a bit messy. Some of it is just Swift syntax (where
clauses and the like), but it really comes down to Identifier not being a very good protocol. Look at the implementations:
struct User.ID: Identifier { let value: Int }
struct Document.ID: Identifier { let value: String }
If you think about any other implementations, they’re going to be almost identical: a struct with a single property called value
. It’s hard to imagine any other way you’d want to implement this protocol. If every instance of a protocol conforms in exactly the same way, it should probably be a generic struct.
// An identifier (of some Value type) that applies to a specific Model type
struct Identifier<Model, Value> where Value: Codable & Hashable {
let value: Value
init(_ value: Value) { self.value = value }
}
extension Identifier: Codable, Hashable {
init(from decoder: Decoder) throws {
self.init(try decoder.singleValueContainer().decode(Value.self))
}
func encode(to encoder: Encoder) throws {
var container = encoder.singleValueContainer()
try container.encode(value)
}
}
Identifier has two type parameters. The Model is the type this identifier applies to. The Value is the kind of identifier it requires (Int, UInt64, String, etc). The Model isn’t actually used anywhere, but it means that Identifier<User, Int>
and Identifier<Document, Int>
are completely different types and can’t be mixed up.
So User becomes:
struct User: Codable, Hashable {
let id: Identifier<User, Int>
let name: String
}
That’s ok, but it’d be nicer to typealias it so I can refer to User.ID as a type:
struct User: Codable, Hashable {
typealias ID = Identifier<User, Int>
let id: ID
let name: String
}
And it can be even a little nicer if I extract a protocol, and apply it to Fetchable:
// Something identified with an Identifier
protocol Identified: Codable {
associatedtype IDType: Codable & Hashable
typealias ID = Identifier<Self, IDType>
var id: ID { get }
}
// Something that can be fetched from the API by ID
protocol Fetchable: Identified {
static var apiBase: String { get } // The part of the URL for this fetchable type
}
// User model object
struct User: Identified {
typealias IDType = Int
let id: ID
let name: String
}
extension User: Fetchable {
static var apiBase: String { return "user" }
}
And finally, fetch
doesn’t need any type parameters. The only thing that could be fetched with a User.ID is a User:
func fetch<Model: Fetchable>(_ id: Model.ID,
completion: @escaping (Result<Model, Error>) -> Void)
client.fetch(User.ID(1), completion: { print($0)} )
There are a lot of people who are a lot better than I am at this, and I’m sure they would have built this (or something better!) all at once on the first try. But I’m not bad at this stuff, and this is how it usually works for me. I want to stress that I’ve shipped the protocol version of Identifier successfully in several products, and have never run into a case where I actually wanted a more powerful Identifier that knew its Model. It’s just that by playing around (and thinking a lot about Brandon Williams’ excellent Protocol Witnesses talk) I discovered another approach.
Of course I’ve never actually shipped this Identified protocol. Maybe I’m wrong. Maybe it has quirks when you try to use it in real code. Maybe it turns out to awkward or limited for some reason. I won’t know until I ship it in a production project.
I’ll now remind you that stdlib’s Collection protocol required a pretty major overhaul in Swift 3, and tweaks in Swift 4.1 and Swift 5. The stdlib team is definitely better at this than I am, and Collection is probably the most foundational and carefully considered protocol in Swift. And still, it’s hard to get it right on the first try. (For another major example, see the four iterations of Protocol-oriented integers.)
Generic code is hard. There are trade-offs. Some things are hard because Swift is still evolving. And some things are hard because generic code is just hard. Build simply and concretely, and extract solutions as you discover problems. Don’t invent problems for yourself. “It isn’t generic enough” is not a problem. Make sure your generic code is solving a problem you really have, and put it off as long as you can get away with. You’ll probably have to redesign it anyway.
Next time I’ll move beyond fetching models. There are so many other things an API can do. What would that look like?
]]>If you’re interested in the future of generics in Swift, Joe Groff has a must-read post called Improving the UI of generics. (You should also read the linked Generics Manifesto for background.) In it, he touches on a common confusion in Swift. If you don’t understand what he’s talking about here, don’t worry. Explaining this paragraph is the point of this article.
We gave existential types an extremely lightweight spelling, just the bare protocol name, partially following the example of other languages like Java and C# where interfaces also serve as value-abstracted types, and partially out of a hope that they would “just work” the way people expect; if you want a type that can hold any type conforming to a protocol, just use the protocol as a type, and you don’t have to know what “existential” means or anything like that. In practice, for a number of reasons, this hasn’t worked out as smoothly as we had originally hoped. Although the syntax strongly suggests that the protocol as a constraint and the protocol as a type are one thing, in practice, they’re related but different things, and this manifests most confusingly in the “Protocol (the type) does not to conform to Protocol (the constraint)” error.
In programming languages, the “spelling” of something is the sequence of characters a programmer would type to represent a concept. This is often the most visible and argued-over part of a language. It’s also often a fairly shallow concern to the design, which is why it’s common to use intentionally bad “straw man” names to discuss a concept without getting bogged down in spelling. Consider the concept “true if x
or y
, otherwise false.” Swift spells that x || y
. In SML the same concept is spelled x orelse y
. But the spelling difference, the difference between the characters ||
and orelse
, isn’t very important. It doesn’t tell you much about how the language works. A more interesting difference, at least to me, is that ||
is a stdlib function in Swift, while orelse
is hard-coded into the SML compiler, which would likely be true no matter how they were spelled.
In English, some spellings have multiple meanings. The same thing happens in programming languages, and it happened in the last article:
final class AddHeaders: Transport { let base: Transport ... }
The spelling “Transport” has two related, but distinct, meanings. The first refers to the protocol Transport. The second refers to the existential of Transport.
The “existential of a protocol” can mean several things, but here it refers to a compiler-generated box that holds a value that conforms to the protocol. To see why Swift needs this box, consider an Array of Transports:
// URLSession and TestTransport both conform to Transport
var transports: [Transport] = [URLSession.shared, TestTransport(...)]
Swift would like to store Arrays contiguously in memory. So for an Array of Ints, the storage looks like this:
+---+---+---+
| 0 | 1 | 2 |
+---+---+---+
There are no pointers or indirection. The Ints are stored one after the other. To find the offset of index 2, you just have to multiply the size of an Int times two. That’s really fast and how you probably expect Arrays to work. Swift does the same thing for structs. It just lays them out field after field (there might be some padding, but that’s not important here).
struct S {
let a: Int
let b: Int
}
++--------+--------++--------+--------++--------+--------++
|| S[0].a | S[0].b || S[1].a | S[1].b || S[2].a | S[2].b ||
++--------+--------++--------+--------++--------+--------++
Again, to find the offset of S[2]
, Swift just has to multiply two times the size of S (which is the same as two Ints). But what happens in a “protocol-typed” Array like [Transport]
? Each element might be a different size. What can Swift do?
It makes a box that’s a fixed size (currently five machine words, with three for storage). If the type can fit in the box, then it’s stored in the box. If it can’t fit, then the compiler allocates some space, copies the data there, and puts a pointer in the box. Reference types are already pointers, so it just puts the pointer in the box. In Swift, that box is called an existential container. The thing in the box is called a witness.
See WWDC 2016: Understanding Swift Performance for more on the implementation details.
This section is a bit more technical; feel free to skip it if you like.
Why “existential?” Because the Transport protocol asserts that there exists some type that satisfies its requirements. By “some type,” I mean “in the universe of all possible types,” not “types that happen to be in your program.” That assertion may be wrong. It’s possible to define a protocol that nothing could ever conform to. For example:
protocol Impossible {
func make<A>() -> A
}
(If you don’t believe me, spend some time trying to implement make
. You need to return an instance of whatever the caller requests.)
An existential container is a placeholder box for some unknown type that satisfies the protocol. It’s possible there is no such type, or there may not be any such type in your program. Nothing can be done with it at runtime until a real, concrete value, a witness, is put in the box. The existence of a witness proves that such a type really does exist.
This implicit box isn’t the only example of an existential in Swift. The “Any” types like AnySequence, AnyHashable, and AnyKeyPath often get called “type-erasers” because they hide the concrete type, but they’re also explicit existentials. In future Swift, we may spell implicit existentials as any Transport
to parallel the explicit spelling.
While protocols create existential (“there exists”) types, generics create universal (“for all”) types. When you write struct Array<Element> {...}
, that’s an assertion that “for all types (Element), there is another type (Array<Element>) with the following attributes….”
Existentials and universals are “duals,” which means that one can be transformed into the other without losing its structure. So AnySequence is a universal type (generic) that’s equivalent to an explicit existential of Sequence (protocol). That’s why when you run into problems with protocols, your solution may be to convert it into generic structs (or vice versa). They solve the same problems in different ways with different trade-offs.
If you have a function with a parameter whose type is a protocol, that really means it requires an existential of that protocol.
protocol Transport { ... }
func transmit(data: Data, over transport: Transport) { ... }
In order to call transmit
with URLSession, Swift needs to copy the URLSession into an existential, and then pass that to transmit
.
What if you used a generic function instead?
func transmit<T: Transport>(data: data, over transport: T) { ... }
This says that the caller gets to decide the type of T. If they pass URLSession, then the compiler creates an implicit overload:
func transmit(data: Data, over transport: URLSession) { ... }
If somewhere else in the code they pass TestTransport, then the compiler creates another overload:
func transmit(data: Data, over transport: TestTransport) { ... }
The entire transmit
function is (in principle) copied, just as if you’d written an overload transmit
for each type. This is an over-simplification, and the compiler may not actually make all the copies, or it may generate an existential version instead (or in addition). It depends on a lot of things, including the optimization flags. But when you call a generic function, you should think of it as creating a new version of the function written specifically for the type you called it with.1
This run-time/compile-time distinction is a key difference between existentials and generics. Existentials are containers that are filled at run-time. Generics are compile-time functions for generating new code.
Existentials are used when you need to store heterogeneous values that are only known at run-time, for example in a heterogeneous collection.
Generics are used to apply algorithms to types that are known at compile-time. Protocols constrain what types can be used with those generics.
You don’t pass “a protocol value” to a function. You pass the existential of the protocol. Because Swift often converts concrete types into existentials for you, it’s easy to forget that they’re not the same thing. So when Swift doesn’t perform the conversion, it comes as a surprise, and we get the “can only be used as a generic constraint” (i.e. “as a protocol”) error.
So couldn’t Swift just create an existential all the time, even for protocols with associated types (PATs)? Yes, but…it’s complicated. For the most common cases, yes, Swift could automatically create an any Collection<.Element == T>
2 implicit existential just like it currently has an AnyCollection<T>
explicit existential. That idea is called generalized existentials, and I’m pretty certain Swift will add it eventually (maybe even soon). That’ll knock off several of protocols’ sharp edges for some of the most common cases.
But it probably won’t solve as many problems as people expect. Many protocol problems I see in the wild are really just design problems that have little to do with missing Swift features. A generalized existential will get you past the compiler error, but in the process it may let you go much further down a wrong road.
And there are many kinds of types that don’t lend themselves to automatically-generated existentials. The compiler can’t fulfill an init
requirement or any static
requirements on its own. It needs help from the programmer to determine what the default implementations are. It’s similar for protocols with a Self requirement. It may not always be possible to create a sensible default implementation. For protocols like Decodable that have no instance methods, an existential may not make sense at all.
As Joe said, the hope was that existentials wouldn’t really matter. They’re created by the compiler, you can’t access them, and you can’t even refer to them directly in the language today. You’d think they’d be an implementation detail. But sometimes when you type the name of a protocol you mean the protocol and sometimes you mean the box, and sometimes that matters. We’d like to ignore reference counting, too, and mostly we can…except when we can’t.
The point of a protocol is algorithms. Protocols express what a type must be able to do in order to be used in certain ways. Ideally, protocols should have a very small number of requirements, and enable a large number of extensions and generic functions. A good protocol is short, but shows up in a lot of where
clauses and extensions. They’re fundamentally about compile-time type concerns. “I want to apply this algorithm to many different concrete types.”
The point of an existential is heterogeneous collections, or “type-erasure” where you want to know less about the specific type and just use it according to an interface. They’re fundamentally about run-time values. “I want to assign values of many different concrete types to this variable.”
They’re not unrelated, but they’re not the same thing. When I say “protocols do not (generally) conform to protocols,” I really mean “existentials do not (generally) conform to protocols.” And when you see “can only be used as a generic constraint,” what the compiler is really telling you is that protocols with associated types (PATs) don’t have an existential.
My hope is that after reading all this, you’ll feel more comfortable reading SE-244, which adds opaque return types in Swift 5.1. I don’t expect opaque return types to be an important feature for most developers. Please don’t assume you need to rewrite your code to use them. The problems they solve impact stdlib much more than day-to-day app development in my opinion. Looking over my code, I haven’t found a single place I want to use one.
The importance of SE-244 isn’t opaque return types. It’s that it lays the groundwork for the future of Swift generic code. If that interests you, then you definitely want to study it, and in particular get comfortable with any P
(an existential) versus some P
(an unknown but concrete type that conforms). Hopefully this article demystifies some of the terminology.
Next time, back to the networking stack and hopefully some more practical concerns.
All these “may” qualifiers are why you shouldn’t assume that protocols or generics are “better for performance” (for whatever meaning you’re attaching to “performance”). It depends on a lot of things. If your code is sensitive to the performance of generics or protocols, you need to profile it and look at what the compiler is actually generating. Do not take away from this discussion that “generics are faster” or “protocols create smaller binaries.” That might be true in certain cases, but it can also be the other way around. Write you code clearly and correctly, and say what you mean in types. The Swift compiler teams works very hard to make sure that kind of code will be performant. Don’t guess what the compiler will do. Test. ↩
For an introduction to that proposed syntax, see the Protocol<.AssocType == T> shorthand forum thread. ↩
// Something that can be fetched from the API
protocol Fetchable: Decodable {
static var apiBase: String { get } // The part of the URL for this type
}
// A client that fetches things from the API
final class APIClient {
let baseURL = URL(string: "https://www.example.com")!
let session: URLSession = URLSession.shared
// Fetch any Fetchable type given an ID, and return it asynchronously
func fetch<Model: Fetchable>(_ model: Model.Type, id: Int,
completion: @escaping (Result<Model, Error>) -> Void)
{
// Construct the URLRequest
let url = baseURL
.appendingPathComponent(Model.apiBase)
.appendingPathComponent("\(id)")
let urlRequest = URLRequest(url: url)
// Send it to URLSession
let task = session.dataTask(with: urlRequest) { (data, _, error) in
if let error = error {
completion(.failure(error))
} else if let data = data {
let result = Result { try JSONDecoder().decode(Model.self, from: data) }
completion(result)
}
}
task.resume()
}
}
This can decode any Fetchable model from an API endpoint that has a URL something like https://<base>/<model>/<id>
. That’s pretty good, but we can do a lot better. A natural first question is “how do I test it?” It relies explicitly on URLSession, which is very hard to test against. A natural approach would be to create a protocol to mock URLSession.
I hope by the time you’re done with this series, hearing “create a protocol to mock” makes you flinch just a little.
The basic premise of a mock is to build a test object that mimics some other object you want to replace. That encourages you to design a protocol that very closely matches the existing interface, and then your “mock object” will also closely match that interface. This makes your “real” object, the protocol, and the mock evolve tightly in lockstep, and it cuts off opportunities for more powerful protocols that aren’t tied to one implementation. If the only reason you can imagine using a protocol is for testing, then you’re not getting all you could out of it. Protocols can be so much more.
So my goal isn’t to “mock” URLSession, but to abstract the functionality I need. What I want is to map a URLRequest to Data, asynchronously:1
// A transport maps a URLRequest to Data, asynchronously
protocol Transport {
func send(request: URLRequest, completion: @escaping (Result<Data, Error>) -> Void)
}
Notice that nothing about that says “HTTP server over the network.” Anything that can map a URLRequest to Data asynchronously is fine. It could be a database. It could be static unit test data. It could be flat files. It could be different routes depending on the scheme.
Now comes the power of retroactive modeling. I can extend URLSession to be a Transport:
extension URLSession: Transport {
func send(request: URLRequest, completion: @escaping (Result<Data, Error>) -> Void)
{
let task = self.dataTask(with: request) { (data, _, error) in
if let error = error { completion(.failure(error)) }
else if let data = data { completion(.success(data)) }
}
task.resume()
}
}
And then anything that requires a Transport can use a URLSession directly. No need for wrappers or adapters. It just works, even though URLSession is a Foundation type and Apple doesn’t know anything about my Transport protocol. A few lines of code and it just works, without giving up any of the power of URLSession.
With that in place, APIClient
can use Transport rather than URLSession.
final class APIClient { let baseURL = URL(string: "https://www.example.com")! let transport: Transport init(transport: Transport = URLSession.shared) { self.transport = transport } // Fetch any Fetchable type given an ID, and return it asynchronously func fetch<Model: Fetchable>(_ model: Model.Type, id: Int, completion: @escaping (Result<Model, Error>) -> Void) { // Construct the URLRequest let url = baseURL .appendingPathComponent(Model.apiBase) .appendingPathComponent("\(id)") let urlRequest = URLRequest(url: url) // Send it to the transport transport.send(request: urlRequest) { data in let result = Result { try JSONDecoder().decode(Model.self, from: data.get()) } completion(result) } } }
By using a default value in init
, callers can still use the APIClient()
syntax if they want the standard network transport.
Transport is a lot more powerful than just “a URLSession mock.” It’s a function that converts URLRequests into Data. That means it can be composed. I can build a Transport that wraps other Transports. For example, I can build a Transport that adds headers to every request.
// Add headers to an existing transport
final class AddHeaders: Transport
{
let base: Transport
var headers: [String: String]
init(base: Transport, headers: [String: String]) {
self.base = base
self.headers = headers
}
func send(request: URLRequest, completion: @escaping (Result<Data, Error>) -> Void)
{
var newRequest = request
for (key, value) in headers { newRequest.addValue(value, forHTTPHeaderField: key) }
base.send(request: newRequest, completion: completion)
}
}
let transport = AddHeaders(base: URLSession.shared,
headers: ["Authorization": "..."])
Now, rather than having every request deal with authorization, that can be centralized to a single Transport transparently. If the authorization token changes, then I can update a single object, and all future requests will get the right headers. But this is still unit testable (even the AddHeaders part). I can swap in whatever lower-level Transport I want.
This means I can extend existing systems in a really flexible way. I can add encryption or logging or caching or priority queues or automatic retries or whatever without intermingling that with the actual network layer. I can tunnel all the network traffic over a custom VPN protocol (I’ve done exactly that with a system like this), all without losing the ability to unit test. So yes, I get mocks, yes, I get unit testing, but I get so much more.
For completeness, here’s a “mock” Transport, but it’s probably the least interesting thing we can do with this protocol.
// A transport that returns static values for tests
enum TestTransportError: Swift.Error { case tooManyRequests }
final class TestTransport: Transport {
var history: [URLRequest] = []
var responseData: [Data]
init(responseData: [Data]) { self.responseData = responseData }
func send(request: URLRequest, completion: @escaping (Result<Data, Error>) -> Void) {
history.append(request)
if !responseData.isEmpty {
completion(.success(responseData.removeFirst()))
} else {
completion(.failure(TestTransportError.tooManyRequests))
}
}
}
And I still haven’t used an associated type or a Self requirement. Transport doesn’t need any of that. It’s not even generic.
The split between a APIClient.fetch
, which is generic, and Transport.send
, which is not, is a common structure that I look for. Transport.send
operates on a small set of concrete types: URLRequest in, Data out. When you’re working with a small set of concrete types, then composition is easy. Anything that can generate a URLRequest or can consume Data can participate. APIClient.fetch
converts Data into any kind of generic Fetchable. When angle-brackets and associated types start creeping in, the code becomes more expressive, but harder to compose because you have to make sure all the types line up.
The power of the Internet is that it mostly operates on just one type: the packet. It doesn’t care what’s in the packet or what the packet “means.” It just moves packets from one place to another; packets in, packets out. And that’s why the Internet is so flexible, and the equipment that makes it work can be implemented by numerous vendors in wildly different ways, and they can all work together.
At each layer above the network layer, additional context and meaning is applied to the information. It’s interpreted as user information or commands to execute or video to display. That’s composition, gluing together independent layers, each with their own concerns. When designing protocols, I try to employ the same approach. Particularly at the lowest layers I look for common, concrete types to work with. URL and URLRequest. Data and Int. Simple functions like () -> Void
. As I move up the stack, then greater meaning is applied to the data in the form of model types and the like. That means it’s easy to write Transports and many different things can use Transports. And that’s the goal.
This network stack still is nowhere near as flexible and powerful as I want. But now it can fetch a wide variety of model types from a particular type of API in a very composable and testable way. That’s great progress. For some very simple APIs, it might even be done. There’s no need to make it more flexible for its own sake. But I think we’ll quickly find more features we need to add.
Next time, I’ll jump back up to the very top of the stack, to the models, and show where a PAT (protocol with associated type) can really shine.
Throughout this series, whenever it’s unambiguous, I’ll refer to Result<Value, Error>
as just “Value.” ↩
I need a new protocol.
protocol Fetchable: Decodable {
static var apiBase: String { get }
}
I need a protocol that requires that the type be Decodable, and also requires that it provide this extra string,
apiBase
.
Read that again. It requires that the type be Decodable and also requires other things. I didn’t say that Fetchable is Decodable. It isn’t.
Protocols (with a few exceptions) do not conform to protocols, not even to themselves. A type that conforms to Fetchable, must also conform to Decodable, but Fetchable is not Decodable. Fetchable is not Fetchable. Decodable is not Decodable. Why do I keep repeating this. Because you will forget, and it will bite you. What would it mean if Decodable were Decodable?
func decode<T>(_ type: T.Type, from data: Data) throws -> T where T : Decodable
Well, remember that JSONDecoder’s decode
method requires a type that conforms to Decodable. If Decodable (or Fetchable) conformed to Decodable, I could write:
let result = try JSONDecoder().decode(Decodable.self, from: data)
And in fact, I see people try to write that all the time. But how could that possibly work? How can JSONDecoder know which of an unbounded number of possible types you want this JSON to be decoded into? Even if you did it, what could you possibly do with result
? It’s only known method would be init(from: Decoder)
. There are no instance methods on Decodable.
And so again: Protocols do not conform to protocols, not even to themselves.
When I say this bites people all the time, here’s a common example: Say you have a protocol and an extension on Array:
protocol MyProtocol {...}
extension Array where Element: MyProtocol {
func doThing() { ... }
}
And then you have some array of MyProtocol:
let things: [MyProtocol] = ...
You might imagine that you could call things.doThing()
. After all, doThing()
applies to any array of MyProtocol, and what’s more MyProtocol than MyProtocol? But that’s exactly what you can’t do. The syntax where Element: MyProtocol
means “Element conforms to MyProtocol.” And as I’ve repeated many times now: MyProtocol does not conform to itself. In order to add an extension on [MyProtocol]
, you would need to write an extension with ==
rather than :
.
extension Array where Element == MyProtocol {
func doThing() { ... }
}
This says that Element is exactly MyProtocol. That doesn’t include types that conform to MyProtocol. It only applies exactly to [MyProtocol]
.
OK, what about the exceptions? There are a some protocols that do conform to themselves. @objc
protocols do unless they have “static” requirements such as init
, or static properties or methods. And in Swift 5, Error conforms to itself so that you can have “untyped error” Results like Result<T, Error>
. If Error didn’t conform to itself, you’d have to use a concrete type for the error. But these are compiler-enforced special cases. You can’t make your protocol conform to itself.
But could they? Yes, some could in principle. The rule is pretty straightforward: if a protocol includes an init
or static
requirement, or includes a Self
method parameter, then self-conformance is tricky. If there is no such requirement, then it it’s much more straightforward (basically the same as for @objc
). There’s no deep reason that Encodable can’t be Encodable. The following could work, and I think would be both sensible and useful, it just doesn’t today:
let encodables: [Encodable] = ...
let json = try JSONEncoder().encode(encodables)
Will this ever work? I don’t know. It’s been brought up a few times on Swift Evolution, and hasn’t been rejected outright. One concern is that adding an init
requirement to an existing protocol could break existing usage (possibly in downstream code) in ways that might surprise developers. I haven’t found a clear statement, but it seems the team wants to make this work someday.
It’s even possible that “challenging” protocols could self-conform if there were default implementations. One could imagine a Swift where Collection(1, 2, 3)
would return an Array in a Collection existential. (I’m not suggesting that would be a good idea; I really don’t know. It’s just that it’s the kind of thing one could imagine.)
In this series I’m generally going to talk about things I know from experience using today’s Swift or can predict about likely-near-term Swift (i.e. there’s an SE in the works). So any time I say something like “that won’t work,” I mean “without adding a significant feature to Swift that I don’t know is planned.” (Hopefully folks will continue to correct me if I’m misleading about how hard something would be.)
I want to talk about this more later, but when I say “a protocol doesn’t conform to itself,” it’s more accurate to say “the existential of a protocol doesn’t conform to that protocol.” But again, that’s for a later sidebar…. The thing to keep in mind is that these two things are different:
func f<T: P>(t: T) // This requires a concrete T that conforms to P
func f(p: P) // This requires a variable of type P (pedantically: "a P existential")
So that’s just a quick side-bar. Next time, I’ll continue expanding the network stack.
]]>In 2015, at WWDC, Dave Abrahams gave what I believe is still the greatest Swift talk ever given, and certainly the most influential. ”Protocol-Oriented Programming in Swift,” or as it is more affectionately known, “The Crusty Talk.”
This is the talk that introduced the phrase “protocol oriented programming.” The first time I watched it, I took away just one key phrase:
Start with a protocol.
And so, dutifully, I started with a protocol. I made a UserProtocol and a DocumentProtocol and a ShapeProtocol and on and on, and then started implementing all those protocols with generic subclasses and eventually I found myself in a corner.
Protocol 'P' can only be used as a generic constraint because it has Self or associated type requirements
And then I started throwing things.
For a couple of years, I was rather annoyed at the phrase “protocol-oriented programming.” If by “protocol” you just mean “interface,” then Go is much more “protocol oriented” than Swift. But the more I’ve wrestled with this new paradigm, the more I realized that protocols are more than just interfaces, and POP isn’t deeply about the protocols anyway. It’s about the extensions. But “extension-oriented programming” would be an even worse name. And more than extensions, it’s really, deeply, about generic algorithms. And “algorithm-oriented programming,” well, aren’t we all?
Naming a paradigm is always fraught with trouble. Most modern “object-oriented” languages aren’t object-oriented at all. They’re class-oriented (vs Smalltalk and JavaScript). And most “functional programming” languages are mostly value-oriented (vs FP and point-free). But the point of the names is shorthand for concepts bigger than a word, so let’s not get too caught up on the “protocol” in protocol-oriented programming. The Holy Roman Empire was in no way holy, nor Roman, nor an empire. Discuss.
The famous “start with a protocol” quote is actually the end of a longer paragraph:
For example, if you want to write a generalized sort or binary search…Don’t start with a class. Start with a protocol.
Or as Dave clarified on Twitter:
Use value types, then if you need polymorphism, make them conform to protocols. Avoid classes.
If you’re reaching for class inheritance, try a protocol and value type instead. That’s very different from “start with a protocol for every problem.” Ben Cohen covered this in much more detail in the WWDC 2018 talk Swift Generics (Expanded).
So notice that we considered a varied number of concrete types first. And now, we’re thinking about a kind of protocol that could join them all together. And, it’s important to think of things as this way around. To start with some concrete types, and then try and unify them with a protocol.
If you take away just one thing from this series, I want it to be this: Write concrete code first. Then work out the generics. Start with concrete types and clear use cases, and find the places that duplication happens. Then find abstractions. The power of protocol-oriented programming is that you don’t have to decide when you create a type exactly how it will be used. When you use class inheritance, you have to design your class hierarchy very early. But with protocols, you can wait until later.
When I most get into trouble with protocols is when I try to write code “as generically as possible.” That doesn’t really mean anything. Abstractions are choices, and when you make a choice to be flexible in one direction, you generally make it harder to be flexible in other directions. Without some clear use cases, you don’t know what abstractions make sense.
So today, I want to come to protocol-oriented programming fresh, with a focus on very every-day problems we face when developing iOS apps in Swift.
Over the next several articles I’ll be developing a very common system, a general-purpose networking stack that can fetch data asynchronously and decode arbitrary types. You may have built a system like this yourself in Swift. You may have used a framework that does it. The point of this exercise isn’t really the end result (though I think it’s quite useful code), but the process. What questions should you ask, and when, and how do you know what good answers look like? And most importantly, how does this “protocol-oriented” thing guide us? How is it different than other approaches?
I expect that you’re somewhat familiar with Swift, and particularly that you understand the syntax of generic functions and types, and have at least seen an associatedtype
before. If you’re just getting started in Swift, maybe bookmark this series for later.
So to get started, I want to show a common starting point that never goes well for me. I’ve made this mistake many times, and I always find myself in a corner eventually. I see a lot of other people make this mistake, too.
// A network Request knows the URLRequest to fetch some data, and then can parse it.
// This will not go well.
protocol Request {
associatedtype Response
func parse(data: Data) throws -> Response
var urlRequest: URLRequest { get }
}
How do I know this won’t go well? I’ll discuss it much more in depth later, but Request is a protocol with associated type (PAT). Any time you create a PAT, you should ask yourself “will I ever want to put this in an Array?” If the answer is yes, you don’t want a PAT. Requests are certainly something you’d want to put in an Array. Lists of pending requests, lists of requests that need to be retried, request priority queues. There are lots of reasons to put a Request in an Array.
You might be tempted to look for a work-around, but don’t. Type-eraser? No. Generalized Existential?!?! …no… Even if you find some “work-around” to the problem at hand you’ll run into other walls very quickly (and I’ve seen that again and again). That “can only be used as a generic constraint” is telling you something important. This isn’t a problem with Swift. This just isn’t what PATs are for. There are other tools for this problem. In later articles I’ll explain why you don’t want these work-arounds, but the basic problem is starting with a protocol before we even know what algorithm we want to write.
So what does “know the algorithm” look like in practice?
A good way to find a generic algorithm is to start with several concrete algorithms, and then make a parameter out of what varies. In this case, I want to fetch several model types from an API and decode them. In order to start concretely, I’ll make some actual types.
struct User: Codable, Hashable {
let id: Int
let name: String
}
struct Document: Codable, Hashable {
let id: Int
let title: String
}
This may not be our final implementations, but they’re good enough to get started. They’re pretty similar, but not identical, and that’s good for the first concrete types. I’ll want to push the envelope a bit more later, but this is good enough for now.
I also want a client to manage my connection to the server. I’m marking classes “final” to remind you that there’s no class inheritance here. I’m not suggesting you need to include “final” on all your class definitions. It’s not usually necessary. I’m making it a reference type because the client might eventually have some shared state. For example, if a login step were required, I’d want all references to the client to be logged in together.
// A client that fetches things from the API
final class APIClient {
let baseURL = URL(string: "https://www.example.com")!
let session = URLSession.shared
// ... methods to come ...
}
And now I want the code to fetch and decode a User, as a method on APIClient.
func fetchUser(id: Int, completion: @escaping (Result<User, Error>) -> Void)
{
// Construct the URLRequest
let url = baseURL
.appendingPathComponent("user")
.appendingPathComponent("\(id)")
let urlRequest = URLRequest(url: url)
// Send it to the URLSession
let task = session.dataTask(with: urlRequest) { (data, _, error) in
if let error = error {
completion(.failure(error))
} else if let data = data {
let result = Result { try JSONDecoder().decode(Model.self, from: data) }
completion(result)
}
}
task.resume()
}
I’m sure many of you have written code kind of like this many times. Construct a URLRequest. Fetch it. Parse it. Pass it to the completion handler. Now, what does the code for fetchDocument
look like?
func fetchDocument(id: Int, completion: @escaping (Result<Document, Error>) -> Void) { // Construct the URLRequest let url = baseURL .appendingPathComponent("document") .appendingPathComponent("\(id)") let urlRequest = URLRequest(url: url) // Send it to the URLSession let task = session.dataTask(with: urlRequest) { (data, _, error) in if let error = error { completion(.failure(error)) } else if let data = data { let result = Result { try JSONDecoder().decode(Document.self, from: data) } completion(result) } } task.resume() }
Unsurprisingly, fetchDocument
is almost identical except for four changes: the function name, the type to pass to the closure, the URL path, and the type to decode. It’s so similar because I copied and pasted it. And when you find yourself copying and pasting, that’s where I know there’s probably some reusable code. So I extract that into a generic function:
func fetch<Model: Decodable>(_: Model.Type, id: Int,
completion: @escaping (Result<Model, Error>) -> Void)
{
...
}
Before going on, it’s worth exploring the signature. Notice that I pass the type of Model as a parameter. It doesn’t even need a name, because the value won’t be used. It’s just there to nail down the type parameter in the function’s parameters rather than in completion handler’s parameters. I’m mostly doing this to show a technique, and because fetch(2) { ... }
is a bit ambiguous to the reader (since all ID types are Int currently). Sometimes this makes sense, sometimes it doesn’t.
A good example where I think it makes a lot of sense is JSONDecoder’s decode
method. It’s called this way:
let value = try JSONDecoder().decode(Int.self, from: data)
It could have been designed this way instead:
let value: Int = try JSONDecoder().decode(data)
It would have even been a little shorter that way. But it forces the caller to add a type annotation on the variable, which is a little ugly, and unusual in Swift. If the only place the type parameter shows up is in the return value, I usually recommend passing it as a parameter. But in any case, try writing some code with it, and focus on making things clear at the call-site. 1
fetch
genericImplementing fetch
is pretty straightforward, except for one small problem:
func fetch<Model>(_ model: Model.Type, id: Int, completion: @escaping (Result<Model, Error>) -> Void) where Model: Fetchable { // Construct the URLRequest let url = baseURL .appendingPathComponent("??? user | document ???") .appendingPathComponent("\(id)") let urlRequest = URLRequest(url: url) // Send it to the URLSession let task = session.dataTask(with: urlRequest) { (data, _, error) in if let error = error { completion(.failure(error)) } else if let data = data { let result = Result { try JSONDecoder().decode(Model.self, from: data) } completion(result) } } task.resume() }
There’s this string that’s either “user” or “document”. That’s something that this algorithm requires, but isn’t part of Decodable. So Decodable isn’t powerful enough to implement this. I need a new protocol.
// Something that can be fetched from the API
protocol Fetchable: Decodable {
static var apiBase: String { get }
}
I need a protocol that requires that the type be Decodable, and also requires that it provide this extra string, apiBase
. (See Protocols are nonconformists for more on the difference between “requires Decodable” and “is Decodable.”) With that, I can finish writing fetch
:
// Fetch any Fetchable type given an ID, and return it asynchronously
func fetch<Model>(_ model: Model.Type, id: Int,
completion: @escaping (Result<Model, Error>) -> Void)
where Model: Fetchable
{
let url = baseURL
.appendingPathComponent(Model.apiBase)
.appendingPathComponent("\(id)")
let urlRequest = URLRequest(url: url)
let task = session.dataTask(with: urlRequest) { (data, _, error) in
if let error = error { completion(.failure(error)) }
else if let data = data {
let result = Result { try JSONDecoder().decode(Model.self, from: data) }
completion(result)
}
}
task.resume()
}
Now to use it, I need to make User and Document conform to Fetchable.
extension User: Fetchable {
static var apiBase: String { return "user" }
}
extension Document: Fetchable {
static var apiBase: String { return "document" }
}
These tiny extensions represent one of the most powerful, and easiest to overlook, aspects of protocol-oriented programming: retroactive modeling. It is quite non-obvious that I can take a type like User that wasn’t designed to be Fetchable, and make it Fetchable in an extension. And that extension doesn’t even have to be in the same module. That’s not something you can typically do with class inheritance. You need to choose a superclass when you define a type.
I can take any type I want and conform it to my own protocols to use it in new and more powerful ways that the original type creator may never have thought of. There’s no need to tie User to this one use case and this one API. That’s why this protocol is called Fetchable rather than something like Model. It isn’t a “model.” It’s “something that can be fetched” and it only provides the methods and properties that allow that. I’m not suggesting that you should create a protocol for every use case, just the opposite. Really good protocols are usable by many algorithms. But you want most uses of the protocol to need most of the requirements. If the protocol is just a copy of the type’s entire API, it’s not doing its job. I’ll talk about that more in later articles.
I know this has been basic so far. I know many of you “know all this.” This article is a warm-up, and the point of the exercise is not what was built, but how it was built. I started with simple, concrete code, and extracted first a generic function, and then a simple (no associated type) protocol. This is exactly the opposite of starting with a Request PAT and then trying to figure out the callers. This was just the first step. This system is nowhere near as flexible and powerful as it could be, but already it’s meeting the goal I set at the beginning: “fetch a several model types from an API and decode them.” Keep the current goal in mind and don’t let the protocols get out in front of you.
Next time, I’ll push this example further, and start seeing what protocol oriented programming can really accomplish. Eventually I’ll even need a PAT!
(I don’t have comments on this site, but if you’re interested in any conversations about it, follow the thread on Twitter.)
A previous version of this post advocated for this approach much more strongly, but some questions on Twitter made me rethink this. ↩
Originally, I wasn’t comfortable being a full “Conditional.” I’m not much of a guitarist. I can play along reasonably if there’s someone to cover my mistakes, and a group probably sounds slightly better with me than without. But listening to me play alone is an act of love and friendship, not something you’d do on purpose. So I kind of wanted to be called a “Provisional Breakpoint” instead. But that was wrong. Someone who’s played at a Breakpoint Jam is a Conditional Breakpoint. That’s what it means. If you’ve done it, you’ve earned it.
Everyone who played guitar that night was a Conditional, but Ellen Shapiro was clearly our leader. She plays with an energy and style that I want to emulate. She’s much better than I am, but what I’ve played in years, she’s played in decades, so maybe that’s natural. Choosing the right people to copy is a worthwhile skill in itself. There’s nothing wrong with being a beginner. There’s nothing wrong with learning and flailing and trying again. There’s nothing wrong with letting someone mentor you. And there are a lot of ways to mentor. You might not even know when you’re doing it.
When I first asked to play with the Breakpoints a few weeks ago, James said yes, and I immediately panicked a little on the inside, and I asked for the whole setlist so I could practice. And he said sure and sent them. But then he said, you know, it’s fine to just play some of the songs, or even drop out of parts if you’re not up for it. It’s better to have a few songs you’re good at than a bunch you stumble through. And I said yeah, yeah, yeah, I know that. I’m cool. And I did kind of know. But not really. I’d been planning to practice them all and just beat myself up a lot everywhere I stumbled. I needed someone who knows more than me to give me permission to be a beginner, but still let me play. Sometimes you’ll never know the impact of a small kindness.
After the show, Eric Knapp asked me a very useful question. “So, that was your dream, and now you’ve done it. What’s your next dream?” It’s easy to lose your direction when you get to a destination. If you want to keep growing, your goals have to grow with you. My next goal is to be good enough that I can play alone and you’d rather stay than leave, even if you’re not my friend. It’s what I call “a mediocre guitarist.” Eric thought it was a good goal, but suggested a more concrete one: Play one song at an open mic night. And he passed on some wisdom he’d received and I needed. “Don’t wait until you’re ready. Go play before you’re ready. There are lessons you can only learn by doing.” Eric has forgotten more about playing than I know (and I don’t believe I’m exaggerating). It’s good advice.
I talk a bit about learning guitar and make a bit of noise about being a beginner. Some of that is because I love to tell stories, and I hope my stories can help other people. But of course saying “I’m a beginner” lowers expectations and overdone is a cheap way of getting praise. It’s a dangerous thing to make too big a deal about. For all my “I’m a beginner and I’m scared,” Laura Savino has been playing guitar for less time than I have, and this wasn’t her first Breakpoint Jam. So sometimes the brave ones don’t make as much noise. You have to look or you’ll never notice.
I may just have been lucky so far, but I’ve found the guitar community to be incredibly welcoming. James isn’t alone in inviting beginners to play, but the Breakpoints has a special history of it. Except for Ellen, the rest of us (Laura, myself, and Josh Smith) all made our first public performances at Breakpoint Jams. Creating that kind of opportunity is a gift and how you create and sustain a community. If we all “only hire the best,” where do you think the next generation will grow? Thankfully, James is willing to play with folks who are just trying their best. Are we as willing to develop software the same way?
There are senior developers and there are junior developers, and there are developers in between. Different people have different experience and skill. You might be a senior developer in one language or platform or style, and just learning another. But there’s no point talking about “real” developers. If you write software and it runs, you’re a real developer. That’s what it means. You develop software. If you’ve done it, you’ve earned it.
And if you develop software in Cocoa, then you really want to listen to Backtrace. Hope to see you at the next show to benefit App Camp For Girls!
]]>The conversation started by referencing the classic Joel piece, Things You should Never Do, Part I. Leading to my thoughts:
Just finished some major refactoring work, moving ObjC to Swift and completely redesigning its state machine. I absolutely stand behind the pieces I rewrote (which were a constant source of subtle race conditions and bugs, with every fix causing two new problems). I absolutely stand behind the pieces that I have delayed rewriting (which are a spaghetti mess, and incredibly difficult to safely modify, but after some minor tweaks are stable enough to leave alone).
I’m a big fan of “radical refactoring.” I’ve refactored several code bases until there was almost nothing left of the original code. But it was done steadily, only doing major rewrites to individual pieces after painstakingly detangling them from the rest of the code (usually over the course of several releases). And at the end, there was always some “ball of mud” part that was a bit crazy, but just worked and didn’t need to be touched that often, so we let it be.
I’ve even refactored a C program into a Go program, by turning it into two independent processes that communicated over sockets, and moving bits from one side of the API to the other.
(So even “we need to switch languages/platforms entirely” doesn’t stop you from evolving towards a goal.)
But there’s an exception that Joel doesn’t mention (but I think Martin Fowler does): if you have incredibly buggy code, that is, if you don’t have working code, then that’s the time to consider a rewrite. Not ugly code. Not badly designed or horrible to work with code. But code that doesn’t actually work, and several attempts to make it work have failed. That’s when a rewrite (at least of those portions) is likely appropriate.
The discussion then turned to unit testing, and particluarly Forgotten Refactorings.
Having had some very successful radical refactors on code without solid unit test coverage, I think it’s worth discussing how that can be done.
First, unit test coverage is absolutely the best first step. That said, sometimes it is impossible in any meaningful way. When all the most likely and common bugs in your system are race conditions and corner cases involving things outside the program (non-trivial networking, bluetooth, version-specific OS interactions, complex animations, etc), I’ve found unit tests rapidly become tests of mocks, and not tests of the system. We can debate whether or not it is possible or profitable to redesign your system so it is more testable. I’ll even concede that it is and leave arguments about TDD for another day (I’m actually a fan of TDD). But redesigning for testablity will itself require massive refactoring without unit tests (because you can’t unit test until you make it testable). Even if you have lots of tests, refactoring often means changing the tests dramatically (which means you’re not really testing the same thing). So at some point, you’re going to find yourself needing to refactor without perfect (or even barely sufficient) unit tests. How do you do it?
Slow down.
I cannot stress this enough. Slow. Down. Expect your refactor to take many releases. Do a small piece of refactoring, and run it through a full QA cycle (whatever that means for you) and ship it. Do it again and again. My “convert a C project to Go” project included a release where we just shipped the Go code alongside the C code, without even calling the Go code, just to prove it would install and not break anything. Then we built one, tiny, new feature in the Go code. It was so minor and impacted so few users, we were ready to declare it unsupported if it didn’t work. We’d been working on the Go code for almost two years before we cut over to it “for real” (and the vast majority of the code was still in C at that point). But at each step along the way, the system was better, and saner, and more reliable. And at each step along the way, it shipped, and got real field exercise. And we built a lot of tests for it, and we still found bugs that we were unable to build automated tests for. “Fails to determine domain on Mac previously joined to AD domain, but then removed, only on OS X prior to 10.8” or “SMB connection fails to Window 2000 server if username contains space” or “fails to determine correct IP address on Mac with case-sensitive file system if on Cisco VPN.” That kind of stuff.
Second point that goes along with this is to keep your refactor steps contained. I’ve had so many experimental refactor branches that I threw away because they spiraled out of control and touched too many pieces of the system in non-trivial ways. Don’t be afraid to throw away several attempts at refactoring until you can get your change focused enough that the risk is contained. Sometimes that means creating “firebreaks,” an object that wraps the thing you’re refactoring and provides the old API for code you don’t want to touch yet. Creating a firebreak often starts as just a pass-through that does nothing but call methods on the original. Tedious, but often invaluable. They make it possible to move to your new API piece by piece rather than having to touch half the system in one go.
I strongly recommend keeping your commits very focused. “Rename FooAdapter to Foo” should be its own commit. Don’t mix it with changes to API. “Rename X to Y” commits are really easy to code review, even if they touch hundreds of files. But if you also changed logic in there, then it’s a monster. Similarly, anything that is an easy win with little risk (like naming things sanely, or moving some duplicated code into a function), do those first and get them into the main code base. That way, when you discover that your ambitious new design is out of control and have to start over, you don’t lose your easy wins.
Testing is great. Testing is critical. Testing is necessary. But unit testing is not sufficient. And when there are hundreds of test cases that need to be rewritten, they can be a hindrance to refactoring. The more important rule in my experience is go slow and steady and keep shipping.
And yes. Write your unit tests. We’re professionals here.
]]>There’s a huge difference when you compare a talk from someone that has been working a lot on the topic and from someone that studied the topic for giving a talk. Why do people do it then? Talks with a lot of value usually come from unknown people. From these people that from the anonymity worked on a topic and they achieved something that they were willing to share. … People don’t care about the company that person had worked for, or the newsletter that the person had written, but instead, what that person wants to share.
While I agree with Pedro’s concern, I disagree that this is the proper ideal.
I’ve learned more about programming from listening to Daniel Steinberg talk about baking cookies than I have from a dozen talks from intelligent, highly experienced people who don’t have his skill for teaching. There are a number of speakers whose sessions I will attend no matter their topic because it’ll always be worth the time. I get inspired every time I listen to Jaimee Newberry, even though almost everything about her life and style is different than mine. If I watch Chris Eidhof live-code, I know I’m going to see some amazing idea that forces me to rethink something in my code.
I absolutely care who’s giving the talk. And if we’re going to ask people to spend hours or days of their time and hundreds or thousands of dollars to attend conferences, I think we owe the best we can offer.
But while I disagree with some of Pedro’s analysis, he raises a very important point. A community with many teachers and broad sharing is better than one with few teachers and a hierarchical flow of knowledge. How do we improve?1
We should recognize that there’s nothing wrong with having a core group of known, skilled speakers who draw a crowd. We should celebrate that. We should grow that group, and we should use them to make things even better.
At dotSwift, Daniel Steinberg reached out to all the speakers and offered assistance refining their talks. Conference organizers should encourage and facilitate that kind of mentorship, and use it to reduce the risks of inexperienced speakers. I hope conference organizers speak to each other and share names of promising speakers whom they didn’t have room for. CocoaHeads and other local groups should strive to video their local talks and make them available. Conference organizers should use those to find and contact promising speakers outside the usual suspects. Local lightning talks have a very low barrier to entry and make an excellent way to get into speaking. Not everyone has a local group, but it’s a start.
For those without a local group to video them, I always recommend blogging. Learning to write well is a major part of learning to speak well, and the barrier to entry for blogs is lower. A blog doesn’t have to be constantly updated to be helpful. A single, well-written article can be a huge value. This is another important place for conference organizers to search for new speakers, and I think those who are already well known have a duty to amplify lesser-known blogs that are well written and insightful.
To those who want to speak, I’d like to offer a little advice. I’m not a top-tier speaker. I don’t get invited to many different conferences (thanks to CocoaConf for making room for me so often), and most of my CfP submissions are rejected, but I’ve spoken at 11 conferences over the last 4 years and some of my talks have been very well received, so take my advice for what it’s worth, remembering that it comes from someone with a lot of privilege. Not all of it applies easily to marginalized groups, but hopefully it can be of some help.
First, I want to quote Pedro again:
New announcements from Apple are the perfect source of topics for talks, grabbing it quickly is crucial: Protocol Oriented, Swift Open Source, Extension, Swift in the server… You can build your developer brand around the topic. After a few conferences talking about it, the community will tag you as the expert in the topic X. You might not have worked in a production environment with that new thing, you might not have faced the real use cases and issues, but documentation is perfect to prepare a talk based on it, isn’t it?
Yes! I totally agree with most of this, except that this is positive. I take exception to a few points, though. First, “grabbing it quickly” is not crucial. Few people talk about Bluetooth or CoreMotion despite them being around for years. If those interest you, there’s plenty of room for new talks. (I sure would love some more on Bluetooth!) Would you be a better speaker if you shipped many production products with them? Of course. But you can still help a lot of people understand what’s possible by spending a few months going a little further than most and coming back and teaching. Don’t think you have to know everything before you’re allowed to say something! Just be honest and don’t pretend to know more than you do. There are many topics to explore. Natalia Berdys gives a brilliant talk on random number generation. You can’t get much more niche than that. You don’t have to chase the “current cool thing.”
I don’t know how it is for most speakers, but for me, preparing a talk is very difficult. It took me nearly five months to develop my talk for try! Swift. I completely rewrote it four times and practiced it for weeks. I think many speakers are much faster at this than I am, so it may not be so hard for you. But if you find it challenging and find yourself throwing away draft after draft because you can’t figure out what you’re trying to say, just know you’re not alone. If it matters to you, keep at it, and don’t be afraid to throw away a draft that isn’t working.2
If you want to give a talk, and think I can help you make it better, please reach out. I’ve critiqued talks before, and I’m happy to keep doing it. (I will start by asking you to evaluate the talk using Goethe’s Three Questions. You have been warned.)
And if you read all this and say “hey, I don’t even want to give a talk,” that is absolutely fine. Most people don’t. As anyone who’s scheduled CocoaHeads talks before knows, public speaking isn’t for everyone and one reason there’s a small group who speaks so often is because they’re the ones willing to do it. It’s hard and it’s scary and it’s rewarding and it’s valuable. And sometimes it’s even fun. I hope we can include everyone who wants to be part of it. And I hope there’s always a venue for those who want to listen.
I’m only talking here about whether and how to broaden the number of unique speakers, because I believe this is Pedro’s point. I think there’s a different, very important discussion about diversity of background and drawing from marginalized groups. What I’m going to discuss can help, but that problem requires and deserves more targeted effort than I’m discussing here. ↩
This advice assumes a lot of privilege. I know there are many people who don’t have the kind of time I do and don’t have the kind of family support I have. I don’t know the answer for that. If I didn’t have much of the privilege I have, I don’t know how I would speak at conferences. Take my advice for what it’s worth; it doesn’t apply to everyone. ↩
I travel pretty well, but sometimes I make mistakes, and this was one of those times. The deodorant I thought I’d packed turned out to be body wash. Now there are a dozen reason that this shouldn’t really matter, and wouldn’t really matter given the A/C and the weather, etc., but I’m a product of my culture, and it was a bit stressful. I tried to find a drug store on the way to the conference, but I was afraid of being late and finally resigned myself to accepting things as they are and moving on.
And then, in the conference rest room, I discovered a small cache of toiletries under a try! Swift sign saying “if you need one, please take one.” I was dumbfounded. It was a very small kindness, but it mattered to me. I spoke to the conference organizer, Natasha, to thank her. She immediately told me it wasn’t her idea, she’d just copied it from Erik Romijn. I went to thank Erik, and he assured me it wasn’t his idea, he’d just copied it from Django conferences he’d been a part of. So I just wanted to say thanks to whomever came up with this very kind idea.
To me, the best part of try! Swift was chatting with people during the breaks and in office hours. Hopefully the videos will be up soon, since I missed a few presentations due to conversations that ran long. So if you only “attend” by watching the videos online (like I did for the first one in Tokyo), you’ll unfortunately miss the best parts. I wish I could change that for you, but I can’t. I’m sorry.
One of the more interesting talks I had was about cut and pasting from Stack Overflow. This practice gets a lot of shaming, and we were discussing that. I can’t promise that this conversation actually took the form of a Platonic dialog, but I can’t promise it didn’t, either. You’ll have to see me at a conference to find out.
Anubis: I’ve only been programming for a little while. Swift is my first language, and I really don’t know anything yet. I find myself just searching Stack Overflow and cutting and pasting code. I know that’s a horrible way to program, but I don’t know what else to do. Before the Internet, I guess people had to figure it out themselves, but we’ve all gotten so lazy.
Kakophonis: Most of my first years programming were just copying BASIC out of Nibble magazine.
Anubis: You mean reading articles and implementing what they taught?
Kakophonis: Oh, no. I mean typing hundreds and hundreds of lines of code for full programs that they included.
Anubis: So you didn’t really learn much those first few years.
Kakophonis: I learned a lot. I learned to use my tools. I learned to debug because I made a lot of typing mistakes. But most of all, I learned what good programs looked like. These were working programs written by good programmers. And I didn’t just read their code, I started changing it. Not a lot. I didn’t know a lot. But I learned to change the colors, or add a trivial feature, or just make random changes that made me happier with it.
Anubis: When did you stop copying other people’s code and become a real programmer?
Kakophonis: I copy code all the time.
Anubis: As a short-cut, right? To make a deadline?
Kakophonis: Not at all. I use others’ code whenever it’s beautiful and useful and they’ve shared it. But you’re right that I copy code a little differently than a beginner does.
Anubis: How so?
Kakophonis: I rarely cut and paste. Maybe it’s a habit from my magazine days, but I usually retype the code by hand. Stack Overflow answers aren’t very long, and retyping gives me a chance to really think about what the code is doing. And then, if the code works and solves my problem, I usually restyle it to match my preferences and make it a little more my own. I almost always rename things. Sometimes I rewrite it from scratch. I make sure I know what each line does and why it’s there.
Anubis: Is that all?
Kakophonis: If the code is more than a line or two, I generally add a comment crediting the source.
Anubis: For politeness?
Kakophonis: Yes, but more importantly, I want future maintainers, which is sometimes me, to know the context around the code, why it does things this way. That keeps them from creating regression bugs if they ever need to rewrite it further.
Anubis: It sounds like you might learn more about programming from copying that way than from doing it yourself.
Kakophonis: Exactly. But copying only improves your code and your understanding if you choose the right things to copy. Many answers on Stack Overflow are incorrect, or they’re presented as magic incantations that may work, but are fragile. A mark of a good programmer is the ability to distinguish what they should copy from what they shouldn’t. Stack Overflow provides beginners some clues about quality, like votes and reputation and comments, but ultimately it’s experience that will help you recognize good code.
Anubis: So Stack Overflow doesn’t make us weak programmers?
Kakophonis: It’s a tool. It is what you make of it. Programmers today have many more resources than they did before the Internet. It’s easy to think that the previous generations had it much harder. But computers and programs were also much, much simpler. No networking. No threading. Not even a GUI. I’m amazed that anyone is able to jump straight to building an iOS app with no programming experience. I had 25 years of experience before I encountered my first multi-threaded program. No, I don’t think beginning programmers today are weak.
Sometimes life is not unlike programming. Who and what we choose to copy says a lot about us. Choosing to use someone’s idea is still a choice. Give credit to your inspirations, but don’t discount the wisdom in picking something beautiful to copy.
]]>This isn’t to say we’re close. We see each other at conferences. We email and tweet. We’re members of a community. So of course I have some idealized picture of him in my mind, without all the this and that of a real person. Even so, he inspires me. When my wife, Janet, and I talk about what’s ahead for our future, we always talk about Daniel and his wife, Kim. I’ve seen them at conferences, traveling together, independent but a team, and I think, hey, we could pull that off when the kids go to school. And Janet and I talk about how to make that work. And we know we only have the shallowest understanding of their real lives, but they inspire us.
Kim died this week. I didn’t know her well. We’d met, and she was nice to me. She and Daniel always seemed so “together” even when they were apart most of the day. Hearing the news unmoored me and scared me and made me think about myself and my family and my plans. And then my heart broke for Daniel.
I’m not kind by nature. But I’m part of a community that is, and part of that is Daniel’s influence on us. Ellen Shapiro has been kind and set up a fund to let us support SmileTrain, where Daniel and his daughter Maggie have asked us to give in Kim’s name. I had never heard of SmileTrain, but it is so perfect. It is kind. And it is practical. It is literally the gift of a smile. It is Daniel and Kim.
]]>Forgive me, NSData. I was running around with that flashy [UInt8], acting like you didn’t have everything I need. I’ve learned my lesson.
— Rob Napier (@cocoaphony) September 28, 2015
I did a lot of writing and rewriting of the Swift version of RNCryptor. I struggled especially with what type to use for data. I gravitated quickly to [UInt8]
with all its apparent Swiftiness. But in the end, after many iterations, I refactored back to NSData
, and I’m really glad I did.
This is the story of why.
First, I want to be clear that I don’t think [UInt8]
is bad. In some places it’s better than NSData
, but there are tradeoffs, and ultimately I found the tradeoffs favored NSData
today. Some of those will improve in Future Swift, and I suspect something more Swifty than NSData
will be the way of the future. But today, in the kinds of projects I work on, there’s a lot going for NSData
.
“In the kinds of projects I work on” is an important caveat. I mostly build things that are used by other developers; frameworks, engines, services, even just snippets of code. I don’t build a lot of full applications that an end user would see. And the systems I build typically have a very small API surface. They typically do just one thing, and they’re built to be easily clicked together with things that I didn’t write.
To achieve that, I try to make most of my code as self-contained as possible, with minimal dependencies. I try to make it easy to plug into whatever system you prefer. That usually means sticking as much as possible to the types provided by the system. Much as I love Result
, none of my systems use it externally (and only a few use it internally). I try to avoid exposing my caller to any cleverness. If they’re familiar with the platform, I want them to find my API obvious, even boring. If they have a preferred error handling system, they probably have a way to convert throws
to it, since that’s what Cocoa generates. So I use throws
.
I see a lot of Swift devs behaving as though Cocoa has somehow disappeared. Cocoa has become the embarrassing uncle that no one wants to acknowledge, even though he’s sitting right there at Thanksgiving dinner passing you the potatoes. And this is crazy. First, Cocoa is a great framework, filled with all kinds of tools that we use every day, implemented well and refined for years. And second, Cocoa is a required framework, filled with tools that we have to use every day if we want to write apps.
Trying to cordon off Cocoa means constantly converting your types and patterns. That’s horrible for programs, and it’s very unswifty. Swift is all about integrating cleanly with Cocoa.
If you’re writing Cocoa apps you wind up with NSData
all the time. You get it when you read or write files, when you download things from the network, when you create PNGs, when you serialize. You can’t escape NSData
. Swift automatically bridges NSString
and String
, NSArray
and Array
, NSError
and ErrorType
. Some day I hope NSData
gets a bridge, but it doesn’t have one today. So the question is what to do in the meantime?
There are basically two options: build the bridge or use NSData
. Building the bridge (without modifying stdlib) is tricky if you want to avoid copying. It’s not hard if you’re not worried about performance, but an NSData
can easily be multiple megabytes and that’s both time and memory. Even temporary copies raise your high water mark, which hurts the whole system. Yes, I know all about premature optimization, but when you’re building frameworks you need to avoid patterns that are reasonably likely to cause performance problems. Even when you’re building just one app, there’s a difference between “build simply, then optimize” and “throw performance out the window until Apple rejects your app, then optimize.” If it were just one copy, and it made everything else really simple, that might be worth discussing. But making a copy every time you move from one part of the system to another is a problem.
With enough work you can solve this problem. It’s not tons of code, but it is a little bit tricky to be certain you’ve done it exactly right and won’t leak or crash (and much trickier to do from outside of stdlib). But is that work and complexity worth it? What problems were we really solving converting NSData
to [UInt8]
?
Array
Even if you don’t care about NSData
interop, using [UInt8]
doesn’t give you a magical unicorn API. Array
has all kinds of little sharp edges that surprise and confuse if you want to very careful of making copies.
Let’s start with a simple function using NSData
and see what happens with [UInt8]
. This function takes a CCCryptorRef
and updates it with some data, writes the resulting encrypted data to a buffer and returns the buffer.
func updateCryptor(cryptor: CCCryptorRef, data: NSData) -> NSData {
let outputLength = CCCryptorGetOutputLength(cryptor, data.length, false)
let buffer = NSMutableData(length: outputLength)!
var dataOutMoved: Int = 0
let result = CCCryptorUpdate(cryptor,
data.bytes, data.length,
buffer.mutableBytes, buffer.length,
&dataOutMoved)
guard result == 0 else { fatalError() }
buffer.length = dataOutMoved
return buffer
}
No problems there IMO. That’s a fine implementation. Easy to read and understand (if you understand CCCryptorRef
). The [UInt8]
implementation is about the same. I don’t think you could say one is really much cleaner than the other.
func updateCryptor_(cryptor: CCCryptorRef, data: [UInt8]) -> [UInt8] {
let outputLength = CCCryptorGetOutputLength(cryptor, data.length, false)
var buffer = [UInt8](count: outputLength, repeatedValue: 0)
var dataOutMoved: Int = 0
let result = CCCryptorUpdate(cryptor,
data, data.count,
&buffer, buffer.count,
&dataOutMoved)
guard result == 0 else { fatalError() }
buffer[dataOutMoved..<buffer.endIndex] = []
return buffer
}
But is it correct? Can we be certain that data
is contiguous memory and isn’t really an NSArray<NSNumber>
under the covers? If it is an NSArray
, will this work or will we get the wrong data? Should we use .withUnsafePointer
here? I studied the docs, and talked to several devs (including Apple devs), and in the end am pretty sure that this will always work. But I’m only “pretty sure.” And that’s only because of kind people at Apple (especially @jckarter) taking time to walk through it with me. Not everyone has that.
This “but is it correct?” came up all over the place. Would this operation cause a copy? Exactly how long is an UnsafeBufferPointer
valid? There’s a lot of bridging magic in Array
, and it’s not always clear what is promised. Testing only gets you so far if the current implementation just happens to work. Sometimes behaviors change just by importing Foundation.
I thought I might avoid the Cocoa-bridging ambiguities of Array
by using ContiguousArray
instead. That way I could be very precise about my expectations. But it turns out that passing ContiguousArray
to C behaves very differently than passing Array
. Array
gets turned into a pointer to the first element, but ContiguousArray
gets turned into a pointer to the struct. So the ContiguousArray
gets corrupted and you crash. Array
is more magical than you think. Magic is wonderful until your program crashes and you don’t know why.
I struggled with copy-on-write behavior. How do I know if an array’s buffer is shared so that a copy will happen on mutation? Will this code allocate 10MB or 20MB?
func makeArray() -> [UInt8] {
return Array(count: 10_000_000, repeatedValue: 0)
}
var array = makeArray() + [1]
What tests would prove that? Is it promised or just the current implementation? Does optimization level matter? Is it the same if makeArray()
is in another module than the caller? Would small changes in my code lead to dramatic and surprising performance changes in apps that use my framework? This was a common problem in Scala before the @tailrec
annotation was added. Very small tweaks to a recursive function could cause your stack to explode because you quietly broke tail call optimization. All your unit tests still pass, but the program crashes.
In the end, I spent hours trying to be certain of the precise behaviors of Array
bridging and copying. And all that to replace NSData
code that is perfectly fine.
When updating the cryptor, it is common that you’ll only want some of the data you were passed. You might want to slice off a header, or you might want to chunk the data up to reduce your encryption buffer size. In either case, you want to pass updateCryptor()
a slice.
For an immutable NSData
that’s easy. Call .subdataWithRange()
and you get another NSData
back with no copying.
But the SubSequence
of Array
is ArraySlice
, and updateCryptor()
doesn’t accept that. Of course you can copy your slice into a new Array
, but unnecessary copying was what we wanted to avoid.
We could make all the functions take ArraySlice
and overload all the functions with an Array
interface that forwards to the ArraySlice
interface. But it’s a lot of duplication.
So I decided to just convert everything to UnsafeBufferPointer
and then pass that around internally. Easier semantics after a one-time conversion. No bridging worries. No unexpected copies. It seemed like a good idea at the time.
The problem is that using UnsafeBufferPointer
everywhere tends to turn your code inside out. Where you used to say:
updateCryptor(cryptor, data: data)
You now have to say:
data.withUnsafeBufferPointer { updateCryptor(cryptor, data: $0) }
Two solutions present themselves. First you decide that you are very clever, and use the UnsafeBufferPointer
constructor:
// Never do this
updateCryptor(cryptor, data: UnsafeBufferPointer(start: data, count: data.count))
Then @jckarter points out that by the time updateCryptor
runs, there’s no promise that the UnsafeBufferPointer
is still valid. ARC could destroy data
before the statement even completes. (If you know that data
is life-extended, then it is possible to know this will work, but it’s very unsafe, fragile, and hard to audit. Coding that way breaks everything Swift was trying to fix.)
So then you start creating function overloads to accept Array
and ArraySlice
and convert them into UnsafeBufferPointer
, and you have even more duplicated code. And then you realize you want to accept NSData
here, too, so you write an extension that adds .withUnsafeBufferPointer()
to NSData
, and now you have four versions of every function, and you realize you really should use a protocol instead. Brilliant!
protocol BufferType {
func withUnsafeBufferPointer<R>(body: (UnsafeBufferPointer<UInt8>) throws -> R) rethrows -> R
}
This really feels like it’ll solve all these problems very elegantly. Except for this one problem. You want [UInt8]
to be a BufferType
, but you don’t want [String]
to be a BufferType
. And then you discover that while you can write extensions that only apply to [UInt8]
, you can’t use those extensions to conform to a protocol. And that’s when the screaming starts. And then the barginning, and then the drinking.
When you get to the muttering, you came back and start building a Buffer
class to wrap Array
, ArraySlice
, NSData
, and even CollectionType
to give it all a consistent interface. It’s ok, but it creates another “thing” for callers to deal with. In almost all cases, they have an NSData
. There is almost no chance they had a [UInt8]
. This is all just an extra layer for callers to deal with and to get in the way of the optimizer.
I want to remind you that all of this, all these many, many hours of struggle, were to avoid the simple NSData
code that took two minutes to write, works great, and is pretty darn Swifty as long as you don’t define “Swifty” as “does not import Foundation.”
So why did I resist using NSData
anyway? Well, even though I believe the Foundation is absolutely a part of Swift, some of it isn’t great Swift. Notably NSData
isn’t a CollectionType
. But fixing that is pretty easy.
End-to-end NSData
also opened up some other opportunities for me, namely dispatch_data
, which had threatened to be another can of worms with [UInt8]
.
For some, none of this will matter. The vast majority of my problems come from trying to dodge unnecessary copies. Much of this is very simple if you’re willing to just copy the data all over the place. For many kinds of problems, that’s fine. Use whatever you like. The copy-on-write system is actually pretty awesome and for most problems you can certainly trust it.
But for those in my situation, where performance is a serious consideration in much of your code, you’re looking for predictability as much as speed, and data can be huge, my hope is we get a Buffer
type (or Data
or whatever) that acts as a bridge to NSData
, supports dispatch_data
, and plays nicely with stdlib. But until that comes, I think NSData
is just fine.
(While this works exactly like AnySequence, this isn’t how AnySequence is implemented. In my next post I’ll discuss why and how to implement type erasers like stdlib does.)
At the time I thought I’d pretty well nailed it down, but every time I dug into it I found another little thing I’d missed, and it never seemed to end. And with stdlib open sourcing soon, you’ll all just be able to read this yourselves, so why embarrass myself getting it all wrong? Over time I kind of hoped you all had forgotten that comment and planned to move on to other things. But then I was busted by Michael Welch, and so I had to finish the spelunking and here you go.
So, here is my annotated implementation of AnyGenerator
and AnySequence
that I believe pretty closely matches Apple’s implementation in stdlib (minus some low-level optimizations in AnySequence
that I’ll call out when we get there). I’ve named my versions of public symbols with a trailing underscore to differentiate from the Swift version. For private symbols (with leading underscore), I’ve used the same name Swift does.
All type information was determined using :type lookup
in the Swift REPL. In a few places I checked the implementation with Hopper, but in most cases, once you have the types, the implementation is obvious. (There’s a deep lesson in there if you pay attention.)
RNCryptor 4 is a complete rewrite of RNCryptor for Swift 2 with full bridging support to Objective-C. It has a streamlined API, simpler installation, and improved internals. It continues to use the v3 data format and is fully interoperable with other RNCryptor implementations.
For users desiring a fully Objective-C solution, v3 is still available. I don’t currently plan to do significant new work on v3, but will consider it if there is strong interest. Going forward, I expect most OS X and iOS projects to be able to accept a mix of ObjC and Swift code. Objective-C continues to be a fully supported language in RNCryptor 4.
I plan to convert this to a final release in a week or so if no major issues are discovered.
I now move onto:
SequenceOf
and GeneratorOf
, and they could type erase stuff. “Type erase?” you may ask. “I thought we loved types.” We do. Don’t worry. Our types aren’t going anywhere. But sometimes we want them to be a little less…precise.
In Swift 2, our little type erasers got a rename and some friends. Now they’re all named “Any”-something. So SequenceOf
became AnySequence
and GeneratorOf
became AnyGenerator
and there are a gaggle of indexes and collections from AnyForwardIndex
to AnyRandomAccessCollection
.
So what are these type erasers? Let’s start with how to use one and we’ll work backwards to why.
let seq = AnySequence([1,2,3])
This creates an AnySequence<Int>
. It’s just a sequence of Ints that we can iterate over. Isn’t [1,2,3]
also a sequence of Ints we can iterate over? Well, yeah. But it’s also explicitly an Array. And sometimes you don’t want to have to deal with that kind of implementation detail.
Let’s consider a little more complicated case:
let xs = [1,2,3]
let ys = ["A","B","C"]
let zs = zip(xs.reverse(), ys.reverse())
// Zip2Sequence<ReverseRandomAccessCollection<Array<Int>>, ReverseRandomAccessCollection<Array<String>>>
That’s quite a type. Imagine it as the return type of a function:
func reverseZip<T,U>(xs: [T], _ ys: [U]) -> Zip2Sequence<ReverseRandomAccessCollection<[T]>, ReverseRandomAccessCollection<[U]>> {
return zip(xs.reverse(), ys.reverse())
}
That’s insane. Let’s not do that. Not only is the type overwhelming, but it ties us to this particular implementation. We might want to refactor the code like this:
return zip(xs, ys).reverse()
Then the return type would change to [(T,U)]
and all the callers would have to be updated. Clearly we’re leaking too much information about our implementation. What’s the point of reverseZip
? Is it to return a Zip2Sequence<...>
? No. It’s to return a sequence of tuples. We want a type that means “a sequence of tuples.” Often we use Array
for that, but there’s an even less restrictive way that doesn’t require making an extra copy: AnySequence
.
func reverseZip<T,U>(xs: [T], _ ys: [U]) -> AnySequence<(T,U)> {
return AnySequence(zip(xs, ys).reverse())
}
Now we can keep our implementation details private. If we have some internal sequence type, we don’t have to share it with our callers. We just give them what they need and no more.
Notice that AnySequence
is not a protocol. It’s a generic struct
that wraps another sequence. You can’t use an [Int]
in a place that expects an AnySequence<Int>
. You still want to use SequenceType
for parameters in most cases.
These “Any” type erasers also aren’t like Any
and AnyObject
, which are protocols that just “hide” the type. You can still as!
an AnyObject
back to its original type. AnySequence
and its kin completely encapsulate the underlying data. You can’t get the original back. This creates a very strong abstraction layer and strengthens type safety by making as!
casting impossible.
The new names worry me a little because they make it look like AnyObject
and AnySequence
are the same kind of thing when they’re not. But the new naming convention is definitely more flexible. You couldn’t have named the AnyIndex
types using the old ...Of
convention. So, I’m getting used to the new names.
Hopefully by now you’re sold on why you’d want to use a type eraser. But would you ever want to build one? Let’s look at an example that comes up pretty often around associated types in protocols.
// An Animal can eat
protocol Animal {
typealias Food
func feed(food: Food)
}
// Kinds of Food
struct Grass {}
struct Worm {}
struct Cow: Animal {
func feed(food: Grass) { print("moo") }
}
struct Goat: Animal {
func feed(food: Grass) { print("bah") }
}
struct Bird: Animal {
func feed(food: Worm) { print("chirp") }
}
So now let’s say we have a bunch of grass available and we’d like to feed it to some grass eaters. Seems easy:
for animal in grassEaters {
animal.feed(Grass())
}
Now we just have to create this array of grass eaters. Should be simple, right? Hmmm…
let grassEaters = [Cow(), Goat()] // error: '_' is not convertible to 'Goat'
That’s a weird error. We probably just need to be explicit about the the type.
let grassEaters: [Animal] = [Cow(), Goat()]
// error: protocol 'Animal' can only be used as a generic constraint because it has Self or associated type requirements
We all know that error, don’t we? OK, let’s try generics.
let grassEaters: [Animal<Grass>] = [Cow(), Goat()]
// error: protocol 'Animal' can only be used as a generic constraint because it has Self or associated type requirements
Still? Oh right. You can’t specialize an associated type using generic syntax. That’s fine, we’ll just make the protocol generic.
protocol Animal<Food> {
func feed(food: Food)
}
// error: Statement cannot begin with a closure expression
Right, protocols can’t be generic. Type-safety is for chumps. Let’s go back to Objective-C.
…Or maybe type erasure is what we need. Let’s build AnyAnimal
. There are several ways to do this, but the easiest in my opinion is with closures.
struct AnyAnimal<Food>: Animal {
private let _feed: (Food) -> Void
init<Base: Animal where Food == Base.Food>(_ base: Base) {
_feed = base.feed
}
func feed(food: Food) { _feed(food) }
}
(While this works exactly like AnySequence
, this isn’t how AnySequence
is implemented. In my next post I’ll discuss why and how to implement type erasers like stdlib does.)
Now we can make grassEaters
:
let grassEaters = [AnyAnimal(Cow()), AnyAnimal(Goat())] // Type is [AnyAnimal<Grass>]
But we still get type safety if we try to incorrectly mix our animals:
let mixedEaters = [AnyAnimal(Cow()), AnyAnimal(Bird())]
// error: type of expression is ambiguous without more context
This kind of type eraser lets us convert a protocol with associated types into a generic type. That means we can put it in properties and return values and other places that we can’t use protocols directly. As you use more protocols in your Swift (and you should be), I think this will become an important tool in your toolbelt.
So get out there and erase some over-specific types. Focus on the protocol, hide the implementation.
]]>A friend of mine is an arborist. He takes care of a large forest, trimming and culling trees. He’s quite good at it and enjoys it, but he’s worried about job security. He thinks cabinet making would be a good career move. He likes to work with wood, and high-end cabinets are very expensive so there’s clearly a lot of money there. I’m a hobbyist woodworker, so we were talking about it.
Me: So, what saws are you looking at?
Him: I have a Husqvarna 3120. That should be all I need.
Me: Husqvarna? Isn’t that a…chainsaw?
Him: It’s the best! I use it every day. I plan to do everything with it.
Me: It’s a good saw, sure, but… um… How are you going to join the pieces?
Him: Dovetail joints. I’ve heard those are best and you just have to cut notches. My chainsaw will make quick work of that.
Me: Dovetails have really tight tolerances. I can barely get them to work with a dovetail saw. Maybe you should start with pocket screws? They’re much easier and make great joints. A chainsaw…
Him: I’ve worked it all out. I modded this router table to hold my chainsaw upright so that’s just like a table saw. And I can make the other cuts freehand. I’m really good with a chainsaw. See, I already have a proof-of-concept.
Me: Um… that’s just a board with a notch in it.
Him: Yeah, but a cabinet is just a bunch of boards with notches. I’ll make some more and they should snap together.
Me: Cabinet tolerances are really tight. I mean, it’s not that hard to make a box, but making a full cabinet is difficult, even with proper tools. Why would you want to use some cobbled-together thing? That table looks terrifying.
Him: Look, I know chainsaws. I’m not going to go learn a bunch of new tools just to build cabinets. Wood is wood. Saws are saws. I’m using the saw I’m comfortable with. If I use the tools I know, I’m sure I can make really great cabinets.
Me: …
In unrelated news, a lot of folks have asked if I’m excited that Go 1.5 can build iOS apps.
My wife’s background is art education. When you’re teaching art, she tells me, it’s important to remember whether you’re doing process art or product art in order to judge its success. Is the end result the goal, or how you got there? Am I trying to make the most beautiful painting I can, or am I trying to make a painting using only recycled materials? Does the audience need to know the process in order to appreciate the art, or does the end result stand on its own?
A lot of great hacks are process art. Can you run VMS on Linux? Sure you can! Is that awesome? It is absolutely awesome. Is there any commercial reason to develop new VMS apps to run on Linux? Of course not. That would be ridiculous.
Is it fun to get the Go runtime working on surprising platforms? Sure, and the process may improve Go. That’s why process art is so common in art education. You put unnecessary restrictions on students so that they have to develop new skills that will be useful when they work on product art. “Draw without looking at your work” is an important exercise, but not usually how you create a masterpiece.
So let’s get to the “unnecessary restrictions” I’m talking about if you want to write excellent iOS apps in Go (or C# or JavaScript or….) I don’t want to spend too much time on the great memory management debate. Do modern concurrent garbage collection engines work well on dual-core, 1GB iPhones? I don’t know. I’m still hearing a lot of complaints about GC pauses on Lollipop with 2x the memory. It’s hard to imagine that Go GC on iOS is going to be faster and more memory efficient than Java GC on Android. But let’s put that question aside for the moment. Let’s assume there is a large genre of apps for which Go performance and memory usage is adequate, even excellent, on iOS devices. (This might even be true.)
The real question is “what’s the hard part of iOS development?”
If you think the answer is “Objective-C” then you don’t understand iOS development at all. Learning Objective-C is the easiest part of iOS development, doubly-so for anyone who is comfortable in a wacky (and awesome) language like Go. And when it finally settles down a bit, Swift will likely be easier to learn than Objective-C for Java-esque developers.
The sort-of hard piece of iOS development is UIKit, CloudKit, HomeKit, SpriteKit, SceneKit, HealthKit, WebKit, MapKit, StoreKit, GameplayKit, and another new Kit every release. It’s power management and backgrounding and handoff and watch integration and multitasking and adaptive fonts and auto layout and animations and app thinning. You can sometimes ignore those pieces, but they’re what separate barely acceptable apps from great apps. If your only goal is something that kind of works, fine. Anything can do that. Web apps can do that. But if you want to make a really great app, integrating with the platform is a must. And the platform is designed for Cocoa. And Cocoa is designed for ObjC and Swift. This stuff is hard when you’re using the tools Apple intends you to use. Working through translation layers is surgery with welding gloves.
And that brings us around to the really hard piece of iOS development. It changes. Constantly. Apple moves forward fast and they deprecate old APIs mercilessly. Clever work-arounds have short half-lives. You have to adapt every release, and find ways to be backward compatible while still compiling against the latest SDK. Once you’ve lived through a few releases of iOS, you discover how hard this piece can be even in ObjC, even when you’ve followed all the rules. I don’t envy anyone trying to deal with it through an unsupported layer. By the time anyone knows what the next iOS version breaks, the clock is already ticking. Whoever supports your extra layer has to fix their piece, and then you have to fix your piece. And with open iOS betas, your customers are already yelling.
So I may celebrate your chainsaw router table as a glorious (and terrifying) hack. But I’ll stick with my table saw for cabinets and my chainsaw for cutting up trees. A good crafter is capable with many tools, not just their favorite. And most iOS apps are about product, not process.
Swift on Android? I’ll take Java, thanks.
Lee Whitney has an interesting take on this subject from the C# side.
]]>“Can we solve this by throwing money at it?” Problems come in two major forms: money problems and not-money problems. Money problems are the easy ones. If you can write a check and the whole problem goes away, that’s the kind of problem you want. Lots of problems aren’t like that. Sometimes it’s because the check would have to be too big, but many times it’s because no amount of money would fix the problem alone. It’s not a “throwing money” problem. In fact, most problems are not-money problems when you dig into them.
Big companies love money problems because they have money. No matter how much they’re cutting budget, I assure you they have money. It’s what big companies are made of. And no matter how much they say “employees are our greatest asset,” I assure you that money is their greatest asset. That’s why when companies can’t find anything to invest in, they horde money and lay off employees. If employees were their greatest asset, they’d burn money to keep them.
And this brings us to music streaming and the App Store and why they’re completely different problems and why Taylor Swift could turn Apple around in 24 hours when tens of thousands of developers have groused for years about the App Store with slow improvements at best. Yes, Taylor Swift is a big name and controls a lot of content that Apple would like access to. That’s not why it all turned around so fast. And no, Taylor Swift demanding faster App Store reviews won’t change anything, no matter how many albums she threatens to withhold.
The difference is that musicians had a money problem and devs have a not-money problem.
What was the musician problem? Apple wasn’t going to pay musicians during free trials (but was paying them after that). What was the solution to the musician problem? Apple writes a bigger check. They were already writing a huge check. Musicians wanted it to be a little bigger and structured a little differently. Throw money and this whole problem goes away. Musicians are happy. Taylor Swift tweets Apple love. That’s a money problem. Money problems are easy and clean.
What is the dev problem? App Store approvals are slow and inconsistent. It is not always clear what is or isn’t allowed, and some decisions seem arbitrary. Apple’s 30% cut may be high. Apple policies make many valid business models difficult or impossible. Apple technology and policies make many useful kinds of apps impossible. Apple created a 99 cent culture that undermines sustainable app development. And the list goes on.
Only one problem on that list is a throw-money-at-it problem: “Apple’s 30% cut may be high.” OK, so Apple switches to 25% cut. Is everyone happy now? We all tweet App Store love? No. We’re a little happier, but we’re still grumbling because most of the problems are about process and policy (and particularly inconsistency and opaqueness), not money. Apple drops it to 20%? 10%? 0%? At what point do we say “yep, that was the problem, thanks Apple! The App Store is fixed now.” There is no answer. This is a not-money problem.
Not-money problems are a pain. There’s no check that Apple writes that fixes their App Store problems. It’s not even obvious exactly what you’d change so it’d be right. You could hire more reviewers, but then it’s harder to keep them consistent. You could hire fewer, more heavily trained reviewers, but then review times go up. You could make your rules fixed and automatic, but then bad actors will more easily to slip through. You can adapt to bad actors as you discover them, but then you’re inconsistent over time. The App Store is an engineering problem, not a money problem. Engineering problems are hard and messy.
If you want a company to do something quickly, turn it into a simple money problem if at all possible. Create situations where “Company wants X. If Company writes a check for Y, they get X and we’re done.” Many problems can’t be solved that way, but when they can, you’re going to have much smoother sailing.
And that’s why one artist can move all of Apple. Yes, she’s big and important, but she also brought a simple problem.
]]>Like last time, we start with map
(mymap
so there’s no confusion with the
built-in).
extension Array {
func mymap<T>(@noescape transform: (Generator.Element) -> T) -> [T] {
var ts: [T] = []
for x in self {
ts.append(transform(x))
}
return ts
}
}
So that’s the simple map
. As we discussed previously, we can’t pass a throwing
closure to it because it would be the wrong type. Let’s rewrite mymap
so it
can throw:
extension Array {
func mymapThrows<T>(@noescape transform: (Generator.Element) throws -> T) throws -> [T] {
var ts: [T] = []
for x in self {
ts.append(try transform(x))
}
return ts
}
}
Now transforms
can throw, and so it needs try
when we call it. And since we
don’t handle the error ourselves, the whole method has to be marked throws
.
Let’s create a couple of functions to check this out:
func double(x: Int) -> Int { return x*2 }
extension NSCalculationError: ErrorType {}
func reciprocal(x: Int) throws -> Double {
guard x != 0 else { throw NSCalculationError.DivideByZero }
return 1.0 / Double(x)
}
The first function, double
, always succeeds. The second function,
reciprocal
, may throw.
let xs = [1,2,3]
let ds = xs.mymap(double) // No problem
let rs = try xs.mymapThrows(reciprocal) // No problem
And if we pass them to the other methods?
let ds = try xs.mymapThrows(double) // No problem
let rs = xs.mymap(reciprocal)
// Invalid conversion from throwing function of type '(Int) throws -> Double' to non-throwing function type '@noescape Int -> `T'
So we can pass a non-throwing closure to the throwing map
, but not vice versa.
Why? Let’s take a step back and talk about subtypes.
A good way to think about types is as a set of promises. In the OOP world, we create types like this:
class Animal {
func eat() {...}
}
class Cat : Animal {
func purr() {...}
}
Every Animal promises it can eat. Every Cat promises it can purr. Since a Cat is an Animal, it also promises it can eat. But not every Animal promises to purr (other Animals may be able to purr, it’s just not promised). You’re used to calling Cat a subclass of Animal, and that’s true. But it’s more generally a subtype. This idea isn’t restricted to classes. After all, the same thing is true of protocols:
protocol Animal {
func eat()
}
protocol Cat : Animal {
func purr()
}
No classes required. The important thing about the type/subtype relationship is that a subtype can only add promises. It can never remove promises. Understanding what promises are being made is very important to understanding your types.
NSArray
doesn’t promise to be immutable. That may surprise you, but you know
it’s true because you copy them when they’re passed as parameters. If NSArray
promised to be immutable (like NSDate
does), you’d never do that. If NSArray
promised to be immutable, then NSMutableArray
couldn’t be its subclass,
because it breaks that promise.
NSArray
only promises to be readable. That’s a completely different thing.
NSMutableArray
also promises to be readable. It keeps the promise NSArray
made. NSMutableArray
also promises to be writable, and any subclass of
NSMutableArray
would also have to keep that promise.
A subtype can only add promises. It can never remove them.
So, what promises does (T) throws -> U
make? It promises to accept a T
. And
it promises that it will either return a U
or it will throw an error.
What promises does (T) -> U
make? It promises to accept a T
. And
it promises that it will return a U
.
How are these types related? Which one makes the stronger promise? A good way to figure this out is to think through some cases.
U
. Keeps both promises.The stronger promise is the one that we broke. It’s the non-throwing function that added a new, stricter promise. “I will do X or Y, and furthermore I will only do X.” Doing X keeps that promise. Doing Y breaks it.
So that tells us that a non-throwing closure can be used anywhere a throwing closure is requested, just like a Cat can be used anywhere an Animal is requested. No conversions necessary. It’s just types.
So great, we have mymapThrows
, and it takes either kind, so we’re done, right?
Well, we could be, but it’d be really annoying. Consider if map
were marked
throws
. That would mean that every map
would have to include a try
, and
somewhere you’d have to catch the error.
let ds: [Int]
do {
ds = try xs.map { $0 * 2 }
} catch {
// Really, Swift? Really? Every time? Even when it can't possibly throw?
// No, not really. Swift is smarter than that.
}
There are two ways out of this annoyance. The obvious way is overloading. We can just have two methods with the same name but different types:
map<T>(@noescape transform: (Generator.Element) throws -> T) throws -> [T]
map<T>(@noescape transform: (Generator.Element) -> T) -> [T]
Since overloading picks the most specific subtype available, this works fine for
the caller. But it’s a serious pain for the dev who has to write map
. There’s
the obvious annoyance of needing two methods to do the job of one, but it gets
worse if you try to share code between the implementations. You’d think you
could just call the throwing version from the non-throwing version like:
func map<T>(@noescape transform: (Generator.Element) -> T) -> [T] {
return try! self.map(transform as (Generator.Element) throws -> T))
}
But that runs afoul of @noescape
, which doesn’t allow the conversion. And even
if that worked (might be a Swift bug), having to use try!
all over the place
is crazy, on top of the madness of having two (or three) methods for everything.
My overload implementation looks like this:
extension Array {
func mymap<T>(@noescape transform: (Generator.Element) throws -> T) throws -> [T] {
return try self._mymap(transform)
}
func mymap<T>(@noescape transform: (Generator.Element) -> T) -> [T] {
return try! self._mymap(transform)
}
func _mymap<T>(@noescape transform: (Generator.Element) throws -> T) throws -> [T] {
var ts: [T] = []
for x in self {
ts.append(try transform(x))
}
return ts
}
}
If Swift had shipped this way, I suspect the stdlib folks would be having words with the compiler folks by now. “Please come over to my desk. I’d like to introduce you to another kind of throws.”
Luckily, Swift is much smarter than that. It’s nice that you can overload based
on throwing, but in many cases we have a better tool. We can mark the method
rethrows
rather than throws
.
func map<T>(@noescape transform: (Generator.Element) throws -> T) rethrows -> [T]
So what promise does rethrows
make? It promises that the only way it will
throw is if a closure it is passed throws. So if it’s passed a closure that
can’t throw, the compiler knows that the function can’t throw either.
(Why isn’t stdlib’s map
marked rethrows
today? Because it’s beta 1, and the
Swift team hasn’t updated all of stdlib yet. They’ve indicated that a lot of
stdlib will be fixed in future betas. Have patience.)
It’s natural to think of rethrows
as a subtype of throws
, and non-throwing
closures as a subtype of rethrows
, but that doesn’t quite seem to be true.
Swift doesn’t treat rethrows
as a full type. For example, you can’t write
overloads with both throws
and rethrows
, and closures can’t include
rethrows
in their type. Instead, rethrows
acts more like a function
attribute (like @noreturn
). It just modifies the rules around what contexts
can call the function. The real types are throwing and non-throwing, and
“rethrowing” can just morph between the two based on context.
A function that accepts a closure has three throwing options:
It can throw. That means that the function may throw errors whether or not the closure throws errors.
It can rethrow, like map
. This means that the function cannot create any
errors of its own, but may propagate errors from the closure it was passed.
It can not throw. That means that it either handles the errors thrown by the closure, or it does not evaluate the closure. For example, a setter on a closure property doesn’t throw just because the closure might throw. It just sets the property and returns.
Which one you use is completely dependent on your situation. There’s no “best”
answer, though you should generally choose the most restrictive one you can. You
shouldn’t just make all your functions throws
for the same reasons you
shouldn’t make all your variables Any
. It’s all about choosing the right type.
So when you use the new Swift error handling system, don’t think “exceptions.” Think types. Your function returns “either X or an error.” And sometimes, you can promise it’ll only return X.
Throw in peace.
]]>throw
stuff in Swift 2. And
say you’re running an early Beta in which many stdlib functions don’t handle
throw closures yet. Or maybe you’re in the future and dealing with some other
piece of code that you wish could handle a throw closure, but doesn’t. What do
you do?
By now you may be asking “what the heck is a throw closure? Talk sense, man!”
Let’s take a small step back and quickly introduce the new throws
feature.
That’s not the point of this article, though. You should go watch the WWDC
videos. But basically, it’s like this. Say you have a function that might fail.
In Swift 2, you can write it this way:
enum Error: ErrorType { case Negative }
let f: (Int) throws -> String = {
guard $0 >= 0 else { throw Error.Negative }
return "".join(Repeat(count: $0, repeatedValue: "X"))
}
First, the throws
in the signature tells us that this function may throw.
Functions may not throw errors unless they explicitly indicate that they can.
You might think of “throwing errors” as equivalent to “exceptions” in languages
you’re familiar with, but it’s a little different. A throw is really just a
fancy return. A throwing function can return either a type or an ErrorType
.
And “throws” is probably best thought of as somewhat opaque sugar around an
Either type.
That tells us something very important:
A function that throws errors is a different type than one that doesn’t.
I don’t mean that it’s “some other kind of thing.” I mean like Int
is a
different type than String
, and String
is a different type than (Int) ->
String
, (Int) -> String
is a different type than (Int) throws -> String
. In
fact, (Int) -> String
is a subtype of (Int) throws -> String
, which is
pretty awesome and a little subtle, but we’ll get to that in another post.
So what does that mean? Let’s think of a simple case of map today (Swift 2 Beta 1):
print([1,2,3].map(f)) // Cannot invoke 'map' with an argument list of type '((Int) throws -> String)'
What’s going on here? Let’s look at the type signature:
func map<T>(@noescape transform: (Self.Generator.Element) -> T) -> [T]
So transform
is of type (Element) -> T
. We’re passing (Element) throws ->
T
. Remember I said that a non-throwing function is a subtype of a throwing
function. So (Element) throws -> T
is a supertype of what what this function
wants. That’s like passing NSObject to something that wants UIView. You can’t do
that.
So what do we do? Well for map, this is easy. We can just implement our own throwing version:
extension Array {
func map<T>(@noescape transform: (Generator.Element) throws -> T) rethrows -> [T] {
var result: [T] = []
for x in self {
result.append(try transform(x))
}
return result
}
}
And now we can use it:
print(try [1,2,3].map(f))
Notice the use of try
. This is pretty different than how try
is used in
other langauges, and another way that Swift’s error handling doesn’t quite match
“exceptions.” Swift uses try
to remind the programmer about functions that
may throw errors. The compiler doesn’t need try
. It doesn’t create scope, or
mark control flow points, or anything like that. It’s not a function or a
constructor. It’s just a keyword that Swift forces you to include so that you
(and your coworkers) remember what’s going on. When you see try
, you should
think “hey, control could suddenly jump somewhere else from this point.” It
reduces surprise when that happens, and conversely tells you where control
can’t suddenly jump (i.e. everywhere without try
). I think that’s pretty
nice.
You may also notice both throws
and rethrows
in the method signature. I’ll
get to that in a later blog post. Just trust me for now. This code would also
work if you used throws
in both places, but this is the better signature.
And one more “also notice.” Also notice that this is an overload of map. The closures have different types, so the compiler can pick the right one. Nice.
OK, that was a lot of setup, and you could probably figure out on your own how to rewrite map this way. And besides, by beta 2, I’m sure there will be a proper (re)throwing version of map. So why bother? For the next step.
I know how map is implemented. It’s really simple. But what if I didn’t know
how it was implemented? How about some function that I’m not sure I could write
correctly? How about a more obscure function that may not get throwing love
quite so quickly? How about Array.withUnsafeBufferPointer
? Ooohhh….
So here’s our signature:
func withUnsafeBufferPointer<R>(@noescape body: (UnsafeBufferPointer<T>) -> R) -> R
We want to accept a body
that can throw, but we want to pass it to the
existing method, which can’t accept a throwing closure. So what do we do? We go
back to our old friend, Result. Here’s a super-simple Result implementation that
can convert to and from throwing closures:
enum Result<T> {
case Success(T)
case Failure(ErrorType)
func value() throws -> T {
switch self {
case .Success(let value): return value
case .Failure(let err): throw err
}
}
init(@noescape f: () throws -> T) {
do { self = .Success(try f()) }
catch { self = .Failure(error) }
}
}
If you’re familiar with Result or Either, this should be pretty self-evident,
but the key pieces are that result.value()
will unwrap the result into either
a value or a thrown error. And init
will take a throwing closure and convert
it into a Result. With that piece, here’s how we build our method:
extension Array {
func withUnsafeBufferPointer<R>(@noescape body: (UnsafeBufferPointer<T>) throws -> R) throws -> R {
return try self.withUnsafeBufferPointer { buf in
return Result{ try body(buf) }
}.value()
}
}
The closure body
is of type (UnsafeBufferPointer<T>) throws -> R
, which we
can’t pass to withUnsafeBufferPointer
. But our closure is of type
(UnsafeBufferPointer<T>) -> Result<R>
, which is just fine (no throws here,
move along).
Let’s walk through the closure from the inside out.
try body(buf)
. Execute our throwing closure using the buf
provided to us by the default implementation.Result{...}
. Capture it into a Result enumreturn Result{...}
. Return the Result, not the underlying value.return try result.value()
R
), or throwsThis method is marked throws
rather than rethrows
because … reasons. (The
final throw doesn’t come directly from body
, but from value()
. That’ll
hopefully make more sense when I explain rethrows
.)
And just for completeness (and because I needed it myself), we can do the same
thing to withUnsafeMutableBufferPointer
, but we need to give the compiler more
type information because of the inout
parameter:
extension Array {
mutating func withUnsafeMutableBufferPointer<R>(@noescape body: (inout UnsafeMutableBufferPointer<T>) throws -> R) throws-> R {
return try self.withUnsafeMutableBufferPointer { (inout buf: UnsafeMutableBufferPointer<T>) in
return Result{try body(&buf)}}.value()
}
}
These particular implementations probably won’t be useful for long. I’m sure the
stdlib team will quickly clean this up (and you probably shouldn’t be using
withUnsafeBufferPointer
very much anyway). But hopefully exploring how this
works can give some insight into Swift’s new error handling system. Result isn’t
dead; it still has interesting use cases like this one. But I expect those use
cases to shrink, and I highly recommend exploring the new error handling and
discover how to build great things with it.