My original plan was that, upon returning from my vacation, I would spend my remaining time at EPFL writing a technical report explaining everything I knew about the problems with Scala Classic. Instead, on my first day back in the office, Martin came by with a draft of a new formal system, DOT (for Dependent Object Types), that he came up with while he was on vacation. After about four weeks I managed to root out pretty much all of the obvious problems with DOT, but another four weeks was not enough to get the proofs and the metatheory into a state that we were happy with. I am not really sure what will happen with DOT in the long term.

There are a few clever things about DOT that avoid some of the problems I encountered with Scala Classic. For example, Scala Classic only had intersection types while DOT has both intersection and union types, which solves some problems with computing the members of types.  However, with Scala Classic I was trying to model a minimal subset Scala language as it exists, without adding any new features that many never be supported by the full language. I have no idea if we will see union types in the actual Scala implementation anytime soon.

The other thing DOT does is give up the goal of trying to be a minimal subset of Scala and throws out quite a few important things from a user perspective. For example, there are no methods (only λ-abstractions), no existentials, no built-in inheritance or mix-in composition (though you can encode it by hand), and object fields must be syntactically values.

This last restriction is particularly important because it solves by fiat the problems that arose in Scala Classic from using paths that have an empty type in dependent type projections. However, it also means that you may not be able to directly encode some Scala programs into DOT without an effect or termination analysis. Therefore, while DOT has the potential to be a useful intermediary step, there will still be more work to be done to provide a semantics for a more realistic subset of the Scala language.

I have been in Copenhagen for a little over a month now, but I am not really ready to comment on the research I have been doing here yet.  So in the meantime I'll probably focus more on fonts and typography.

However, I figured I should take some time now, before I leave for my first proper vacation in a year, to announce that I have accepted a postdoc position at ITU in Copenhagen with Carsten Schürmann.  This has been in the works for a while now, but this week I finally was able to make it official.  My plan is to start at ITU at the beginning of March, where I will be working on things relating to the LF family of logical frameworks, Twelf, and Delphin.

There are probably a vanishingly small number of you that care about what this means for Scala Classic, but I hope to write something much more detailed about it when I get back from my vacation.  I'll leave you in suspense with the short answer: despite all my efforts, it simply is not possible to prove it sound without making it a very different language.  Which is rather unfortunate.

And there are plenty of other Scalas about. In San Francisco:

Alas, we did not succeed in getting Martin to eat there while he was at JavaOne. Photo courtesy of Robby Findler.

For example, consider the following SML signature:

 
signature Nat = sig
  type t
  val z: t
  val s: t -> t
end
 

This signature can be translated in to Scala as:

 
type Nat = {
  type T
  val z: T
  def s(arg: T): T
}
 

It is then possible to create an implementation of this type, and opaquely seal it (hiding the definition of T). In SML:

 
structure nat :> Nat = struct
  type t = int
  val z = 0
  fun s n = n + 1
end
 

In Scala:

 
val nat : Nat = new {
  type T = Int
  val z = 0
  def s(arg: Int) = arg + 1
}
 

In many cases when programming with SML modules it is necessary or convenient to give a module that reveals the definition of an abstract type. In the above example, this can be done by adding a where type clause to the first line:

 
structure nat :> Nat where type t = int = struct
...
 

We can do the same thing in Scala using refinements:

 
val nat : Nat { type T = Int } = new {
...
 

Great, right? Well, almost. The problem is that structural types are still a bit buggy in Scala compiler at present. So, while the above typechecks, you can't quite use it yet:

 
scala> nat.s(nat.z)
java.lang.NoSuchMethodException: $anon$1.s(java.lang.Object)
        at java.lang.Class.getMethod(Class.java:1581)
        at .reflMethod$Method1(<console>:7)
        at .<init>(<console>:7)
        at .<clinit>(<console>)
        at RequestResult$.<init>(<console>:3)
        at RequestResult$.<clinit>(<console>)
        at RequestResult$result(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflec...
 

There were some issues raised about how faithful an encoding of SML functors, and well-known extensions for higher-order functors, one can get in Scala. Indeed, off the top of my head it is not entirely clear. So I need to think more about that before I write some examples.

First we can define an infinite family of traits to represent n-ary universal quantification, much like Scala represents n-ary functions types:

 
trait Univ1[Bound1,Body[_]] {
   def Apply[T1<:Bound1] : Body[T1]
}
 
trait Univ2[Bound1,Bound2,Body[_,_]] {
   def Apply[T1<:Bound1,T2<:Bound2] : Body[T1,T2]
}
 
// ... so on for N > 2
 

Really, the key to this encoding is the use of higher-kinded quantification to encode the binding structure of the quantifier.

Now it is possible to write some examples similar to what I gave previously, but more concisely:

 
object Test extends Application {
   def id[T](x : T) = x
 
   type Id[T] = T => T
   val id = new Univ1[Any,Id]
     { def Apply[T <: Any] : Id[T] = id[T] _ }
 
   val idString = id.Apply[String]
   val idStringList = id.Apply[List[String]]
 
   println(idString("Foo"))
   println(idStringList(List("Foo", "Bar", "Baz")))
 
   type Double[T] = T => (T, T)
   val double = new Univ1[Any,Double]
     { def Apply[T <: Any] : Double[T] = (x : T) => (x, x) }
 
   val doubleString = double.Apply[String]
   val doubleStringList = double.Apply[List[String]]
 
   println(doubleString("Foo"))
   println(doubleStringList(List("Foo", "Bar", "Baz")))
}
 

As I mentioned previously, this example would be much improved by support for anonymous type functions in Scala. I am pretty sure Scala will eventually support them, as they would not require any deep changes in the implementation. They could be just implemented by desugaring to a higher-kinded type alias with a fresh name, but depending on when that desugaring is performed, it is possible that it would result in poor error messages. Supporting curried type functions is also quite desirable, but given my current knowledge of the internals, that seems like adding them will require some more elaborate changes.

I think Vincent Cremet was the first person to suggest this sort of encoding, and I vaguely recall reading about it on one of the Scala mailing lists, but I could not find the message after a little bit of time spent searching.

I also found it interesting that their dot product operator sounds more like the traditional Cartesian product type (one of each type), and their Cartesian product type sounds much more like an intersection type (the value is a structure having both types). Perhaps it is a notational holdover from previous work on species. It also seem to make for some nice concrete examples of the "locative" aspects of Girard's Ludics.

Now back to reviewing journal and ICFP papers...

Programming in OMeta would be very frustrating if all productions were defined in the same namespace: two grammars might unknowingly use the same name for two productions that have different purposes, and one of them would certainly stop working! (Picture one sword-wielding grammar decapitating another, Highlander-style: “There can be only one!”)

From Warth and Piumarta's OMeta: an Object-Oriented Language for Pattern Matching.

So after thinking about it further, I was incorrect, and it is possible to explicitly unpack existentials in Scala. As with some other languages, it is done via pattern matching. The following example illustrates how:

val x : List[T] forSome { type T } = List(42)
val w = x match { case y : List[u] => ((z : u) => z)(y.head) }

However, in practice this functionality seems to be rather fragile. For example, the following two variations are rejected:

val x : T forSome { type T } = 42
val w = x match { case y : u => ((z : u) => z)(y) }

and

val x : List[List[T]] forSome { type T } = List(List(42))
val w = x match { case y : List[List[u]] => ((z : u) => z)(y.head.head) }

In both cases it reports that it cannot find the type u. In the second case, it could be attributed to erasure, as there is no way dynamically to guarantee that the contents of List are in turn also a List. However, the first seems reasonable, so it should probably be reported as a bug.

Later, while working on Featherweight Scala, it occurred to me that subtyping is very likely to be undecidable in Scala because its language of types is essentially as expressive as full F_<:. Benjamin agreed that there existed a straightforward reduction to languages with only bounded existential quantification rather than bounded universal quantification, but did not know offhand if there is a paper that gives it.

So I spent a bit of time thinking about encoding universal quantification using existential quantification, and as alluded to in TAPL (the answer to exercise 24.3.3), it can be done, but there is no completely local encoding of universals into existentials. To obtain the classical equivalences between the negation of universal quantification and existential quantification (∀α.τ implies ¬∃α.¬τ) in a constructive setting you have to use the double-negation encoding, which essentially means CPS converting your program. The other classical equivalence under negation, ∃α.τ implies ¬∀α.¬τ, is constructively valid which is why it is existential quantification is macro-expressible in languages with universal quantification.

Since I am not aware of any papers that illustrate the encoding of universals into existentials, I figured I would explain it here. The encoding of the universal type pretty much follows from the equivalence:

|∀α.τ| = (∃α.|τ|→0)→0

Here I am using 0 for the void or empty type and using the standard constructive encoding of negation:

|¬τ| = |τ| → 0

The term level encodings are not so difficult, but does require a little thought:

|Λα.e : ∀α.τ|_k = k (λx:∃α.|τ|→0.let (α,k') = unpack x in |e : τ|_k')

The term encoding has the form |e : τ|_k, where τ is the type of the term and k is a continuation threaded through the encoding. The first thing to note is that because type abstractions are values, we always immediately pass them to the continuation. Given that,
if we look at the type we need to give the encoding of type abstraction, it is a function that that takes an existentially quantified continuation, and never returns. The only way to never return is to use the continuation. This tells us just about everything else we need to know about the encoding.

Because the continuation is existentially quantified, it is not possible to invoke it as is, so we must unpack it first. This is how we simulate the introduction of a type variable without using type abstraction. The next problem is that in order to call the continuation, we need to apply it to something with the correct type. The continuation takes values of type |τ|, possibly referencing a free type variable α, which is fortunately what the encoding of the type abstraction's body gives us. Therefore, we can use the unpacked continuation k' as the continuation argument of the recursive call to the encoding.

Given that we know how to encode type abstractions, encoding type applications is straightforward:

|e[τ] : τ'|_k = |e : ∀α.τ'{α/τ}|_{(λv:(∃α.τ'{α/τ}→0)→0.v (pack(τ,k) as ∃α.(τ'{α/τ}→0)))}

We encode the term to be applied and pass it a new continuation, that packs up the original continuation and applies it to function to which the term evaluates. I am writing {α/τ} to mean replace τ with α. I am using curly braces rather than the usual square brackets to avoid confusion with the square brackets that is commonly used for type application.

While I am almost certain that the encoding above is correct, it would be good at some point to prove it. This is the sort of proof for which Twelf is very well suited, so that is probably the route I would take.

Okay, so that is the theoretical half out of the way. Now, given the above, can we implement higher-rank impredicative polymoprhism in Scala? Yes, with a minor caveat.

First, I'll define an abstract class that defines the type of bounded universally quantified values, and methods for creating and destructing them.

abstract class Universal {
  type Univ[Bound,Body[_]]
 
  def createUniv[Bound,Body[_]]
      (impl : ((Body[A] => Nothing) forSome { type A <: Bound }) => Nothing) :
        Univ[Bound,Body]
 
  def applyUniv[Bound,Body[_],A<:Bound](arg : Univ[Bound,Body]) : Body[A]
}

This should be fairly straightforward given the encoding described above. The only tricky bit is the use of a higher-kinded type parameter (Body[_]) to give the body of the universal type. The other important point to note is that it is not necessary to completely CPS convert the implementation because Scala provides primitives for non-local control transfers.

One possible implementation of the above abstract class is the following:

class UniversalImpl extends Universal {
  type Univ[Bound,Body[_]] =
     ((Body[A] => Nothing) forSome { type A <: Bound }) => Nothing
 
  def createUniv[Bound,Body[_]]
      (impl : ((Body[A] => Nothing) forSome { type A <: Bound }) => Nothing) :
        Univ[Bound,Body] = impl
 
  def applyUniv [Bound,Body[_],A<:Bound](univ : Univ[Bound,Body]) : Body[A] = {
    case class Control(arg : Body[A]) extends Throwable
    val res = try {
      univ((arg:Body[A]) => throw new Control(arg))
    } catch {
      case Control(arg) => arg
      case e => throw e
    }
    res
  }
}

The implementation of the abstract type and the code for createUniv are trivial. The implementation of applyUniv is a little more interesting. Here, we create a local case class Control, extending Throwable so that it may be thrown as an exception, that will hold a value of the desired result type. We then just pass the representation of the type abstraction a continuation that throws a new instance of Control. We immediately catch it and return the value stored within. If we happen to catch some other sort of Throwable we just re-throw it.

And that's it. It is worth looking at a few examples of how this might be used. The first thing that is necessary is to create a concrete implementation of Universal:

val u : Universal = new UniversalImpl

Given u, we can create an instance of the polymorphic identity function:

type Id[T] = T => T
val id =
    u.createUniv[AnyRef,Id](
      (k: (Id[A] => Nothing) forSome { type A }) =>
         k(x => x))
val idString = u.applyUniv[AnyRef,Id,String](id)
val idStringList = u.applyUniv[AnyRef,Id,List[String]](id)
println(idString("Foo"))
println(idStringList(List("Foo", "Bar", "Baz")))

The first thing that needs to be done is to define a type function for representation the body of the universally quantified type. For the polymorphic identity function we call it Id[T]. The rest is pretty straightforward. It ought to be possible to make the above code a little less verbose if Scala allowed type parameters to methods to be curried: it should be possible for it to infer that the first two arguments to u.applyUniv are AnyRef and Id from the argument id. However, because it will not be able to infer the last type parameter, we have to give all three. It might also be desirable in some case to not have to give a type definition to define the body of the a universal type; this could be achieve by extending Scala with support for anonymous type functions.

Now for that caveat I mentioned. When we defined the type abstraction,

u.createUniv[AnyRef,Id](
  (k: (Id[A] => Nothing) forSome { type A }) =>
    k(x => x))

a significant difference between how this code is written, and how it would have been written in terms of my original explanation of the encoding, is that there is no explicit unpacking of the existential. All existential unpacking in Scala is implicit. This is nice in some ways, but it means that we have no way to give a name to type that the existential is hiding. Therefore, when constructing a function to pass to the continuation,

k(x => x)

it must be the case that Scala can infer the domain of this function's type, because it is not possible to write a type annotation for x without having a name for the existentially hidden type. I think in most cases this should not be possible, as Scala knows the type that k is expecting as an argument, but there may be some corner cases I have not considered.

(Does anyone know how to prevent WordPress from completely screwing up less than and greater than inside of <pre> tags?)