My original plan was that, upon returning from my vacation, I would spend my remaining time at EPFL writing a technical report explaining everything I knew about the problems with Scala Classic. Instead, on my first day back in the office, Martin came by with a draft of a new formal system, DOT (for Dependent Object Types), that he came up with while he was on vacation. After about four weeks I managed to root out pretty much all of the obvious problems with DOT, but another four weeks was not enough to get the proofs and the metatheory into a state that we were happy with. I am not really sure what will happen with DOT in the long term.

There are a few clever things about DOT that avoid some of the problems I encountered with Scala Classic. For example, Scala Classic only had intersection types while DOT has both intersection and union types, which solves some problems with computing the members of types.  However, with Scala Classic I was trying to model a minimal subset Scala language as it exists, without adding any new features that many never be supported by the full language. I have no idea if we will see union types in the actual Scala implementation anytime soon.

The other thing DOT does is give up the goal of trying to be a minimal subset of Scala and throws out quite a few important things from a user perspective. For example, there are no methods (only λ-abstractions), no existentials, no built-in inheritance or mix-in composition (though you can encode it by hand), and object fields must be syntactically values.

This last restriction is particularly important because it solves by fiat the problems that arose in Scala Classic from using paths that have an empty type in dependent type projections. However, it also means that you may not be able to directly encode some Scala programs into DOT without an effect or termination analysis. Therefore, while DOT has the potential to be a useful intermediary step, there will still be more work to be done to provide a semantics for a more realistic subset of the Scala language.

I have been in Copenhagen for a little over a month now, but I am not really ready to comment on the research I have been doing here yet.  So in the meantime I'll probably focus more on fonts and typography.

As soon as I started rewriting the code, I wondered why the original author hadn't just used an off the shelf searching algorithm. However, a quick search reminded me why: algorithms like Knuth-Morris-Pratt and Boyer-Moor construct a table based upon the sequence to search for. However, Scala sequences may be infinite so it is not possible to blindly go ahead and attempt to construct a table, because doing so may diverge.

Furthermore, there is no way to test whether a sequence is finite without potentially diverging. So it is not possible to first test whether the argument is finite, because if the receiver object is finite then indexOf should return that there is no match. Alternately, testing whether the receiver object is finite would be incorrect because it is possible the argument is finite an could potentially match.

However, it seems like it should still be possible to do better than O(nm), where n is the length of the receiver and m the length of the argument. For example if you start out with the sequence 1, 2, 3, 1 ... and the pattern 1, 3, 4 ... it seems like it should be possible to exploit the fact that you've looked ahead and know that there is no point and comparing 2 with 1. Alternately it seems like it might be possible to lazily build a table from the argument, but I would need to think longer to see whether it is always possible, in Knuth-Morris-Pratt for example, to fill in a complete prefix of the table without having processed the entire pattern.

In any event, searching with combinations of keywords like "string", "searching", "lazy", "infinite", etc. did not really turn anything up. One possible direction might be to look at "incremental" search algorithms like those used in text editors, etc. However, I expect that because they are geared to interactive use that the pattern will usually be quite small and therefore much thought has not been put into optimizing them.

Overall it was far less work than I was expecting.  Scala already internally supports what it calls "constant types", there is just no way to write them in Scala source code presently.  Consequently, most of the effort was in extending the parser.

Given my modifications, it is now possible to write code like the following:

 
scala> val x : "foo".type = "foo"
x: "foo".type = foo
 

What I was not expecting was that out-of-the-box things like the following would work:

 
scala> val x : "foobar".type = "foo" + "bar"
x: "foobar".type = foobar
scala> val y : 10.type = 2 * 5
y: 10.type = 10
scala> def frob(arg : 10.type) : 6.type = arg - 4
frob: (10.type)6.type
 

Unfortunately the excitement soon passes when you realize all the things you can't do with singleton literals (yet). Even if we turn on the experimental dependent method support, you can't write things like

 
def add(arg : Int) : (arg + 5).type = arg + 5
 

because these are exactly what they are called, singleton literals, not full-blown dependent types.

One cute example, based on a use suggested by Sean McDirmid, would be that some people might do something like the following with implicits:

 
implicit stringToColor(arg : String) : java.awt.Color = java.awt.Color.getColor(arg)
 

However, with singleton literals you can tighten it up by ensuring that for some strings it will never fail:

 
implicit redToColor(arg : "red".type) : java.awt.Color = java.awt.Color.RED
implicit blueToColor(arg : "blue".type) : java.awt.Color = java.awt.Color.BLUE
 

Happily, the type inferencer already chooses the most specific implicit conversions.

In any event, they will hopefully become more useful as the type system continues to grow. I am also sure someone will probably come up with a clever use for them that hasn't occurred to me yet. If so, let me know.

Several weeks ago I decided that instead of calling the core calculus that I have been working on Featherweight Scala, which could be confusing given that there is already a different calculus with that name, that I would call it Scala Classic.

I had hoped to submit a paper on Scala Classic to POPL 2009, but it has been taking too long to work out the metatheory. Partly because I am developing a mechanized proof of type soundness in Twelf, and on my first attempt at representing heaps I think tried to be too clever.  However, a more traditional treatment of heaps leads to the need to explicitly represent typing contexts, rather than the implicit treatment of contexts more commonly used in Twelf.

This afternoon I finally finished the proof of the theorem that is traditionally called "substitution".  However, the proof of type preservation will also require another theorem formalizing a substitution-like property for singleton types.  I am hoping that I can prove that theorem more quickly now that I've built up all the explicit context theory.  I have not thought much about whether there are any unusual theorems needed to prove progress.

In any event, I would ideally have the proofs finished by the beginning of August.  My current plan is to try submitting the paper to PLDI (which is even in Europe this year), unless some obviously better venue comes to my attention.

One of the reasons I would really like to have the proofs done by August is that I will be teaching a PhD level course here at EPFL this fall on mechanized sepcifications and proofs, with a strong emphasis on deductive systems. It also looks like I may fill in some for a masters level course based on TAPL.  So between those two things, I am not going to have nearly as much time for research as would be ideal.

And there are plenty of other Scalas about. In San Francisco:

Alas, we did not succeed in getting Martin to eat there while he was at JavaOne. Photo courtesy of Robby Findler.

So here is my example of how the encoding works. First, the SML version:

 
signature Eq = sig
  type t
  val eq: t -> t -> bool
end
 
signature RicherEq = sig
  include Eq
  val neq: t -> t -> bool
end
 
functor mkRicherEq(arg : Eq) :> RicherEq where type t = arg.t = struct
  type t = arg.t
  val eq = arg.eq
  fun neq x y = not (eq x y)
end
 

We can transliterate this example into Scala as:

 
type Eq = {
  type T
  def eq(x: T, y: T): Boolean
}
 
type RicherEq = {
  type T
  def eq(x: T, y: T): Boolean
  def neq(x: T, y: T): Boolean
}
 
def mkRicherEq(arg: Eq) : RicherEq { type T = arg.T } = new {
  type T = arg.T
  def eq(x: T, y: T) = arg.eq(x, y)
  def neq(x: T, y:T) = !eq(x, y)
}
 

The only problem I discovered is that it is not possible to define RicherEq in terms of Eq as we could in SML:

 
scala> type RicherEq = Eq { def neq(x: T, y: T): Boolean }
<console>:5: error: Parameter type in structural refinement may
not refer to abstract type defined outside that same refinement
       type RicherEq = Eq { def neq(x: T, y: T): Boolean }
                                       ^
<console>:5: error: Parameter type in structural refinement may
not refer to abstract type defined outside that same refinement
       type RicherEq = Eq { def neq(x: T, y: T): Boolean }
                                             ^
 

Why this restriction exists I don't know. In fact, this sort of refinement should work in the current version of Featherweight Scala, so perhaps it can be lifted eventually.

I still need to think about higher-order functors, and probably spend a few minutes researching existing proposals. I think this is probably something that cannot be easily supported in Scala if it will require allowing method invocations to appear in paths. However, off hand that only seems like it should be necessary for applicative higher-order functors, but again I definitely need to think it through.

For example, consider the following SML signature:

 
signature Nat = sig
  type t
  val z: t
  val s: t -> t
end
 

This signature can be translated in to Scala as:

 
type Nat = {
  type T
  val z: T
  def s(arg: T): T
}
 

It is then possible to create an implementation of this type, and opaquely seal it (hiding the definition of T). In SML:

 
structure nat :> Nat = struct
  type t = int
  val z = 0
  fun s n = n + 1
end
 

In Scala:

 
val nat : Nat = new {
  type T = Int
  val z = 0
  def s(arg: Int) = arg + 1
}
 

In many cases when programming with SML modules it is necessary or convenient to give a module that reveals the definition of an abstract type. In the above example, this can be done by adding a where type clause to the first line:

 
structure nat :> Nat where type t = int = struct
...
 

We can do the same thing in Scala using refinements:

 
val nat : Nat { type T = Int } = new {
...
 

Great, right? Well, almost. The problem is that structural types are still a bit buggy in Scala compiler at present. So, while the above typechecks, you can't quite use it yet:

 
scala> nat.s(nat.z)
java.lang.NoSuchMethodException: $anon$1.s(java.lang.Object)
        at java.lang.Class.getMethod(Class.java:1581)
        at .reflMethod$Method1(<console>:7)
        at .<init>(<console>:7)
        at .<clinit>(<console>)
        at RequestResult$.<init>(<console>:3)
        at RequestResult$.<clinit>(<console>)
        at RequestResult$result(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflec...
 

There were some issues raised about how faithful an encoding of SML functors, and well-known extensions for higher-order functors, one can get in Scala. Indeed, off the top of my head it is not entirely clear. So I need to think more about that before I write some examples.

Probably what makes working out the design of Featherweight Scala so difficult is that I am trying my best to commit to it being a valid subset of Scala. It seems like whenever I try to simplify one part of the language, it just introduces a complication elsewhere. Still, I think so far that I have been able to ensure that it is a subset with two exceptions.

The first exception is that some programs in Featherweight Scala will diverge in cases where the Scala compiler will generate a terminating program. This is because in programs that attempt to access an uninitialized field, Scala will return null, while Featherweight Scala diverges. Since the program many never attempt to use the null value, it may simple terminate without raising a NullPointerException.

The second exception arises because type checking in Featherweight Scala's type system is most likely undecidable. Given that the design is in flux, I haven't precisely verified this, but it seems quite plausible given its expressive power is similar to languages where type checking is known to be undecidable. Because type checking is undecidable, and the Scala language implementation attempts to have its type checking phase be terminating, there should be some programs that can be shown to be well-typed in Featherweight Scala that the Scala compiler will reject.

I think the other thing I've determined is that I do not understand F-bounded quantification as well as I thought I did. At least, I find myself worried about the cases where the typing rules presently require  templates, that have not yet been verified to be well-formed, to be introduced as hypotheses while verifying their own correctness. So I need to do some additional reading to see how some other languages deal with this issue. Certainly I can put my mind at ease by making the type system more restrictive, but I do not want to be so restrictive that it is no longer reasonable to call the language Featherweight Scala.

Update: Okay, so after looking at Featherweight Generic Java and νObj again more closely, they both do essentially the same kind of thing. In FGJ, the class table is well-formed if all its classes are well-formed, but checking a class's well-formedness may require presupposing that that class tale is well-formed. In νObj, the well-formedness of recursive record types is checked under the assumption that the record's self-name has the type of the record. Certainly the same sort of things come up when verifying the well-formedness of recursive functions and recursive types, but at least there what is being assumed is at a higher level of abstraction: when checking a recursive function you assume that the function has some well-formed type, or when checking a recursive type you assume that the type has some well-formed kind.

First we can define an infinite family of traits to represent n-ary universal quantification, much like Scala represents n-ary functions types:

 
trait Univ1[Bound1,Body[_]] {
   def Apply[T1<:Bound1] : Body[T1]
}
 
trait Univ2[Bound1,Bound2,Body[_,_]] {
   def Apply[T1<:Bound1,T2<:Bound2] : Body[T1,T2]
}
 
// ... so on for N > 2
 

Really, the key to this encoding is the use of higher-kinded quantification to encode the binding structure of the quantifier.

Now it is possible to write some examples similar to what I gave previously, but more concisely:

 
object Test extends Application {
   def id[T](x : T) = x
 
   type Id[T] = T => T
   val id = new Univ1[Any,Id]
     { def Apply[T <: Any] : Id[T] = id[T] _ }
 
   val idString = id.Apply[String]
   val idStringList = id.Apply[List[String]]
 
   println(idString("Foo"))
   println(idStringList(List("Foo", "Bar", "Baz")))
 
   type Double[T] = T => (T, T)
   val double = new Univ1[Any,Double]
     { def Apply[T <: Any] : Double[T] = (x : T) => (x, x) }
 
   val doubleString = double.Apply[String]
   val doubleStringList = double.Apply[List[String]]
 
   println(doubleString("Foo"))
   println(doubleStringList(List("Foo", "Bar", "Baz")))
}
 

As I mentioned previously, this example would be much improved by support for anonymous type functions in Scala. I am pretty sure Scala will eventually support them, as they would not require any deep changes in the implementation. They could be just implemented by desugaring to a higher-kinded type alias with a fresh name, but depending on when that desugaring is performed, it is possible that it would result in poor error messages. Supporting curried type functions is also quite desirable, but given my current knowledge of the internals, that seems like adding them will require some more elaborate changes.

I think Vincent Cremet was the first person to suggest this sort of encoding, and I vaguely recall reading about it on one of the Scala mailing lists, but I could not find the message after a little bit of time spent searching.

trait Ordered[A] {
def compare(that: A): Int
def <  (that: A): Boolean = (this compare that) <  0
def >  (that: A): Boolean = (this compare that) >  0
def <= (that: A): Boolean = (this compare that) <= 0
def >= (that: A): Boolean = (this compare that) >= 0
def compareTo(that: A): Int = compare(that)
}

However, the Ordered trait does not provide a representation of a total ordering. Therefore, the new trait Ordering:

 
trait Ordering[T] extends PartialOrdering[T] {
  def compare(x: T, y: T): Int
  override def lteq(x: T, y: T): Boolean = compare(x, y) <= 0
  override def gteq(x: T, y: T): Boolean = compare(x, y) >= 0
  override def lt(x: T, y: T): Boolean = compare(x, y) < 0
  override def gt(x: T, y: T): Boolean = compare(x, y) > 0
  override def equiv(x: T, y: T): Boolean = compare(x, y) == 0
}
 

The tricky part however, was writing description of the properties required of something that implements the Ordering trait. When one normally thinks of a total ordering one thinks of a relation that is

• anti-symmetric,
• transitive,
• and total.

The problem is that Ordering is not defined in terms of a binary relation, but a binary function producing integers (compare). If the first argument is less than the second the function returns a negative integer, if they are equal in the ordering the function returns zero, and if the second argument is less than the first the function returns a positive integer. Therefore, it is not straightforward to express these same properties. The best I could come up with was

• compare(x, x) == 0, for any x of type T.
• compare(x, y) == z and compare(y, x) == w then Math.signum(z) == -Math.signum(w), for any x and y of type T and z and w of type Int.
• if compare(x, y) == z and lteq(y, w) == v and Math.signum(z) >= 0 and Math.signum(v) >= 0 then
  compare(x, w) == u and Math.signum(z + v) == Math.signum(u),
  for any x, y, and w of type T and z, v, and u of type Int.

Where Math.signum returns -1 if its input is negative, 0 if its input is 0, and 1 if its input is positive.

The first property is clearly reflexivity. I call the third property transitivity. I am not sure what to call the second property. I do not think a notion of totality is required because it is assumed you will always get an integer back from compare rather than it throwing an exception or going into an infinite loop.

It would probably be a good exercise to prove that given these properties on compare hold if and only if lteq (defined above in terms of compare) has all the normal properties of a total ordering.