Improve stub error messages (SCP-009 proposal)

The following commit message is a squash of several commit messages.

- This is the 1st commit message:

Add position to stub error messages

Stub errors happen when we've started the initialization of a symbol but
key information of this symbol is missing (the information cannot be
found in any entry of the classpath not sources).

When this error happens, we better have a good error message with a
position to the place where the stub error came from. This commit goes
into this direction by adding a `pos` value to `StubSymbol` and filling
it in in all the use sites (especifically `UnPickler`).

This commit also changes some tests that test stub errors-related
issues. Concretely, `t6440` is using special Partest infrastructure and
doens't pretty print the position, while `t5148` which uses the
conventional infrastructure does. Hence the difference in the changes
for both tests.

- This is the commit message #2:

Add partest infrastructure to test stub errors

`StubErrorMessageTest` is the friend I introduce in this commit to help
state stub errors. The strategy to test them is easy and builds upon
previous concepts: we reuse `StoreReporterDirectTest` and add some
methods that will compile the code and simulate a missing classpath
entry by removing the class files from the class directory (the folder
where Scalac compiles to).

This first iteration allow us to programmatically check that stub errors
are emitted under certain conditions.

- This is the commit message #3:

Improve contents of stub error message

This commit does three things:

* Keep track of completing symbol while unpickling

  First, it removes the previous `symbolOnCompletion` definition to be
  more restrictive/clear and use only positions, since only positions are
  used to report the error (the rest of the information comes from the
  context of the `UnPickler`).

  Second, it adds a new variable called `lazyCompletingSymbol` that is
  responsible for keeping a reference to the symbol that produces the stub
  error. This symbol will usually (always?) come from the classpath
  entries and therefore we don't have its position (that's why we keep
  track of `symbolOnCompletion` as well). This is the one that we have to
  explicitly use in the stub error message, the culprit so to speak.

  Aside from these two changes, this commit modifies the existing tests
  that are affected by the change in the error message, which is more
  precise now, and adds new tests for stub errors that happen in complex
  inner cases and in return type of `MethodType`.

* Check that order of initialization is correct

  With the changes introduced previously to keep track of position of
  symbols coming from source files, we may ask ourselves: is this going to
  work always? What happens if two symbols the initialization of two
  symbols is intermingled and the stub error message gets the wrong
  position?

  This commit adds a test case and modifications to the test
  infrastructure to double check empirically that this does not happen.
  Usually, this interaction in symbol initialization won't happen because
  the `UnPickler` will lazily load all the buckets necessary for a symbol
  to be truly initialized, with the pertinent addresses from which this
  information has to be deserialized. This ensures that this operation is
  atomic and no other symbol initialization can happen in the meantime.

  Even though the previous paragraph is the feeling I got from reading the
  sources, this commit creates a test to double-check it. My attempt to be
  better safe than sorry.

* Improve contents of the stub error message

  This commit modifies the format of the previous stub error message by
  being more precise in its formulation. It follows the structured format:

  ```
  s"""|Symbol '${name.nameKind} ${owner.fullName}.$name' is missing from the classpath.
      |This symbol is required by '${lazyCompletingSymbol.kindString} ${lazyCompletingSymbol.fullName}'.
  ```

  This format has the advantage that is more readable and explicit on
  what's happening. First, we report what is missing. Then, why it was
  required. Hopefully, people working on direct dependencies will find the
  new message friendlier.

Having a good test suite to check the previously added code is
important. This commit checks that stub errors happen in presence of
well-known and widely used Scala features. These include:

* Higher kinded types.
* Type definitions.
* Inheritance and subclasses.
* Typeclasses and implicits.

- This is the commit message #4:

Use `lastTreeToTyper` to get better positions

The previous strategy to get the last user-defined position for knowing
what was the root cause (the trigger) of stub errors relied on
instrumenting `def info`.

This instrumentation, while easy to implement, is inefficient since we
register the positions for symbols that are already completed.

However, we cannot do it only for uncompleted symbols (!hasCompleteInfo)
because the positions won't be correct anymore -- definitions using stub
symbols (val b = new B) are for the compiler completed, but their use
throws stub errors. This means that if we initialize symbols between a
definition and its use, we'll use their positions instead of the
position of `b`.

To work around this we use `lastTreeToTyper`. We assume that stub errors
will be thrown by Typer at soonest.

The benefit of this approach is better error messages. The positions
used in them are now as concrete as possible since they point to the
exact tree that **uses** a symbol, instead of the one that **defines**
it. Have a look at `StubErrorComplexInnerClass` for an example.

This commit removes the previous infrastructure and replaces it by the
new one. It also removes the fields positions from the subclasses of
`StubSymbol`s.

- This is the commit message #5:

Keep track of completing symbols

Make sure that cycles don't happen by keeping track of all the
symbols that are being completed by `completeInternal`. Stub errors only
need the last completing symbols, but the whole stack of symbols may
be useful to reporting other error like cyclic initialization issues.

I've added this per Jason's suggestion. I've implemented with a list
because `remove` in an array buffer is linear. Array was not an option
because I would need to resize it myself. I think that even though list
is not as efficient memory-wise, it probably doesn't matter since the
stack will usually be small.

- This is the commit message #6:

Remove `isPackage` from `newStubSymbol`

Remove `isPackage` since in 2.12.x its value is not used.

src/compiler/scala/tools/nsc/Global.scala

@@ -96,6 +96,20 @@ class Global(var currentSettings: Settings, var reporter: Reporter)
 
   def erasurePhase: Phase = if (currentRun.isDefined) currentRun.erasurePhase else NoPhase
 
+  /* Override `newStubSymbol` defined in `SymbolTable` to provide us access
+   * to the last tree to typer, whose position is the trigger of stub errors. */
+  override def newStubSymbol(owner: Symbol,
+                             name: Name,
+                             missingMessage: String,
+                             isPackage: Boolean = false): Symbol = {
+    val stubSymbol = super.newStubSymbol(owner, name, missingMessage, isPackage)
+    val stubErrorPosition = {
+      val lastTreeToTyper = analyzer.lastTreeToTyper
+      if (lastTreeToTyper != EmptyTree) lastTreeToTyper.pos else stubSymbol.pos
+    }
+    stubSymbol.setPos(stubErrorPosition)
+  }
+
   // platform specific elements
 
   protected class GlobalPlatform extends {

src/compiler/scala/tools/nsc/symtab/classfile/ClassfileParser.scala

@@ -1030,8 +1030,11 @@ abstract class ClassfileParser {
       val sflags      = jflags.toScalaFlags
       val owner       = ownerForFlags(jflags)
       val scope       = getScope(jflags)
-      def newStub(name: Name) =
-        owner.newStubSymbol(name, s"Class file for ${entry.externalName} not found").setFlag(JAVA)
+      def newStub(name: Name) = {
+        val stub = owner.newStubSymbol(name, s"Class file for ${entry.externalName} not found")
+        stub.setPos(owner.pos)
+        stub.setFlag(JAVA)
+      }
 
       val (innerClass, innerModule) = if (file == NoAbstractFile) {
         (newStub(name.toTypeName), newStub(name.toTermName))
@@ -1152,7 +1155,11 @@ abstract class ClassfileParser {
         if (enclosing == clazz) entry.scope lookup name
         else lookupMemberAtTyperPhaseIfPossible(enclosing, name)
       )
-      def newStub = enclosing.newStubSymbol(name, s"Unable to locate class corresponding to inner class entry for $name in owner ${entry.outerName}")
+      def newStub = {
+        enclosing
+          .newStubSymbol(name, s"Unable to locate class corresponding to inner class entry for $name in owner ${entry.outerName}")
+          .setPos(enclosing.pos)
+      }
       member.orElse(newStub)
     }
   }

src/partest-extras/scala/tools/partest/StubErrorMessageTest.scala

@@ -0,0 +1,47 @@
+package scala.tools.partest
+
+trait StubErrorMessageTest extends StoreReporterDirectTest {
+  // Stub to feed to partest, unused
+  def code = throw new Error("Use `userCode` instead of `code`.")
+
+  val classpath = List(sys.props("partest.lib"), testOutput.path)
+    .mkString(sys.props("path.separator"))
+
+  def compileCode(codes: String*) = {
+    val global = newCompiler("-cp", classpath, "-d", testOutput.path)
+    val sourceFiles = newSources(codes: _*)
+    withRun(global)(_ compileSources sourceFiles)
+  }
+
+  def removeClasses(inPackage: String, classNames: Seq[String]): Unit = {
+    val pkg = new File(testOutput.path, inPackage)
+    classNames.foreach { className =>
+      val classFile = new File(pkg, s"$className.class")
+      assert(classFile.exists)
+      assert(classFile.delete())
+    }
+  }
+
+  def removeFromClasspath(): Unit
+  def codeA: String
+  def codeB: String
+  def userCode: String
+  def extraUserCode: String = ""
+
+  def show(): Unit = {
+    compileCode(codeA)
+    assert(filteredInfos.isEmpty, filteredInfos)
+
+    compileCode(codeB)
+    assert(filteredInfos.isEmpty, filteredInfos)
+    removeFromClasspath()
+
+    if (extraUserCode == "") compileCode(userCode)
+    else compileCode(userCode, extraUserCode)
+    import scala.reflect.internal.util.Position
+    filteredInfos.map { report =>
+      print(if (report.severity == storeReporter.ERROR) "error: " else "")
+      println(Position.formatMessage(report.pos, report.msg, true))
+    }
+  }
+}

src/reflect/scala/reflect/internal/Symbols.scala

@@ -182,6 +182,17 @@ trait Symbols extends api.Symbols { self: SymbolTable =>
 
   private[reflect] case class SymbolKind(accurate: String, sanitized: String, abbreviation: String)
 
+  protected def newStubSymbol(owner: Symbol,
+                              name: Name,
+                              missingMessage: String,
+                              isPackage: Boolean = false): Symbol = {
+    name match {
+      case n: TypeName  => if (isPackage) new StubPackageClassSymbol(owner, n, missingMessage)
+                           else new StubClassSymbol(owner, n, missingMessage)
+      case _            => new StubTermSymbol(owner, name.toTermName, missingMessage)
+    }
+  }
+
   /** The class for all symbols */
   abstract class Symbol protected[Symbols] (initOwner: Symbol, initPos: Position, initName: Name)
           extends SymbolContextApiImpl
@@ -505,9 +516,9 @@ trait Symbols extends api.Symbols { self: SymbolTable =>
      *  failure to the point when that name is used for something, which is
      *  often to the point of never.
      */
-    def newStubSymbol(name: Name, missingMessage: String, isPackage: Boolean = false): Symbol = name match {
-      case n: TypeName  => if (isPackage) new StubPackageClassSymbol(this, n, missingMessage) else new StubClassSymbol(this, n, missingMessage)
-      case _            => new StubTermSymbol(this, name.toTermName, missingMessage)
+    def newStubSymbol(name: Name, missingMessage: String, isPackage: Boolean = false): Symbol = {
+      // Invoke the overriden `newStubSymbol` in Global that gives us access to typer
+      Symbols.this.newStubSymbol(this, name, missingMessage, isPackage)
     }
 
     /** Given a field, construct a term symbol that represents the source construct that gave rise the field */
@@ -3491,7 +3502,7 @@ trait Symbols extends api.Symbols { self: SymbolTable =>
     private def fail[T](alt: T): T = {
       // Avoid issuing lots of redundant errors
       if (!hasFlag(IS_ERROR)) {
-        globalError(missingMessage)
+        globalError(pos, missingMessage)
         if (settings.debug.value)
           (new Throwable).printStackTrace
 

src/reflect/scala/reflect/internal/pickling/UnPickler.scala

@@ -266,13 +266,15 @@ abstract class UnPickler {
               adjust(mirrorThatLoaded(owner).missingHook(owner, name)) orElse {
                 // (5) Create a stub symbol to defer hard failure a little longer.
                 val advice = moduleAdvice(s"${owner.fullName}.$name")
+                val lazyCompletingSymbol = completingStack.headOption.getOrElse(NoSymbol)
                 val missingMessage =
-                  s"""|missing or invalid dependency detected while loading class file '$filename'.
-                      |Could not access ${name.longString} in ${owner.kindString} ${owner.fullName},
-                      |because it (or its dependencies) are missing. Check your build definition for
-                      |missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
+                  s"""|Symbol '${name.nameKind} ${owner.fullName}.$name' is missing from the classpath.
+                      |This symbol is required by '${lazyCompletingSymbol.kindString} ${lazyCompletingSymbol.fullName}'.
+                      |Make sure that ${name.longString} is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
                       |A full rebuild may help if '$filename' was compiled against an incompatible version of ${owner.fullName}.$advice""".stripMargin
-                owner.newStubSymbol(name, missingMessage)
+                val stubName = if (tag == EXTref) name else name.toTypeName
+                // The position of the error message is set by `newStubSymbol`
+                NoSymbol.newStubSymbol(stubName, missingMessage)
               }
             }
           }
@@ -717,11 +719,18 @@ abstract class UnPickler {
       new TypeError(e.msg)
     }
 
+    /** Keep track of the symbols pending to be initialized.
+      *
+      * Useful for reporting on stub errors and cyclic errors.
+      */
+    private var completingStack = List.empty[Symbol]
+
     /** A lazy type which when completed returns type at index `i`. */
     private class LazyTypeRef(i: Int) extends LazyType with FlagAgnosticCompleter {
       private val definedAtRunId = currentRunId
       private val p = phase
       protected def completeInternal(sym: Symbol) : Unit = try {
+        completingStack = sym :: completingStack
         val tp = at(i, () => readType(sym.isTerm)) // after NMT_TRANSITION, revert `() => readType(sym.isTerm)` to `readType`
 
         // This is a temporary fix allowing to read classes generated by an older, buggy pickler.
@@ -744,7 +753,10 @@ abstract class UnPickler {
       }
       catch {
         case e: MissingRequirementError => throw toTypeError(e)
+      } finally {
+        completingStack = completingStack.tail
       }
+
       override def complete(sym: Symbol) : Unit = {
         completeInternal(sym)
         if (!isCompilerUniverse) markAllCompleted(sym)

test/files/neg/t5148.check

@@ -1,11 +1,7 @@
-error: missing or invalid dependency detected while loading class file 'Imports.class'.
-Could not access type Wrapper in class scala.tools.nsc.interpreter.IMain.Request,
-because it (or its dependencies) are missing. Check your build definition for
-missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
-A full rebuild may help if 'Imports.class' was compiled against an incompatible version of scala.tools.nsc.interpreter.IMain.Request.
-error: missing or invalid dependency detected while loading class file 'Imports.class'.
-Could not access type Request in class scala.tools.nsc.interpreter.IMain,
-because it (or its dependencies) are missing. Check your build definition for
-missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
-A full rebuild may help if 'Imports.class' was compiled against an incompatible version of scala.tools.nsc.interpreter.IMain.
-two errors found
+t5148.scala:4: error: Symbol 'type <none>.Request.Wrapper' is missing from the classpath.
+This symbol is required by 'value scala.tools.nsc.interpreter.Imports.wrapper'.
+Make sure that type Wrapper is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
+A full rebuild may help if 'Imports.class' was compiled against an incompatible version of <none>.Request.
+class IMain extends Imports
+            ^
+one error found

test/files/run/StubErrorBInheritsFromA.check

@@ -0,0 +1,6 @@
+error: newSource1.scala:4: Symbol 'type stuberrors.A' is missing from the classpath.
+This symbol is required by 'class stuberrors.B'.
+Make sure that type A is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
+A full rebuild may help if 'B.class' was compiled against an incompatible version of stuberrors.
+      new B
+          ^

test/files/run/StubErrorBInheritsFromA.scala

@@ -0,0 +1,22 @@
+object Test extends scala.tools.partest.StubErrorMessageTest {
+  def codeA = """
+    package stuberrors
+    class A
+  """
+
+  def codeB = """
+    package stuberrors
+    class B extends A
+  """
+
+  def userCode = """
+    package stuberrors
+    class C {
+      new B
+    }
+  """
+
+  def removeFromClasspath(): Unit = {
+    removeClasses("stuberrors", List("A"))
+  }
+}

test/files/run/StubErrorComplexInnerClass.check

@@ -0,0 +1,6 @@
+error: newSource1.scala:9: Symbol 'type stuberrors.A' is missing from the classpath.
+This symbol is required by 'class stuberrors.B.BB'.
+Make sure that type A is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
+A full rebuild may help if 'B.class' was compiled against an incompatible version of stuberrors.
+        new b.BB
+            ^

test/files/run/StubErrorComplexInnerClass.scala

@@ -0,0 +1,42 @@
+object Test extends scala.tools.partest.StubErrorMessageTest {
+  def codeA = """
+    package stuberrors
+    class A
+  """
+
+  def codeB = """
+    package stuberrors
+    class B {
+      def foo: String = ???
+
+      // unused and should fail, but not loaded
+      def unsafeFoo: A = ???
+      // used, B.info -> BB.info -> unpickling A -> stub error
+      class BB extends A
+    }
+  """
+
+  def userCode = """
+    package stuberrors
+    class C {
+      def aloha = {
+        val b = new B
+        val d = new extra.D
+        d.foo
+        println(b.foo)
+        new b.BB
+      }
+    }
+  """
+
+  override def extraUserCode = """
+    package extra
+    class D {
+      def foo = "Hello, World"
+    }
+  """.stripMargin
+
+  def removeFromClasspath(): Unit = {
+    removeClasses("stuberrors", List("A"))
+  }
+}

test/files/run/StubErrorHK.check

@@ -0,0 +1,6 @@
+error: newSource1.scala:4: Symbol 'type stuberrors.A' is missing from the classpath.
+This symbol is required by 'type stuberrors.B.D'.
+Make sure that type A is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
+A full rebuild may help if 'B.class' was compiled against an incompatible version of stuberrors.
+      println(new B)
+                  ^

test/files/run/StubErrorHK.scala

@@ -0,0 +1,22 @@
+object Test extends scala.tools.partest.StubErrorMessageTest {
+  def codeA = """
+    package stuberrors
+    class A
+  """
+
+  def codeB = """
+    package stuberrors
+    class B[D <: A]
+  """
+
+  def userCode = """
+    package stuberrors
+    object C extends App {
+      println(new B)
+    }
+  """
+
+  def removeFromClasspath(): Unit = {
+    removeClasses("stuberrors", List("A"))
+  }
+}

test/files/run/StubErrorReturnTypeFunction.check

@@ -0,0 +1,6 @@
+error: newSource1.scala:13: Symbol 'type stuberrors.A' is missing from the classpath.
+This symbol is required by 'method stuberrors.B.foo'.
+Make sure that type A is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
+A full rebuild may help if 'B.class' was compiled against an incompatible version of stuberrors.
+      b.foo
+      ^

test/files/run/StubErrorReturnTypeFunction.scala

@@ -0,0 +1,37 @@
+object Test extends scala.tools.partest.StubErrorMessageTest {
+  def codeA = """
+    package stuberrors
+    class A
+    class AA
+  """
+
+  def codeB = """
+    package stuberrors
+
+    abstract class B {
+      def bar: String = ???
+      def foo: A = new A
+      def baz: String = ???
+    }
+  """
+
+  def userCode = """
+    package stuberrors
+
+    abstract class C extends App {
+      val b = new B {}
+
+      // Use other symbols in the meanwhile
+      val aa = new AA
+      val dummy = 1
+      println(dummy)
+
+      // Should blow up
+      b.foo
+    }
+  """
+
+  def removeFromClasspath(): Unit = {
+    removeClasses("stuberrors", List("A"))
+  }
+}

test/files/run/StubErrorReturnTypeFunction2.check

@@ -0,0 +1,6 @@
+error: newSource1.scala:13: Symbol 'type stuberrors.A' is missing from the classpath.
+This symbol is required by 'method stuberrors.B.foo'.
+Make sure that type A is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
+A full rebuild may help if 'B.class' was compiled against an incompatible version of stuberrors.
+      b.foo
+      ^