Compiler API

[Peter Niederwieser]

Hi,

I'm looking for a way to invoke the Scala compiler programmatically. Ideally, I'd pass it some source code and receive java.lang.Class objects for all classes defined in the source. I've digged around in the scala.tools.nsc package, but haven't found a clear entry point that would seem to satisfy my needs. Can anyone point me in the right direction?

Cheers
Peter

[sean.mcdirmid]

Its not that easy, but not really hard either. Depending on what you are doing, you can look at the Scala
Interpreter front end, which compiles lines of Scala code and then loads the resulting class in a class
loader to execute, or TestPad in the IDE (similar), or the IDE builder, which provides an alternative front
end to Scalac.

Basically, you have to create a global object, a run object in Global, and then pass it a list of source files
names that you want compiled. Afterwards, you'll get CompilationUnits back, not java.lang.Class
objects. If you want, you can then load the generated class files in a ClassLoader then instantiate and
execute.

Sean

Quoting Peter Niederwieser <[hidden email]>:

>
> Hi,
>
> I'm looking for a way to invoke the Scala compiler programmatically.
> Ideally, I'd pass it some source code and receive java.lang.Class
> objects
> for all classes defined in the source. I've digged around in the
> scala.tools.nsc package, but haven't found a clear entry point that
> would
> seem to satisfy my needs. Can anyone point me in the right direction?
>
> Cheers
> Peter
> --
> View this message in context:
> http://www.nabble.com/Compiler-API-tf4235472.html#a12050645
> Sent from the Scala mailing list archive at Nabble.com.
>
>

[Eric Willigers]

[hidden email] wrote:

> Its not that easy, but not really hard either. Depending on what you are doing, you can look at the Scala
> Interpreter front end, which compiles lines of Scala code and then loads the resulting class in a class
> loader to execute, or TestPad in the IDE (similar), or the IDE builder, which provides an alternative front
> end to Scalac.
>
> Basically, you have to create a global object, a run object in Global, and then pass it a list of source files
> names that you want compiled. Afterwards, you'll get CompilationUnits back, not java.lang.Class
> objects. If you want, you can then load the generated class files in a ClassLoader then instantiate and
> execute.
>
> Sean

Hi Peter,

Here's some sample code that was informed by scala.tools.ant.Scalac . A
couple of times I've had to change a line here or there for a new Scala
release.  Sorry, it doesn't return you a list of Class objects.

Eric.





         // We currently call the compiler directly
         // To reduce coupling, we could instead use ant and the scalac
ant task


         import scala.tools.nsc.{Global, Settings}
         import scala.tools.nsc.reporters.ConsoleReporter

         {
             // called in the event of a compilation error
             def error(message: String): Nothing = ...

             val settings = new Settings(error)
             settings.outdir.value = classesDir.getPath
             settings.deprecation.value = true // enable detailed
deprecation warnings
             settings.unchecked.value = true // enable detailed
unchecked warnings

             val reporter = new ConsoleReporter(settings)

             val compiler = new Global(settings, reporter)
             (new compiler.Run).compile(filenames)

             reporter.printSummary
             if (reporter.hasErrors || reporter.WARNING.count > 0)
             {
                 ...
             }
         }


         val mainMethod: Method =
         {
             val urls = Array[URL]( classesDir.toURL )

             val loader = new URLClassLoader(urls)

             try
             {
                 val clazz: Class = loader.loadClass(...)

                 val method: Method = clazz.getMethod("main",
Array[Class]( classOf[Array[String]] ))
                 if (Modifier.isStatic(method.getModifiers))
                 {
                     method
                 }
                 else
                 {
                     ...
                 }
             }
             catch
             {
                 case cnf: ClassNotFoundException => ...
                 case nsm: NoSuchMethodException => ...
             }
         }

         mainMethod.invoke(null, Array[Object]( args ))

[Peter Niederwieser]

Thanks for your answers. Do I always have to go through the file system to load the classes that I've just compiled? I'd rather want to stay in memory, but I don't see how this can be achieved.

I guess what I'm really asking for is a flexible, easy-to-use and well-documented public compiler API... :-)

Cheers
Peter

Eric Willigers wrote

[e-mail] wrote:
> Its not that easy, but not really hard either. Depending on what you are doing, you can look at the Scala
> Interpreter front end, which compiles lines of Scala code and then loads the resulting class in a class
> loader to execute, or TestPad in the IDE (similar), or the IDE builder, which provides an alternative front
> end to Scalac.
>
> Basically, you have to create a global object, a run object in Global, and then pass it a list of source files
> names that you want compiled. Afterwards, you'll get CompilationUnits back, not java.lang.Class
> objects. If you want, you can then load the generated class files in a ClassLoader then instantiate and
> execute.
>
> Sean


Hi Peter,

Here's some sample code that was informed by scala.tools.ant.Scalac . A
couple of times I've had to change a line here or there for a new Scala
release.  Sorry, it doesn't return you a list of Class objects.

Eric.





         // We currently call the compiler directly
         // To reduce coupling, we could instead use ant and the scalac
ant task


         import scala.tools.nsc.{Global, Settings}
         import scala.tools.nsc.reporters.ConsoleReporter

         {
             // called in the event of a compilation error
             def error(message: String): Nothing = ...

             val settings = new Settings(error)
             settings.outdir.value = classesDir.getPath
             settings.deprecation.value = true // enable detailed
deprecation warnings
             settings.unchecked.value = true // enable detailed
unchecked warnings

             val reporter = new ConsoleReporter(settings)

             val compiler = new Global(settings, reporter)
             (new compiler.Run).compile(filenames)

             reporter.printSummary
             if (reporter.hasErrors || reporter.WARNING.count > 0)
             {
                 ...
             }
         }


         val mainMethod: Method =
         {
             val urls = Array[URL]( classesDir.toURL )

             val loader = new URLClassLoader(urls)

             try
             {
                 val clazz: Class = loader.loadClass(...)

                 val method: Method = clazz.getMethod("main",
Array[Class]( classOf[Array[String]] ))
                 if (Modifier.isStatic(method.getModifiers))
                 {
                     method
                 }
                 else
                 {
                     ...
                 }
             }
             catch
             {
                 case cnf: ClassNotFoundException => ...
                 case nsm: NoSuchMethodException => ...
             }
         }

         mainMethod.invoke(null, Array[Object]( args ))

[sean.mcdirmid]

Don't worry, the filesystem will cache the just written file, so there shouldn't be
any performance implications (its as if you took the bytes directly and did the
defineClass). The JVM doesn't support native dynamic code generation (without
going through defineClass), so this is how it must be done in Java land. I think
the DLR in .NET has something a bit better, but I haven't looked at it closely
enough yet.

Quoting Peter Niederwieser <[hidden email]>:

>
> Thanks for your answers. Do I always have to go through the file system
> to
> load the classes that I've just compiled? I'd rather want to stay in
> memory,
> but I don't see how this can be achieved.
>
> I guess what I'm really asking for is a flexible, easy-to-use and
> well-documented public compiler API... :-)
>
> Cheers
> Peter
>
>
> Eric Willigers wrote:
> >
> > [hidden email] wrote:
> >> Its not that easy, but not really hard either. Depending on what you
> are
> >> doing, you can look at the Scala
> >> Interpreter front end, which compiles lines of Scala code and then
> loads
> >> the resulting class in a class
> >> loader to execute, or TestPad in the IDE (similar), or the IDE
> builder,
> >> which provides an alternative front
> >> end to Scalac.
> >>
> >> Basically, you have to create a global object, a run object in
> Global,
> >> and then pass it a list of source files
> >> names that you want compiled. Afterwards, you'll get
> CompilationUnits
> >> back, not java.lang.Class
> >> objects. If you want, you can then load the generated class files in
> a
> >> ClassLoader then instantiate and
> >> execute.
> >>
> >> Sean
> >
> >
> > Hi Peter,
> >
> > Here's some sample code that was informed by scala.tools.ant.Scalac .
> A
> > couple of times I've had to change a line here or there for a new
> Scala
> > release.  Sorry, it doesn't return you a list of Class objects.
> >
> > Eric.
> >
> >
> >
> >
> >
> >          // We currently call the compiler directly
> >          // To reduce coupling, we could instead use ant and the
> scalac
> > ant task
> >
> >
> >          import scala.tools.nsc.{Global, Settings}
> >          import scala.tools.nsc.reporters.ConsoleReporter
> >
> >          {
> >              // called in the event of a compilation error
> >              def error(message: String): Nothing = ...
> >
> >              val settings = new Settings(error)
> >              settings.outdir.value = classesDir.getPath
> >              settings.deprecation.value = true // enable detailed
> > deprecation warnings
> >              settings.unchecked.value = true // enable detailed
> > unchecked warnings
> >
> >              val reporter = new ConsoleReporter(settings)
> >
> >              val compiler = new Global(settings, reporter)
> >              (new compiler.Run).compile(filenames)
> >
> >              reporter.printSummary
> >              if (reporter.hasErrors || reporter.WARNING.count > 0)
> >              {
> >                  ...
> >              }
> >          }
> >
> >
> >          val mainMethod: Method =
> >          {
> >              val urls = Array[URL]( classesDir.toURL )
> >
> >              val loader = new URLClassLoader(urls)
> >
> >              try
> >              {
> >                  val clazz: Class = loader.loadClass(...)
> >
> >                  val method: Method = clazz.getMethod("main",
> > Array[Class]( classOf[Array[String]] ))
> >                  if (Modifier.isStatic(method.getModifiers))
> >                  {
> >                      method
> >                  }
> >                  else
> >                  {
> >                      ...
> >                  }
> >              }
> >              catch
> >              {
> >                  case cnf: ClassNotFoundException => ...
> >                  case nsm: NoSuchMethodException => ...
> >              }
> >          }
> >
> >          mainMethod.invoke(null, Array[Object]( args ))
> >
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Compiler-API-tf4235472.html#a12057351
> Sent from the Scala mailing list archive at Nabble.com.
>
>

[Lex Spoon-3]

Peter Niederwieser <[hidden email]> writes:
> Thanks for your answers. Do I always have to go through the file system to
> load the classes that I've just compiled? I'd rather want to stay in memory,
> but I don't see how this can be achieved.

It would be nice to keep it in memory, but nobody has ever put
together a full solution to that problem.  It would be great, though,
if you could specify an output directory such as "memory://foo".

The interactive shell and the script runner could both make good use
of such a facility, if it were available.  Any volunteers??  :)


-Lex

[Jim White-2]

sean.mcdirmid wrote

...
The JVM doesn't support native dynamic code generation (without
going through defineClass), so this is how it must be done in Java land.
...

Many languages (certainly the ones I use like Kawa and Groovy) do dynamic code generation on the JVM without using the file system.

And yes, ClassLoader.defineClass is the route for all class definitions (static and dynamic) and it doesn't use files (or even java.io), it uses byte[].

Jim

[√iktor Klang]

On Nov 18, 2007 3:02 PM, Jim White <[hidden email]> wrote:

sean.mcdirmid wrote:
>
> ...

> The JVM doesn't support native dynamic code generation (without
> going through defineClass), so this is how it must be done in Java land.

> ...
>

Many languages (certainly the ones I use like Kawa and Groovy) do dynamic
code generation on the JVM without using the file system.

And yes, ClassLoader.defineClass is the route for all class definitions
(static and dynamic) and it doesn't use files (or even java.io), it uses
byte[].

Jim

I have to agree with Jim on this one. There are actually a few solutions that do not use the file system for dynamic class-creation/definition.
For instance: http://en.wikipedia.org/wiki/ObjectWeb_ASM

Best regards
-Viktor
 

--
View this message in context: http://www.nabble.com/Compiler-API-tf4235472.html#a13819448

Sent from the Scala mailing list archive at Nabble.com.

[sean.mcdirmid]

True. But so what? Performance shouldn't be much different as newly
written files are cached in memory. Is there another good reason to
have scalac write out to a byte buffer and not a temp file?

Quoting Viktor Klang <[hidden email]>:

> On Nov 18, 2007 3:02 PM, Jim White <[hidden email]> wrote:
>
> >
> >
> > sean.mcdirmid wrote:
> > >
> > > ...
> > > The JVM doesn't support native dynamic code generation (without
> > > going through defineClass), so this is how it must be done in Java
> land.
> > > ...
> > >
> >
> > Many languages (certainly the ones I use like Kawa and Groovy) do
> dynamic
> > code generation on the JVM without using the file system.
> >
> > And yes, ClassLoader.defineClass is the route for all class
> definitions
> > (static and dynamic) and it doesn't use files (or even java.io), it
> uses
> > byte[].
> >
> > Jim
>
>
> I have to agree with Jim on this one. There are actually a few
> solutions
> that do not use the file system for dynamic class-creation/definition.
> For instance: http://en.wikipedia.org/wiki/ObjectWeb_ASM
>
> Best regards
> -Viktor
>
>
> >
> >
> > --
> > View this message in context:
> > http://www.nabble.com/Compiler-API-tf4235472.html#a13819448
> > Sent from the Scala mailing list archive at Nabble.com.
> >
> >
>

[√iktor Klang]

On Nov 18, 2007 4:13 PM, <[hidden email]> wrote:

True. But so what? Performance shouldn't be much different as newly
written files are cached in memory. Is there another good reason to
have scalac write out to a byte buffer and not a temp file?

From my perspective it has more to do with: Why write it to a file, if there's no advantage of doing that?

Best regards
-Viktor
 

Quoting Viktor Klang <[hidden email]>:

> On Nov 18, 2007 3:02 PM, Jim White <[hidden email]> wrote:
>
> >
> >
> > sean.mcdirmid wrote:
> > >
> > > ...
> > > The JVM doesn't support native dynamic code generation (without
> > > going through defineClass), so this is how it must be done in Java
> land.
> > > ...
> > >
> >
> > Many languages (certainly the ones I use like Kawa and Groovy) do
> dynamic
> > code generation on the JVM without using the file system.
> >
> > And yes, ClassLoader.defineClass is the route for all class
> definitions
> > (static and dynamic) and it doesn't use files (or even java.io), it
> uses
> > byte[].
> >
> > Jim
>
>
> I have to agree with Jim on this one. There are actually a few
> solutions
> that do not use the file system for dynamic class-creation/definition.
> For instance: http://en.wikipedia.org/wiki/ObjectWeb_ASM

>

> Best regards
> -Viktor
>
>
> >
> >
> > --
> > View this message in context:
> > http://www.nabble.com/Compiler-API-tf4235472.html#a13819448
> > Sent from the Scala mailing list archive at Nabble.com.
> >
> >
>

[Jim White-2]

sean.mcdirmid wrote

True. But so what? Performance shouldn't be much different as newly
written files are cached in memory. Is there another good reason to
have scalac write out to a byte buffer and not a temp file?

Besides performance there is security and installation complexity.

By requiring file access your compiler/interpreter must have permission to write and read files, *and* your user must figure out some acceptable place for you to do temporary file things.  

For command line usage it is not hard to come up with acceptable defaults, but in container environments and other managed applications those things are not so easy.

Also I think this thread started regarding JSR-223 (which is something I'm going to use so that my WYSIWYG literate scripting tool "Wings"  <http://www.ifcx.org/> can support many languages and how I got on this thread), and so if your language engine doesn't play nicely with the standard scripting API then it won't get used as widely as it might otherwise.

Jim

[sean.mcdirmid]

Quoting Viktor Klang <[hidden email]>:

> On Nov 18, 2007 4:13 PM, <[hidden email]> wrote:
>
> From my perspective it has more to do with: Why write it to a file, if
> there's no advantage of doing that?

If you are a compiler writer, you have to justify a new back-end that
will output classes to a buffer rather than file. Its not so huge of a
change (someone in the community could do it as a patch), but it
requires development effort nonetheless. All scripting solutions up
until now have used the file system well enough. Not sure about
the security issues (having access to temp files) but thats only a
big deal in on-client web applications.

[√iktor Klang]

On Nov 18, 2007 8:54 PM, <[hidden email]> wrote:

Quoting Viktor Klang <[hidden email]>:

> On Nov 18, 2007 4:13 PM, <[hidden email]> wrote:
>

> From my perspective it has more to do with: Why write it to a file, if
> there's no advantage of doing that?

If you are a compiler writer, you have to justify a new back-end that
will output classes to a buffer rather than file. Its not so huge of a
change (someone in the community could do it as a patch), but it
requires development effort nonetheless. All scripting solutions up
until now have used the file system well enough. Not sure about
the security issues (having access to temp files) but thats only a
big deal in on-client web applications.

Hi Sean,
I understand your point of view, definately.
But outputting it into a buffer is a Good Thing(tm).
Just think about it, what if you'd use Terracotta and replicate the buffer across a grid and have the possibility to use classes generated by one node, in another?
Without pesky file management.

As I think about it I feel file systems have gone the same route as RDBMSs, they're used as  standard storage, no matter the specific needs of the task.

I apologize if it seems like my tone of voice seems blunt, that is not my intention at all.
I've just promised myself to look at stuff at different angles than I have gotten used to.

Best regards
-Viktor

[Lex Spoon-3]

Jim White wrote:
> Many languages (certainly the ones I use like Kawa and Groovy) do dynamic
> code generation on the JVM without using the file system.

I missed the beginning of this thread, but let me say I agree that it would
be nice if the Scala interpreter generated its classfiles into memory
instead of onto the underlying file system.

Somebody has to implement it, though.  It's not just a matter of writing a
class loader, because you have to also convince the compiler to write into
this in-memory virtual file system.  It's a lot of details.

-Lex

[Lex Spoon-3]

Lex Spoon wrote:
> I missed the beginning of this thread, but let me say I agree that it
> would be nice if the Scala interpreter generated its classfiles into
> memory instead of onto the underlying file system.
>
> Somebody has to implement it, though.  It's not just a matter of writing a
> class loader, because you have to also convince the compiler to write into
> this in-memory virtual file system.  It's a lot of details.

Let me try and sketch a solution, for anyone who wants to hack on this.  I
thought a lot about this exact feature, and the smoothest approach I could
think of was to come up with some string like "<mem>" that designates
in-memory file systems.  Then the interpreter could invoke the compiler
like this:

        scalac -d '<mem>/scalac123' line123.scala

It could read out of this virtual directory, too:

        scala -cp '<mem>/scalac123'

(Of course, the interpreter actually uses the programmatic API instead of
invoking the compiler via the command-line interface.)

The advantage of this approach is that the logic for classpath management
stays the same, and this is a fragile part of the compiler's behavior!  
The compiler just has to be careful to interpret these '<mem>' filenames
wherever it sees them, whether in a -d argument or a classpath argument.
Likewise, the interpreter just has to swap its "temporary directory" for
one of these <mem> filenames, and otherwise it can specify class paths in
the same way.


Also, be aware that the compiler already has an abstraction for files and
directories; it does not use the Java IO classes directly.  It needs this
for reading from jars.  Thus a lot of the necessary work is already done.

So overall, keeping classfiles in memory requires several implementation
details, but it seems perfectly doable.

-Lex

[Alex Cruise-2]

Lex Spoon wrote:

> Let me try and sketch a solution, for anyone who wants to hack on this.  I
> thought a lot about this exact feature, and the smoothest approach I could
> think of was to come up with some string like "<mem>" that designates
> in-memory file systems.  Then the interpreter could invoke the compiler
> like this:
>
>         scalac -d '<mem>/scalac123' line123.scala
>
> It could read out of this virtual directory, too:
>
>         scala -cp '<mem>/scalac123'
>  

Why not use URIs, which were invented precisely for use cases like this one?

e.g. scala-classpath:mem:/my/name/space vs.
scala-classpath:file:/my/name/space

-0xe1a

[Alex Cruise-2]

Alex Cruise wrote:
> e.g. scala-classpath:mem:/my/name/space vs.
> scala-classpath:file:/my/name/space
Oops, obviously file:/foo is a better choice for the usual file-based
approach; one should try to avoid polluting a global namespace if one
can help it. :)

-0xe1a