IKVM.NET

I've been doing lots of modularization work, but I'm not quite ready to check it in yet. I've now split the class library assembly into ten assemblies and I've got "shared class loader" support working (at least for the core class ... [More] library scenario). I've been positively surprised by how many scenarios can be supported while only loading IKVM.OpenJDK.Core.dll. As the graph below shows, it has more (necessarily circular) dependencies than I would like, but for a number of scenarios I've carefully tweaked the set of classes in Core to enable lazy loading the dependencies only when they are really needed. I'm not yet ready to commit to supported "Core-only" scenarios, but here's a flavor of some things I've been able to do: Running "Hello, World!" in dynamic mode Serialization Reflection File I/O & Socket I/O (both classic and nio) Packages included in Core: java.io java.lang java.lang.annotation java.lang.ref java.lang.reflection java.math java.net java.nio java.nio.channels java.nio.channels.spi java.nio.charset java.nio.charset.spi java.security java.security.cert java.util java.util.concurrent java.util.concurrent.atomic java.util.concurrent.locks java.util.regex javax.net Here is the assembly dependency graph (click the image to enlarge): Here are the current assembly file sizes: IKVM.OpenJDK.Core.dll 3,278,848 IKVM.OpenJDK.Security.dll 2,646,016 IKVM.OpenJDK.Util.dll 1,111,040 IKVM.OpenJDK.Xml.dll 8,497,664 IKVM.OpenJDK.SwingAWT.dll 3,040,768 IKVM.OpenJDK.Charsets.dll 5,017,088 IKVM.OpenJDK.Corba.dll 2,335,232 IKVM.OpenJDK.Management.dll      1,180,160 IKVM.OpenJDK.Misc.dll 2,668,032 IKVM.OpenJDK.Text.dll 628,736   30,403,584 It's interesting to see that the total is slightly less than the previous size of IKVM.OpenJDK.ClassLibrary.dll (30,472,704). [Less]

Recently Mark Reinhold started blogging about modularizing the JDK. I've been getting requests to split up IKVM.OpenJDK.ClassLibrary.dll for a long time. It's good to see that the Java platform is also moving in that direction, ... [More] but we needn't wait for that. I've been working on making ikvmc support "multi target" mode for a while now. Last week I used this mode to compile the OpenJDK classes into 743 different assemblies (one per package) . I then tried to use NDepend to dig through the dependencies, but it turns out that 743 assemblies is a bit too much to handle (NDepend handled it resonably well, but the dependency graph was way too large to be useful). So I started moving some obvious stuff together. The result was this NDepend report. Still a lot of data and a lot of dependencies, but some patterns are starting to emerge. Here's my preliminary view of how things could be split: IKVM.OpenJDK.Core.dll 5 MB IKVM.OpenJDK.Security.dll 3 MB IKVM.OpenJDK.Util.dll 1 MB IKVM.OpenJDK.Xml.dll 8 MB IKVM.OpenJDK.SwingAWT.dll 3 MB IKVM.OpenJDK.Charsets.dll 5 MB IKVM.OpenJDK.Corba.dll 2 MB IKVM.OpenJDK.Management.dll      2 MB IKVM.OpenJDK.Misc.dll 3 MB (The sizes are only approximate.) I had originally hoped to make IKVM.OpenJDK.Core.dll smaller by keeping java.util, java.net and java.io out of it, but it looks like you won't be able to run anything non-trivial without requiring classes from these packages, so it makes more sense to put them into the core assembly. The IKVM.OpenJDK.Security.dll and  IKVM.OpenJDK.Util.dll assemblies contain other util and security related packages that shouldn't be needed as often. It is possible to split packages across assemblies (e.g. java.awt.AWTPermission will be in IKVM.OpenJDK.Core.dll because java.lang.SecurityManager depends on it), but given the potential for confusion my current thinking is that it is probably best to only move individual classes into Core should it be necessary (because, realistically, you can't develop without having a reference to Core you're less likely to be confused when trying to locate the class). To avoid confusion or expectations that are too high: I haven't yet built the runtime infrastructure to support this. So while I can compile the class library into all these parts, the runtime won't actually be able to work correctly, because it still expects all the boot class loader classes in a single assembly. As always, feedback on the proposed split is very welcome. NDepend I really like NDepend's ability to show the dependencies in different ways and the interactive dependency matrix that allows you to drill down into a dependency to see exactly where it comes from. Beyond dependency analysis it also has a power SQL like query language to allows use to query dependencies and compute all the code metrics you can come up with. It also includes a number of code metrics out of the box, but those aren't really my cup of tea, so I can't comment how useful they are. One other small but really nice thing about it is that you can run it without installing. IMO this is very nice compared with installers that do who knows what to your system (and require administrator access). An evaluation copy can be downloaded from their website. Full disclosure: I was given a free copy of NDepend Professional Edition. [Less]

More ikvmc testing revealed a couple of IKVM.Reflection.Emit bugs and also some bugs that have been lurking in ikvmc for a long time, that only show up under fairly uncommon scenarios when lots of dependencies are missing (and hence the ... [More] resulting assembly has little chance of working anyway). The fixes are small and relatively low risk, but I'm not sure whether it is worthwhile to back port them to 0.36 and 0.38. Feedback on this is appreciated. Changes: Added support for SpecialNameAttribute and TypeForwardedToAttribute pseudo custom attributes to IKVM.Reflection.Emit. Fixed IKVM.Reflection.Emit relocation table bug. Fixed IKVM.Reflection.Emit TypeDefOrRef and HasCustomAttribute column size calculations. A couple of ikvmc fixes in dealing with missing classes. This fixes #2351524. Fixed regression introduced in 0.39.3240 that caused failure when reflection triggered a call to a static initializer on a dynamically loaded class. As always with a development snapshot, don't use this in production, but please do try it out and let me know about it. The sources are available in cvs and the binaries here: ikvmbin-0.39.3257.zip [Less]

One of the really cool things about the CLR, when it was introduced, was the notion of custom attributes. Custom attributes provide an extensible way to add custom metadata. However, since .NET compilers virtually have to support custom ... [More] attributes, it's tempting to reuse this capability for specifying non-custom metadata. It isn't efficient to encode all metadata in a generic way, hence the pseudo custom attributes were born. From most compilers they appear as normal custom attributes, but the compiler stores them in the metadata in a way that is specific to each pseudo custom attribute. To recap, my definition of pseudo custom attributes is: Any custom attribute that modifies the metadata, but isn't stored in the CustomAttribute metadata table. While implementing ikvmc (and later IKVM.Reflection.Emit) I looked for a definitive list of pseudo custom attributes, but there doesn't appear to be one, so I thought I'd make one:   ECMA CLI IKVM.Reflection.Emit System.Reflection.Emit AssemblyAlgorithmIdAttribute Yes* No No AssemblyCultureAttribute No No No AssemblyFlagsAttribute Yes* No No AssemblyKeyFileAttribute No No No AssemblyKeyNameAttribute No No No AssemblyVersionAttribute No No No CodeAccessSecurityAttribute** Yes No No ComImportAttribute No Yes Yes DefaultParameterValueAttribute No Yes No DllImportAttribute Yes Yes Yes FieldOffsetAttribute Yes Yes Yes InAttribute Yes Yes Yes MarshalAsAttribute Yes Yes Yes MethodImplAttribute Yes Yes Yes NonSerializedAttribute No Yes Yes OptionalAttribute No Yes Yes OutAttribute Yes Yes Yes PreserveSigAttribute No Yes Yes SerializableAttribute No*** Yes Yes SpecialNameAttribute No Yes**** Yes StructLayoutAttribute Yes Yes Yes TypeForwardedToAttribute No Yes**** No * Reserved only. ** Its subclasses. *** Not part of the spec, but the spec does mention it. **** Not in most recent development snapshot, but code is in cvs. (System.Reflection.Emit values based on .NET 3.5 SP1) For completeness and clarity, here is a list of non-pseudo custom attributes that are sometimes mistakenly referred to as pseudo custom attributes: AllowPartiallyTrustedCallersAttribute Understood by the runtime, but stored normally. ClassInterfaceAttribute Understood by the runtime and compiler, but stored normally. ComCompatibleVersionAttribute Understood by the runtime (?) and compiler, but stored normally. DebuggableAttribute Understood by the runtime, but stored normally. GuidAttribute Understood by the runtime and compiler, but stored normally. InterfaceTypeAttribute Understood by the runtime and compiler, but stored normally. SecurityCriticalAttribute Understood by the runtime, but stored normally. SecurityTransparentAttribute Understood by the runtime, but stored normally. SecurityTreatAsSafeAttribute Understood by the runtime, but stored normally. TypeLibVersionAttribute Understood by the compiler, but stored normally.   Clarifications, corrections and additions to both lists are very much appreciated. Update: Removed incorrect claim that the runtime only supports the DebuggableAttribute (bool,bool) constructor. Thanks to Kevin M. Owen for the correction. [Less]

I've now enabled "multi target" support. This means that you can compile multiple assemblies at once. One of the advantages this has is that it makes dealing with circular dependencies much easier, but even if there aren't any circular ... [More] dependencies, it can make building much easier because you don't have to do any dependency analysis. As an example I compiled the jars from the JBoss lib directory: ikvmc -target:library { commons-codec.jar } { commons-httpclient.jar } { commons-logging.jar } { concurrent.jar } { getopt.jar } { jboss-common.jar } { jboss-jmx.jar } { jboss-system.jar } { jboss-xml-binding.jar } { log4j-boot.jar } The curly braces define the targets. This is all it takes to compile these jars into seperate assemblies that automatically reference eachother, as required. Using NDepend (which Patrick Smacchia kindly gave me a free copy of) you can easily graph the dependencies of the resulting assemblies: In this case there aren't any cycles, but had there been, it would have made no differences. The multi target feature is by no means done. I want to add a -sharedclassloader option to enable multiple assemblies to share a single Java class loader. I'm also considering some options to select which classes/packages go in which assembly. This would be helpful for splitting the core class library into multiple assemblies. Changes: Added support to ikvmstub for automatically exporting non-vector array types. Implemented/fixed support for pointers, by ref and non-vector arrays in IKVM.Reflection.Emit. Fixed pointer detection to work for types with multiple indirection levels. Changed proxy stub name mangling to work around ildasm bug. Fixed IKVM.Reflection.Emit to not write token of FAULT pseudo exception that is used as a marker for fault blocks. Fixed class loading to take ClassLoader lock. Changed ikvmc not to generate warning when it loads ikvmstub generated classes. Removed our version of System.Runtime.CompilerServices.ExtensionAttribute and instead add System.Core.jar to the compilation, this will allow us to reference the real ExtensionAttribute when it is available and yet the build will still work (albeit with a warning and without the ExtensionAttribute) when it is not available (i.e. when building on .NET 2.0). Did some clean up and restructuring in IKVM.Reflection.Emit. Fixed sorting of InterfaceImpl table in IKVM.Reflection.Emit. Removed hard coded public keys from JniInterface.cs and Configuration.java. Changed build process so that version number for all assemblies only has to be specified in CommonAssemblyInfo.cs. Enabled multi target support. Fixed typo in IKVM.Reflection.Emit that caused assembly minor version to be set to major version. Added assembly version to ikvmstub. As always with a development snapshot, don't use this in production, but please do try it out and let me know about it. The sources are available in cvs and the binaries here: ikvmbin-0.39.3245.zip [Less]

One of the lesser known features of the .NET runtime is the support for multidimensional and/or non-zero-based arrays. The typical arrays that you use (and that are the same as the arrays in Java) are called vectors in the CLI ... [More] specification, but unfortunately the non-vector arrays don't have a specific name (they are simply called arrays in the CLI specification). In C# you can easily create a non-vector array: int[,] arr1 = new int[4, 4] This creates a two dimensional array of integers with 16 elements. This array is zero based. If you want to create a non-zero-based array, you have to use the API, because C# doesn't directly support that: int[,] arr2 = (int[,])Array.CreateInstance(typeof(int), new[] { 4, 4 }, new[] { 1, 1 }); This also creates a two dimensional array of integers with 16 elements, but in this case the indexes run from 1 through 5. So far, so good. Now let's look at how things look at the IL level. If you use ildasm to look at the first C# based instantation you'll see: ldc.i4.1 ldc.i4.1 newobj instance void int32[0...,0...]::.ctor(int32, int32) Most of this looks straightforward if you're familiar with IL, except for the [0...,0...] part. It looks like the lower bounds are part of the type, but a simple experiment shows that this is not the case: Console.WriteLine(arr1.GetType() == arr2.GetType()); This prints out True. This implies that the lower bounds are not part of the type (and the CLI specification confirms this). So why are they part of the signature? I don't know for sure, but it does have an interesting consequence. You can overload methods based on this: .assembly extern mscorlib { } .assembly Test { } .module Test.exe .class public Program {   .method public static void Foo(int32[0...,0...] foo)   {     ret   }   .method public static void Foo(int32[1...,1...] foo)   {     ret   }   .method private static void Main()   {     .entrypoint     ldc.i4.1     ldc.i4.1     newobj instance void int32[0...,0...]::.ctor(int32, int32)     call void Program::Foo(int32[0...,0...])     ret   } } This is a valid and verifiable application. Unfortunately, using the  .NET reflection API it is impossible to see the difference in signatures between the two Foo methods. This limitation in reflection means that code compiled with the new IKVM.Reflection.Emit based ikvmc won't be able to call methods that have non-zero-based array parameters. It is also impossible to override these methods, but that was already the case with the previous System.Reflection.Emit based implementation as well. Finally, in the above text I talk about the lower bounds, but the same thing applies to the upper bounds of the array. [Less]

Over the past two months I've been working on reimplementing a large portion of the Reflection.Emit API from scratch. After finally growing tired of the System.Reflection.Emit bugs and limitations and not finding Mono.Cecil satisfactory ... [More] either, I decided to build my own implementation. I started out with these design goals: API compatible with System.Reflection.Emit (as much as possible). Implement only the Emit part and be compatible with System.Reflection. Only implement functionality required by ikvmc, but not implemented functionality shouldn't silently fail (i.e. it should throw a NotSupportedException or NotImplementedException). This also means that the API is mostly write-only. Efficient implementation, optimized for ikvmc specific scenarios. I think I've met or exceeded all of the design goals. Without doing any significant performance work on my Ref.Emit implementation (other than the design), ikvmc became so much faster that it is rather emberassing for the Microsoft System.Reflection.Emit implementation. I've only had to make a couple of minor changes to the ikvmc sources (apart from changing using System.Reflection.Emit; to using IKVM.Reflection.Emit; in every file) to account for the fact that IKVM.Reflection.Emit.ModuleBuilder and IKVM.Reflection.Emit.AssemblyBuilder unfortunately cannot extend System.Reflection.Module and System.Reflection.Assembly. However, it looks like this is fixed in the .NET 4.0 CTP. Here are some random statistics about compiling IKVM.OpenJDK.ClassLibrary.dll on .NET 2.0 SP2 x64:   System.Reflection.Emit     IKVM.Reflection.Emit   File size 31,645,696 30,480,896   bytes CPU time used 272 35   seconds Peak virtual memory 1,433,399,296 1,035,018,240   bytes Generation 0 GCs 770 896   Generation 1 GCs 201 240   Generation 2 GCs 11 8   (The huge memory usage is not because it actually needs that much memory, but simply the result of the fact that garbage collection is more efficient if you have more memory available and that my system had about 1.5GB of free memory while running these tests.) The smaller file size is because System.Reflection.Emit always uses fat method headers and IKVM.Reflection.Emit uses tiny method headers whenever possible. There is still some work left to do, I've only spent limited time on debugging support and there is no support for Mono's .mdb format yet. I also haven't done any testing on Mono yet. BTW, thanks to Sebastien Pouliot for code I lifted from Mono to parse strong name CAPI key blobs. Other changes in this snapshot: Dropped support for Visual Studio 2005. Added error message when map.xml references non-existing constructor. Added more statistics to ikvmc -time option output. As always with a development snapshot, don't use this in production, but please do try it out and let me know about it. The sources are available in cvs and the binaries here: ikvmbin-0.39.3240.zip [Less]

A codegen bug was found (not a regression, so there will be a 0.36 update as well) so here's another release candidate. Changes: Changed version to 0.38.0.2. Fixed openjdk.build BOM issue on Linux. ... [More] Fixed jsr verifier bug that caused incorrect codegen under very specific circumstances (thanks to Brian Schwallier for tracking down a repro). Binaries available here: ikvmbin-0.38.0.2.zip Sources ( binaries): Sources: ikvm-0.38.0.2.zip, classpath-0.95-stripped.zip, openjdk6-b12-stripped.zip [Less]