Module descriptors under META-INF not found

[jafarre-bi]

Affected version

4.0.0-beta-2

Bug description

jackson-core 3.0.0-RC9 has the module-info.class located at META-INF/versions/17/module-info.class.
This is correctly recognised when compiling with the plugin version 3.14.0, but it isn't recognised with plugin version 4.0.0-beta-2, and the build crashes because tools.jackson.core module can't be found by other modules that depend on it.
Is this a bug in the 4.x versions or an intentional change?

[desruisseaux]

Can you post the <dependency> block in the pom.xml file that declare this dependency? I would like to see in particular if a <type> is declared.

[jafarre-bi]

It's a transitive dependency of:

        <dependency>
            <groupId>tools.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
        </dependency>

The dependency management is:

            <dependency>
                <groupId>tools.jackson</groupId>
                <artifactId>jackson-bom</artifactId>
                <version>3.0.0-rc9</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>

Inside the BOM I see:

      <dependency>
        <groupId>tools.jackson.core</groupId>
        <artifactId>jackson-core</artifactId>
        <version>3.0.0-rc9</version>
      </dependency>
      <dependency>
        <groupId>tools.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>3.0.0-rc9</version>
      </dependency>

[desruisseaux]

Can you try to add the following line in the Jackson dependency?

      <type>modular-jar</type>

If the module-info.class file was ignored, it suggests that the Jackson JAR file was placed on the class-path rather than the module-path. When the type is not specified, or when the type is jar, Maven uses heuristic rules for deciding where to place the JAR file. There is maybe a regression in the heuristic rules of compiler plugin 4.0.0-beta-2. But when classpath-jar or modular-jar type is specified, there is no heuristic rules: the plugin will obey strictly to what have been specified. I would recommend to always specify these dependency types in a Maven 4 project for making the build more deterministic.

Nevertheless, there is maybe a regression in 4.0.0-beta-2 if we forgot to check in the META-INF/services/ directory. I will investigate. In the meantime, if you have a chance to confirm whether <type>modular-jar</type> works, that would be nice to know since avoiding this kind of issue was the whole point of allowing explicit control on class-path versus module-path in Maven 4.

[jafarre-bi]

Ok, the results require some detailed explanation. As jackson-core is a transitive dependency of jackson-databind, I tried several things:

1. Making jackson-databind dependency modular-jar type -> No effect.
2. Making jackson-core dependency modular-jar type -> Compiles correctly, but there is a different error in testCompile.
3. Making both jackson-core and jackson-databind dependencies modular-jar type -> Compiles correctly, but there is a different error in testCompile.
   The error in testCompile is:

[ERROR] error: --patch-module specified more than once for module my.module.using.jackson

So, it seems to be a problem with the heuristics. Please note that in this case module-info.class is under META_INF/versions, not META-INF/services.

By the way, now that I tried it, these new dependency types make dependency management much more complicated, because, potentially, you could need to manage each dependency three times: one for type jar, one for type classpath-jar and one for type modular-jar.

[desruisseaux]

META-INF/services/ was my typo. Indeed we mean META-INF/version/.

To my knowledge, the types declarations are not transitive, so it is normal that setting the type of jackson-databind has no effect on the type of jackson-core dependency. Maybe the type should be transitive, that would be an open question.

Regarding the error about --patch-module, I would like to know if this error is still present on 4.0.0-beta-3. Testing would require downloading Maven 4.0.0-rc-4, running mvn install on the master branch of the compiler plugin and setting the plugin version to 4.0.0-beta-3-SNAPSHOT in your project. Additionally, if you have a chance to test with the #959 pull request as well, that would be great.

There is probably a regression in the heuristic regarding the scanning of META-INF/version/, but fixing that heuristic would only make the plugin behaves as if <type>modular-jar</type> has been specified. The --patch-module issue is a different problem, not caused by the heuristic.

[desruisseaux]

By the way, now that I tried it, these new dependency types make dependency management much more complicated, because, potentially, you could need to manage each dependency three times: one for type jar, one for type classpath-jar and one for type modular-jar.

I'm not sure to understand. Why declaring the same dependency 3 times? If the idea is to build and test the same project once with the dependency on the class-path, and once again with the dependency on the module-path, this is not something that we did before and not the goal of those types neither. The use of those types should not change the way that we manage dependencies, except making the JAR placement explicit.

[jafarre-bi]

I'm managing a parent pom with generic dependencies to be reused by projects developed by independent teams. The intention is to standardise the software versions across modules and teams. We can't control the type of dependency people will use in their poms, but we need to manage the version of the artefact in all cases. This is where the need for three entries arises. Adding a dependency to the module path or the class path feels like an orthogonal concept to the dependency type. I have a jar dependency with one artefact, that's the dependency, and I want it added to this or that path. Perhaps it would be more suitable to set that at scope level, or with a totally new concept like true/false</modular). Just thinking out loud.

[desruisseaux]

I think that as single parent POM would still work as today, after we fix the regression. As a bonus, there will be an extra level of control when desired:

• <type>jar</type> (the default) should continue to behave as today (pending regression fix).
• <type>modular-jar</type> means "we know that this JAR doesn't work on the class-path (or we don't want it on the class-path for whatever reason). Unconditionally place it on the module-path, no matter if the project using that dependency is modular or not".
• <type>classpath-jar</type> is the converse of the above.

[jafarre-bi]

I think that as single parent POM would still work as today, after we fix the regression. As a bonus, there will be an extra level of control when desired:

Thank you, I misunderstood your previous message. This is fine for me, then.
About the problem with the --patch-module, it may be due to the code under test being modular, but the tests not declaring a module themselves. They are unit tests in the same package as the code under test, and that module is being automatically patched with the test classes added to it. May it be that when specifying the dependencies as module-jar there is another module patching over the previous one and that's the reason why --patch-module is duplicated?

[desruisseaux]

About the problem with the --patch-module, it may be due to the code under test being modular, but the tests not declaring a module themselves. They are unit tests in the same package as the code under test, and that module is being automatically patched with the test classes added to it.

Yes, it is the way that it should work. The --patch-module options should be emitted automatically by the compiler plugin, and the plugin is supposed to manage duplications itself (for the options emitted by the plugin itself. The plugin will not check if the user has put --patch-module options manually in a <compilerArgs> plugin configuration).

The code that manages the --patch-module option has been improved in 4.0.0-beta-3, and improved again in #959, which is why I would like to know if this bug is still present in those versions. The problem may also be due to apache/maven#11118, which will be fixed in Maven 4.0.0-rc-5. In short: the plugin uses an HashMap for detecting duplicated --patch-module options. But because of missing hashCode and equals methods in JavaPathType.Modular, those duplications were never detected.

[jafarre-bi]

With beta-3:

• It still can't find the dependency module, unless I specify module-jar.
• If I specify module-jar, the duplicate --patch-module doesn't happen anymore, but I get a different error. It's a bit unsettling because in this case, it doesn't even mention the dependency, only the module I'm compiling and unit testing.
• The PR doesn't make a difference.

I attach the log of the new error:

log.txt

[desruisseaux]

It still can't find the dependency module, unless I specify module-jar.

It is normal. As said in above comment, what has been improved in 4.0.0-beta-3 is the handling of --patch-module option. The search for module-info in META-INF/versions is a separated issue.

For the error that you get, would it be possible to send a test case?

[jafarre-bi]

While trying to create a minimal reproducer (not yet done) I realised about this message while compiling main sources:

[WARNING] Filename-based automodules detected on the module path:
  - jackson-core-3.0.0-rc9.jar
Please don't publish this project to a public artifact repository.

Looks like it isn't detecting the module-info.class, it's just considering Jackson as an automodule.