Skip to content

Conversation

@wadoon
Copy link
Member

@wadoon wadoon commented Apr 16, 2023

This PR replaces recoder by javaparser, or more precisely by key-javaparser.
The special version has support for ProofJava, SchemaJava and Java 17+, all in one grammar.

Notes on the grammar can be found in the key-docs

This MR requires a complete overhaul of the KeY-Infrastructure.

Status:

  • Migration of KeY/Java transformations from recoder to javaparser
    • Get compile-error free for testing
    • Check JavaInfo for unused methods and delete them
    • Testing (wip)
      • Currently there is one transformation example
      • JavaRedux is parsed and translated.
      • Tests for translation from JP-AST to KeY-AST
  • Reimplementing the old infrastructure on top of Javaparser
  • Tests
    • What to do with TestProofJavaParser and TestSchemaJavaParser?
    • TestKeYRecoderMapping
  • Activate the new implementation.

@wadoon wadoon changed the title Weigl/key javaparser3 The removal of Recoder Apr 18, 2023
@wadoon wadoon added this to the v3.0.0 milestone Apr 18, 2023
@wadoon wadoon added Feature New feature or request Java Parser labels Apr 18, 2023
@github-actions
Copy link

github-actions bot commented Apr 18, 2023

Thank you for your contribution.

The test artifacts are available on Artiweb.
The newest artifact is here.

@wadoon
Copy link
Member Author

wadoon commented Apr 20, 2023

@jwiesler Sources of key-javaparser are now available via maven.

@jwiesler
Copy link
Contributor

jwiesler commented May 10, 2023

About NameInfo: How do we find what is actually needed by KeY from this interface, I've checked a few methods and many of them are just used by Recoder.

Edit: Took a look around and I didn't find any usages outside of the XX2KeY-converters.

@jwiesler jwiesler force-pushed the weigl/key-javaparser3 branch from e9e9885 to b2bd6c5 Compare May 22, 2023 12:40
@wadoon
Copy link
Member Author

wadoon commented May 28, 2023

contraposition loads again.

Syntax changes to catch #cs were necessary.

@jwiesler
Copy link
Contributor

jwiesler commented Jun 2, 2023

@wadoon tests are not able to resolve java.lang.Object. Since you wrote this dynamic symbol solver, could you take a look?

Edit: This seems to be the issue everywhere. I'm not sure how to fix this or continue with this error.

@wadoon
Copy link
Member Author

wadoon commented Jun 3, 2023

@wadoon tests are not able to resolve java.lang.Object. Since you wrote this dynamic symbol solver, could you take a look?

Is this the error message of

Caused by: UnsolvedSymbolException{context='null', name='$initialized', cause='null'}

This is not caused by unresolving java.lang.Object. It is caused because the wrong java.lang.Object is resolved. It is a ReflectionClassDeclaration and so it is not the Redux class.

And this is correct as useSystemClassLoaderInResolution was set. I think we should disable this option

@jwiesler
Copy link
Contributor

jwiesler commented Jun 3, 2023

Nope, what I meant is this one (thrown by most tests):

Unsolved symbol in de.uka.ilkd.key.java.JavaParserFactory$DynamicTypeSolver@3856d0cb : java.lang.Object
UnsolvedSymbolException{context='de.uka.ilkd.key.java.JavaParserFactory$DynamicTypeSolver@3856d0cb', name='java.lang.Object', cause='null'}
	at com.github.javaparser.resolution.TypeSolver.solveType(TypeSolver.java:71)
	at de.uka.ilkd.key.java.JP2KeYTypeConverter.getKeYJavaType(JP2KeYTypeConverter.java:125)
	at de.uka.ilkd.key.java.JP2KeYTypeConverter.<init>(JP2KeYTypeConverter.java:89)
	at de.uka.ilkd.key.java.JavaService.<init>(JavaService.java:967)
	at de.uka.ilkd.key.java.JavaService.<init>(JavaService.java:958)

However, I did see the one you mentioned and thought it would be the same error.

@wadoon
Copy link
Member Author

wadoon commented Jun 3, 2023

I am stuck at the circular problem:

  1. The symbol solver loads CompilationUnits via JavaParserTypeSolver requested upon type resolution.
  2. KeY loads Redux in one chunk in parseLibraryClasses0, incl. pre-transformation.

During the pre-transformation in (2), we resolve java.lang.Object which triggers (1), and which resolves into a different java.lang.Object entity.

Options:

  1. We hack parseLibraryClasses0 s.t. the JavaParserTypeSolver is replaced with a special type solver, which operates on the list of compilation units.

  2. We avoid the ahead-of-time parsing of JavaRedux. Implement our pre-transformation as post-processors for the JavaParser and hope that everything works just-in-time. This would require, that type resolution is used to obtain classes.

    • Disadvantage: Closed-program analyses not possible.
    • Advantage: Speed. Only necessary, reachable parts of JavaRedux are hit.

@wadoon
Copy link
Member Author

wadoon commented Jun 3, 2023

Nope, what I meant is this one (thrown by most tests):

Could you point me to one test case?

@jwiesler
Copy link
Contributor

jwiesler commented Jun 3, 2023

E.g. TestDeclParser::testGenericSortDecl1, TestTermParser::testNotEqual

But you're right the remaining tests have the problem with $initialized. The two mentioned classes use a weird setup method, maybe the issue is initialization.

@wadoon
Copy link
Member Author

wadoon commented Jun 3, 2023

We have an error in the pre-transformation pipeline. arg0 can not be resolved in the code below. Looking at the generated (excerpt below), we see the generation has produced garbage code: super(...) is only allowed at the beginning of a constructor.

 @($classInitializationInProgress) = true;
        try {
            @(java.lang.RuntimeException.$clinit());
            {
                super();
            }
            {
                super(arg0);
            }
        }//Created by ClassInitializeMethodBuilder.java:188
         catch (java.lang.Error err) {
            //Created by ClassInitializeMethodBuilder.java:148
            @($classInitializationInProgress) = true;
            //Created by ClassInitializeMethodBuilder.java:149
            @($classErroneous) = true;
            throw err;
        } catch (java.lang.Throwable twa) {
            //Created by ClassInitializeMethodBuilder.java:148
            @($classInitializationInProgress) = true;
            //Created by ClassInitializeMethodBuilder.java:149
            @($classErroneous) = true;
            throw new java.lang.ExceptionInInitializerError(twa);
        }

Complete:

package java.lang;

public class ArithmeticException extends java.lang.RuntimeException {

    public ArithmeticException() {
        super();
    }

    public ArithmeticException(java.lang.String arg0) {
        super(arg0);
    }

    @javax.annotation.processing.Generated()
    static private boolean $classInitializationInProgress;

    @javax.annotation.processing.Generated()
    static private boolean $classErroneous;

    @javax.annotation.processing.Generated()
    static private boolean $classInitialized;

    @javax.annotation.processing.Generated()
    static private boolean $classPrepared;

    static private void $clprepare() {
    }

    static public void $clinit() {
        if (!@($classInitialized)) {
            if (!@($classInitializationInProgress)) {
                if (!@(this.$classPrepared)) {
                    //Created by ClassInitializeMethodBuilder.java:213
                    @($clprepare());
                }
                if (@($classErroneous)) {
                    throw new java.lang.NoClassDefFoundError();
                }
                //Created by ClassInitializeMethodBuilder.java:237
                @($classInitializationInProgress) = true;
                try {
                    @(java.lang.RuntimeException.$clinit());
                    {
                        super();
                    }
                    {
                        super(arg0);
                    }
                }//Created by ClassInitializeMethodBuilder.java:188
                 catch (java.lang.Error err) {
                    //Created by ClassInitializeMethodBuilder.java:148
                    @($classInitializationInProgress) = true;
                    //Created by ClassInitializeMethodBuilder.java:149
                    @($classErroneous) = true;
                    throw err;
                } catch (java.lang.Throwable twa) {
                    //Created by ClassInitializeMethodBuilder.java:148
                    @($classInitializationInProgress) = true;
                    //Created by ClassInitializeMethodBuilder.java:149
                    @($classErroneous) = true;
                    throw new java.lang.ExceptionInInitializerError(twa);
                }
                //Created by ClassInitializeMethodBuilder.java:243
                @($classInitializationInProgress) = false;
                //Created by ClassInitializeMethodBuilder.java:245
                @($classErroneous) = false;
                //Created by ClassInitializeMethodBuilder.java:247
                @($classInitialized) = true;
            }
        }
    }

    protected void $prepare() {
    }

    private void $prepareEnter() {
    }

    public ArithmeticException $create() {
        //Created by CreateBuilder.java:53
        this.$initialized = false;
        $prepareEnter();
        return this;
    }

    public static ArithmeticException $createObject() {
        ArithmeticException __NEW__;
        //Created by CreateObjectBuilder.java:67
        __NEW__ = ArithmeticException.$allocate();
        __NEW__.$create()@ArithmeticException
        return __NEW__;
    }
}

@jwiesler
Copy link
Contributor

jwiesler commented Jun 7, 2023

@wadoon fixed it.

New issues are:

  • Lots of duplicate registration of XX that should not happend (many of them are definitely not the same)
  • TestJP2KeY: Unsolved symbol java.lang.Object, int (?!) and parsing errors Illegal left hand side of an assignment
  • testTypeNotInScopeShouldNotBeFound: Something with type resolution in method frames is corrupt (no idea what the issue is here)
  • TestJavaInfo: Some fields are not found, e.g. Did not find locally declared attribute $created. I suspect this is because of my first point.

@wadoon
Copy link
Member Author

wadoon commented Jun 7, 2023

  • Lots of duplicate registration of XX that should not happend (many of them are definitely not the same)

This may arise by two things: (1) I believe that the SchemaJavaParser has not used Recoder2KeY mapping, (2) there might be differences in #equals.

@wadoon
Copy link
Member Author

wadoon commented Jun 7, 2023

  • TestJP2KeY: Unsolved symbol java.lang.Object, int (?!) and parsing errors Illegal left hand side of an assignment

int problem fixed. Use of wrong class new ClassOrInterfaceType("int"); instead of new PrimitiveType(Primtive.INT).

Test case fails, as it is expected that i in { i = 2; } is the same instance as provided in the creation of the context.

Unknown if this is really required.

@wadoon
Copy link
Member Author

wadoon commented Jun 7, 2023

  • Illegal left hand side of an assignment

There are some simple checks inside JavaParser, look into CommonValidators.java.
There are some strange expression. How should the precedence?
We have ... && i+=j && i=j && .... The JavaParser interpretes this as
(... && i) += (j && i=j && ...).

Case fixed with parenthesis.

@wadoon wadoon removed this from 3rd HacKeYthon Jan 21, 2026
@Drodt
Copy link
Member

Drodt commented Jan 29, 2026

RAP: simpleTest (2026-01-29)

0 failing, 27 passing.

RAP: '_arrays' (2026-01-30)

0 failing, 8 passing.

RAP: '_javadl' (2026-01-30)

11 failing, 64 passing

RAP: '_switch' (2026-01-30)

1 failing, 8 passing

RAP: '_redux' (2026-01-30)

0 failing, 12 passing

Drodt and others added 11 commits January 29, 2026 11:54
Method expand was used for instance creation instead of the taclet.
Reason was that allocate had a non-null body (also it was empty)
In particular:
-  source code order of initializers respected
- correct appending of initializer expressions to normalform body
- fixes for recognizing implicit elements
@unp1
Copy link
Member

unp1 commented Jan 30, 2026

  1. The fix 6313a62 allows standard_key/instanceCreation/instanceCreation1.key to be provable. The problem standard_key/instanceCreation/instanceCreation2.key fails at the moment due to fields of the super type not being accessed correctly. Most likely a consequence of JP2KeYConverter method public Object visit(FieldAccessExpr n, Void arg) creating different program variables for same field accesses.
  2. The change 898fd0e should allow the last failing "_switch" example to be proven, but the strategy settings in the .key file are ignored.

unp1 added 3 commits January 30, 2026 03:04
This makes at least reverseArray2.key to be provable again.
@Drodt
Copy link
Member

Drodt commented Jan 30, 2026

Very curious behavior of switch_in_switch.key: When proven in the UI, at some point a label is removed (correctly) via unusedLabel. The saved proof has this rule in the correct line (3620) and can be loaded.

The automatic tests save a proof without this rule app, creating an unloadable proof.

@PiIsRational PiIsRational mentioned this pull request Jan 30, 2026
5 tasks
@Drodt
Copy link
Member

Drodt commented Jan 30, 2026

RAP javadl Tests

`testlocalDeclaredMethod`: Model Element not known:(type, sort): (B,B))
Line 133, goal 2, rule impRight not available or not applicable in this context. (file: /home/daniel/Documents/key/key/key.core/../key.ui/examples/standard_key/staticInitialisation/localDeclaredMethod.key.proof; caused by: de.uka.ilkd.key.util.AssertionFailure: 
Assertion failure: Model Element not known:(type, sort): (B,B))
	at de.uka.ilkd.key.proof.io.IntermediateProofReplayer.reportError(IntermediateProofReplayer.java:861)
	at de.uka.ilkd.key.proof.io.IntermediateProofReplayer.replay(IntermediateProofReplayer.java:274)
	at de.uka.ilkd.key.proof.io.IntermediateProofReplayer.replay(IntermediateProofReplayer.java:189)
	at de.uka.ilkd.key.proof.io.AbstractProblemLoader.replayProof(AbstractProblemLoader.java:663)
	at de.uka.ilkd.key.proof.io.AbstractProblemLoader.loadSelectedProof(AbstractProblemLoader.java:343)
	at de.uka.ilkd.key.proof.io.AbstractProblemLoader.load(AbstractProblemLoader.java:279)
	at de.uka.ilkd.key.proof.io.AbstractProblemLoader.load(AbstractProblemLoader.java:251)
	at de.uka.ilkd.key.control.AbstractUserInterfaceControl.load(AbstractUserInterfaceControl.java:227)
`testfor_ReferenceArray`, `testvararg`, `testfor_Iterable`, `testfor_Array`: EnumDeclaration not supported
message='Unsupported element detected given by Java Parser: EnumDeclaration. Please extend the KeY-Java-Hierarchy'}]})
	at de.uka.ilkd.key.control.AbstractUserInterfaceControl.load(AbstractUserInterfaceControl.java:240)
	at de.uka.ilkd.key.control.KeYEnvironment.load(KeYEnvironment.java:299)
	at de.uka.ilkd.key.control.KeYEnvironment.load(KeYEnvironment.java:267)
	at de.uka.ilkd.key.control.KeYEnvironment.load(KeYEnvironment.java:239)
	at de.uka.ilkd.key.control.KeYEnvironment.load(KeYEnvironment.java:198)
	at de.uka.ilkd.key.control.KeYEnvironment.load(KeYEnvironment.java:309)
	at de.uka.ilkd.key.proof.runallproofs.ProveTest.load(ProveTest.java:239)
	at de.uka.ilkd.key.proof.runallproofs.ProveTest.runKey(ProveTest.java:113)
`testclasspath`, `testexceptions3`, `testexceptions`: Unsolved Symbol
Unsolved symbol : C1
UnsolvedSymbolException{context='null', name='C1', cause='null'}
	at com.github.javaparser.symbolsolver.javaparsermodel.declarations.JavaParserClassDeclaration.toReferenceType(JavaParserClassDeclaration.java:501)
	at com.github.javaparser.symbolsolver.javaparsermodel.declarations.JavaParserClassDeclaration.getSuperClass(JavaParserClassDeclaration.java:203)
	at com.github.javaparser.symbolsolver.javaparsermodel.declarations.JavaParserClassDeclaration.getAncestors(JavaParserClassDeclaration.java:360)
	at com.github.javaparser.resolution.declarations.ResolvedReferenceTypeDeclaration.getAncestors(ResolvedReferenceTypeDeclaration.java:79)
	at de.uka.ilkd.key.java.transformations.pipeline.ClassInitializeMethodBuilder.createInitializerExecutionTryBlock(ClassInitializeMethodBuilder.java:182)
	at de.uka.ilkd.key.java.transformations.pipeline.ClassInitializeMethodBuilder.createInitializeMethodBody(ClassInitializeMethodBuilder.java:247)
	at de.uka.ilkd.key.java.transformations.pipeline.ClassInitializeMethodBuilder.createInitializeMethod(ClassInitializeMethodBuilder.java:285)
	at de.uka.ilkd.key.java.transformations.pipeline.ClassInitializeMethodBuilder.apply(ClassInitializeMethodBuilder.java:297)
`testobjectOfErroneousClass`: AssertionError
java.lang.AssertionError
	at de.uka.ilkd.key.java.JavaInfo.buildNameCache(JavaInfo.java:212)
	at de.uka.ilkd.key.java.JavaInfo.getTypeByName(JavaInfo.java:190)
	at de.uka.ilkd.key.java.JavaInfo.getTypeByClassName(JavaInfo.java:1416)
	at de.uka.ilkd.key.java.JavaInfo.getTypeByClassName(JavaInfo.java:271)
	at de.uka.ilkd.key.nparser.builder.ExpressionBuilder.isClass(ExpressionBuilder.java:1162)
	at de.uka.ilkd.key.nparser.builder.ExpressionBuilder.visitAccesstermAsJava(ExpressionBuilder.java:1234)
	at de.uka.ilkd.key.nparser.builder.ExpressionBuilder.visitAccessterm(ExpressionBuilder.java:1438)
	at de.uka.ilkd.key.nparser.builder.ExpressionBuilder.visitAccessterm(ExpressionBuilder.java:63)

Fails w/o throwing exception

  • testtestcontext
  • teststaticInitialisersAreNonSimple
  • testcompileTimeConstants

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Feature New feature or request Java Parser

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants