NIFI-15365: Allow Temporary Processor and Controller Services to verify.

[bobpaulin]

• Use new InstanceClassLoader to load additional classpath urls
• Initialize component using reload component to set references from existing component
• Allow Instance and InstanceClassLoader to be GC and closed by calling OnRemove

Summary

NIFI-15365

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

• Apache NiFi Jira issue created

Pull Request Tracking

• Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
• Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

• Pull Request based on current revision of the main branch
• Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Verified by calling verification api on DBCPConnectionPool Controller Service

#!/bin/bash
COUNT=100
controllerServiceId="<enter id>"
accessToken='<accessToken>'
for ((i=1; i<=COUNT; i++)); do
    echo "Request #$i"
    curl -k --location 'https://localhost:8443/nifi-api/controller-services/'"$controllerServiceId"'/config/verification-requests' --header 'Content-Type: application/json' --header 'Authorization: Bearer '"$accessToken"'' --data '{    "request": {        "properties": {            "Database Connection URL": "jdbc:postgresql://localhost:5432/postgres",            "Database Driver Class Name": "org.postgresql.Driver",            "Database Driver Locations": "/Users/rpaulin/Downloads/postgresql-42.7.7.jar",            "Database User": "postgres",            "Password": "postgres"        },        "componentId": "'"$controllerServiceId"'",        "attributes": {}    }}'

Then verifying classes are unloaded via a heap dump and jmx.

Build

• Build completed using ./mvnw clean install -P contrib-check
  • JDK 21
  • JDK 25

Licensing

• New dependencies are compatible with the Apache License 2.0 according to the License Policy
• New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

• Documentation formatting appears as expected in rendered files

[exceptionfactory]

Thanks for working on this improvement @bobpaulin.

On a structural review, I'm not sure that ReloadComponent is the right place to add the new methods. It might make sense to implement some the loading logic in the ProcessorNode and ControllerServiceNode classes themselves. That should also avoid the need for duplicative implementations in the Stateless and Standard ReloadComponents.

One fundamental note is that the ExtensionManager interface and corresponding implementation have methods for creating temporary components, including logic for handling Python Processors and working with the bundle ClassLoader. Decoupling the creation logic from the initialize call also seems important for clarity of behavior.

I can take a closer look soon, but those are some initial elements to consider.

[bobpaulin]

Thanks for working on this improvement @bobpaulin.

On a structural review, I'm not sure that ReloadComponent is the right place to add the new methods. It might make sense to implement some the loading logic in the ProcessorNode and ControllerServiceNode classes themselves. That should also avoid the need for duplicative implementations in the Stateless and Standard ReloadComponents.

I am also skeptical about ReloadComponent. However that is what the temporary component is doing. Its create a temporary instance to verify that will eventually be replace when the configuration is applied and the component is reloaded. It has all the object references required to swap in all the parts of the existing component.
The issue with StandardProcessorNode and StandardControllerServiceNode is:

1. It does not have access to the StateManager so we'd need to wire that in only for this purpose.
2. It does not have access to the KeberosConfig. This can be built with the NiFiProperties class however that is also not currently wired in.
3. It does not have access to the FlowController (NodeTypeProvider).

These would all need to be wired into both of the above classes which in my opinion might introduce unintended coupling in the future. So while I do agree ReloadComponent is a little awkward I think it is better than opening up access to all these objects directly in the ProcessorNode and ControllerServiceNode classes.

One fundamental note is that the ExtensionManager interface and corresponding implementation have methods for creating temporary components, including logic for handling Python Processors and working with the bundle ClassLoader. Decoupling the creation logic from the initialize call also seems important for clarity of behavior.

Yes I noticed the TempComponents there first. My original plan was to go in that direction. The ExtensionManager faces similar challenges as StandardProcessorNode and StandardControllerServiceNode where there are additional classes that would need to be wired in to get access to state, keberos and FlowController. The current "TempComponents" use a MockControllerServiceInitializationContext for these pieces which might produce inconsistent behavior in the verify method as all 3 can be accessed within the ControllerService or Processor verify methods. This seems like a more appropriate place for the behavior but I was concerned with confusing these TempComponents with the existing ones. Not sure if just better naming could do the trick here but this seemed like a more significant change than using the ReloadComponent.

I can take a closer look soon, but those are some initial elements to consider.

Appreciate the feedback @exceptionfactory! Not sure if there is another path to consider here but I think there are trade offs with all 3 possibilities above:

1. ReloadComponent - a bit awkward since reload's purpose is generally for reloading the existing component
2. ExtensionManager - introduces additional object dependencies and confusion over existing TempComponents.
3. StandardProcessorNode and StandardControllerServiceNode - exposing additional object dependencies might encourage coupling in the future.

The options above are in my current preferred order but I'm open to changing.