diff --git a/README.md b/README.md index 5a2a0162..1b9f24dd 100644 --- a/README.md +++ b/README.md @@ -32,9 +32,11 @@ cd openespi-authserver && mvn spring-boot:run ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ Third Party │───▶│ Authorization │───▶│ Data Custodian │ │ (Java 21+Jakarta)│ │ Server (SB 3.5) │ │ Server (SB 3.5) │ +│ │ │ (Independent) │ │ │ └─────────────────┘ └─────────────────┘ └─────────────────┘ - │ │ │ - └───────────────────────┼───────────────────────┘ + │ │ + │ │ + └──────────────────────────────────────────────┘ ▼ ┌─────────────────┐ │ OpenESPI Common │ @@ -42,6 +44,12 @@ cd openespi-authserver && mvn spring-boot:run └─────────────────┘ ``` +**Module Dependencies:** +- **openespi-common**: Foundation library (no dependencies on other modules) +- **openespi-datacustodian**: Depends on openespi-common +- **openespi-authserver**: **Independent** (no dependency on openespi-common) +- **openespi-thirdparty**: Depends on openespi-common + ## ✨ Migration Achievements **All modules now support:** @@ -118,22 +126,117 @@ mvn test mvn test -pl openespi-common ``` -### Integration Tests +### Integration Tests with TestContainers + +**Prerequisites:** +- Docker Desktop installed and running +- Minimum 4GB RAM allocated to Docker +- Ports 3306 (MySQL) and 5432 (PostgreSQL) available + +**Quick Start:** ```bash -# TestContainers integration tests +# Verify Docker is running +docker --version && docker ps + +# Run all integration tests (H2, MySQL, PostgreSQL) mvn verify -pl openespi-common -Pintegration-tests +# Run specific database tests +mvn test -Dtest=ComplexRelationshipMySQLIntegrationTest -pl openespi-common +mvn test -Dtest=ComplexRelationshipPostgreSQLIntegrationTest -pl openespi-common + # Cross-module integration mvn verify -Pfull-integration ``` +**What Gets Tested:** +- ✅ **H2**: In-memory database with vendor-neutral migrations +- ✅ **MySQL 8.0**: TestContainers with BLOB column types +- ✅ **PostgreSQL 15**: TestContainers with BYTEA column types +- ✅ **Flyway Migrations**: Vendor-specific migration paths +- ✅ **JPA Relationships**: Complex entity hierarchies +- ✅ **Transaction Boundaries**: Data consistency across transactions +- ✅ **Bulk Operations**: saveAll, deleteAll operations + +**TestContainers Configuration:** +- MySQL container: `mysql:8.0` +- PostgreSQL container: `postgres:15-alpine` +- Container reuse enabled for faster test execution +- Automatic cleanup after tests complete + +**Troubleshooting:** +```bash +# If containers fail to start +docker system prune # Clean up Docker cache +docker pull mysql:8.0 +docker pull postgres:15-alpine + +# View running containers during tests +docker ps + +# Check container logs +docker logs +``` + +## 🗄️ Database Migrations + +### Vendor-Specific Migration Structure + +The project uses Flyway with a vendor-specific migration strategy to support H2, MySQL, and PostgreSQL: + +``` +openespi-common/src/main/resources/db/ +├── migration/ +│ ├── V1__Create_Base_Tables.sql # Vendor-neutral base tables +│ └── V3__Create_additiional_Base_Tables.sql # Additional shared tables +└── vendor/ + ├── h2/ # (Future: H2-specific migrations) + ├── mysql/ # (Future: MySQL-specific migrations) + └── postgres/ # (Future: PostgreSQL-specific migrations) +``` + +**Current Implementation:** +- All migrations are currently in the base `db/migration` directory +- Tables use vendor-neutral SQL syntax compatible with all three databases +- Future enhancements will move vendor-specific tables to `/vendor/{database}/` paths + +**Migration Locations by Profile:** +- `test` (H2): `classpath:db/migration` +- `test-mysql`: `classpath:db/migration,classpath:db/vendor/mysql` +- `test-postgresql`: `classpath:db/migration,classpath:db/vendor/postgres` +- `dev-mysql`: `classpath:db/migration,classpath:db/vendor/mysql` +- `dev-postgresql`: `classpath:db/migration,classpath:db/vendor/postgres` + +**Vendor-Specific Column Types:** +| Type | H2 | MySQL | PostgreSQL | +|------|------|-------|------------| +| Binary Data | `BINARY` | `BLOB` | `BYTEA` | +| UUID | `UUID` | `CHAR(36)` | `UUID` | +| Timestamps | `TIMESTAMP` | `DATETIME` | `TIMESTAMP` | + +**Running Migrations:** +```bash +# Migrations run automatically on application startup +mvn spring-boot:run + +# Test migrations with specific database +mvn test -Dtest=ComplexRelationshipMySQLIntegrationTest +mvn test -Dtest=ComplexRelationshipPostgreSQLIntegrationTest + +# Validate migration status +mvn flyway:info -pl openespi-common +``` + ## 🚀 Deployment Each module has independent deployment capabilities: -- **Common**: Maven Central library -- **DataCustodian**: Kubernetes/Docker deployment -- **AuthServer**: Kubernetes/Docker deployment -- **ThirdParty**: WAR deployment or future containerization +- **Common**: Maven Central library (shared dependency) +- **DataCustodian**: Kubernetes/Docker deployment (Spring Boot 3.5 JAR) +- **AuthServer**: Kubernetes/Docker deployment (Spring Boot 3.5 JAR) +- **ThirdParty**: WAR deployment to Tomcat/Jetty ⚠️ **Temporary** - awaiting Spring Boot 3.5 migration for containerization + +**Note on ThirdParty Deployment:** +ThirdParty currently uses legacy WAR packaging because it runs on Spring Framework (not Spring Boot) with JSP/JSTL views. Once the Spring Boot 3.5 migration completes, it will switch to executable JAR packaging with embedded server and Docker/Kubernetes deployment like the other modules. ### Docker Build ```bash diff --git a/openespi-common/src/test/java/org/greenbuttonalliance/espi/common/migration/DataCustodianApplicationPostgresTest.java b/openespi-common/src/test/java/org/greenbuttonalliance/espi/common/migration/DataCustodianApplicationPostgresTest.java deleted file mode 100644 index ac914628..00000000 --- a/openespi-common/src/test/java/org/greenbuttonalliance/espi/common/migration/DataCustodianApplicationPostgresTest.java +++ /dev/null @@ -1,165 +0,0 @@ -/* - * - * Copyright (c) 2025 Green Button Alliance, Inc. - * - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package org.greenbuttonalliance.espi.common.migration; - -import org.greenbuttonalliance.espi.common.TestApplication; -import org.junit.jupiter.api.DisplayName; -import org.junit.jupiter.api.Test; -import org.springframework.boot.test.context.SpringBootTest; -import org.springframework.test.context.ActiveProfiles; -import org.springframework.test.context.DynamicPropertyRegistry; -import org.springframework.test.context.DynamicPropertySource; -import org.testcontainers.containers.PostgreSQLContainer; -import org.testcontainers.junit.jupiter.Container; -import org.testcontainers.junit.jupiter.Testcontainers; - -import java.sql.Connection; -import java.sql.ResultSet; -import java.sql.SQLException; - -import static org.junit.jupiter.api.Assertions.assertEquals; -import static org.junit.jupiter.api.Assertions.assertTrue; - -/** - * Integration test for the OpenESPI Data Custodian Spring Boot application with PostgreSQL Test Container. - * - * This test verifies that the application context loads successfully with a real PostgreSQL database - * running in a Docker container, and that Flyway migrations execute correctly with the new - * vendor-specific migration structure. - */ -@SpringBootTest(classes = { TestApplication.class }) -@ActiveProfiles("test-postgres") -@Testcontainers -@DisplayName("PostgreSQL Test Container Integration Tests") -class DataCustodianApplicationPostgresTest { - - @Container - static PostgreSQLContainer postgresContainer = new PostgreSQLContainer<>("postgres:15") - .withDatabaseName("openespi_test") - .withUsername("testuser") - .withPassword("testpass") - .withReuse(true); - - @DynamicPropertySource - static void configureProperties(DynamicPropertyRegistry registry) { - registry.add("spring.datasource.url", postgresContainer::getJdbcUrl); - registry.add("spring.datasource.username", postgresContainer::getUsername); - registry.add("spring.datasource.password", postgresContainer::getPassword); - registry.add("spring.datasource.driver-class-name", () -> "org.postgresql.Driver"); - - // Configure Flyway locations for PostgreSQL vendor-specific migrations - registry.add("spring.flyway.locations", () -> "classpath:db/migration,classpath:db/vendor/postgres"); - registry.add("spring.flyway.baseline-on-migrate", () -> "true"); - registry.add("spring.flyway.validate-on-migrate", () -> "true"); - - // JPA/Hibernate configuration for PostgreSQL - registry.add("spring.jpa.database-platform", () -> "org.hibernate.dialect.PostgreSQLDialect"); - registry.add("spring.jpa.hibernate.ddl-auto", () -> "validate"); - registry.add("spring.jpa.show-sql", () -> "true"); - } - - /** - * Test that the Spring Boot application context loads successfully with PostgreSQL Test Container. - * This verifies that all configuration classes, beans, and dependencies - * are properly configured and can be instantiated with a real PostgreSQL database. - */ - @Test - @DisplayName("Application context loads with PostgreSQL container") - void contextLoads() { - // Verify that the PostgreSQL container is running - assertTrue(postgresContainer.isRunning(), "PostgreSQL container should be running"); - - // Verify that the container has the expected configuration - assertEquals("openespi_test", postgresContainer.getDatabaseName()); - assertEquals("testuser", postgresContainer.getUsername()); - assertEquals("testpass", postgresContainer.getPassword()); - - // This test passes if the application context loads without errors - // It validates the entire Spring Boot configuration including: - // - PostgreSQL Test Container configuration - // - JPA configuration with PostgreSQL dialect - // - Flyway migration configuration - // - Service layer beans - // - Repository layer beans - } - - /** - * Test that database migrations execute successfully with the new vendor-specific structure. - * This verifies that both base migrations and PostgreSQL-specific migrations created the expected tables. - */ - @Test - @DisplayName("Database migrations execute successfully") - void databaseMigrationsExecute() throws SQLException { - // Verify that the PostgreSQL container is running - assertTrue(postgresContainer.isRunning(), "PostgreSQL container should be running"); - - // Connect to the database and verify that expected tables exist - try (Connection connection = postgresContainer.createConnection("")) { - - // Verify base migration tables exist (from V1__Create_Base_Tables.sql) - assertTrue(tableExists(connection, "application_information"), - "application_information table should exist from base migration"); - assertTrue(tableExists(connection, "retail_customers"), - "retail_customers table should exist from base migration"); - assertTrue(tableExists(connection, "reading_types"), - "reading_types table should exist from base migration"); - assertTrue(tableExists(connection, "subscriptions"), - "subscriptions table should exist from base migration"); - assertTrue(tableExists(connection, "batch_lists"), - "batch_lists table should exist from base migration"); - - // Verify PostgreSQL-specific migration tables exist (from V2__PostgreSQL_Specific_Tables.sql) - assertTrue(tableExists(connection, "time_configurations"), - "time_configurations table should exist from PostgreSQL-specific migration"); - assertTrue(tableExists(connection, "usage_points"), - "usage_points table should exist from PostgreSQL-specific migration"); - assertTrue(tableExists(connection, "meter_readings"), - "meter_readings table should exist from PostgreSQL-specific migration"); - assertTrue(tableExists(connection, "interval_blocks"), - "interval_blocks table should exist from PostgreSQL-specific migration"); - - // Verify that BYTEA columns exist in PostgreSQL-specific tables - assertTrue(columnExists(connection, "time_configurations", "dst_end_rule"), - "dst_end_rule BYTEA column should exist in time_configurations"); - assertTrue(columnExists(connection, "time_configurations", "dst_start_rule"), - "dst_start_rule BYTEA column should exist in time_configurations"); - assertTrue(columnExists(connection, "usage_points", "role_flags"), - "role_flags BYTEA column should exist in usage_points"); - } - } - - /** - * Helper method to check if a table exists in the database. - */ - private boolean tableExists(Connection connection, String tableName) throws SQLException { - try (ResultSet rs = connection.getMetaData().getTables(null, null, tableName.toLowerCase(), null)) { - return rs.next(); - } - } - - /** - * Helper method to check if a column exists in a table. - */ - private boolean columnExists(Connection connection, String tableName, String columnName) throws SQLException { - try (ResultSet rs = connection.getMetaData().getColumns(null, null, tableName.toLowerCase(), columnName.toLowerCase())) { - return rs.next(); - } - } -} \ No newline at end of file diff --git a/openespi-common/src/test/java/org/greenbuttonalliance/espi/common/repositories/integration/ComplexRelationshipPostgreSQLIntegrationTest.java b/openespi-common/src/test/java/org/greenbuttonalliance/espi/common/repositories/integration/ComplexRelationshipPostgreSQLIntegrationTest.java index 84d79c69..792ce9f5 100644 --- a/openespi-common/src/test/java/org/greenbuttonalliance/espi/common/repositories/integration/ComplexRelationshipPostgreSQLIntegrationTest.java +++ b/openespi-common/src/test/java/org/greenbuttonalliance/espi/common/repositories/integration/ComplexRelationshipPostgreSQLIntegrationTest.java @@ -20,10 +20,16 @@ import org.greenbuttonalliance.espi.common.domain.customer.entity.CustomerEntity; import org.greenbuttonalliance.espi.common.domain.customer.entity.StatementEntity; -import org.greenbuttonalliance.espi.common.domain.usage.*; +import org.greenbuttonalliance.espi.common.domain.usage.IntervalBlockEntity; +import org.greenbuttonalliance.espi.common.domain.usage.MeterReadingEntity; +import org.greenbuttonalliance.espi.common.domain.usage.RetailCustomerEntity; +import org.greenbuttonalliance.espi.common.domain.usage.UsagePointEntity; import org.greenbuttonalliance.espi.common.repositories.customer.CustomerRepository; import org.greenbuttonalliance.espi.common.repositories.customer.StatementRepository; -import org.greenbuttonalliance.espi.common.repositories.usage.*; +import org.greenbuttonalliance.espi.common.repositories.usage.IntervalBlockRepository; +import org.greenbuttonalliance.espi.common.repositories.usage.MeterReadingRepository; +import org.greenbuttonalliance.espi.common.repositories.usage.RetailCustomerRepository; +import org.greenbuttonalliance.espi.common.repositories.usage.UsagePointRepository; import org.greenbuttonalliance.espi.common.test.BaseTestContainersTest; import org.greenbuttonalliance.espi.common.test.TestDataBuilders; import org.junit.jupiter.api.DisplayName; @@ -39,7 +45,7 @@ import java.util.List; import java.util.Optional; -import static org.assertj.core.api.Assertions.*; +import static org.assertj.core.api.Assertions.assertThat; /** * Complex relationship integration tests for JPA entities using PostgreSQL TestContainer. @@ -238,8 +244,9 @@ void shouldHandleBulkSaveOperations() { flushAndClear(); // Assert - assertThat(savedReadings).hasSize(5); - assertThat(savedReadings).allMatch(reading -> reading.getId() != null); + assertThat(savedReadings) + .hasSize(5) + .allMatch(reading -> reading.getId() != null); // Verify all readings were saved long count = meterReadingRepository.count(); @@ -273,4 +280,87 @@ void shouldHandleBulkDeleteOperations() { assertThat(finalCount).isEqualTo(initialCount - 3); } } + + @Nested + @DisplayName("Flyway Migration Verification") + class MigrationVerificationTest { + + @Test + @DisplayName("Should create all required base migration tables") + void shouldCreateBaseMigrationTables() throws Exception { + // Verify base migration tables exist (from V1__Create_Base_Tables.sql) + try (var connection = postgres.createConnection("")) { + assertThat(tableExists(connection, "application_information")) + .as("application_information table should exist from base migration") + .isTrue(); + assertThat(tableExists(connection, "retail_customers")) + .as("retail_customers table should exist from base migration") + .isTrue(); + assertThat(tableExists(connection, "reading_types")) + .as("reading_types table should exist from base migration") + .isTrue(); + assertThat(tableExists(connection, "subscriptions")) + .as("subscriptions table should exist from base migration") + .isTrue(); + assertThat(tableExists(connection, "batch_lists")) + .as("batch_lists table should exist from base migration") + .isTrue(); + } + } + + @Test + @DisplayName("Should create PostgreSQL vendor-specific tables") + void shouldCreatePostgreSQLSpecificTables() throws Exception { + // Verify PostgreSQL-specific migration tables exist (from V2__PostgreSQL_Specific_Tables.sql or V3__Create_additiional_Base_Tables.sql) + try (var connection = postgres.createConnection("")) { + assertThat(tableExists(connection, "time_configurations")) + .as("time_configurations table should exist from vendor-specific migration") + .isTrue(); + assertThat(tableExists(connection, "usage_points")) + .as("usage_points table should exist from vendor-specific migration") + .isTrue(); + assertThat(tableExists(connection, "meter_readings")) + .as("meter_readings table should exist from vendor-specific migration") + .isTrue(); + assertThat(tableExists(connection, "interval_blocks")) + .as("interval_blocks table should exist from vendor-specific migration") + .isTrue(); + } + } + + @Test + @DisplayName("Should create BYTEA columns for PostgreSQL vendor-specific types") + void shouldCreateBYTEAColumns() throws Exception { + // Verify that BYTEA columns exist in PostgreSQL-specific tables + try (var connection = postgres.createConnection("")) { + assertThat(columnExists(connection, "time_configurations", "dst_end_rule")) + .as("dst_end_rule BYTEA column should exist in time_configurations") + .isTrue(); + assertThat(columnExists(connection, "time_configurations", "dst_start_rule")) + .as("dst_start_rule BYTEA column should exist in time_configurations") + .isTrue(); + assertThat(columnExists(connection, "usage_points", "role_flags")) + .as("role_flags BYTEA column should exist in usage_points") + .isTrue(); + } + } + + /** + * Helper method to check if a table exists in the database. + */ + private boolean tableExists(java.sql.Connection connection, String tableName) throws java.sql.SQLException { + try (var rs = connection.getMetaData().getTables(null, null, tableName.toLowerCase(), null)) { + return rs.next(); + } + } + + /** + * Helper method to check if a column exists in a table. + */ + private boolean columnExists(java.sql.Connection connection, String tableName, String columnName) throws java.sql.SQLException { + try (var rs = connection.getMetaData().getColumns(null, null, tableName.toLowerCase(), columnName.toLowerCase())) { + return rs.next(); + } + } + } } \ No newline at end of file diff --git a/openespi-common/src/test/resources/application-test-postgres.yml b/openespi-common/src/test/resources/application-test-postgres.yml deleted file mode 100644 index b1a770f9..00000000 --- a/openespi-common/src/test/resources/application-test-postgres.yml +++ /dev/null @@ -1,47 +0,0 @@ -# Test Configuration for PostgreSQL compatibility -spring: - datasource: - # url: jdbc:h2:mem:testdb;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE;MODE=PostgreSQL;DATABASE_TO_LOWER=TRUE;DEFAULT_NULL_ORDERING=HIGH - # username: sa - # password: - driver-class-name: org.postgresql.Driver - - jpa: - hibernate: - ddl-auto: validate - show-sql: false - properties: - hibernate: - dialect: org.hibernate.dialect.PostgreSQLDialect - generate_statistics: false - - flyway: - enabled: true - locations: - - classpath:db/migration - - classpath:db/vendor/postgres - baseline-on-migrate: true - baseline-version: 0 - validate-on-migrate: false - - - security: - oauth2: - resourceserver: - jwt: - issuer-uri: http://localhost:8080 - jwk-set-uri: http://localhost:8080/.well-known/jwks.json - -logging: - level: - org.greenbuttonalliance.espi: DEBUG - org.springframework.security: WARN - org.hibernate: WARN - org.flywaydb: debug - -espi: - datacustodian: - base-url: http://localhost:8081/DataCustodian - authorization-server: - issuer-uri: http://localhost:8080 - jwk-set-uri: http://localhost:8080/.well-known/jwks.json \ No newline at end of file