feat(datafusion): Add DDL support with PRIMARY KEY constraint syntax#237
feat(datafusion): Add DDL support with PRIMARY KEY constraint syntax#237JingsongLi merged 5 commits intoapache:mainfrom
Conversation
| match &statements[0] { | ||
| Statement::CreateTable(create_table) => self.handle_create_table(create_table).await, | ||
| Statement::AlterTable { | ||
| name, operations, .. |
There was a problem hiding this comment.
Statement::AlterTable has an if_exists flag, but it is ignored here.
|
|
||
| let identifier = resolve_identifier(&cmd.name)?; | ||
|
|
||
| // Build Paimon schema from the CREATE EXTERNAL TABLE command. |
There was a problem hiding this comment.
My understanding is that both CREATE TABLE and CREATE EXTERNAL TABLE create managed Paimon tables here, rather than true external tables. It looks like two SQL entry points for creating the managed Paimon tables. Is that right?
There was a problem hiding this comment.
Good point, let' ban EXTERNAL.
|
+1 |
leaves12138
left a comment
leaves12138
left a comment
There was a problem hiding this comment.
Nice work. The DDL support is well-structured and the mock-based tests in ddl.rs are thorough.
Highlights:
-
PaimonDdlHandlercleanly intercepts CREATE TABLE and ALTER TABLE while delegating everything else to the innerSessionContext. The three-part name resolution (catalog.db.table→ strip catalog prefix) is correct. -
Arrow↔Paimon type conversion in
crates/paimon/src/arrow/mod.rscovers all the common types with roundtrip tests. The mapping ofUtf8/LargeUtf8/Utf8View → VarCharandBinary/LargeBinary/BinaryView → VarBinaryis sensible. -
Integration tests in
ddl_tests.rsusingFileSystemCatalog+TempDirprovide good end-to-end coverage for CREATE TABLE with PK, partitions, and WITH options.
Minor notes (non-blocking):
sql_data_type_to_arrowdoesn't handle ARRAY, MAP, or ROW types — those will hit the_ => Err(...)fallback. Users would need to use programmatic API for complex types in DDL. Worth a doc mention.- The
register_tableimpl incatalog.rsjust acknowledges (Ok(Some(table))). This is fine since the actual table creation happens viaPaimonDdlHandler, but a short comment explaining the lifecycle (DDL handler creates → DataFusion calls register_table → we ack) would help future readers.
+1, good to merge.
- Add PaimonDdlHandler for CREATE TABLE, ALTER TABLE (ADD/DROP/RENAME COLUMN, RENAME TABLE) - Add PaimonTableFactory for CREATE EXTERNAL TABLE via DataFusion TableProviderFactory - Extend PaimonCatalogProvider/SchemaProvider with CREATE/DROP SCHEMA and DROP TABLE - Add arrow_to_paimon_type and arrow_fields_to_paimon to paimon::arrow for Arrow-to-Paimon type conversion - Support PRIMARY KEY (col, ...) constraint syntax in CREATE TABLE DDL
3 participants
Purpose
Brief change log
Tests
API and Format
Documentation