Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/source/user-guide/latest/expressions.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,12 @@ Expressions that are not Spark-compatible will fall back to Spark by default and
| Substring | Yes | |
| Upper | No | Results can vary depending on locale and character set. Requires `spark.comet.caseConversion.enabled=true` |

## JSON Functions

| Expression | Spark-Compatible? | Compatibility Notes |
| ------------- | ----------------- | --------------------------------------------------------------------------------------------- |
| GetJsonObject | No | Spark allows single-quoted JSON and unescaped control characters which Comet does not support |

## Date/Time Functions

| Expression | SQL | Spark-Compatible? | Compatibility Notes |
Expand Down
2 changes: 1 addition & 1 deletion docs/spark_expressions_support.md
Original file line number Diff line number Diff line change
Expand Up @@ -245,7 +245,7 @@
### json_funcs

- [ ] from_json
- [ ] get_json_object
- [x] get_json_object
- [ ] json_array_length
- [ ] json_object_keys
- [ ] json_tuple
Expand Down
1 change: 1 addition & 0 deletions native/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion native/spark-expr/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,8 @@ chrono-tz = { workspace = true }
num = { workspace = true }
regex = { workspace = true }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
# preserve_order: needed for get_json_object to match Spark's JSON key ordering
serde_json = { version = "1.0", features = ["preserve_order"] }
thiserror = { workspace = true }
futures = { workspace = true }
twox-hash = "2.1.2"
Expand Down
4 changes: 4 additions & 0 deletions native/spark-expr/src/comet_scalar_funcs.rs
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,10 @@ pub fn create_comet_physical_fun_with_eval_mode(
let func = Arc::new(crate::string_funcs::spark_split);
make_comet_scalar_udf!("split", func, without data_type)
}
"get_json_object" => {
let func = Arc::new(crate::string_funcs::spark_get_json_object);
make_comet_scalar_udf!("get_json_object", func, without data_type)
}
_ => registry.udf(fun_name).map_err(|e| {
DataFusionError::Execution(format!(
"Function {fun_name} not found in the registry: {e}",
Expand Down
Loading
Loading