[SPARK-55089][CONNECT] Correct the output schema of toJSON

[zhengruifeng]

What changes were proposed in this pull request?

Correct the output schema of toJSON

Why are the changes needed?

classic:

scala> import org.apache.spark.sql.functions._ 
import org.apache.spark.sql.functions._

scala> val df = spark.range(10).withColumn("s", col("id").cast("string")) 
val df: org.apache.spark.sql.DataFrame = [id: bigint, s: string]

scala> df.toJSON.show() 
+----------------+                                                              
|           value|
+----------------+
|{"id":0,"s":"0"}|
|{"id":1,"s":"1"}|
|{"id":2,"s":"2"}|
|{"id":3,"s":"3"}|
|{"id":4,"s":"4"}|
|{"id":5,"s":"5"}|
|{"id":6,"s":"6"}|
|{"id":7,"s":"7"}|
|{"id":8,"s":"8"}|
|{"id":9,"s":"9"}|
+----------------+

connect (scala client)

scala> df.toJSON.show() 
+----------------------+
|to_json(struct(id, s))|
+----------------------+
|      {"id":0,"s":"0"}|
|      {"id":1,"s":"1"}|
|      {"id":2,"s":"2"}|
|      {"id":3,"s":"3"}|
|      {"id":4,"s":"4"}|
|      {"id":5,"s":"5"}|
|      {"id":6,"s":"6"}|
|      {"id":7,"s":"7"}|
|      {"id":8,"s":"8"}|
|      {"id":9,"s":"9"}|
+----------------------+

Does this PR introduce any user-facing change?

Yes, the field name will be value which is the same with spark classic

How was this patch tested?

updated golden file

Was this patch authored or co-authored using generative AI tooling?

no

[github-actions]

JIRA Issue Information

=== Bug SPARK-55089 ===
Summary: Correct the output schema of toJSON
Assignee: None
Status: Open
Affected: ["4.2.0"]

---

This comment was automatically generated by GitHub Actions