[SPARK-55090][PYTHON][CONNECT] Implement DataFrame.toJSON in Python Client

[zhengruifeng]

What changes were proposed in this pull request?

Implement DataFrame.toJSON in Python Client

Why are the changes needed?

to provide a similar way for creating json string

Does this PR introduce any user-facing change?

Yes, new API

In [2]: df = spark.range(10).withColumn("s", col("id").cast("string"))

In [3]: df.toJSON().show()
+----------------+
|           value|
+----------------+
|{"id":0,"s":"0"}|
|{"id":1,"s":"1"}|
|{"id":2,"s":"2"}|
|{"id":3,"s":"3"}|
|{"id":4,"s":"4"}|
|{"id":5,"s":"5"}|
|{"id":6,"s":"6"}|
|{"id":7,"s":"7"}|
|{"id":8,"s":"8"}|
|{"id":9,"s":"9"}|
+----------------+

How was this patch tested?

added tests

Was this patch authored or co-authored using generative AI tooling?

no

[github-actions]

JIRA Issue Information

=== New Feature SPARK-55090 ===
Summary: Implement DataFrame.toJSON in Python Client
Assignee: None
Status: Open
Affected: ["4.2.0"]

---

This comment was automatically generated by GitHub Actions

[zhengruifeng]

follow scala client

spark/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala

Lines 1229 to 1232 in e75ca57

	/** @inheritdoc */
	def toJSON: Dataset[String] = {
	select(to_json(struct(col("*")))).as(StringEncoder)
	}

with additional fix #53850