Method public org.apache.spark.sql.streaming.DataStreamReader org.apache.spark.sql.SQLContext.readStream() is not whitelisted on class class org.apache.spark.sql.SQLContext

[jgranadosH]

Hello I'm having problems reading in streams using evenhubs inside azure databricks, the read works using an interactive cluster but when i try to use the same code inside of a high currency cluster with the role access control enabled (only accepts sql and python code),i got the following error:

py4j.security.Py4JSecurityException: Method public org.apache.spark.sql.streaming.DataStreamReader org.apache.spark.sql.SQLContext.readStream() is not whitelisted on class class org.apache.spark.sql.SQLContext

Featured used :
Runtime 6.4(scala 2.11, Spark 2.4.5)

Spark configuration:
spark.databricks.cluster.profile serverless
spark.databricks.io.cache.maxMetaDataCache 1g
spark.databricks.io.cache.maxDiskUsage 80g
spark.databricks.acl.dfAclsEnabled true
spark.databricks.delta.preview.enabled true
spark.databricks.io.cache.compression.enabled false
spark.databricks.repl.allowedLanguages python,sql

Library installed :
com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.6

[robinsonmhj]

+1
I have the same issue when trying to read event hub as kafaka when calling readStream

[tross46863]

I was struggling with this issue, it was because the cluster I was using was too new, I downgraded my cluster to 5.5 LTS (includes Apache Spark 2.4.3, Scala 2.11) and structured streaming started working again, I don't think it is an issue with the version of scala but rather with Apache Spark.

[alexeyegorov]

The issue is with the table access control. :/
https://kb.gcp.databricks.com/streaming/readstream-is-not-whitelisted.html