Apache Spark as a connector for federated SQL query against a Spark Cluster using Spark Connect

Copy

datasets:
  - from: spark:spiceai.datasets.my_awesome_table
    name: my_table
    params:
      spark_remote: sc://my-spark-endpoint

Unquoted identifiers are normalized to lowercase. To reference a table with mixed-case characters, wrap each case-sensitive part in double quotes: spark:my_catalog."MySchema"."MyTable". See Identifier Case Sensitivity.

Configuration

• spark_remote: A spark remote connection URI. Refer to spark connect client connection string for parameters in URI.

Limitations

• Correlated scalar subqueries are only supported in filters, aggregations, projections, and UPDATE/MERGE/DELETE commands. Spark Docs

• The Spark connector does not yet support streaming query results from Spark.

Last updated 3 months ago