HTTPS | Spice.ai Cloud Documentation

For the complete documentation index, see llms.txt. This page is also available as Markdown.

The HTTP(s) Data Connector enables federated SQL query across supported file formats stored at an HTTP(s) endpoint.

datasets:
  - from: http://my-http-api.com/report.csv
    name: local_report
    params:
      http_password: ${env:MY_HTTP_PASS}

Configuration

`from`

The from field must contain a valid URI to the location of a supported file. For example, http://static_username@my-http-api/report.csv.

`name`

The dataset name. This will be used as the table name within Spice.

Example:

datasets:
  - from: http://static_username@my-http-api.com/report.csv
    name: cool_dataset
    params: ...

SELECT COUNT(*) FROM cool_dataset;

+----------+
| count(*) |
+----------+
| 6001215  |
+----------+

`params`

The connector supports Basic HTTP authentication via param values.

Parameter Name	Description
`http_port`	Optional. Port to create HTTP(s) connection over. Default: 80 and 443 for HTTP and HTTPS respectively.
`http_username`	Optional. Username to provide connection for HTTP basic authentication. Default: None.
`http_password`	Optional. Password to provide connection for HTTP basic authentication. Default: None. Use the secret replacement syntax to load the password from a secret store, e.g. `${secrets:my_http_pass}`.
`client_timeout`	Optional. Specifies timeout for HTTP operations. Default value is `30s` E.g. `client_timeout: 60s`

Examples

Basic example

datasets:
  - from: https://github.com/LAION-AI/audio-dataset/raw/7fd6ae3cfd7cde619f6bed817da7aa2202a5bc28/metadata/freesound/parquet/freesound_parquet.parquet
    name: laion_freesound

Using Basic Authentication

datasets:
  - from: http://static_username@my-http-api.com/report.csv
    name: local_report
    params:
      http_password: ${env:MY_HTTP_PASS}