Spark instance parameters
When creating or editing a Cloud Spark service instance, you can configure its parameters. They define how the Cloud Spark instance performs tasks and interacts with other services. These include:
- Apache Spark version: affects functionality, performance, and stability when executing Spark tasks.
- Connection parameters: define the services that the Cloud Spark instance will connect to for reading and writing information.
These parameters can be set via the VK Cloud management console or via Terraform.
Parameters for configuring a Spark instance:
Parameter in the management console | Parameter in Terraform | Description |
|---|---|---|
Spark version |
| Spark version used when running tasks |
Parameters for adding data sources to a Spark instance:
Parameter in the management console | Parameter in Terraform | Description |
|---|---|---|
|
| Connection to the VK Object Storage cloud storage in the same project |
|
| Connection to the Cloud Iceberg Metastore cloud storage |
|
| Connection to an external S3 cloud storage |
|
| Connection to an external Iceberg Metastore cloud storage |
|
| Connection to a PostgreSQL database |
Parameters for connecting the S3 VK Cloud data source to a Spark instance:
Parameter in the management console | Parameter in Terraform | Description |
|---|---|---|
Internal S3 connection name |
| Name of the connection to VK Object Storage displayed in the management console |
S3 URL |
| Domain of the S3 storage. Example: |
Region |
| Region where the S3 storage is located |
Bucket |
| Unique name of the VK Object Storage bucket |
Path in the S3 bucket |
| Logical structure inside the bucket for organizing and grouping objects. Examples: |
Access Key |
| Identifier of the access key to the S3 storage |
Secret Key |
| Secret key for accessing the S3 storage |
Parameters for connecting the Iceberg Metastore VK Cloud data source to a Spark instance:
Parameter in the management console | Parameter in Terraform | Description |
|---|---|---|
Internal Iceberg Metastore connection name |
| Name of the connection to Cloud Iceberg Metastore displayed in the management console |
Bucket name in S3 VK Cloud |
| Unique name of the VK Object Storage bucket. This bucket is used only for storing internal application data |
Path in the Iceberg Metastore S3 bucket |
| Logical structure inside the bucket for organizing and grouping objects. Examples: |
Iceberg database name |
| Name of the Iceberg Metastore database that Spark connects to |
Iceberg DB hostname |
| Host and port of the Iceberg Metastore database in the |
Iceberg username |
| Login of the account used to connect Spark to the Iceberg Metastore |
User password |
| Password of the account used to connect Spark to the Iceberg Metastore |
Name of the catalog Spark will connect to |
| Default directory for connection. Used in the Spark settings |
Parameters for connecting the External S3 data source to a Spark instance:
Parameter in the management console | Parameter in Terraform | Description |
|---|---|---|
External S3 connection name |
| Name of the connection to the external S3 displayed in the management console |
S3 URL |
| Domain of the S3 storage |
Region |
| Region where the S3 storage is located |
Bucket |
| Unique bucket name |
Path in the S3 bucket |
| Logical structure inside the bucket for organizing and grouping objects. Examples: |
Access Key |
| Identifier of the access key to the S3 storage |
Secret Key |
| Secret key for accessing the S3 storage |
Parameters for connecting the External Iceberg Metastore data source to a Spark instance:
Parameter in the management console | Parameter in Terraform | Description |
|---|---|---|
External Iceberg Metastore connection name |
| Name of the connection to the external Iceberg Metastore displayed in the management console |
S3 URL |
| Domain of the S3 storage |
S3 region |
| Region where the S3 storage is located |
Bucket name in your S3 service |
| Unique name of the S3 bucket |
Path in the S3 bucket |
| Logical structure inside the bucket for organizing and grouping objects. Examples: |
Access Key |
| Identifier of the access key to the S3 storage |
Secret Key |
| Secret key for accessing the S3 storage |
Iceberg database name |
| Name of the Iceberg Metastore database that Spark connects to |
Iceberg DB hostname |
| Host and port of the Iceberg Metastore database in the |
Iceberg username |
| Login of the account used to connect Spark to the Iceberg Metastore |
User password |
| Password of the account used to connect Spark to the Iceberg Metastore |
Name of the catalog Spark will connect to |
| Default directory for connection. Used in the Spark settings |
Parameters for connecting the PostgreSQL data source to a Spark instance:
Parameter in the management console | Parameter in Terraform | Description |
|---|---|---|
PostgreSQL |
| Name of the connection to PostgreSQL displayed in the management console |
DB hostname |
| Host and port of the PostgreSQL database in the |
Database name |
| Name of the PostgreSQL database that Spark connects to |
Username |
| Login of the account used to connect Spark to PostgreSQL |
User password |
| Password of the account used to connect Spark to PostgreSQL |
TLS certificate |
| TLS certificate for a secure connection |