FAQ
The choice of tools depends on the selected data ingestion model
Model | Tools |
|---|---|
PULL model | Cloud Airflow, Cloud Spark (batch processing), Cloud Trino |
PUSH model | Cloud Spark (stream processing) |
The main advantage of ingesting data using the PULL model is that data is requested on demand, when it is actually needed.
Disadvantages:
- Delayed data delivery to the storage, since data is updated only on schedule or manually.
- Risk of overloading the source, since constant data requests can slow down the system.
For more details on the specifics of the PULL model, see the DLH architecture article.
The main advantage of ingesting data using the PUSH model is that there is no need to request data — it arrives without delays in real time.
The main disadvantage is the need to develop a component for sending data (Producer API) when connecting a new data source to the storage. This increases integration costs.
For more details on the specifics of the PUSH model, see the DLH architecture article.
The main difference lies in their purpose and the way they process data.
Criterion | Cloud Spark | Cloud Trino |
|---|---|---|
Data processing model | PULL model (batch processing), PUSH model (stream processing, micro-batches) | PULL model |
Resource intensity | Requires long-lived clusters | Temporary resource allocation — worker nodes (Workers) are automatically removed after processing is completed |
Data handling | Can modify data | Does not modify the source data |
Tasks | Used for complex ETL and ELT processes | Specializes in optimizing and executing SQL queries |
Primary purpose | Machine learning | Interactive analytics |
In the Data Lakehouse, the Cloud Spark and Cloud Trino components complement each other: Spark prepares the data, while Trino provides fast access to it.