Python Read Parquet File

Read Parquet file and write it to an SQL or ClickHouse database.

Create Python Virtual Environment

# Linux/macOS
python3 -m venv venv
source venv/bin/activate

# Windows
python -m venv venv
venv\Scripts\activate

# Conda
conda create --name myenv python=3.11
conda activate myenv

Install Required Libraries

pip install pandas pyarrow sqlalchemy

For ClickHouse:

pip install clickhouse-connect

Config

Update config.py file. Keep parquet or any gz.parquet files in the data directory.

Read Parquet file

python read.py

Write to Database

python write.py

Read from Database

python query.py