Add a script to run whole ingestion pipeline

[zhiwei531]

Currently we need to manually run update_data.py, load_chroma.py, load_redis.py, load_postgres.py everytime.
We need a main entry script to run them in pipeline.

[Mil3sYu]

@zhiwei531
Maybe I can take that one, shall I assign myself right now?

[zhiwei531]

@zhiwei531 Maybe I can take that one, shall I assign myself right now?

Ok. Let me know if you have any question.
Apart from pipeline, you probably need to dig into new update_data.py logic, since it required "access_type" and "role". The data in one folder may have different attributes on them.
In this case, you may need to develop a mechanism to auto attributes data on different access/role.

[Mil3sYu]

@Ar-temis @zhiwei531
Hi, quick update on my recent progress:

1. I completed the first version of the Box sync + run_pipeline workflow. The pipeline can now sync selected Box folders and run the ingestion/loading process into Chroma/Redis/Postgres automatically.

2. I restored and fixed the structure-aware PDF chunking pipeline. I fixed the compatibility/loading issues so the generated nodes can now be successfully loaded and retrieved through the existing retrieval pipeline.

3. I also ran retrieval-level comparisons between fixed chunking and structure-aware chunking. Current observation is that:

• fixed chunking performs better for some broad recall-style queries
• structure-aware chunking performs better for targeted semantic retrieval and produces cleaner/coherent chunks

4. My next step is to continue evaluating the retrieval tradeoffs and work toward a more comprehensive/adaptive chunking strategy instead of relying on only fixed or only structure-aware chunking.