Skip to content

Update migrate_to_pinecone.py#1

Open
ericx00 wants to merge 1 commit intotoonshi:masterfrom
ericx00:patch-1
Open

Update migrate_to_pinecone.py#1
ericx00 wants to merge 1 commit intotoonshi:masterfrom
ericx00:patch-1

Conversation

@ericx00
Copy link
Copy Markdown
Collaborator

@ericx00 ericx00 commented Nov 29, 2024

Pinecone Initialization:

Uses pinecone.init() and checks if the specified index exists.
Adds an environment variable for the Pinecone environment (e.g., us-west1-gcp).

Metadata Preservation:

Includes all metadata from ChromaDB in the migration process and appends the document text to the metadata.

Batching Robustness:

Handles cases where the dataset is not evenly divisible by the batch size.

Error Handling:

Validates data consistency and raises errors for missing index or improper configurations.

Progress Feedback:

Adds a progress bar using tqdm for better user experience.

Pinecone Initialization:

    Uses pinecone.init() and checks if the specified index exists.
    Adds an environment variable for the Pinecone environment (e.g., us-west1-gcp).

Metadata Preservation:

    Includes all metadata from ChromaDB in the migration process and appends the document text to the metadata.

Batching Robustness:

    Handles cases where the dataset is not evenly divisible by the batch size.

Error Handling:

    Validates data consistency and raises errors for missing index or improper configurations.

Progress Feedback:

    Adds a progress bar using tqdm for better user experience.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant