Skip to content

Conversation

@tcnichol
Copy link
Contributor

This moves the listeners to an async service so that there is no longer a need for the time.sleep hack after indexing a file to get it to work.

To fix this, I used an async service and added a different type of event - a 'file indexed' event that the message_listener is listening to.

In order to make all this work, I did change the rabbitmq dependency from a blockingChannel to an AbstractChannel. I tested to make sure that files trigger extractors when added/indexed and also, that manual submission of datasets and files still works.

To test this, do the following.

First, comment out in docker-compose.dev.yml the part related to the message_listener.

Second, run the dependencies using docker-dev.sh up as usual.

Third, run the message_listener using pycharm, in debug mode.

Fourth, run the backend in debug mode. Run the front end as usual.

Test the following:

Create a dataset and add a file - it should create an event that is picked up by the message_listener when the file is indexed, and this should trigger the check_feed_listeners and any extractors that should run on the file.

Also, to verify that everything extractor related works as before, manually submit files and datasets to extractors.

@tcnichol tcnichol linked an issue Sep 22, 2025 that may be closed by this pull request
@tcnichol tcnichol requested review from lmarini and max-zilla and removed request for lmarini and max-zilla September 22, 2025 20:26
@lmarini lmarini added this to the v2.1 milestone Oct 1, 2025
@tcnichol
Copy link
Contributor Author

Last few commits were to fix the pytest issues. Changed made in conftest.py for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Move check_feed_listeners to async service

3 participants