Conversation
asatwal
force-pushed
the
add-airbyte-deployment-tasks
branch
from
2915d28 to
88f47ce
Compare
…l services to use this service
asatwal
force-pushed
the
add-airbyte-deployment-tasks
branch
from
2ca5cce to
5788ad0
Compare
asatwal
force-pushed
the
add-airbyte-deployment-tasks
branch
from
5fe01f7 to
3ad0e8e
Compare
asatwal
force-pushed
the
add-airbyte-deployment-tasks
branch
from
bb9cf10 to
a496950
Compare
There was a problem hiding this comment.
Pull request overview
This PR adds automation for Airbyte deployment tasks and makes the Airbyte internal dataset configurable. The changes include significant refactoring to improve code organization and maintainability.
Key Changes
- Introduces a new
AirbyteDeployJobthat orchestrates the complete deployment workflow including waiting for migrations, refreshing connections, starting syncs, and applying policy tags - Adds new Airbyte service classes (
WaitForSync,StartSync,JobStatus,JobLast,ApiServer) to handle sync job management and API communication - Refactors the job and service structure by moving classes into proper namespaces (
DfE::Analytics::JobsandDfE::Analytics::Services)
Reviewed changes
Copilot reviewed 59 out of 59 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| lib/dfe/analytics/jobs/airbyte_deploy_job.rb | New orchestration job that automates all deployment tasks in sequence |
| lib/services/airbyte/wait_for_sync.rb | New service to poll Airbyte sync job status with timeout handling |
| lib/services/airbyte/start_sync.rb | New service to initiate Airbyte sync jobs with conflict handling |
| lib/services/airbyte/job_status.rb | New service to retrieve job status from recent jobs list |
| lib/services/airbyte/job_last.rb | New service to fetch the last sync job for a connection |
| lib/services/airbyte/api_server.rb | New centralized HTTP client wrapper for Airbyte API calls |
| lib/dfe/analytics/services/wait_for_migrations.rb | New service to wait for pending migrations before deployment |
| lib/dfe/analytics/services/apply_airbyte_internal_tables_policy_tags.rb | New service to apply policy tags to internal Airbyte tables |
| lib/dfe/analytics/services/apply_airbyte_final_tables_policy_tags.rb | New service to apply policy tags to final Airbyte tables |
| lib/dfe/analytics/jobs/big_query_apply_policy_tags_job.rb | Refactored from top-level to Jobs namespace with dataset parameter |
| lib/dfe/analytics/jobs/analytics_job.rb | Moved to Jobs namespace as base class for all analytics jobs |
| lib/dfe/analytics/jobs/entity_table_check_job.rb | Moved to Jobs namespace for consistency |
| lib/services/airbyte/connection_refresh.rb | Updated to accept optional parameters for more flexible usage |
| lib/services/airbyte/access_token.rb | Refactored error handling and separated HTTP logic |
| lib/services/airbyte/connection_list.rb | Refactored to use new ApiServer wrapper |
| lib/services/airbyte/discover_schema.rb | Simplified by delegating to ApiServer |
| lib/services/airbyte/connection_update.rb | Simplified by delegating to ApiServer |
| lib/dfe/analytics/big_query_api.rb | Updated apply_policy_tags to accept dataset parameter |
| lib/dfe/analytics/config.rb | Added airbyte_internal_dataset configuration option |
| lib/dfe/analytics/tasks/airbyte_deploy_tasks.rake | New rake task to trigger deployment job |
| lib/dfe/analytics/tasks/big_query_apply_policy_tags.rake | Updated to use new service structure |
| docs/airbyte.md | Added documentation for deployment tasks workflow |
| spec/* | Comprehensive test coverage for all new and refactored services and jobs |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
lib/dfe/analytics/services/apply_airbyte_final_tables_policy_tags.rb
Outdated
Show resolved
Hide resolved
lib/dfe/analytics/services/apply_airbyte_internal_tables_policy_tags.rb
Outdated
Show resolved
Hide resolved
goodviber
reviewed
Nov 26, 2025
goodviber
approved these changes
Nov 26, 2025
Contributor
goodviber
left a comment
goodviber
left a comment
There was a problem hiding this comment.
Approved with some comments - one from me, a few from copilot
avinhurry
added a commit
to DFE-Digital/register-early-career-teachers-public
that referenced
this pull request
Dec 2, 2025
- Update production task `send_entity_table_checks_to_bigquery` to use `DfE::Analytics::Jobs::EntityTableCheckJob`, matching the new `dfe-analytics` namespace so the job runs. - Change came from: DFE-Digital/dfe-analytics#216
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Add a rake task to automate all deployment tasks
Trello tickets:
Changes proposed in this pull request: