2.0.0 - First Release Of Data Output

Release Notes - Version 2.0.0 - First Release Of Data Output

It is time to move on from the 1.x that tried to keep a link between cosmotech API 3.x and 5.x and fully embrace the non retro-compatibility.

Now for a full breakdown of the release content by our favorite redactor : Mr. Art Ifficial.

Overview

This major release represents a significant evolution of the CosmoTech Acceleration Library (CoAL), introducing CoAL 2.0 with breaking changes that remove retrocompatibility with cosmotech-api versions below 5.0. The release includes comprehensive architectural refactoring, new features for data export, and removal of deprecated functionality.

🚨 Breaking Changes

API Compatibility

• Removed retrocompatibility with cosmotech-api < 5.0
• This is a major version bump requiring updates to dependent projects

Removed Deprecated Commands

The following deprecated commands have been removed from csm-data:

• rds-load-csv - RDS CSV loading functionality
• rds-send-csv - RDS CSV sending functionality
• rds-send-store - RDS store sending functionality
• tdl-load-files - Twin Data Layer file loading
• tdl-send-files - Twin Data Layer file sending
• runtemplate-load-handler - Run template handler loading

Removed Modules

• cosmotech.coal.azure.functions - Deprecated Azure functions module
• cosmotech.coal.cosmotech_api.twin_data_layer - Legacy Twin Data Layer module
• cosmotech.coal.cosmotech_api.dataset/* - Legacy dataset submodules (converters, download, upload, utils)
• cosmotech.coal.cosmotech_api.runner/* - Legacy runner submodules (data, datasets, download, metadata, parameters)
• cosmotech.coal.utils.postgresql - Moved to cosmotech.coal.postgresql.utils
• cosmotech.coal.utils.semver - Removed deprecated semver utilities
• Legacy connection, parameters, run, run_data, run_template, and workspace modules from cosmotech_api

✨ New Features

Output Channel System

A new flexible output channel system for data export operations:

• Channel Interface: Base interface for output operations (channel_interface.py)
• AWS S3 Channel: Export data directly to S3 buckets (aws_channel.py)
• Azure Storage Channel: Export data to Azure Blob Storage (az_storage_channel.py)
• PostgreSQL Channel: Export data to PostgreSQL databases (postgres_channel.py)
• Channel Splitter: Multi-destination output support (channel_spliter.py)

Configuration Management

• New configuration.py module in cosmotech.coal.utils for centralized configuration loading and management
• Enhanced configuration handling across the library

New csm-data Command

• output - New command for flexible data export functionality

🔧 Refactoring & Improvements

CosmoTech API Architecture

Complete restructuring of the API client with modular design:

New /apis/ Directory Structure

• dataset.py - Dataset management operations
• meta.py - Metadata operations
• organization.py - Organization management
• run.py - Run operations
• runner.py - Runner management
• solution.py - Solution operations
• workspace.py - Workspace management

New /objects/ Directory

• connection.py - Enhanced connection objects and handling
• parameters.py - Improved parameter management objects

Cloud Provider Modules

Azure Improvements

• Enhanced blob.py and storage.py modules
• Updated ADX modules (auth, runner, store, tables)
• Improved error handling and authentication

AWS S3 Enhancements

• Refactored s3.py for better performance and maintainability
• Improved error handling and retry logic

PostgreSQL Integration

• New utils.py module with comprehensive PostgreSQL utilities
• Enhanced runner.py with improved functionality
• Updated store.py with better operations handling
• Moved utilities from cosmotech.coal.utils.postgresql to cosmotech.coal.postgresql.utils

Updated Commands

All existing csm-data commands have been updated to work with the new architecture:

• postgres-send-runner-metadata
• run-load-data
• wsf-load-file and wsf-send-file
• S3 commands (s3-bucket-delete, s3-bucket-download, s3-bucket-upload)
• Store commands (dump-to-s3, store)

🧪 Testing

Test Suite Restructure

Major reorganization of tests to match the new code structure:

New Test Modules

• test_apis/test_dataset.py - Dataset API tests
• test_apis/test_runner.py - Runner API tests
• test_apis/test_simple_apis.py - Simple API tests
• test_apis/test_workspace.py - Workspace API tests
• test_objects/test_connection.py - Connection object tests
• test_objects/test_parameters.py - Parameters object tests
• test_store/test_output/* - Complete test suite for output channels
• test_utils/test_utils_configuration.py - Configuration utility tests

Reorganized Tests

• Updated AWS S3 tests for new structure
• Enhanced Azure ADX tests (store, tables, utils)
• Improved PostgreSQL tests with better organization
• Removed tests for deprecated modules

Test Statistics

• 166 files changed
• 5,548 insertions(+), 12,552 deletions(-)
• Net reduction of ~7,000 lines while improving functionality

📚 Documentation

Updated Documentation

• Removed documentation for deprecated RDS and TDL commands
• Removed runtemplate-load-handler documentation
• Updated csm-data tutorial to reflect command changes
• Updated API documentation to match new structure

Removed Tutorial Files

• tutorial/csm-data/tdl_load_files.bash
• tutorial/csm-data/tdl_send_files.bash

🛠️ Development & Tooling

Configuration Updates

• Enhanced .coveragerc for better test coverage tracking
• Added .flake8 configuration for code linting
• Updated .pre-commit-config.yaml with new hooks
• Updated pyproject.toml with current project metadata

Dependencies

• Consolidated requirements files (removed requirements.extra.txt and requirements.past.txt)
• Updated requirements.all.txt, requirements.dev.txt, and requirements.txt
• Added new development dependencies for improved tooling

🔍 Translation Files

Added and updated translation files for internationalization:

• New translations for output modules
• Updated dataset service translations
• Added configuration utility translations
• Removed translations for deprecated commands

📋 Design Documentation

• Added designs/data_outputs/requirements.md documenting the output system requirements and architecture

🐛 Bug Fixes

• Fixed missing lint on files (commit 58fcb76)
• Various bug fixes and improvements throughout the refactoring process

📦 Migration Guide

For Users of Deprecated Commands

If you were using any of the removed commands, you'll need to:

1. Update to the new output command for data export operations
2. Migrate from RDS/TDL commands to the new modular API structure
3. Review the updated documentation for equivalent functionality

For API Users

If you were using the CoAL API directly:

1. Update imports from legacy modules to new /apis/ and /objects/ structure
2. Replace cosmotech_api.twin_data_layer usage with new modular APIs
3. Update connection and parameter handling to use new objects
4. Migrate PostgreSQL utilities from coal.utils.postgresql to coal.postgresql.utils

🎯 Requirements

• Python: 3.11+
• CosmoTech API: 5.0+

🔗 Links

• Documentation: https://cosmo-tech.github.io/CosmoTech-Acceleration-Library
• Source Code: https://github.com/Cosmo-Tech/CosmoTech-Acceleration-Library
• Issues: https://github.com/Cosmo-Tech/CosmoTech-Acceleration-Library/issues

👥 Contributors

This release includes contributions from the CosmoTech development team, representing a significant collaborative effort to modernize and improve the library architecture.

---

Full Changelog: 1.1.0...2.0.0

1.1.0 - Prepare for cosmotech-api 5.0

Prepare for cosmotech-api 5.0 compatibility

Key Areas of Changes:

• New Features:
  • Dataset upload functionality
• API Compatibility:
  • Support for CosmoTech API v5.0 with backward compatibility
• Bug Fixes:
  • Multiple fixes for dataset download
  • file processing
  • credential handling
• CI/CD Improvements:
  • Dependency updates
  • GitHub Actions improvements
• Code Quality:
  • Black formatting,
  • test improvements,
  • code cleanup

What's Changed

• Create track_dependencies.yml by @Leopold-Cramer in #59
• ci: Update checkout action version to v5 by @sellisd in #60

New Contributors

• @Leopold-Cramer made their first contribution in #59

Full Changelog: 1.0.1...1.1.0

0.10.2 - Python 13 compatibility

Python 13 compatibility

With the release of Debian 13, some of our users got issues setting up installs of CoAL 0.10, this release update the incriminated dependencies to python 13 compatible ones

Full Changelog: 0.10.1...0.10.2

0.10.1 - patching use of shutils on mounts

Patch shutil.copytree usage

When used on mounted volumes shutil.copytree got permissions errors, this patch should avoid direct use of copytree on known mount points

Full Changelog: 0.10.0...0.10.1

1.0.1 - The missing pieces

We forgot stuff so this release is made to make it look like we did not

New Features

• Added missing encode password option for PostgreSQL commands

Dependencies and Build System

• Removed locked dependency to setuptools (now handled as build dependency)
• Updated pyarrow to version 20
• Updated dev requirements
• Updated documentation requirements

Documentation

• Updated scanner configuration documentation
• Removed documentation for deprecated commands

Internationalization (i18n)

• Implemented translations for csm-data main command
• Separated and updated translation files for csm-data
• Updated translation keys from "csm-data" to "csm_data"

General Improvements

• Corrected workflows
• Applied linter corrections after black update

This release includes security improvements with PostgreSQL password encoding, dependency updates including pyarrow v20, enhanced internationalization support for the csm-data component, and various documentation and code quality improvements.

What's Changed

• Update pyarrow to version 20 by @sjoubert in #54
• docs: update scanner config docs by @sellisd in #55
• feat: add missing encode password option for psql commands by @neomatamune in #56

New Contributors

• @sellisd made their first contribution in #55

Full Changelog: 1.0.0...1.0.1

1.0.0 - Open Core Extension Layer for Operational Transform (OCELOT)

Open Core Extension Layer for Operational Transform (OCELOT)

Finally a major release of CoAL ! we get out of the 0.x hell and welcome the suffering of following Major.Minor.Patch <3
This version comes with a really big overhaul of the code base, no more Cosmotech_Acceleration_Library.Accelerator welcome to cosmotech.coal ! Yeah :D !

Lots of changes were made for this version, most of the code base got reworked to get ready for future evolutions.

now for the AI slob because we love our new computer friends <3

The first major release of the CoAL library marks a significant milestone in providing a comprehensive, enterprise-ready framework for cloud operations and data transformation. This release introduces a complete overhaul of the core architecture while adding extensive new capabilities for operational transformations.

Core Enhancements

• Complete project structure reorganization for improved modularity
• Enhanced development infrastructure with comprehensive testing suite
• Streamlined CI/CD workflows for reliable deployments
• New plugin architecture supporting extensible operations

Cloud Integration

• Robust AWS S3 integration for cloud storage operations
• Comprehensive Azure platform support:
  • Blob storage management
  • Azure Functions integration
  • Azure Data Explorer (ADX) connectivity with advanced query capabilities
• Enterprise database support:
  • PostgreSQL integration with runner and store capabilities
  • SingleStore support for high-performance operations

Data Management

• Enhanced store module implementations:
  • Optimized CSV handling
  • Advanced Pandas integration
  • High-performance PyArrow support
  • Native Python store capabilities
• Comprehensive dataset functionality:
  • Advanced data converters
  • Streamlined download capabilities
  • Extended utility functions

Operational Features

• Extended CosmoTech API capabilities:
  • Enhanced connection management
  • Workspace operations
  • Run template handling
  • Parameter management
• Runner module implementation:
  • Sophisticated data handling
  • Dataset management
  • Download capabilities
  • Comprehensive metadata support

Enterprise Ready

• Multi-language support with translation infrastructure
  • English (US) language pack
  • Extensible translation framework
• Comprehensive documentation and tutorials
• Orchestrator plugin templates for custom extensions

What's Changed

• Prerelease 1.0.0 by @neomatamune in #53

Full Changelog: 0.10.0...1.0.0

0.10.0 - orc 2.0 is the new orc

Update dependency to csm-orc to use 2.0.0

This update of the dependency to csm-orc mostly allow to use the new logging and translation systems added in csm-orc and correct a few issues in loggers where tags in text "[...]" could end up crashing the commands

Full Changelog: 0.9.5...0.10.0

0.9.5 - Conditional encoding

Conditional Encoding

now with env vars to control psql password encoding

Full Changelog: 0.9.4...0.9.5

0.9.4.1 - POSTGRESQL !!!

Full Changelog: 0.9.4...0.9.4.1

0.9.4 - POSTGRESQL !!!

Ensure Postgresql schema is used on data append

During inejction of data store results into Postgresql no verifications on the datatypes were executed, now we add a check and tentative conversion of types to existing schema if target table exists on append mode

• 0.9.4 by @neomatamune in #52

Full Changelog: 0.9.3...0.9.4