Skip to content

using file-based metadata#102

Draft
pkdash wants to merge 15 commits into
masterfrom
use-file-based-metadata
Draft

using file-based metadata#102
pkdash wants to merge 15 commits into
masterfrom
use-file-based-metadata

Conversation

@pkdash
Copy link
Copy Markdown
Member

@pkdash pkdash commented Apr 19, 2026 •
edited
Loading

Summary of this WIP PR:

  • Most of the hsclient functions have been updated to use s3 protocol replacing the use of hydroshare rest endpoints.
  • To keep the resource level metadata editing interface not to change due to the new schema.org metadata file based metadata, metadata mapping has been implemented between the legacy metadata format and the schema.org format.
  • TODO:
    • Similar to resource level metadata mapping, metadata mapping needs to be implemented for each content type/aggregation.
    • Tests are currently failing which need to be fixed
    • New Pydantic models for file-based metadata in schema.org format used in hsclient may have to be moved to hsmodels

pkdash added 15 commits March 24, 2026 10:35
@pkdash
Adding schema files for core metadata, dataset, and data variable bas…
caa1300 
…ed on schema org (later these should be part of hsmodels)
@pkdash
Adding adapter module for resource metadata conversion between the ne…
2aae91c 
…w schema org based pydnatic models and the legacy pydantic models
@pkdash
Adding load_json helper function to handle JSON metadata loading into…
8a57a7d 
… schema models
@pkdash
Initial S3 integration for file-based json metadata handling
97272e1 
- Introduced S3 client setup and user credential retrieval for accessing S3 resources.
- Updated Aggregation and Resource classes to support file-based metadata operations using S3.
- Refactored metadata retrieval and saving methods to utilize JSON-LD format.
- Enhanced file handling with size and checksum attributes in the File class.
@pkdash
Additional S3 integration in resource and aggregation operations/func…
d790662 
…tionalities
@pkdash
test: fxing hydroshare fixture to use s3 url path
463d519
@pkdash
Refactor DefinedTerm classes to use Literal types for name fields
72ed4f5
@pkdash
fixing 'relation' field of CoreMetadata - deleting the comma
63f52d1
@pkdash
Adding metadata adapters and resource models for legacy resource meta…
e90815d 
…data and schema.org based resource metadata convertion
@pkdash
using metadata adapter for resource metadata conversion
26b28e2
@pkdash
Using the new metadata adapter in the helper load_json function
03ab44e
@pkdash
Adding tests for legacy and schema.org resource metadata adapters
f67e9e4
@pkdash
Adding tests for resource metadata editing using hsclient
cbb322b
@pkdash
Adding s3fs to requirements for working with s3 buckets
ef891b9
@pkdash
Fix type hint to use Union[str, None]
5394f15
@pkdash pkdash changed the title using file-based metada using file-based metadata Apr 20, 2026
@pkdash
Copy link
Copy Markdown
Member Author

pkdash commented Apr 30, 2026

@sblack-usu Have a look at this initial implementation to make hsclient to work with file-based metadata. There is still quite a bit of work to do. It would be good to get your feedback on what I have implemented so far before I do more work on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@pkdash