v0.16.0

What's Changed

This Release includes several large updates:

Reporting
updated template and readme for rule run status by @gerrycampion in #1635
update for execution status spaces by @gerrycampion in #1643

New Operations
1329: Add DatasetIsCustom operation to determine if a dataset is custom by @alexfurmenkov in #1580
Add minus operation for set difference to support CORE-000334 by @RakeshBobba03 in #1613

New Metadata & Rule Types
1316: New rule - value against library metadata by @alexfurmenkov in #1680
1047 define variable has method by @DmitryMK in #1664
Add define_dataset_variable_order metadata (#791) by @RakeshBobba03 in #1508
library_variable_has_codelist new metadata by @SFJohnson24 in #1628
1198: Add variable_max_size to variables metadata for enhanced data insights by @alexfurmenkov in #1679

CSV Data Readers
1558 csv data reader by @alexfurmenkov in #1640
datasets.csv open rules support by @SFJohnson24 in #1688

Operator Improvements
value_does_not_have_multiple_references operator rework by @SFJohnson24 in #1638
fix equal string parts operator value_is_literal functionality by @SFJohnson24 in #1620
#1654 added regex to is_inconsistent_across_dataset operation by @alexfurmenkov in #1658
#1652 wildcard replacement with domain on grouping_variables by @alexfurmenkov in #1674
make library_column_order a shared parent class as a move toward consistency by @gerrycampion in #1671

Bug Fixes
#1121 fix empty alias creating raising IndexError by @alexfurmenkov in #1646
#1442 fix dataset filtering when -dp is provided by @alexfurmenkov in #1645
1305: Fix type error in row number assignment to handle NaN values by @alexfurmenkov in #1673
1675: fix colliding file path by @alexfurmenkov in #1695
Fix jsonata execution status by @gerrycampion in #1707
fix codelist operations type inference for empty dataframes by @gerrycampion in #1709
Fix get_dataset by @gerrycampion in #1691
1669: stop execution when multiple --data options are provided by @alexfurmenkov in #1684
#1267 changed full path to original path in stdm reporting by @alexfurmenkov in #1618
whitespace column header removal by @SFJohnson24 in #1663
#1660 added numeric fields normalization for supp datasets by @alexfurmenkov in #1676
1699: get_parent_model_column_order calculation made only for unique rdomains and merged by @alexfurmenkov in #1711
984: define subversion workaround by @alexfurmenkov in #1697
1681: added @cached decorator to get_dataset by @alexfurmenkov in #1687
fixes by @SFJohnson24 in #1619
error handling and removed if conditional by @SFJohnson24 in #1670

Error Handling
#1616: User-friendly errors for missing/wrong Datasets tab and columns, standard validation, and library metadata lookup failures by @RakeshBobba03 in #1626
1182: Add custom exceptions for unsupported XPT format and enhance error handling in XPTReader by @alexfurmenkov in #1644
983 handle missing external dictionaries by @RakeshBobba03 in #1611
Resolve rule execution errors by @gerrycampion in #1653

CI/CD & Infrastructure
1604 automated ci workflows should run on push to main by @gerrycampion in #1642
Add GitHub Action to merge schema files with markdown descriptions by @Copilot in #1603
merge schema only runs manually. add a check schema to run in general by @gerrycampion in #1648
add pull-request perm to check schema markdown by @gerrycampion in #1668
schemas updated by @gerrycampion in #1662
Dispatch by @SFJohnson24 in #1693
dispatch action by @SFJohnson24 in #1704
make test suite more secure by @gerrycampion in #1698
revert test suite and prohibit fork runs by @gerrycampion in #1706
remove define_version by @gerrycampion in #1683
#1248 Specify define xml path in operations by @alexfurmenkov in #1649

Documentation
Update readme check_parameter.md by @SFJohnson24 in #1637
inconcistent_across_dataset readme by @SFJohnson24 in #165
Update .pre-commit-config.yaml by @SFJohnson24 in #1647
688: update tests by @alexfurmenkov in #1666

Full Changelog: v0.15...v0.16.0

v0.15

What's Changed

Several large changes come with this release that may impact end-users. TIG standard now requires the use-case argument outlined in the readme and this PR:

• TIG use case by @SFJohnson24 in #1556

The report template has changed and may impact apps that ingest it. The changes to the report are outlined in these PRs:

• 471 improved issue reporting by @gerrycampion in #1551
• converted issue templates to forms by @gerrycampion in #1631

Other updates include bug fixes, operators and operation updates, new rule type metadata. We hope you consult the reference guide on these updates

• #1197 Fix domain_is_custom operation for TIG standards and refactor domain custom check logic by @RakeshBobba03 in #1511

• #1180 Fix output file extension handling and normalize extensions to lowercase by @RakeshBobba03 in #1510

• #710 Add CORE logo icon to executable builds and update README logo by @RakeshBobba03 in #1504

• autoupdate git action by @SFJohnson24 in #1523

• run test suite and editor preview for all PRs--internal and external by @SFJohnson24 in #1524

• permissions for autoupdate by @SFJohnson24 in #1527

• Autoupdate by @SFJohnson24 in #1528

• autoupdate by @SFJohnson24 in #1529

• Test fake API key for outside collaborators making PRs by @SFJohnson24 in #1519

• Autoupdate by @SFJohnson24 in #1530

• update by @SFJohnson24 in #1512

• concurrency fix by @SFJohnson24 in #1531

• Concurrency by @SFJohnson24 in #1532

• ci by @SFJohnson24 in #1533

• e2e fix by @SFJohnson24 in #1539

• CI Pipeline Update by @SFJohnson24 in #1538

• 1501: fix JSON format report creation when -rr flag is provided by @alexfurmenkov in #1534

• Revert "#1197 Fix domain_is_custom operation for TIG standards and re… by @SFJohnson24 in #1541

• 1487: Fix engine crash when selected rule is not applicable for version by @alexfurmenkov in #1526

• Date Comparison with different precision by @SFJohnson24 in #1544

• 1174: json load fallback logic. update-cache encoding parameter. by @alexfurmenkov in #1540

• resolve double testing with CI by @SFJohnson24 in #1553

• readme by @SFJohnson24 in #1549

• FB1501 Regex string conversion by @SFJohnson24 in #1550

• 1345: correctly preprocess supp and relrec datasets by @alexfurmenkov in #1542

• Python by @SFJohnson24 in #1547

• Fb0906 by @SFJohnson24 in #1555

• update the expected json in e2e because of update to rule CG0006 by @RamilCDISC in #1579

• 1516: Refactor target name handling to preserve order by @alexfurmenkov in #1570

• make CLI work on Linux and other minor fixes by @raamana in #1496

• Allow updating cache without apikey by @nhaydel in #1506

• 1402 cg0546 support multi key param update (Execution error fix) by @RakeshBobba03 in #1447

• #1348 fix not as head condition provides proper AST by @alexfurmenkov in #1574

• preprocessor exception by @SFJohnson24 in #1572

• Move VALIDATION_FORMATS_MESSAGE from core.py to constants package by @RakeshBobba03 in #1576

• Update core.py by @SFJohnson24 in #1586

• 1188 by @SFJohnson24 in #1584

• #635 Add documentation for executability by @RakeshBobba03 in #1507

• vscode hover glitches with anyof vs oneof by @gerrycampion in #1592

• FB0405 by @SFJohnson24 in #1589

• Update README.md by @SFJohnson24 in #1609

• Update README.md by @SFJohnson24 in #1610

• 1022 enhance utf handling by @RakeshBobba03 in #1456

• move rule schema files to subfolder by @gerrycampion in #1602

• update expected json and workflow to resolve import issue by @RamilCDISC in #1612

• Update core.py by @SFJohnson24 in #1607

• test-validate updates by @SFJohnson24 in #1585

• new operation get_library_class_domains by @SFJohnson24 in #1595

• Custom Domain Logic for Library by @SFJohnson24 in #1577

• #1513 fix TypeError when generating validation reports with None entity or core id values by @alexfurmenkov in #1569

• Fb0405 by @SFJohnson24 in #1615

• Is_contained_by list wildcard logic by @SFJohnson24 in #1590

• 1212: Implement case-insensitive containment checks for column comparisons by @alexfurmenkov in #1624

• Cg0418 by @SFJohnson24 in #1614

• new titlecase operator by @SFJohnson24 in #1593

• Variable is null operation by @SFJohnson24 in #1627

New Contributors

• @raamana made their first contribution in #1496

Full Changelog: v0.14.2...v0.15

Please reach out in the discussion board https://github.com/cdisc-org/cdisc-rules-engine/discussions with questions and/or concerns.

• the CDISC Open Rules Dev Team

v0.14.2

What's Changed

• manager shutdown by @SFJohnson24 in #1490
• TIG Library Changes by @SFJohnson24 in #1480
• 1308: Fix grouping in max_date operation by @alexfurmenkov in #1459
• Codelist att by @SFJohnson24 in #1503
• AP library changes by @SFJohnson24 in #1497
• 1325: extend define xml readers _get_metadata_representation by @alexfurmenkov in #1483
• update invalid date docs by @SFJohnson24 in #1514
• 1045 define dataset variables missing documentation by @RakeshBobba03 in #1509
• fix release tests by @SFJohnson24 in #1518

Full Changelog: v0.14.1...v0.14.2

v0.14.1

What's Changed

• AP-- schema by @SFJohnson24 in #1471
• new report template by @SFJohnson24 in #1461
• record count optimize by @SFJohnson24 in #1472
• 1421: load variables ccode correctly by @alexfurmenkov in #1449
• added domain failure to find exception by @SFJohnson24 in #1481
• Codelist att by @SFJohnson24 in #1477
• 1332: correctly substitute AP domains placeholders by @alexfurmenkov in #1468
• hotfix by @SFJohnson24 in #1494
• 1442: add filetype option to validate type of data by @alexfurmenkov in #1482
• test data datasetjson 1.1.0 by @SFJohnson24 in #1499

Full Changelog: v0.14.0...v0.14.1

v0.14.0

What's Changed

Mac with Intel chipset has been incremented to macOS15 as macOS13 runner has been deprecated

• intel mac runner macOS15 by @SFJohnson24 in #1473
• 897: check that values of specified variable is valid XHTML by @alexfurmenkov in #1371
• 239 validate interval parts by @RakeshBobba03 in #1398
• 1379 fb4607,fb4608 rule sensitivity by @RakeshBobba03 in #1405
• DDF report template working by @gerrycampion in #1397
• cg0370 by @SFJohnson24 in #1404
• CG0001 by @SFJohnson24 in #1411
• trailing zeroes by @SFJohnson24 in #1414
• look for datasets recursively by @RamilCDISC in #1322
• report template updates by @gerrycampion in #1416
• list-rule-set update by @SFJohnson24 in #1415
• 715: usdm schema validation by @alexfurmenkov in #1375
• 1387 improve error by @RakeshBobba03 in #1407
• Update docs for referenced_variable_metadata by @gerrycampion in #1420
• Cg0288 by @SFJohnson24 in #1419
• version by @SFJohnson24 in #1427
• Add JSON Schema Check to Rule_Type.json by @ASL-rmarshall in #1431
• 720: ignore not existing domains by @alexfurmenkov in #1432
• 890: column should consist of delimited codelists by @alexfurmenkov in #1430
• reorganize readme by @gerrycampion in #1434
• 1402 CG0546 support multi-key param update by @RakeshBobba03 in #1429
• Cg0562 by @SFJohnson24 in #1422
• Hotfix for get_xhtml_errors operation by @ASL-rmarshall in #1436
• 1403: Update the cache location to be relative to the folder containing core.py by @alexfurmenkov in #1438
• Update README.md with cache documentation by @SFJohnson24 in #1444
• 1204: improve error message if dataset files are not found by @alexfurmenkov in #1441
• Multiidvar by @SFJohnson24 in #1437
• 1372 datetime comparison by @RakeshBobba03 in #1406
• #1017 Fix is_column_of_iterables to check all values instead of just first row by @RakeshBobba03 in #1446
• Errorhandling preprocessor by @SFJohnson24 in #1445
• 1433: Fix report status handling for JSON schema check rules by @alexfurmenkov in #1455
• 1327: Fix dataset metadata handling in DatasetMetadataDefineDatasetBuilder by @alexfurmenkov in #1440
• Schema by @SFJohnson24 in #1453
• Add documentation and remove unused functions by @ASL-rmarshall in #1457
• #1194 Add domain suffix extraction and standard domains operation for domain validation by @RakeshBobba03 in #1439
• 939 rule blocked corerules 9609 codelist terms operation does not retrieve correct terms by @gerrycampion in #1462
• readme by @SFJohnson24 in #1465
• update all commands and docs by @SFJohnson24 in #1460
• 1017 ddf00102 by @RakeshBobba03 in #1469
• CG0562 by @SFJohnson24 in #1454
• Error Tracing by @SFJohnson24 in #1470

Full Changelog: v0.13.0...v0.14.0

v0.13.0

What's Changed

NOTE: changes to all long form validation arguments were made to attain consistency. The use of underscores with all long form validation arguments is now in place. See the below PR or the github repo README

• 1351 Validation argument consistency by @DmitryMK in #1355

##Other updates include:

• new validation arguments max_report_rows by @SFJohnson24 in #1359
• max_errors refactor by @SFJohnson24 in #1385
• Docker release trigger by @SFJohnson24 in #1284
• CG0303 by @SFJohnson24 in #1285
• dockerfile updates by @SFJohnson24 in #1291
• "DDF performance fix and formats" by @pendingintent in #1290
• mac intel executable by @SFJohnson24 in #1292
• local rules command multiple arguments by @SFJohnson24 in #1301
• Test Suite summary reports by @SFJohnson24 in #1294
• Performance 1233 by @pendingintent in #1302
• cg0314 by @SFJohnson24 in #1306
• e2e testing @RamilCDISC in #1207
• reference guide formatting by @SFJohnson24 in #1314
• cg0371 by @SFJohnson24 in #1313
• Dockerbuild by @SFJohnson24 in #1315
• update builder by @SFJohnson24 in #1318
• reformat doc by @SFJohnson24 in #1323
• 1335: Fixing CT version in JSON report by @DmitryMK in #1336
• 1266: avoid crashes when reading empty xpt files by @alexfurmenkov in #1347
• 1326: codelist preferred term check by @alexfurmenkov in #1352
• null logic by @SFJohnson24 in #1354
• None/NaN handling by @SFJohnson24 in #1357
• Pympler fallback bug fix by @SFJohnson24 in #1358
• output variables numpy scrubber by @SFJohnson24 in #1361
• Adding --exclude-rules parameter by @DmitryMK in #1365
• list rules uppercase fix by @SFJohnson24 in #1368
• 1104 dataservice and datareader for arbitrary json files by @gerrycampion in #1304
• Converted representation interface classes to dataclasses by @gerrycampion in #1376
• Update jsonata_processor.py by @gerrycampion in #1377
• 1333: sync requirements.txt and pyproject.toml by @alexfurmenkov in #1381
• 1366 update exit code by @DmitryMK in #1369
• 1211-fix-na-vs-nan-excel by @RakeshBobba03 in #1383
• rounding parameter by @SFJohnson24 in #1389
• ADaM cache and scope by @SFJohnson24 in #1300
• Update Operator.md by @SFJohnson24 in #1400
• update release by @SFJohnson24 in #1391
• update release by @SFJohnson24 in #1401

New Contributors

• @pendingintent made their first contribution in #1290
• @RakeshBobba03 made their first contribution in #1383

Full Changelog: v0.12.0...v0.13.0

v0.12.0

What's Changed

• Cg0371 operator update by @SFJohnson24 in #1201
• updated for supp by @SFJohnson24 in #1209
• new operator, updated record count logic by @SFJohnson24 in #1215
• added aprelsupp logic by @SFJohnson24 in #1216
• Logging Bug by @SFJohnson24 in #1226
• column_prefix bug by @SFJohnson24 in #1229
• added custom list functionality by @SFJohnson24 in #1230
• Record column by @SFJohnson24 in #1224
• Update README.md by @SFJohnson24 in #1234
• Supp-- match dataset by @SFJohnson24 in #1231
• Meddra bug by @SFJohnson24 in #1235
• Update null handling by @SFJohnson24 in #1227
• map, codelist_extensible and codelist_terms for usdm by @gerrycampion in #1199
• Lib var data type by @SFJohnson24 in #1220
• filter wildcard by @SFJohnson24 in #1236
• removed unused code in data_processor by @SFJohnson24 in #1238
• Added filter to distinct operation by @gerrycampion in #1218
• added ccode parsing logic by @SFJohnson24 in #1239
• added datset name and rule id to error details by @gerrycampion in #1242
• updated subclasses by @SFJohnson24 in #1240
• Fix cache failure breaking rule validation in multiprocessing environment by @Copilot in #1246
• CG0235 by @SFJohnson24 in #1244
• added missing var by @SFJohnson24 in #1257
• update null handling by @SFJohnson24 in #1260
• dask readme by @SFJohnson24 in #1250
• updated logger by @SFJohnson24 in #1262
• bug fix by @SFJohnson24 in #1270
• inconsistent_across_dataset oeprator update by @SFJohnson24 in #1269
• update commands by @SFJohnson24 in #1254
• update tests by @SFJohnson24 in #1275
• update reporting by @SFJohnson24 in #1279
• updated to_dict dask by @SFJohnson24 in #1280
• Cg00014 by @SFJohnson24 in #1281
• fixed ref guide nesting, removed operators/ations by @SFJohnson24 in #1282
• Define metadata by @SFJohnson24 in #1273
• operator rework by @SFJohnson24 in #1276

Full Changelog: v0.11.0...v0.12.0

v0.11.0

What's Changed

• update docker trigger by @SFJohnson24 in #1165
• added get_parent_model_column_order error handling by @gerrycampion in #1166
• New Operator: is_ordered_subset_of by @SFJohnson24 in #1172
• add usdm rule coverage in test suite by @RamilCDISC in #1157
• Update valid_codelist_dates (#1161) by @ASL-rmarshall in #1176
• Python 3.12 increment by @SFJohnson24 in #1178
• Fix TestRule Bug by @ASL-rmarshall in #1183
• Rename is_consistent_across_dataset to is_inconsistent_across_dataset by @ASL-rmarshall in #1186
• Update meddra_code_references_validator.py by @SFJohnson24 in #1191
• Dask 3.12 compatibility by @SFJohnson24 in #1184
• end2end testing for engine by @RamilCDISC in #1137
• New Rule Types with Value Check and Metadata by @SFJohnson24 in #1148

Full Changelog: v0.10.0...v0.11.0

v0.10.0

What's Changed

• deprecate ubuntu 20.04 automated release by @SFJohnson24 in #1120
• Direct excel datasets added by @gerrycampion in #1114
• added rule core_id to operations cache key by @gerrycampion in #1117
• bugfix: update null flavors for JSON output by @SFJohnson24 in #1127
• simplify and fix lib metadata by @gerrycampion in #1123
• docker image gitaction by @SFJohnson24 in #1131
• readme overhaul by @SFJohnson24 in #1138
• updated validate to run tig standard by @SFJohnson24 in #1139
• updated medrt errorhandling by @SFJohnson24 in #1143
• Use case rule filter for TIG by @SFJohnson24 in #1142
• Debugging for cache keyerror by @gerrycampion in #1145
• local rule std and version check by @SFJohnson24 in #1151
• updated rule tester actions by @gerrycampion in #1152
• Custom standards and validation via cache by @SFJohnson24 in #1156
• Relrec fixed with idvarvals populated by @gerrycampion in #1158

New Contributors

• @chowsanthony made their first contribution in #1136

Full Changelog: v0.9.3...v0.10.0

v0.9.3

What's Changed

• custom rules readme by @SFJohnson24 in #1046
• External Dictionary readme by @SFJohnson24 in #1067
• Custom schema by @SFJohnson24 in #1069
• updated operations readme and examples by @SFJohnson24 in #1072
• usdm to use datasemetada class by @gerrycampion in #1081
• Bug fix for expanding list of columns when column doesn't exist by @gerrycampion in #1082
• Automatically run full test suite on PRs by @gerrycampion in #1053
• Get library class and domain only if standard and version provided by @gerrycampion in #1095
• Tig reporting by @SFJohnson24 in #1077
• ruletester library fix by @SFJohnson24 in #1099
• pypi readme and fix by @SFJohnson24 in #1092
• Test command merged into validate by @gerrycampion in #1094
• Update README.md by @SFJohnson24 in #1101
• Update README.md by @gerrycampion in #1102
• Update az login creds to use org-level creds and fix ruletester deploy by @gerrycampion in #1103
• NaN bug in JSON result by @SFJohnson24 in #1113
• missing all outcome vars logic changed by @SFJohnson24 in #1110

Full Changelog: v0.9.2...v0.9.3