[API] Filter refactor

[SchrodingersGat]

This PR is a major refactor of the enable_filter functionality which allows optional fields to be added (or removed) dynamically to API serializers.

Problem Description

The old code was written in such a way that the optional serializer class was instantiated on definition.

Consider the following code

    category_detail = enable_filter(
        CategorySerializer(
            source='category', many=False, read_only=True, allow_null=True
        ),
        prefetch_fields=['category'],
    )

The CategorySerializer class is instantiated (which is quite expensive) - and then this is also chained down for any multi-level child serializers.

In many cases, the fields are later removed, so instantiating them early is a waste of resources.

Also, due to the (wasteful?) way that DRF deep-copies all the fields (multiple times throughout the lifespan of the serializer) this can be very prohibitive.

Some particularly bad API endpoints which had deep nested serializers, saw hundreds of thousands of serializer objects created, and then later deleted.

Solution

The PR introduces an OptionalField approach, which lazily evaluates the optional fields only after we have decided that they should definitely be included in the serializer:

    category_detail = OptionalField(
        serializer_class=CategorySerializer,
        serializer_kwargs={
            'source': 'category',
            'many': False,
            'read_only': True,
            'allow_null': True,
        },
        prefetch_fields=['category'],
    )

Justification

The new API endpoints are significantly faster, we have been introducing a huge amount of wasted overhead (for years now in the codebase) due to the unnecessary serializer evaluation

Benchmarks

Benchmarking shows that serializers which are deeply nested have the most benefit from this PR. Both GET and OPTIONS requests are improved substantially by deferring serializer instantiation.

Methodology

• Perform external requests via python API bindings
• Average across 100 requests per test
• All times specified in ms

GET

• limit=50
• offset=0

URL	Master	PR
/api/part/	151	89
/api/part/category/	43	43
/api/stock/	173	74
/api/stock/location/	64	42
/api/company/	47	38
/api/build/	86	47
/api/build/line/	646	351
/api/build/item/	276	41
/api/order/so/	75	50
/api/order/so/shipment/	90	79
/api/order/po/	89	50
/api/order/po-line/	187	62
/api/user/roles/	32	37
/api/parameter/	42	36
/api/parameter/template/	34	33

OPTIONS

URL	Master	PR
/api/part/	85	71
/api/part/category/	43	33
/api/stock/location/	42	37
/api/company/	43	36
/api/build/	72	61
/api/build/line/	266	44
/api/build/item/	210	31
/api/order/so/	67	60
/api/order/so/shipment/	70	50
/api/order/po/	66	57
/api/order/po-line/	185	60
/api/user/roles/	28	30
/api/parameter/	36	35
/api/parameter/template/	30	33

Search

URL	Master	PR
/api/search/	381	294

[netlify]

✅ Deploy Preview for inventree-web-pui-preview canceled.

Name	Link
🔨 Latest commit	19c2d34
🔍 Latest deploy log	https://app.netlify.com/projects/inventree-web-pui-preview/deploys/699af5b7ffb9570007b5e7cc

[Copilot]

Pull request overview

This PR refactors the enable_filter functionality to use a new OptionalField dataclass approach for lazy evaluation of optional serializer fields, addressing performance issues caused by premature serializer instantiation.

Key changes:

• Replaced enable_filter() function with OptionalField dataclass for declarative field definitions
• Refactored FilterableSerializerMixin to lazily instantiate optional fields only when needed
• Updated all serializers across the codebase to use the new OptionalField pattern

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
InvenTree/serializers.py	Introduced OptionalField dataclass and refactored FilterableSerializerMixin to support lazy field initialization via build_unknown_field()
InvenTree/test_serializers.py	Updated tests to use new OptionalField API; removed obsolete enable_filter validation test; corrected spelling from "failiure" to "failure"
users/serializers.py	Migrated GroupSerializer fields (permissions, roles, users) from enable_filter to OptionalField
stock/serializers.py	Converted multiple optional detail fields including user_detail, template_detail, location_path, part_detail, supplier_part_detail, tests, item_detail to use OptionalField
part/serializers.py	Refactored optional fields across CategorySerializer, PartBriefSerializer, PartSerializer, BomItemSerializer, and related serializers to use OptionalField pattern
order/serializers.py	Updated order-related serializers including purchase order, sales order, and return order serializers to use OptionalField for detail fields
company/serializers.py	Migrated company, manufacturer part, and supplier part serializers' optional fields to OptionalField
common/serializers.py	Converted parameter serializer detail fields to use OptionalField
common/filters.py	Updated filter helper functions to return OptionalField instances instead of enable_filter wrapped fields
build/serializers.py	Refactored build serializer optional fields including part, user, and item detail fields to use OptionalField

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov]

Codecov Report

❌ Patch coverage is 97.62712% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.00%. Comparing base (6c58cc7) to head (19c2d34).
⚠️ Report is 9 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #11073      +/-   ##
==========================================
- Coverage   88.00%   88.00%   -0.01%     
==========================================
  Files        1295     1295              
  Lines       58821    58855      +34     
  Branches     1943     1943              
==========================================
+ Hits        51764    51793      +29     
- Misses       6573     6578       +5     
  Partials      484      484              

Flag	Coverage Δ
backend	89.19% <97.62%> (+<0.01%)	⬆️
migrations	42.10% <69.15%> (-0.07%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
Backend Apps	91.70% <99.33%> (+<0.01%)	⬆️
Backend General	93.37% <100.00%> (+<0.01%)	⬆️
Frontend	70.87% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[matmair]

I might be overlooking something, but are we not completely loosing static introspection with this change? I am not sure if there is much actual runtime performance to be gained by this, have you benchmarked it?

[SchrodingersGat]

I am not sure if there is much actual runtime performance to be gained by this, have you benchmarked it?

Benchmarks added to the top comment.

I might be overlooking something, but are we not completely loosing static introspection with this change?

We should not lose any introspection as the fields are still created - they are just defered until we determine they are actually needed. This is most important in the case of deeply nested serializer fields which will never be exposed to the final serializer tree.

The OPTIONS endpoints still show all the optional fields, and I believe the API docs should be the same. LMK if you spot any items which are no longer included.

[codspeed-hq]

Merging this PR will improve performance by 12.44%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 1 improved benchmark
✅ 1 untouched benchmark
⏩ 42 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	WallTime	test_api_stock_list	428.3 ms	380.9 ms	+12.44%

---

_{Comparing SchrodingersGat:filter-refactor (19c2d34) with master (6c58cc7)}

Footnotes

1. 42 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[SchrodingersGat]

@matmair any idea why the codespeed performance checks do not have any baseline data from master to compare against?

[matmair]

No, looking into it now

[SchrodingersGat]

I have also found drf-flex-fields, similar in approach to drf-shapeless-serializeres.

However from my reading of both the nested fields are not available as class attributes until after the top level serializer class is instantiated - meaning that any type hinting / introspection would also not be available?

Unless you have any experience with these libraries that shows working type hinting?

[SchrodingersGat]

No, looking into it now