[feature] Allow to create parameterized tests

[dmajere]

Hello,
Its common to have tests with similar logic but different inputs. For such situations most testing frameworks support some kind of data attributes (DataRow in MSTest or InlineData in XUnit).
I think it would be useful to have a similar feature for GoDotTest.
Thank you.

[blewis-web]

After chatting with @jolexxa and @wlsnmrk on Discord, we agreed that running the equivalent of xUnit Theory to do parameterized tests with InlineData and MemberData would be a welcome enhancement, but we do not prefer to ever use ClassData.

To learn more about parameterized tests in xUnit, see this helpful resource:
https://andrewlock.net/creating-parameterised-tests-in-xunit-with-inlinedata-classdata-and-memberdata/

On my team, we lean into strongly typed MemberData in particular with xUnit. Here is a helpful resource on how to do that:
https://andrewlock.net/creating-strongly-typed-xunit-theory-test-data-with-theorydata/

I've been approved to contribute towards this feature on behalf of my company and have created a PBI on our team's backlog for follow up whenever we have the time to bring it onto a sprint. We are pretty busy right now, but I hope I could start looking into this by 2025 Q4 at the latest (possibly spending some time as an individual looking into this before then).

If anyone wants to point out some inner workings of GoDotTest to consider or share pointers in advance that would be much appreciated whenever I spend time looking at this.

[jolexxa]

@blewis-web I think you'll be interested in the TestExecutor. It flattens all the methods out into a single sequence of methods that should be executed. You'd essentially want to modify this sequencing to unroll test methods with their inline data. This could get kind of involved depending on how the data is provided, but there may be clever ways to approach it.

GoDotTest/Chickensoft.GoDotTest/src/TestExecutor.cs

Line 143 in 22e814b

var allMethods = suite.SetupAllMethods.Concat(

[bclewi]

Worth noting is that gdUnit4Net just added parameterized tests via [TestCase] and [DataPoint].

[blewis-web]

@jolexxa @wlsnmrk ,
I have ~8hrs or so to start looking into this and play with some ideas until ~9/16. I'll have some time to implement/test features in separate work items on future sprints, so right now I'm just doing research/planning/experimenting. Apologies for any confusion on the two GH accounts, I'm kind of figuring it out on the fly if I like using a "for work" GH account or just want to stick to my personal one.

I'm starting by looking into what was done here:
#169

and then will start playing around with some draft ideas to run by you both.

-Brandon

Edit: Updated window for planning/research.

[blewis-web]

@jolexxa @wlsnmrk,
I noticed that some of the implementations out there for parameterized tests rely on System.Reflection, such as https://github.com/xunit/xunit/blob/main/src/xunit.v3.core/InlineDataAttribute.cs

Am I correct to assume that it would be ok to do the same in GoDotTest without any restrictions since the tests will not run in release builds? I see that GoDotTest already used System.Reflection in some places, such as
https://github.com/chickensoft-games/GoDotTest/blob/main/Chickensoft.GoDotTest/src/GoTest.cs
but I wasn't sure if maybe there were some considerations to keep in mind around that.

[jolexxa]

@blewis-web it is fine to use reflection since we can expect tests to be executed from a desktop environment. That may change someday in the distant future if there is ever a need for on-device end to end test suites, but this is not that day yet.

[blewis-web]

@jolexxa @wlsnmrk,

Idea Drafting: blewis-web#1

What do you all think about these ideas in terms of:

1. the development approach (as noted in code comments)
2. class names
3. where new classes might fit in the existing code base
4. testing approach
5. if any more idea drafting / planning would be helpful prior to adding features one by one?

Copied from code comments (since I thought it was a bit buried in there):

My team is interested in contributing the following features (but probably not more than that):

1. InlineDataAttribute
2. MemberDataAttribute with Static Methods that return IEnumerable<object?[]>
   3. MemberDataAttribute with Static Methods that return TheoryData (strongly typed)

[wlsnmrk]

@blewis-web thanks!

@jolexxa is more familiar with this code base than I am, but so far this all makes sense to me. The main suggestion I might offer before starting on implementation is that it might be worth filling in some additional detail on handling both untyped and strongly-typed MemberDataAttribute cases.

[blewis-web]

@jolexxa @wlsnmrk
I updated my draft with some more details (and some hopefully good questions). I put examples back in for how strongly-typed data might look in terms of example tests and classes. However, since I only have ~8hrs for this first increment focused on high level planning (and I'm a bit over that already), I'm thinking that figuring out how strongly-typed data would be used throughout GoDotTest will need a separate Spike if more details are desired prior to "going for it" (and discussing those details over the actual PR). If there is a concern that adding strongly-typed data would make various parts of GoDotTest too complex, then that extra Spike might be nice so that we can spend the time focused on finding a simple solution that everyone is comfortable with - or else maybe we decide to stick with untyped data if there isn't a viable solution. Would a Spike to follow up more on strongly-typed data be helpful?

Since development would be incremental, are there are enough ideas in place for me to create implementation PBIs for the first couple of untyped data features? Could I please get some high-level feedback (such as the points noted in my prior comment above: #116 (comment), and in the draft PR comments)?

[blewis-web]

@jolexxa and I discussed the scope and landed on it being better for GoDotTest to leave the strong typing out for now. It would be awesome but would also significantly increase complexity and lower maintainability, which doesn't quite fit with the goals for GoDotTest. GDUnit4Net doesn't have strong typing either, so this would be better for keeping it simple and on par with the common features that are expected for a testing framework. I agreed that the strong typing is more of a "nice to have", and that getting the errors when running the test isn't too much worse than getting errors at compile time in the context of a testing framework.