Add plugins for handling meson subprojects

[BuildStream-Migration-Bot]

See original issue on GitLab
In GitLab by [Gitlab user @coldtom] on May 16, 2019, 14:47

Background

Meson has an idea of subprojects, which seem to be like git submodules but in meson. As a result meson elements can try to download these subprojects at build time and cause errors. It would be nice for BuildStream to integrate nicely with this feature. Thankfully it is possible to just depend on the subprojects as additional sources if necessary and stage them in the correct place.

Task description

Actually adding this support could be complicated by the fact that meson is an Element, and not a Source, so doesn't touch fetch or track. I don't know how it could be achieved in a nice way for users.

Acceptance Criteria

Meson subprojects work nicely with BuildStream.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @sstriker] on May 16, 2019, 21:13

I don't see this as a particularly compelling issue. It breaks the
separation of sources and elements. I don't think we want to compromise
there.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @willsalmon] on May 20, 2019, 15:07

Build stream tries to make everything that could effect a cache key to be defined in the yml, there are lots of reasons for this, not least for performance, as it makes caching as easy as possible, but is also important from a reproducibility point of view.

That said this sounds quite cool.

There has also been talk around having dependencies inferred from there kind, but with the details defined in the yml but out side of the element to keep it declarative, but reduce total yml.

A combination/extension/alternative of dep from element and dep from kind would be to have something like track that updates the yml, but not part of the build command.

By having a track-deps command that updates the yml rather than have this done at build time would seem to be the pure solution to both problems, both from a dependency boundary point of view and a declarative point of view.

This could also be expanded to looking at unused/redundant deps.

This would be a pretty big change.

And I'm not sure its the best way to do the dependencies inferred from kind but it would be the most expandable to things like this.

Another alternative, that would be wired but very do able, would be to have a source transform that had all the extra sources it had gone and down loaded in its ref, you can put a lot in a ref but it would be a pretty mega source plug-in. But this would only work if meson was happy find all these extras in the source area of the sand box as the source transform can only go get stuff, it cant build what it has fetched.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @tristanvb] on May 21, 2019, 08:36

I don't see this as a particularly compelling issue. It breaks the separation of sources and elements. I don't think we want to compromise there.

I agree about not wanting to compromise and/or break that separation.

That said, I still wonder if there is some parallel to be found between this feature request, and the existing work we did for projects like pip or potentially cargo.

Currently we have Sources which have the ability to view previously staged Sources at fetch and track time, and so I wonder if:

• We could first stage the (abstract) source which stages a meson project
• We could then stage a meson aware source which inspects the staged meson project in order to track the subproject sources and record them in the ref
• We could then use that ref to deterministically/reliably fetch the exact same version of those subproject sources in a deterministic way

I say all of this without any deep understanding of what meson subprojects are or how they work, but at face value this does appear to be exactly the kind of use case we developed the ability to view previously staged sources at source fetch and track time.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @sstriker] on May 21, 2019, 09:34

To be frank, I think that this is more a matter of a motivated person with
meson expertise to contribute source/build element plugins. These plugins
wouldn't need modifications to the core. And can take inspiration from say
the pip source/build plugins.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @tristanvb] on May 21, 2019, 10:02

To be frank, I think that this is more a matter of a motivated person with meson expertise to contribute source/build element plugins. These plugins wouldn't need modifications to the core. And can take inspiration from say the pip source/build plugins.

I don't disagree, that is quite the same with pip or cargo as well.

Are you suggesting we should not track it in this issue tracker for now ? I think it would make sense to track it here, and welcome the development of such a plugin in the gitlab BuildStream group somewhere, as a separate repository.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @sstriker] on May 21, 2019, 20:58

I think we need to be clear to submitters of issues whether or not the
buildstream developers will pick it up. I think that if we triage this the
answer is likely that we will not do anything with it in the foreseeable
future.
The responses on the issue so far can definitely suggest otherwise.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @tristanvb] on May 21, 2019, 22:03

Certainly agree.

I think issues are always valuable as a location for things to be discussed, as a focal point for anyone who is interested in picking up the work and as a record of potentially worthwhile things to work on for future contributors. I don't think we should ever throw away issues unless they are clearly out of scope for the project entirely or don't make any sense at all.

I don't think that the existence of an open issue should ever be taken as a statement that anyone in particular is going to work on something, I've always taken this to be implied by any open, free software project - but maybe it's worth mentioning somewhere front and center in the contributing guide or website, so that people don't start to get unrealistic expectations that other people might magically start implementing their wishlists for them.

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @cs-shadow] on Oct 10, 2019, 19:21

changed title from Add {-meson subproject support-} to Add {+plugins for handling meson subprojects+}

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @cs-shadow] on Oct 10, 2019, 19:24

I've renamed the issue to denote that this is about adding new plugins, and not about making any fundamental changes to BuildStream itself.

Are you suggesting we should not track it in this issue tracker for now ? I think it would make sense to track it here, and welcome the development of such a plugin in the gitlab BuildStream group somewhere, as a separate repository.

I personally think we do need to have a process for discussing new plugin proposals. But GitLab issues don't seem like the best way for that. Perhaps a mailing list discussion would be more useful for such plugin requests. That way, we can redirect the issue either to the core repo or some other plugin repository dependending on the nature of the request.

[nanonyme]

So is this specifically about handling wrap files? It seems not overly complex to handle. Maybe we should put this plugin into bst-plugins-experimental first though. https://mesonbuild.com/Wrap-dependency-system-manual.html Meson wrap files are reasonably simple to parse. Seems we could have two plugins, one for wrap-git and one for wrap-file. Former could subclass git with custom track and latter could implement DownloadableSource with custom track so it would pull data from Meson project. Tracking in this case must also determine directory to unpack sources to.

[nanonyme]

IMHO this has nothing to do with Meson build element. This is about adding source plugin with dependency on previous source which provides Meson project sources.