'Input string is longer than 4000 character(s)' error

[LMacchi]

Associated issue: choria-io/choria-mcorpc-support#217

TL;DR: in order to increase the maxlength of the input string, there will be 2 places to update: the choria-mcorpc-support gem, issue listed above, and this module.

This issue will track the update of the Puppet module.

The maxlength for input is 4000 chars. My users started using helper scripts to make tasks more modular, and the combination of the task script + helper scripts sometimes go over 4000 chars. We've been advising people to only import the files they require instead of blanket files/*, but the library of helper scripts is growing every day and I fear that soon importing just the necessary helpers won't be enough.

[ripienaar]

In my opinion with how tasks work - this is a really bad idea. Way too many files to make sense with how tasks are designed.

Do you at least copy only the relevant parts of that repo into your side or just all the things?

Seems like you should perhaps install those helpers into your system python via puppet so its always there?

[LMacchi]

We thought about installing the helpers outside of tasks, but we've just inherited them when migrating to Choria, so we're still scoping the effort and trying to keep parity with PE tasks as much as possible in the mean time. We're also encouraging best practices like syncing only the helpers you need instead of the whole dir as short term action but bumping the max length to make it more similar to PE tasks will help as well.

TL;DR as we take over and scope out what we have and how to improve the tasks, we're dealing with "this used to work with PE tasks, why doesn't it work now?"

[ripienaar]

Would you be ok with bumping that locally? I am mainly thinking bumping this will just make people not discover they are doing bad things

[LMacchi]

For both the gem and this module or just this module?

[ripienaar]

these files are never used in the gem afaik - so just the module

[LMacchi]

Interesting, I checked updating the gem and not the module but not the other way around. I'll give it a try and see how it goes and comment back for anyone running into the same issue.

[LMacchi]

Confirmed, just changing the maxlength in /opt/puppetlabs/mcollective/plugins/mcollective/agent/bolt_tasks.* is enough. I'm going to close the associated gem issue since it seems those files are not in use.

One last question: do you see the helpers pattern as an anti-pattern for tasks? Our helpers are all written to interact with the software that we have running in all servers (splunk) and are not used outside of tasks, that's why they ended up in that repo. Splunk and its API are pretty complex, so the helpers are all to allow users to write modular tasks instead of doing repetitive code (auth, passwd retrieval, data collection, etc).

As I mentioned in slack, this issue was driven by users without PE/Choria/Tasks knowledge going from "my task worked yesterday (with PE), code didn't change but now it doesn't work (with Choria)". I imagine some other people might run into this as well because PE has a 4kb (I think) limit, and helpers are a documented tasks pattern.

It feels to me like a good use of the tasks, but willing to start looking into alternatives if it's going to bring us issues in the future, so asking for your 2 cents.

[ripienaar]

The gem is essentially the source of truth for these files. The modules get them from there at release

[ripienaar]

I think the reason I say its a anti pattern is just down to how tasks are implemented, how they copy files, cache those files and how having a task with many many files is going to impact daily life with that task.

Like if it has a ton of files every invocation is going to be slow, if you have many machines and you run this task the overhead cost of syncing the task can impact normal catalog runs etc.

Dependencies and reuse is good, the way that collides with how tasks are built is not great.

So a happy medium might be to install the library dependencies once and then call them from tasks perhaps just ensuring you put only the files the dependency need (just the libs not the whole repo) in the task or something along those lines.