Optimizer code deduplication and kwargs exploitation

[pavelkomarov]

Once all the modules are moved to the new interface, improvements to code that makes use of that interface can commence. I've done some poking around and am seeing a lot of duplicated code, which I'll do my best to deduplicate.

[pavelkomarov]

#124 began to address this. More to be done.

[pavelkomarov]

The optimization happening in this code right now is scipy.minimize over numeric parameters only. Is this what we want? There are also boolean parameters (like whether a method is using a sliding window) and string parameters (like the kind of kernel to use), heretofore called "options". In reworking the code, I've made some effort to flatten the distinction between numeric parameters and categorical ones, because they're all ultimately just choices. Should we use an optimization method that can handle this more complicated search space, like Bayesian optimization? If we want to automate best-method finding, then the choice of method itself becomes a categorical choice in the optimization, and a method that can natively handle this may be more efficient at querying the search space. Or should we try to use a numerical minimization scheme over numerical parameters where possible?

[pavelkomarov]

I've been understanding the inclination to use a list today. Because scipy.optimize.minimize requires all parameters to be continuous numbers, it doesn't handle bools, ints, and especially not strings. Indeed, encoding and decoding from a parameter dict mapping to a "point" scipy can use is pretty annoying. But it's ultimately just encoding and decoding, and better it be done here, deep in the bones of the code, than that the user have to encode every time they ever have to call things. I've managed to get some basic encoding/decoding working. It doesn't handle strings yet, but it can handle everything else, which is more than before. Strings are enough of a pain that including them may have to be its own issue.

[pavelkomarov]

Honestly, an alternative like scikit-optimize might be the way to go ultimately, since there are more sophisticated ways to solve black-box optimization problems than numerical-walking-downhill-from-a-bunch-of-starting-points, but #125 at least brings the optimizer code up to speed with the rest of the code base.