Fix helm on eks

[xieandrew]

Currently, there are issues with the helm chart in kubernetes/helm when running on EKS (some tests from python/tests fail when run using --profile=kubernetes). This mainly had to do with connection issues with ice-rest-catalog and authentication.

Fixes the helm chart when using the default values.yaml for EKS to pass all tests in python/tests. Improves documentation for using the helm chart on EKS.

[hodgesrm]

This looks good to me with a cosmetic change to remove commented-out fields.

Question 1: how does Helm match the manifest directory contents now?

[hodgesrm]

This can wait for a later PR but we should use secrets rather than expose passwords. See https://altinity.com/blog/clickhouse-confidential-using-kubernetes-secrets-with-the-altinity-operator#using-secrets-to-protect-clickhouse-user-passwords for more information.

[xieandrew]

Will look into using secrets. But this section was added so the test_vector_sever_version auth header here works. If the test is changed to not use the auth header, this root password is not needed. Not sure if that's an issue with the test or if we do need a root password specified, because I don't see this password in the docker compose setup.

[xieandrew]

The helm chart is almost identical to the manifest directory when values.yaml is applied, I believe the swarm in helm has a shardsCount of 1 instead of 4 though.