ARO (pathfinder): Pod logs are truncated at the head on large streams

[patricksimonian]

Is your feature request related to a problem? Please describe.

When trying to view pod logs, it appears if you aren't there to catch the start of the log stream, you will lose the start of the logs. This isn't a visual thing, even running oc logs is unable to catch the head of the log stream.

I'm not sure how/why this is happening but it makes debugging challenging if there is important info at the start of a log stream. Specifically I was having issues with Jenkins.

Work Around

If you access the pod logs quickly you will be able to catch the start of the log stream and its preserved.

[stewartshea]

There is a finite amount of log lines that are available (up to 1000 lines) in the console. While the EFK logging stack is a a bit heavy for the size of our clusters, I'm experimenting with a single node (non-HA) EFK stack to balance resource cost with the ability to have some longer persisted logs. This is currently being evaluated in the platform-services lab cluster and then will be deployed in the pathfinder cluster when appropriate.