Update ChanReader / ChanWriter for use inside toxics

[xthexder]

While working on the bidirectional toxics PRs I discovered some problems with how ChanReader worked when used ephemerally inside a toxic.
These turned out to be non-trivial to fix, with the end result being this PR.

The problem with ChanReader is that in some cases it may buffer data. This means if you ever stop using the reader and switch to another, there's a possibility of losing data.
Further problems arise when ChanReader is read using other readers such as bufio.Reader as required by functions such as http.ReadRequest().

These problems are solved in 3 ways:

1. Use a non-ephemeral reader object on the ToxicStub instead of creating it every time Pipe() is called.
2. Create a TransactionalReader that can be rolled back to specific points so that bufio.Reader can effectively be un-read.
3. Allow all buffers inside the reader to be explicitly flushed at any point so that no data is lost.

Example TransactionalReader usage:

// Input is "hello world"
txnReader := stream.NewTransactionalReader(input)

{ // This section is ephemeral (and would be similar to usage in `Pipe()`)
    bufReader := bufio.NewReader(txnReader)
    msg, _ := bufReader.ReadString(' ')
    // msg == "hello "
    // bufReader has "world" buffered (bufReader.Buffered() == 5)

    // Mark everything up to "hello " as read.
    txnReader.Checkpoint(-bufReader.Buffered())
    // Rollback txnReader to the checkpoint, we never read "world"
    txnReader.Rollback()
    // bufReader can now be thrown away
}

// Any further reads from txnReader will start from the checkpoint
buf := make([]byte, 5)
txnReader.Read(buf)
// buf == "world"

Actual usage can be found in my WIP redis toxic here: https://github.com/Shopify/toxiproxy/blob/redis-wip/toxics/redis.go

The documentation for this is pretty tricky to write without actual code examples, so I'll likely leave the detailed docs until we have at least 1 protocol-aware toxic to point at.

@sirupsen @jpittis / anyone else interested in Golang

[xthexder]

Went through this again and cleaned some things up now that it's not fresh in my mind.
Should be good for a review @sirupsen

[jpittis]

Why Checkpoint(0)? Should it not be Checkpoint(-stub.Reader.Buffered())?

[xthexder]

The example where I used .Buffered() was for a bufio.Reader attached to the TransactionalReader.
In this case, the amount of buffered data is always 0. Since everything read is being used, we can set the checkpoint to the last read byte, rather than the (last read) - (amount buffered by bufio.Reader)

[jpittis]

Did you choose 32K sized buffer for a specific reason?

[xthexder]

I chose 32K because it's the size used by io.Copy(), however bytes.Buffer will grow the size of the buffer as necessary.
This is just a nice default size so in most cases it doesn't need to be reallocated.

[jpittis]

Initializing n to current seems to be pointless here? Why not var n int?

[xthexder]

You're right, looks like this is redundant. I'll see if I can refactor this function and make it a little simpler.

[xthexder]

Actually found another bug while looking at this. if bufReader != nil, t.buffer.Reset() shouldn't be called unless the offset is big enough.
I fixed this, and added a test for it.

[jpittis]

I'm trying to get my head around this buffered stuff so we can get this out the door too.Do say if you want to ship this yourself @xthexder. These PR comments are for my own understanding so that I can keep reviewing later.

[jpittis]

I don't understand this Interupt monkey business.

[jpittis]

So current represents how much of the buffer we've already read.

• 0 will say... okay we've read everything! Lock it in!
• -5 will say we've read everything except for the last 5 bytes. Let's start reading again from the start of those last 5.
• 5 will say we successfully read 5 bytes but not the rest. Let's start reading from the rest.
• t.buffer.Len() will do the same thing is 0.

Okay I found this confusing but LGTM

[jpittis]

Is this correct? Why are we trying to flush the reader? It's not buffered is it? I'll look at this again later and probably understand right away.