Use of ByteBuffer in Bufferable

[johnynek]

Bufferable is a somewhat under used typeclass, but with macros, this could change. The idea was a composable binary serialization type that could be used in conjunction with Injections and Bijections which are often related.

The problem is ByteBuffer is mutable, and working with it is highly error prone. As a result, we do a lot of defensive copying.

We could instead have something like:

final case class Buffer(bytes: Array[Byte], position: Int, size: Int) {
  def put(i: Int): Try[Buffer]
  def getInt: Try[(Buffer, Int)]
  // do all the puts/gets using the fastest java methods
  def reset: Buffer
  def toByteBuffer: ByteBuffer
}

In this case, the underlying bytes array is clearly still mutable, but the pointer information around it is not. Maybe this is more confusing since there is the illusion of immutability but really the underlying object is mutable. If we come to this conclusion, perhaps we should change Bufferable from returning new ByteBuffers and have an explicit contract that it will mutate. If you don't want that, you call with buf.duplicate.

[mosesn]

Check out the com.twitter.io.Buf abstraction maybe? It needs a dependency
on util, but you can just put it in bijection util?

On Wed, Dec 31, 2014, 20:30 P. Oscar Boykin [email protected]
wrote:

Bufferable is a somewhat under used typeclass, but with macros, this could
change. The idea was a composable binary serialization type that could be
used in conjunction with Injections and Bijections which are often related.

The problem is ByteBuffer is mutable, and working with it is highly error
prone. As a result, we do a lot of defensive copying.

We could instead have something like:

final case class Buffer(bytes: Array[Byte], position: Int, size: Int) {
def put(i: Int): Try[Buffer]
def getInt: Try[(Buffer, Int)]
// do all the puts/gets using the fastest java methods
def reset: Buffer
def toByteBuffer: ByteBuffer
}

In this case, the underlying bytes array is clearly still mutable, but the
pointer information around it is not. Maybe this is more confusing since
there is the illusion of immutability but really the underlying object is
mutable. If we come to this conclusion, perhaps we should change Bufferable
from returning new ByteBuffers and have an explicit contract that it will
mutate. If you don't want that, you call with buf.duplicate.

—
Reply to this email directly or view it on GitHub
#192.

[johnynek]

Buf is not bad. Honestly, since we don't have anything like the ST monad here, I tend to think we should just own up to the fact that we are mutating the ByteBuffers/Arrays underlying any of these Buffers.

Also, our goal is to use this in scalding, and I'm pretty reluctant to invite util into a diamond dependency with scalding.

It's sad that the JVM module/dependency situation is what it is.