Describe the enhancement requested
Apache Arrow currently calculates chunk sizes based on the assumption that Netty's PooledByteBufAllocatorL uses a 16MB page size. If size is blow 16MB, will use nextPowerOfTwo of requestSize.
public long getRoundedSize(long requestSize) {
return requestSize < chunkSize ? CommonUtil.nextPowerOfTwo(requestSize) : requestSize;
}
However, Netty's default maximum buffer size has been reduced from 16MB to 4MB in recent versions (see netty/netty#12108). This mismatch leads to memory inefficiency in scenarios where chunk sizes fall between 4MB and 16MB.
Describe the enhancement requested
Apache Arrow currently calculates chunk sizes based on the assumption that Netty's PooledByteBufAllocatorL uses a 16MB page size. If size is blow 16MB, will use nextPowerOfTwo of requestSize.
However, Netty's default maximum buffer size has been reduced from 16MB to 4MB in recent versions (see netty/netty#12108). This mismatch leads to memory inefficiency in scenarios where chunk sizes fall between 4MB and 16MB.