You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
On row format V2, TableCodec.decodeObjects returns a String (with silent
U+FFFD replacement) for BINARY/VARBINARY columns, instead of byte[].
Any non-UTF-8 byte value (e.g. 0xFF) is irreversibly corrupted. The V1 path
returns byte[] correctly for the same column, so V1 and V2 disagree.
Root cause:
DataTypeFactory.convertType remaps TypeString → TypeBlob for
binary-charset string columns — but this only selects the DataType
subclass (BytesType); it does not change the instance's type field.
DataType(InternalTypeHolder holder) re-derives the field from the raw
code: this.tp = MySQLType.fromTypeCode(holder.getTp()) (DataType.java),
so the created instance is of class BytesType but carries getType() == TypeString for a BINARY column.
RowDecoderV2.decodeCol switches on tp.getType() (the field, not the
class) → falls into case TypeString: → new String(colData, UTF_8)
(RowDecoderV2.java:54-57), silently replacing invalid sequences.
The V1 path (TableCodecV1) dispatches polymorphically on the class
(DataType.decode(...) → BytesType.decodeNotNull → byte[]), hence the
V1/V2 divergence.
convertType exists so the class handles binary content, but the V2 switch
ignores the class entirely: BINARY (field=TypeString) and VARBINARY
(field=TypeVarchar) both land in the string branch. BLOB family is unaffected
(field stays a blob code → case TypeBlob: return colData).
2. Minimal reproduce step (Required)
-- TiDB, row format V2 (default for new installs)CREATETABLEt (id INTPRIMARY KEY, c BINARY(1));
INSERT INTO t VALUES (1, 0xFF);
// value = TableCodec-encoded row value bytes (e.g. from a CDC event / TiKV read),// handle = row handle, tableInfo = session.getCatalog().getTable("db", "t")Object[] vals = TableCodec.decodeObjects(value, handle, tableInfo);
System.out.println(vals[1].getClass()); // class java.lang.StringSystem.out.println(vals[1]); // "\uFFFD" (0xEFBFBD) — original 0xFF lost
3. What did you see instead (Required)
The BINARY(1) column value is a java.lang.String containing U+FFFD
(EF BF BD). The original byte 0xFF is irreversibly lost.
4. What did you expect to see? (Required)
byte[] { (byte) 0xFF } — raw column bytes, as the V1 path
(BytesType.decodeNotNull) already returns for the same column.
5. What are your Java Client and TiKV versions? (Required)
Client Java: 3.2.0, 3.3.5 and master (all affected — RowDecoderV2.java:54-57
identical in all three; both files last touched 2021-12-30)
TiKV: TiKV 8.5.x (observed via TiDB 8.5.6, tidb_row_format_version=2)
6. Suggested fix
Check charset before the string branch in RowDecoderV2.decodeCol:
caseTypeString:
caseTypeVarString:
caseTypeVarchar:
if (Charset.CharsetBin.equals(tp.getCharset())) {
returncolData;
}
returnnewString(colData, StandardCharsets.UTF_8);
Activity
ti-chi-bot commentedon Nov 3, 2026
Welcome @juefengyi! It looks like this is your first issue to tikv/client-java 🎉