Skip to content

[fix](timestamptz) Report TIMESTAMPTZ as string to MySQL clients#63292

Open
zclllyybb wants to merge 1 commit into
apache:masterfrom
zclllyybb:codex-doris-25709-repro
Open

[fix](timestamptz) Report TIMESTAMPTZ as string to MySQL clients#63292
zclllyybb wants to merge 1 commit into
apache:masterfrom
zclllyybb:codex-doris-25709-repro

Conversation

@zclllyybb
Copy link
Copy Markdown
Contributor

TIMESTAMPTZ result rows are serialized as timezone-aware strings in the MySQL binary protocol. FE metadata used to advertise them as MYSQL_TYPE_DATETIME, so Connector/J decoded the length-encoded string bytes as a MySQL datetime binary payload and failed with Invalid length (32) for type TIMESTAMP when ResultSet.getString read a server-prepared result.

Change PrimitiveType.toMysqlType() to return MYSQL_TYPE_STRING for TIMESTAMPTZ, adjust the field packet length and decimals to string semantics, and make the FE local binary result path write TIMESTAMPTZ through the string fallback instead of the datetime binary layout. BE serialization already uses push_timestamptz -> push_string, so no BE payload change is needed.

Add a field-packet unit test and a regression suite that forces ServerPreparedStatement and compares direct ResultSet.getString(ts) with CAST(ts AS STRING).

TIMESTAMPTZ result rows are serialized as timezone-aware strings in the MySQL binary protocol. FE metadata used to advertise them as MYSQL_TYPE_DATETIME, so Connector/J decoded the length-encoded string bytes as a MySQL datetime binary payload and failed with Invalid length (32) for type TIMESTAMP when ResultSet.getString read a server-prepared result.

Change PrimitiveType.toMysqlType() to return MYSQL_TYPE_STRING for TIMESTAMPTZ, adjust the field packet length and decimals to string semantics, and make the FE local binary result path write TIMESTAMPTZ through the string fallback instead of the datetime binary layout. BE serialization already uses push_timestamptz -> push_string, so no BE payload change is needed.

Add a field-packet unit test and a regression suite that forces ServerPreparedStatement and compares direct ResultSet.getString(ts) with CAST(ts AS STRING).

Verification:

- ./run-fe-ut.sh --run org.apache.doris.mysql.MysqlSerializerVarbinaryTest

- ./run-regression-test.sh --run -d datatype_p0/timestamptz -s test_timestamptz_jdbc_binary_protocol
@hello-stephen
Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zclllyybb
Copy link
Copy Markdown
Contributor Author

/review

@zclllyybb
Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-H: Total hot run time: 31294 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ecac99691a8c6cf44c8227369eceae6afb3cabb7, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17678	3935	3869	3869
q2	q3	10773	1393	805	805
q4	4684	467	349	349
q5	7550	2250	2105	2105
q6	233	174	141	141
q7	929	791	626	626
q8	9355	1675	1611	1611
q9	5152	4911	4871	4871
q10	6405	2057	1768	1768
q11	456	279	242	242
q12	639	425	302	302
q13	18111	3431	2761	2761
q14	271	253	232	232
q15	q16	825	778	707	707
q17	907	1012	861	861
q18	6970	5884	5546	5546
q19	1550	1282	1090	1090
q20	525	406	410	406
q21	6183	2823	2629	2629
q22	450	373	451	373
Total cold run time: 99646 ms
Total hot run time: 31294 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4749	4541	4493	4493
q2	q3	4893	5252	4616	4616
q4	2169	2215	1404	1404
q5	5013	4678	4703	4678
q6	240	178	144	144
q7	1929	1712	1581	1581
q8	2533	2128	2118	2118
q9	7709	7388	7292	7292
q10	4495	4385	3961	3961
q11	546	394	352	352
q12	705	713	517	517
q13	3016	3382	2802	2802
q14	276	271	252	252
q15	q16	672	710	616	616
q17	1288	1270	1264	1264
q18	7364	6858	6728	6728
q19	1108	1107	1081	1081
q20	2217	2214	1911	1911
q21	5359	4748	4605	4605
q22	522	469	421	421
Total cold run time: 56803 ms
Total hot run time: 50836 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-DS: Total hot run time: 169528 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ecac99691a8c6cf44c8227369eceae6afb3cabb7, data reload: false

query5	4347	659	526	526
query6	328	214	196	196
query7	4240	547	315	315
query8	336	236	217	217
query9	8851	4085	4053	4053
query10	447	338	305	305
query11	5807	2348	2199	2199
query12	188	128	126	126
query13	1285	653	433	433
query14	5973	5380	5047	5047
query14_1	4383	4375	4353	4353
query15	215	207	183	183
query16	1041	470	451	451
query17	1174	750	609	609
query18	2738	508	366	366
query19	234	216	174	174
query20	140	135	132	132
query21	223	140	120	120
query22	13579	13591	13331	13331
query23	17316	16466	16009	16009
query23_1	16257	16253	16155	16155
query24	7462	1783	1304	1304
query24_1	1303	1316	1299	1299
query25	592	504	448	448
query26	1322	330	174	174
query27	2680	593	353	353
query28	4457	1978	1974	1974
query29	1002	654	524	524
query30	309	246	202	202
query31	1117	1077	960	960
query32	89	79	81	79
query33	557	368	311	311
query34	1175	1179	654	654
query35	754	783	660	660
query36	1346	1344	1200	1200
query37	153	106	97	97
query38	3224	3157	3048	3048
query39	934	919	911	911
query39_1	887	877	865	865
query40	227	146	128	128
query41	68	63	62	62
query42	111	115	110	110
query43	324	327	277	277
query44	
query45	211	207	200	200
query46	1067	1190	714	714
query47	2323	2380	2194	2194
query48	396	405	295	295
query49	635	483	379	379
query50	959	346	252	252
query51	4325	4212	4221	4212
query52	112	107	97	97
query53	265	279	209	209
query54	334	294	258	258
query55	94	96	91	91
query56	305	308	314	308
query57	1431	1406	1326	1326
query58	301	270	277	270
query59	1554	1631	1407	1407
query60	312	314	315	314
query61	163	158	161	158
query62	670	644	562	562
query63	254	203	210	203
query64	2389	802	656	656
query65	
query66	1663	464	383	383
query67	30062	29891	29877	29877
query68	
query69	457	346	301	301
query70	992	985	995	985
query71	303	283	267	267
query72	3034	2774	2435	2435
query73	870	769	441	441
query74	5059	4902	4729	4729
query75	2692	2608	2247	2247
query76	2308	1115	756	756
query77	409	411	330	330
query78	12180	12091	11611	11611
query79	1484	1067	715	715
query80	662	558	467	467
query81	464	281	249	249
query82	1355	159	122	122
query83	348	280	245	245
query84	259	139	115	115
query85	890	540	459	459
query86	400	362	327	327
query87	3408	3388	3188	3188
query88	3542	2694	2667	2667
query89	443	390	333	333
query90	1987	182	183	182
query91	184	167	141	141
query92	77	76	70	70
query93	1463	1471	941	941
query94	516	352	316	316
query95	707	490	368	368
query96	1039	812	321	321
query97	2726	2669	2563	2563
query98	234	232	235	232
query99	1131	1093	1002	1002
Total cold run time: 253580 ms
Total hot run time: 169528 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 2.94% (1/34) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants