Questions regarding the evaluation process and requests for relevant data files

[Qianhao-Ren]

Dear project maintainer,

Thank you for your valuable contribution and for making your dataset publicly available. I am currently utilizing your dataset for my research on dialectal translation and would be very grateful if you could provide some clarification regarding the evaluation process.

I am keen to understand the specifics of how you conducted the evaluation using the provided data. Could you please share the evaluation script, or alternatively, provide a detailed description of the evaluation methodology? This would be immensely helpful for me to better comprehend your work and ensure that my own models are evaluated under the same conditions.

Furthermore, as the dataset pertains to dialect conversion, the evaluation necessitates access to the corresponding database files for both the source and target dialects. To facilitate my research, would it be possible for you to provide the database file for the source dialect as well as the database file for the target dialect?

Thank you for your time and consideration. I look forward to your response.

Sincerely,
Qianhao Ren

[sgdgp]

Thanks for raising this question. I am also interested in the benchmark and its evaluation methodology. It would be great if the benchmark authors could explain evaluation setup with source and target files for tables reported in the paper.

[Beliefuture]

Hi @Qianhao-Ren and @sgdgp ! Thanks for your interests, and sorry for the late response. I have just seen this issue while preparing our latest version for the leaderboard.

As mentioned in the paper, there exist two metrics for the evaluations:

• (1) Dialect Compatibility ($Acc_{EX}$): The ratio of the translated queries that are executable (i.e.,
  syntactically correct) in the target database without raising an incompatibility error (e.g., incorrect data
  types or functions);
• (2) Result Consistency ($Acc_{RES}$): The ratio of the translated queries that return the strictly identical
  results (i.e., semantically consistent) in the target database as the source queries in the source database,
  including the returned data format, precision, and displayed order.

For $Acc_{EX}$, it can be directly assessed by the separate ANTLR dialect-specific parsers (e.g., the ones provided in https://github.com/weAIDB/PARROT/tree/main/validator) or the parser in SQLGlot with a snippet such as sqlglot.parse_one(sql read="oracle").

For $Acc_{RES}$, you should set up the corresponding database, whose database files are provided in the original benchmark, and additional efforts are required for the evaluation (i.e., using commercial tools such as Navicat to convert the data files, and accommodate the corresponding databases).

Finally, you can also refer to another repository of our work, which proposes an LLM-based dialect translation system (https://github.com/weAIDB/CrackSQL).

@article{zhou2025cracksql,
  author       = {Wei Zhou and
                  Yuyang Gao and
                  Xuanhe Zhou and
                  Guoliang Li},
  title        = {{Cracking SQL Barriers:} {An}  LLM-based Dialect Transaltion System},
  journal      = {Proc. {ACM} Manag. Data},
  volume       = {3},
  number       = {3 (SIGMOD)},
  year         = {2025}
}

[sgdgp]

Hi @Beliefuture Thanks for the response.

• Since the results for ACC_Ex and ACC_Res are already calculated in both here and the CrackSQL paper, could you please provide the exact scripts (and setup) to reproduce the results?
• Additionally, the huggingface link (https://huggingface.co/datasets/weizhoudb/PARROT) only mentions a test split with 28k rows. How to access the eval splits of 598 pairs (referred in the paper) and also the 5k subset of the data?

[sgdgp]

Hi @Beliefuture , just wanted to follow up on the above query. Thanks in advance!