test: add tokenizer regression test for granite-4.0-micro#948

kndtran wants to merge 2 commits intogenerative-computing:maingenerative-computing/mellea:mainfrom

kndtran:test/tokenizer-regression-947kndtran/mellea:test/tokenizer-regression-947Copy head branch name to clipboard