error: (segment): index 3/4 is out of bounds for axis 0 with size 3

[mrtigermeat]

Hello!

I'm very interested in training a custom HFA model, and this project is super interesting! I am curious about this error I am getting that appears to cause issues with training.

Upon trying again with loguru's @logger.catch() decorator, it appears to be an issue with a tensor. I'm not a torch expert, so I'm not sure what the issue is. The data being utilized has been verified accurate, and is the proper format (DiffSinger training data).

It only happens to some segments, not all of them.

Please let me know if I'm looking over anything!

[wolfgitpr]

I guess it's because in the nll model configuration, 'None' occupies index 0, and AP and EP are indices 1 and 2, so the default size is set to 3. I'm not sure if you have modified this content. The complete absence of self-added phonemes in the dataset leads to this error.
If the issue cannot be identified through troubleshooting, you can try changing cvnt_arg: mask_ratio: 0.3 to 0.0 in the nll configuration. The recognition performance may slightly decrease, but this can avoid the issue (provided that the training results correctly recognize most of the targets)

[mrtigermeat]

I included glottal stops (q/GS) and vocal fry (vf) in the nll configuration, and confirmed they're present in the vocab of the model. Changing cvnt_arg: mask_ratio to 0.0 did not fix the issue and I received the same error when binarizing.

Here is a copy of my configuration file. I believe the only things I changed were the dataset config paths, the non-lexical phonemes included and the cvnt_arg you suggested. Binarizing works fine for the forced alignment model.

model_name: nll_exp_01

float32_matmul_precision: high
random_seed: 123456

# Phonemes that have no lexical meaning, do not require annotation, and are automatically recognized,
# such as AP (breath), EP (tail breath), etc
non_lexical_phonemes:
  - AP
  - EP
  - q
  - vf

datasets_config_paths: # datasets
  - configs/datasets_config_en.yaml
  - configs/datasets_config_en_eval.yaml
  - configs/datasets_config_ja.yaml
  - configs/datasets_config_ja_eval.yaml
  - configs/datasets_config_fr.yaml

binary_folder: data/binary_nll # Preprocessing Results Folder
valid_set_size: 20 # When valid_set_Size is greater than 0, randomly select by quantity; otherwise, use the validation set in the dataset
max_length: 45 # Maximum duration of a single audio stream

# It is not recommended to enable the number of preprocessing threads and the size of the temporary queue if the data volume is too small (within 5 hours).
# Every 6-8g of memory and 6g of video memory can open one worker.
multiprocess_works: 0
multiprocess_max_size: 200  # size * num_works. Too large to ensure sufficient memory
multiprocess_start_size: 100  # Minimum startup data volume

# Number of training data loading processes
dataloader_workers: 2
dataloader_prefetch_factor: 2
batch_max_length: 200
binning_length: 1000
drop_last: False

num_valid_plots: 20 # Maximum drawing quantity for val and evaluate items
draw_evaluate: True

augmentation_args: # Code defect, prohibited from use
  enabled: false
  random_pitch_shifting:
    range: [ -5., 5. ]
    num: 0
  blank_padding:
    range: [ 0, 5 ] # seconds
    num: 2

cvnt_arg:
  mask_ratio: 0.0
  encoder_conform_attention_drop: 0.05
  num_layers: 4

  encoder_conform_dim: 128
  encoder_conform_ffn_latent_drop: 0.05
  encoder_conform_ffn_out_drop: 0.05
  encoder_conform_kernel_size: 31

optimizer_config:
  lr: 0.0003
  gamma: 0.9999
  total_steps: 10000
  muon_args:
    weight_decay: 0.1
  adamw_args:
    weight_decay: 0.0

loss_config:
  losses:
    weights: [ 0.5, 0.3, 0.2, 1.0 ]

# trainer
accelerator: auto
devices: auto # num_devices
precision: bf16-mixed # bf16-mixed , 32-true
gradient_clip_val: 0.5
gradient_clip_algorithm: norm
val_check_interval: 1000

save_top_k: 5
save_every_steps: 1000

# Normally not changed
hubert_config:
  encoder: cn_hubert
  model_path: dependencies/chinese-hubert-base
  sample_rate: 16000
  hop_size: 320
  channel: 768

mel_spec_config:
  n_mels: 128
  sample_rate: 44100
  window_size: 882
  hop_size: 441
  n_fft: 1764
  f_min: 40
  f_max: 16000
  clamp: 0.00001

[wolfgitpr]

Please try the fix-nll-binarize branch, with the default mask_ratio of 0.3. For preliminary testing, you can perform preprocessing and training. Please evaluate the phoneme recognition performance beyond AP on your own

[mrtigermeat]

I will test the fork and let you know when I am able, thank you!