Adding support for Bria's FIBO model

[kfirgoldberg]

Adding support for Bria's FIBO model, including training (with low VRAM flags) and inference.

[jaretburkett]

Sorry it took so long, I am talking a look at this today.

[jaretburkett]

I am running into a few potential challenges with this model since it is unique in the face that it is JSON-native and trained on long structured captions. Which is awesome, but AI Toolkit is not really setup to handle that currently. I am curious how you guys recommend handling this.

1. The sample captions in AI toolkit are designed to be single line and short. The default sample prompts alone do not produce good results because of this. I am curious if there is a low information json template we could inject the string prompts into that are not detected to be json, to make the model function the best it can with minimum information for sampling?
2. There is a similar issue with captions on the dataset. Many users when training in a new concept or character, do a simple trigger word, or very short descriptions of the images. Is there a bare minimum json template we could apply to simple captions to keep them from breaking the model when training. I know long json captions would be ideal, but as a bare minimum workaround, this could be helpful?
3. What is the best method to caption a dataset to get this extremely descriptive json caption?
4. For the unconditional, is that json formatted as well, or just a blank prompt?

[kfirgoldberg]

I am running into a few potential challenges with this model since it is unique in the face that it is JSON-native and trained on long structured captions. Which is awesome, but AI Toolkit is not really setup to handle that currently. I am curious how you guys recommend handling this.

1. The sample captions in AI toolkit are designed to be single line and short. The default sample prompts alone do not produce good results because of this. I am curious if there is a low information json template we could inject the string prompts into that are not detected to be json, to make the model function the best it can with minimum information for sampling?
2. There is a similar issue with captions on the dataset. Many users when training in a new concept or character, do a simple trigger word, or very short descriptions of the images. Is there a bare minimum json template we could apply to simple captions to keep them from breaking the model when training. I know long json captions would be ideal, but as a bare minimum workaround, this could be helpful?
3. What is the best method to caption a dataset to get this extremely descriptive json caption?
4. For the unconditional, is that json formatted as well, or just a blank prompt?

A general note on FIBO - it's trained on long, structured JSON prompts and this is what gives the model its strong disentanglement and controllability.

1. As part of the FIBO open source release we've made a small example dataset for finetuning available here, I can adapt it to this repo's format and add it if you'd like

2+3. To generate captions for a dataset we suggest using either Gemini or a local VLM we've finetuned for the FIBO JSON schema, and in order to support "trigger words" you can just add them to the short description field in the JSON, like the examples from the FIBO repo I linked to which all start with A charming, cartoon-style brown bear

4. This is the recommended negative prompt:

def get_default_negative_prompt(existing_json: dict) -> str:
    negative_prompt = ""
    style_medium = existing_json.get("style_medium", "").lower()
    if style_medium in ["photograph", "photography", "photo"]:
        negative_prompt = """{'style_medium':'digital illustration','artistic_style':'non-realistic'}"""
    return negative_prompt

so it's possible to use short, unstructured prompts or JSONs with only some of the fields