[Modular] Add Custom Blocks guide to doc #12339
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
looking great! thanks @DN6
Does not need to be in this PR but relevant on the topic:
for the remote custom code, I think maybe it's easier to work without the cache system. e.g. if I want to make my own version of florence node, I would probably want to download the folder locally first (with local_dir_use_symlinks=False) so I can go in and edit the code
let me know what you think about what should be the best process for this?
| --> | ||
|
|
||
|
|
||
| # Building Custom Blocks |
There was a problem hiding this comment.
| # Building Custom Blocks | |
| # Building custom blocks |
|
|
||
| Modular Diffusers allows you to create custom blocks that can be plugged into Modular Pipelines. This guide will show you how to create and use a custom block. | ||
|
|
||
| First let's take a look at the structure of our custom block project: |
There was a problem hiding this comment.
| First let's take a look at the structure of our custom block project: | |
| The project should be structured as shown below. The custom block code is contained in `block.py` and the `modular_config.json` file contains metadata for loading the block. |
| The code to define the custom block lives in a file called `block.py`. The `modular_config.json` file contains metadata for loading the block with Modular Diffusers. | ||
|
|
||
| In this example, we will create a custom block that uses the Florence 2 model to process an input image and generate a mask for inpainting |
There was a problem hiding this comment.
| The code to define the custom block lives in a file called `block.py`. The `modular_config.json` file contains metadata for loading the block with Modular Diffusers. | |
| In this example, we will create a custom block that uses the Florence 2 model to process an input image and generate a mask for inpainting | |
| This example creates a custom block that uses the [Florence 2](https://huggingface.co/docs/transformers/model_doc/florence2) model to process an input image and generate a mask for inpainting. |
|
|
||
| In this example, we will create a custom block that uses the Florence 2 model to process an input image and generate a mask for inpainting | ||
|
|
||
| ```py |
There was a problem hiding this comment.
It'd be nice to break up this big code example into several smaller chunks that explain whats happening. Maybe break it up into:
- inputs
- outputs
- computation logic
which also nicely mirrors https://huggingface.co/docs/diffusers/main/en/modular_diffusers/pipeline_block#inputs-and-outputs
| Now that we have defined our custom block, we can save it as a model repository on the Huggingface Hub so that it is easy to share and reuse. | ||
|
|
||
| There are two ways to save the block: |
There was a problem hiding this comment.
| Now that we have defined our custom block, we can save it as a model repository on the Huggingface Hub so that it is easy to share and reuse. | |
| There are two ways to save the block: | |
| Save the custom block to the Hub, from either the CLI or with the [`push_to_hub`] method, so you can easily share and reuse it. |
| 1. From the CLI | ||
|
|
||
| ```shell | ||
| # In the folder with the `block.py` file, run: | ||
| diffusers-cli custom_block | ||
| ``` | ||
|
|
||
| Then upload the block to the Hub: | ||
|
|
||
| ```shell | ||
| hf upload <your repo id> . . | ||
| ``` | ||
|
|
||
| 2. From Python | ||
|
|
||
| ```py | ||
| from block import Florence2ImageAnnotatorBlock | ||
| block = Florence2ImageAnnotatorBlock() | ||
| block.push_to_hub("<your repo id>") | ||
| ``` |
There was a problem hiding this comment.
Maybe wrap this in <hfoptions>:
<hfoptions id="share">
<hfoption id="hf CLI">
...
</hfoption>
<hfoption id="push_to_hub">
...
</hfoption>
</hfoptions>
@yiyixuxu this is a good point. I can add a section for this. |
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
What does this PR do?
Simple guide for creating custom blocks. Will try to add some more advanced examples in a follow up.
Fixes # (issue)
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.