After thinking about it for a minute i realized you can already use some of the new and old pre_V6 features to get part what to what i was thinking about for a new colorization option.
Friday, December 2, 2022
Resynthesis Bansky Photo Mosaic
Subscribe to:
Post Comments (Atom)
4 comments:
I'd like you to take a look at this:
https://github.com/1ort/img2mosaic
Is it possible to load external Stable Diffusion libraries in the next version?
some good ideas for A.I.
https://publicprompts.art/
Just to be clear, the resynthesis technique i'm using is not stable diffusion. It's the new unified diffusion architecture.
The original resynthesis experiments i was doing before i switched to this approach were using a retrained stable diffusion model (retrained so it can take a CLIP image embedding as the attention input rather than the normal text embedding). If you look at those first examples you will see the results are interesting, but not at all like what you get with this new resynthesis approach.
This resynthesis approach is not at all the same as the image2image thing in stable diffusion, where you are feeding an image (via VAE encoder) into the U-net latent input while still running a text embedding into the attention input of the U-Net.
Since you mentioned text prompts i'm also going to point out that what i'm doing here with the resynthesis has nothing to do with text prompting. It's all based on image only input into the system. You can add additional text prompting if you want that in the unified approach, but as i keep pointing out here, just working with text prompting is super limiting, you are only able to cover a tiny fraction of the vast range the image synthesis system is capable of generating via more expressive modulation. There's my soapbox, ha,ha.
You could of course feed a series of source image blocks into any of the existing stable diffusion animation programs and get image2image generative output to then use in a construction of a cycle forward photo mosaic. Or use them any other way you can use an image folder brush. So you can certainly do a lot of what you want to do today.
Yes, it would be nice if the integration was easier. So we'll see...
Re: img2mosaic
I like the callout to GPT3 Codex as a contributor.
If you look at the low level way all of my code is written from raw bits in C++, you can certainly appreciate these newer approaches written in Python.
The extensibility with how Automatic1111 is architected is also pretty interesting.
So definitely food for thought.
Post a Comment