CLIP Integrator Dazzler

 

Roundabout resynthesis approach using CLIP Integrator to go from an image input to a long algorithmically generated text prompt that then gets input to stable diffusion 2.0 to generate resynthesized generative image output.  Not as good as the more straightforward direct input image embedding resynthesis i normally work with, but better than i thought it would be for the particular input image i used for the experiment.

ochun's paired energies

 

Paired energy in this case being ochun's positive vs negative latent space attractor.

natural form multidiffusion 2

 

natural form multidiffusion

 

control net resynthesis experiment 2

 

Resynthesis of the last 'society of chaos 3' image using the CLIP embedding as the driver of the resynthesis shown in a few examples above.   A few different ControlNet resynthesis examples shown below using different feature vector options as the driver of the resynthesis (no text prompting involved).  HED (last one below) does the best, but it still lacks what is cool about the above images.

the terror from beyond space multidiffusion

 

control net resynthesis experiment

 

Abstract MSG - vectorizer hybrid procedural image above.  Used that as the only input to a StableDiffusion ControlNet model with different feature control guides derived from above (output shown below). So no text prompting, just the different feature vector options driving the ControlNet SD synthesis.  I was curious how the different feature vector control nets option might work for pure abstract image resynthesis.  As compared to the more typical CLIP image latent embedding used as a resynthesis guide, which in general works really well for this kind of thing.

society of chaos multidiffusion 3

 

society of chaos multidiffusion 2

 

society of chaos multidiffusion

 

new wave multidiffusion

 

north shore multidiffusion

Experimenting with the new MultiDiffusion SD algorithm.

excavation

 

waikiki beach synthesis

 

Synthesis (via stable diffusion) to contrast the resynthesis examples. Difference between above and below is associated with a lot of negative text prompting in the examples below.

climb

 

Experimenting with the SD 2.1 model with an additional CLIP guidance integration.  The hallucinated watermark issue is a big problem with this.  Really needs some kind of watermark suppression, maybe associated with the additional CLIP modulation?  Latent Majesty Diffusion way back in the dark ages of the summer of 22 used a modified CLIP aesthetic model that was specifically trained for watermark suppression and was very good at not generating them.

the masque of youth

 

underwater ballet

 

All of these are using the quantized keras stable diffusion code on a mac.  For the second run below i ran 2 different experiments, the first always working off of the first cycle into the U-Net latent for the subsequent frames, the second feeds back recursively into the U-Net latent.  If you squint and look at the frames in the first animation you can see the spatial position modulation and associated spatial anchoring derived off of that first frame.  The second one is always self-evolving through the different feedback cycles.

woman dancing -part 2

 

In part 2 of this 'woman dancing' series, we break away from the notion of viewing the system as a video processor, and think about it as a pure generative image synthesis system. Again, all of the posts in this series are using the exact same static text prompting.  Above starts with a random initialization of the u-net input, then recursively feeds back the last output as the new input.

Above is using a video generated with CLIP guided RGB optimization (run with a different prompt) as the input to the u-net.  The source video is pretty noisy and abstract, so not representational of a woman dancing in any way.

Above starts with a random initialization of the u-net input, then recursively feeds back a locally adaptive affine transformation of the last output as the new input.