Running some injection modulation tests in the Studio Artist V5.5.4 paint synthesizer.
Start by making a Transition Context from a hand picked set of source images. Build a paint action sequence for the paint style you want. Run the Path : Convert to Paint Synthesizer : Convert PASeq Context keyframes to Embedded Autodraw menu command.
You now have a set of embedded bezier paint action keyframes in a single new paint action step in your PASeq. Mute the old paint steps you used to build the beziers (unless you don't want to). Use Action : PASeq Timeline : Time Compress / Expand menu command to time expand the keyframes and associated embedded bezier paint keyframes.
At this point you have a key-framed paint animation that is independent of whatever is inside the source area. But you can route transition effects frames based off of the original keyframes into the source area. And then build new paint presets that take that transition effected source and paint it on top of the embedded bezier painting.
So you have doubled up your paint transition animation.
Threshold Ip Op followed by Flat Region Colorize Biggest Split approach to building the regions, Flat Input Vectorizer to convert to Bezier, ran the menu command to convert the set to Transition Context keys to a single embedded bezier paint action step.Ran the old slow minimum distance sort menu command to re-index them for this second output render.
Ran an automated command to take a set of Transition Context keyframes and convert them into keyframe bezier paint animation in Studio Artist V5.5.
Testing some style modulation of Mondo Simplify vector output using an old jazz poster as the loaded style image.
It's really an experiment testing Flow based modulation of the Threshold IpOp effect ib nStudio Artist V5.5. But i got a little bit carried away with it.
I've been beating my head against a conceptual wall trying to figure out some behavior associated with a certain Studio Artist internal algorithm. Then i used Studio Artist to configure a very simple visualization of what was going on. Voila.
Same PASeq used for a different source movie with different visual characteristics below. Again, everything makes total sense now.
The first 2 action steps are really the entire thing. Everything else is finishing sparkle dust.
I took one of the style transfer paint animations based of of processing a movie file with a very simple paint action sequence in Studio Artist V5.5.4 with some style modulation turned on (used Starry Night as the loaded style image), then loaded the resulting paint animation movie file as the source image in Studio Artist, and riffed off some different slit scan effects using the Temporal Ip Op effects.
Quick test of the new 'Alg4' algorithm option in the Studio Artist V5.5.4 Transition Contexts. No manually drawn bezier paths or anything, you just keyframe the images and the Transition Context then builds whatever kind of transition you specified with the Algorithm option.
Vectorizer experiment involving processing every 10th frame with an extreme vectorizer effect (you can set Skip Source Frame in the movie tab in the main preferences dialog), then using a Transition Context with Virtual Expand Keyframes to fill in the missing pieces.
Same paint action sequence as the last post. The only thing i did different was load Starry Night as the Style, and turn on 1 style visual attribute modulation option for each of the 2 action steps in the paint action sequence.
I wanted some more 'movement' in the stylized background, so i tweaked things a little bit and got the following.
Paint animation test in Studio Artist V5.5.4 using another example of the world's simplest paint action sequence.
Two steps. A dispersive action step based on the SmartBlu IpOp. And a slop down paint action step that is the factory Init Paint Synthesizer QuickEdit painting a single paint nib that is size modulated based on Flow magnitude.
Many people make the mistake of piling on a ton of stuff into the paint action sequence. Because they are totally missing the point that the paint animation effect occurs over time. Occurs over multiple frames. And by working with recursive processing, you can setup a very simple paint strategy that plays out over multiple frames (creating behavior over time that can be visually rich and complex).
And suddenly the world 'most boring' paint preset (as our factory preset designer likes to refer to the Init QuickEdit) suddenly becomes quite interesting.
Putting together some new presets to aid in visualizing Flow modulation in Studio Artist V5.5. They are also very useful for tweaking some of the finer points of the implementation details.
So this is a pretty simple PASeq that uses an Outline Vectorizer action step followed by 2 paint action steps. the arrows are MSG Live Brush effects. I'm running injection modulation to modulate the angle and size of the MSG Live Brush. I'm using a Tile Spacing Grid Scan approach in the paint synthesizer for the arrow positioning.
Different take on what i was trying in that last post. Used soft threshold to process the flow magnitude (Threshold IpOP that routes selection to selection). Forgot to turn on a fixed seed for the Color Simplify IpOp that is motion matted in before the Flow modulated smear paint (would improve the regonization coloring dropout).
And we need to add smart clip modulation to Color Simplify, turn on raster EFX like we did in some of the other IpOp effects, and have a no-background option so the old canvas shows though if there is not complete coverage with an EFX.
Paint animation experiment using lack of motion to inhibit painting. The jerky stuff at the end is associated with the hand held camera being moved a little bit. It would be interesting to add a motion compensation component to get rid of that kind of stuff.
Trying out some Flow Warp ideas using the paint synthesizer as the warp engine. I reversed the direction on the flow for this one.
World's simplest 2 step paint action sequence. Both of the paint action steps use the Init QuickEdit with very minor additional adjustments. I turned on facial feature detection as an inhibitor option in Path Start control panel for the first paint step. I used Flow Angle for Path Angle in Path Angle control panel and Flow Magnitude for Blend in Paint Fill Apply control panel for the second paint step. I used the new injection modulation to do that routing for the second paint step.
Inaugural run paint animation test of the new injection modulation features in the Studio Artist V5.5 paint synthesizer. Injection modulation is like bus routing in a DAW. In this particular example, i'm routing the optical flow angle to the direction of the paint, and the optical flow magnitude to the path length of the paint. World's simplest paint preset, use the Init QuickEdit and then the 2 manual edits i just said, and i switched it to straight lines rather than curved. And i'm clipping any paint strokes less than 20 pixels long, so they don't get painted.
I slopped down the world's simplest Vectorizer outline technique black overlay for the edging effect. Then let it run. You could of course finesse the presets to make this whole thing much more artistic.
Which i took a quick stab at below, adding a 10%Mix fade to white and a SmartBlur action step to introduce some dissipation into the recursive processing over time. You put those at the beginning, before the painting. So they process the previous output canvas that is never erased.
This is obviously a work in progress. But the very first output from the system shows the promise. Because it probably would not have occurred to me to dial in the particular generative preference settings it auto-created for itself (using the 'ArtStrategy - TestDrive' Gallery Show QuickEdit). It auto-configured an image folder photo mosaic paint effect that includes intelligent path randomization based on visual attributes in the source image.
Here are a few more examples of what it spat out in the inaugural test run of 'ArtStrategy - TestDrive'.
The output is obviously too heavily weighted in certain directions, but that will be vastly improved by the end of the day.
Working right now on full words as text nibs in the paint synth. We should have done that years ago.
Another paint animation experiment. It's using a Watershed DM path start regionization paint preset. I jacked up the brush size, and then use software and hard nib clipping in conjunction with local region brush size modulation to restrict the brush sizes based on the local region size. The nib clipping gives it an image processing kind of look in some respects, but it is painted.
I have it setup so that the edges of the individual regions get painted, but the interior is not painted in most cases (except for really small regions). This was an experiment to see how that would work for adding some temporal continuity in the animation. And it does the job, so add that to the stack of different approaches one can use to generate temporal continuity in paint animations.
However, i forgot to turn off color randomization. So you get some strobing because of that. And if i forgot, you probably will as well. So i'll add a new QuickEdit command specifically targeted at paint animation prep. There already is one to turn off color randomization. But having a specific one to optimize paint for animation output would probably be a good idea.
Different approach for processing the same source footage as the last post. I used a Vectorizer action step for this. Some addition 'finishing' IpOp action steps are in the paint action sequence (PASeq) to generate the result seen here.
And then if you redo that last one with virtual sub-nested keyframes in a Transition Context to re-time the animation output, you get below.
So if you want a more stylized thing, you can adjust the controls in the Vectorizer a little bit.
And then if you redo that last one with virtual sub-nested keyframes in a Transition Context to re-time the animation output, you get below.
I've got additional IpOp action steps in the re-timed versions that also change the overall aesthetic of the output. You could of course dump those into the original PASeq if you wanted that grading kind of effect in the original non-re-timed animation.
Fun little experiment. I don't consider it a success, but it does point in the right direction.
Single paint synthesizer preset. I then used a Transition Context with virtual sub-nested keyframes to re-time the paint animation. I'm doing a few clever IpOp effects afterwards to post process it. One of the clever things i'm going to build into Transition Contexts as an option there you can turn on or off as it suits you.
This is a buggy version of an alternative approach. Different non-dynamic brush paint preset.
I then ran the automatic conversion to embedded bezier paths. So you end up with a different single paint action step, where all of the paint strokes are bezier embedded inside of that action step.
The automatic generation works off of each of the individual virtual sub-nest keyframe positions derived off of the Transition Context. The Transition Context contains one real keyframe reference to the original source movie. It's Destination (where it routes its output) is set to Source (as opposed to canvas or style).
The bug is that the original bezier path sizings are not being passed through in the automatic conversion for this particular paint synthesizer parameter setup. Going to dive into the code and clean that up now. Fix will be in V5.5.4.
Xcode is crashing on me multiple times a day. It just can't keep up with Studio Artist 's AI cranking out cpu cycles at lightning speed. We're going to bail on it at some point and move to a unified cross platform development environment that i hope is better able to handle the stress.
Single paint action step that uses an intelligent dynamic brush paint synthesizer preset. I'm using a Transition Context to re-time the paint animation output as an experimental alternative to using a lower frame rate for the output. This output is totally painted in by the AI and associated visual modeling in the paint synthesizer, no source image mixing involved in this effect at all.
-drop down some paint,
-apply dispersion (via water preset in this case)
Followed by some Style manipulation of what is in the canvas, where the style is derived off of a second movie file being pumped from a Transition Context action step into the Style.
The trick with the Flat Region Colorize IpOp effect is that the regions in the image used as input need to be flat. No anti-aliased edges (because the individual smoothing pixels in the anti-aliased edge will be considered individual regions by the effect). You can use the min size filter thing at the top to help deal with that, but better just to not generate them in the first place.
The particular approach i used is shown below. You want a 50% mix for the Threshold IpOp action step.
I took a different tack for this experiment, using it in combo with some orientation modulated scale space processing to build an oriented halftone mask for the inherent noise in the image itself. I used the denoiser as a part of this process.
So if you auto-mask the flat part of the face, you can build the artificial structure in just the textured areas.
Proof of concept. Obviously i need to pay more attention to the overall coloring of the face for a generically useful preset for people. But i think this very quick experiment gets the idea across. Build the artificial structure where you want it, don't do it in other areas. So for audio people, think of side chain compressor modulation, then do it visually.
So something like this.
I added a new generative deep multi-octave gradient lighting option to gallery show start and end cycle processing in Studio Artist V5.5.3. I streamed 12 cycles of it's output (the deep gradient lighting is processing a single generative paint preference setting) to a movie stream. The movie is only 12 frames long, so it blips by super fast, and flickers because of the large differences as you move from frame to frame.
Wow, i thought to myself, why not modify the new Virtual Keyframe Sub-Nesting feature to support time expansion in Studio Artist. It was a no-brainer code wise, QED.
Here's an example of the output. I used a 20X time expansion ration.
Transition Contexts in Studio Artist V5.5 are really turning into a swiss army knife of visual effect options.
Cam Track selection is doing something interesting, but i don't think it's actually the correct behavior. Will dive into the code and see what the deal is. I haven't tested this at all in V5.5, so maybe something got broken or changed in the port from V5 to V5.5.
I added a TG temporal generator to the MSG preset that is playing back in her hair as it tracks her movement.
This is more like it. The 'fixed' version of the previous programming bug post. And i even learned something useful in the process.
Sometimes programming bugs end up as Studio Artist features. Not this one. But i thought i'd document it here for posterity. Now on to fixing it.
Now that i fixed the Dense Optical Flow in Studio Artist V5.5, it's actually useful for something more than giggles. The fix is in Studio Artist V5.5.3. Quick animation test here. You will be seeing this as a modulator basically everywhere as time moves on.
Magical Horn Pivot Optical Flow. Yes you saw it here first. The mysterious shotgun marriage of Dense Optical Flow and Horn Optical flow, working its magic. Yes it is mysterious.
And boy does it make custom designing a generative paint strategy easier (at least one that uses MSG Brush Load in it). Point it at a folder of woodcut MSG presets and you get endless woodcut paint variations.
I pointed it at the entire VV5 Collection in the MSG factory set for Studio Artist V5.5 in the inaugural image painted above using it. You really need to think this through for designing custom texture synthesis algorithms that paint, because it is a powerful tool to do just that.
So what is Studio Artist anyway?
It's a digital art program with a 20 year long history at this point. When Studio Artist V1 was released at MacWorld New York in 1999 (is that a long time ago or yesterday, i get confused), it was a pretty radical rethink of what a digital paint program could be. It was not PhotoShop, it was not Painter, it was not Illustrator, it was its own thing. With its own internal structure, its own way of working, its own way of thinking about how to work.
It allowed you to work with still images or video, merging those 2 very separate application categories at the time into one unified system. It introduced the concept of a 'source' and a working 'canvas', again a very unique distinction at the time. It introduced the concept of incorporating a human visual model (based on research into how the human visual cortex works at the neural level) into a graphics program for digital artists. It incorporated conceptual ideas from music synthesis, music synthesizers, and digital audio software, and then incorporated them into a digital art program for visual artists. It introduced a new hybrid raster-vector model for digital paint effects (bezier paths that define a drawing path, raster paint nibs that a relaid down on that editable vector paint path).
It also introduced what i call the first 3 levels of ways of working and interacting with digital imagery.
Level 1 is all about working at the pixel level. Down in the basement. Literally pushing pixels around on the canvas with an interactive stylus (wacom pen at the time) or a mouse cursor.
I would also consider the use of simple image processing effects (like an edge or blur filter) to be working at level 1 to some extent. At least for the sake of this discussion.
Level 2 moves up a conceptual notch with intelligent automatic actions. Intelligence based on the underlying human visual modeling built into the program. That takes a center stage in the Studio Artist paint synthesizer with its extensive use of visual attribute modulation internally. But intelligence that was also distributed throughout the program in a wide variety of adaptive image processing effects. Also based on visual attribute modulation.
Studio Artist is built from the ground up on the concept that the program tries to look at a visual 'source' just like a human artist would look at that 'source'. That internal visual representation is then made available internally throughout the program. Visually modulating what the paint synthesizer is doing, visually modulating what adaptive image processing effects are doing.
You can think of the 'source' as a model the artist is painting, or a photograph of a scene the artist is reinterpreting in a painting. The artist looks at the source to influence their work, but so does Studio Artist. Using its internal human visual modeling. Trying to perceive the source in a similar way to the artist perceiving it, reacting to it, etc.
This process is not about the machine replacing the artist. Far from it.
Studio Artist always tries to incorporate the human artist into the automatic action creative loop. At whatever level of interaction the artist is comfortable with. The artist can do all the work if they want to. Or the artist can manually manipulate the stylus, while Studio Artist intelligently assists in the manual drawing (intelligent assisted painting). Or the artist can press the Action button, and Studio Artist does all of the work automatically (fully automatic action painting).
But even in that last case of a fully automatic action, the artist is still involved in the loop, making decisions about what they like and don't like, how to proceed next, etc.
Level 3 in our taxonomy of ways of working i would define as working with scripts of actions (which could be manual or automatic actions). Studio Artist includes a Paint Action Sequence (PASeq) Editor, that allows you to build more expansive visual effects composed of multiple actions that work in sequence. Combining together different kinds of brush sizes, different kinds of digital media emulation, different kinds of artistic techniques (charcoal, ink pen, watercolor, water or acid wash, canvas smear etc) into a sequence of different processing steps that work together in synergy to build up a final visual effect.
Since automatic actions intelligently analyze the 'source', they can be built using just one specific image as the source when they are initially created, but then applied to an infinite variety of different source images later, while achieving the same stylistic visual effect output for those new input images as was designed using the first source.
Intelligent actions meant that you could also work at level 3 building PASeq presets in Studio Artist, and then process movies with them. Creating dynamic visual effects in processed movie files that looked like hand painted or hand animated art styles in moving art.
That was a lot of innovation introduced in old SA V1. Innovation that is still missing in other digital art programs 20 years later to be honest. I say that because in every other digital art program i sit down at, you start with a blank canvas, and then you have to do all of the work. The other programs do nothing, you have to explicitly make everything happen in them yourself.
Here's the thing, that's a really hard process to create something from nothing. Especially if you have not taken the extensive time and training required to develop the muscular and neural motor skills required to draw well. Some people are very good at it, many, many more are not.
Now there's a direct analogy in the world of music composition, where in the old days a composer would compose a piece of music (as marks on paper literally), and then go into a very expensive recording studio, and tell a bunch of musicians they hired at union scale wages to play their piece of music for them, and a recording engineer you also had to pay hourly would record them playing the music, and then another person you had to pay would mix and master the recorded tracks into a finished piece of music. And the expense and organizational complexity of all of this was really prohibitive to musical creativity.
So the home recording studio revolution created by digital audio workstation software running on personal computer hardware (a revolution i was heavily involved in created back in the day), was truly liberating for individual musicians and composers. Because they could do the work on their kitchen table if they wanted to, and they could do all of the work if they so desired themselves.
Studio Artist was trying to do a very similar thing in the visual art and digital video world. So that if an individual had a great idea for an animated film, they could do it all themselves on their kitchen table. As opposed to hiring a team of animators at great expense to make that happen.
David Kaplan pursed that very vision using Studio artist to great effect, winning an award at the Sundance Film Festival for his feature length Studio Artist animated film 'Year of the Fish'. Literally created at his kitchen table in his apartment in NYC.
And the innovation continued over the years as new versions of Studio Artist were released. Vector effects were introduced via the Vectorizer, new vector paint options in the paint synthesizer, and vector output from some Ip Op effects. You could output these new vector effects as resolution independent svg files, an interesting alternative to the traditional raster image file output.
Temporal image processing effects were introduced. These were great for processing video with time based effects, but also opened up a whole new way of working called 'stack filtering'. Stack filtering involves taking a collection of still images, and then using that collection of individual images as the input to temporal image processing effects. The end results can be visually amazing.
MSG (modular synthesized graphics) was introduced in SA V2, and has been extensively expanded over the years. With over 600 individual image processing and generative modular effects that can be used to construct an infinite variety of different modular preset effects. MSG can be totally generative (creating visual imagery from nothing but editable parameters), or it can process a source image into an effected output image, or a source can be used to modulate a generative process in some way internally.
MSG presets can also be embedded into the paint synthesizer, providing a way to modularly expand different internal paint synthesizer components (like path start generation, path shape generation, brush load processing, source brush generation). Studio Artist V5.5 also lets you embed Ip Op and Vectorizer effects directly into the paint synthesizer (once again super-charging what you can do with paint synthesizer presets).
Movie brushes (using a movie as a paint brush) were introduced fairly early on in Studio Artist history. Movie brush capabilities have expanded over the years with image folder brushes, as well as movie and image folder background textures. These can be incorporated along with visual attribute modulation to build photo mosaic and other visual effects built off of artist curated sets of multiple visual images loaded into a digital paintbrush.
Keyframe animation in the PASeq Timeline allows for the construction of interpolated hand painted strokes that then move dynamically over time in an animation, automatic dynamic visual transformations, morphing, warping, etc. Bezier paths derived from automatic actions can also be embedded into single paint actions and then key-framed as well on the PASeq Timeline.
So Studio Artist has always been a generative digital art system from its very beginning. And i want to draw a distinction between how the word 'generative art' is oftentimes used, because many artists use it to refer to digital art output from software coding systems created for artists like Processing, Open Frameworks, or perhaps mucking about in neural net Colab notebooks these days, etc.
I'm all for artists learning how to work with software code if they so desire. But my experience working with a lot of visual artists over the last 30 years indicates that many of them are not really interested in software coding. It requires a certain level of 'left brain' analytical thinking they aren't necessarily comfortable with (especially within their working methodology which is much more intuitive or 'right brain' in nature).
These 'left brain' - 'right brain' metaphors are much over-worked (and technically incorrect) at this point in time, but i think they are still useful on some level. We have traditionally made the distinction between 'left brain' building your set of tool presets, vs 'right brain' using your set of tool presets within the Studio Artist universe. So you would set apart specific times to build working tools (left brain activity), and then use those tools creatively in your artwork (right brain activity).
The previous distinction between these 2 ways of working ( 'building the tools and then 'using the tools' ) is a good lead in to where Studio Artist is heading. Both in terms of the new features we are introducing in Studio Artist V5.5, and where we see those new features heading in the future. Because we believe we have staked out a whole new way for digital artists to work in Studio Artist V5.5. And we are going to fully develop that out and expand the nature of what it truly means in future Studio Artist releases.
Studio Artist V5.5 introduces 2 whole new higher levels to our taxonomy of 'levels of working' for digital visual artists.
The 4th level of working involves expanding the whole concept of what the 'source' even means in a digital art program. Traditional digital art programs are very religiously rooted to the concept of working with a single image, a single movie file. The single digital photo you took, and you now want to enhance or effect in some way. The single movie file that you want to effect in some way. The empty blank canvas that you are supposed to turn into a sketch of a woman's face, or a bowl of fruit on a table.
So this new 4th level of working i call 'source abstraction'. Abstracting and expanding the whole notion of what the source even means. So that it is no longer tied to just a single image or movie file. It might be a representation of a collection of images, a collection of movies. You might start to think of it as more like a data model, filled with visual attributes, all available for modulation within Studio Artist.
You can also start to think about 'style' mixing in the context of the 'source'. Playing one abstracted 'source' off against another abstracted 'source'.
My personal viewpoint on the whole notion of what a source is, or even style for that matter, has evolved and expanded quite a bit over the years. Just like our notion of what the source is and can do has expanded in Studio Artist V5.5. With the new Gallery Show source options, and the new Load Style features. Both of which are really just teasers for even cooler things to come, but still incredibly useful in their current limited forms.
It's fascinating when you start thinking about the 'source' for a piece of artwork as being a collection of images instead of a single image. It's very much like working with a database of visual attributes. It's a very different way of thinking for most digital artists (at least that is my perception). Different, but really expansive, liberating perhaps, definitely worth getting a grasp on, worth trying out and exploring. It's new territory waiting to be charted out, waiting for you to find your own personal niche within it.
As i pointed out earlier, the Studio Artist universe dived into this water when we got heavily into stack filtering. But it's always existed since V1 if you really wanted to explore it, via loading a movie (which could be any old collection of images as individual frames within the movie container), and then riffing with it while painting.
I've been very heavily involved in getting up to speed on the latest developments in deep learning neural nets during this last pandemic year. Most people are probably unaware that i did neural net research in the 90s, including some very basic work on using convolutional neural networks for learning artistic stylistic transformations on images. Not really practical back then to be honest, so we quickly moved on to adaptive fuzzy logic systems and other things that ran a lot faster on the ancient computer hardware available at the time.
But it's a very different world now days. So it's been fascinating to take my background in all things neural net and bring it all up to date with the latest and greatest developments in the field. So i'm heavily influenced by what people have been doing with recent research areas like neural style transfer, generative models like GANs and VAE systems, etc.
And the notion of a database of visual images is very central to how these deep learning neural net systems work. Generative imaging transformations based on the collective visual statistical properties of a collection of images. So all of that exciting new research work has heavily influenced my thinking, and is going to continue to do so in the future as we move forward.
This whole notion of source abstraction (expanding the whole notion of what a 'source' even means), is the new 4th level of working.
And the 'build a tool' - 'use a tool' discussion mentioned earlier is really what the new 5th level of working hopes to alleviate.
Through the use of AI Generative Systems. Implementing automatic intelligent Generative Art Strategies.
