A new work by the author of ControlNet, it is so fun to play that it has received 1.2k stars since it was open sourced.For manipulating image lighting effectsof IC-Lightthe full name is lmposing Consistent Light.
How to play is very simple:
Advertisement
Upload any picture, the system will automatically separate the characters and other subjects, select the light source position, fill in the prompt words, and you can integrate into the new environment without any flaws!
Hurry up and do some Wong Kar-Wai style lighting:
dislike?
It doesn't matter, it only takes a few minutes to switch to the natural light coming in from the window.
Advertisement
Currently, IC-Light provides two types of models:Text conditional relighting model,besidesbackground condition model.
Both models require a foreground image as input.
In view of the fact that Controlnet was so fun before, IC-Light attracted a lot of attention when it appeared this time, and some netizens quickly made the ComfyUI plug-in.
(Confused, everyone is working so hard and they don’t even sleep??)
No matter in terms of expectations or after-use experience, netizens give high praise:
Nice! Can’t wait to get started and play hehehehe.
Can anyone help me change the background of this picture?
From ancient MCN to Tieba and now Xiaohongshu, in every era, there is no shortage of help posts like “Can anyone help me change the background?”
But the help from enthusiastic netizens often looks like this:
Just outrageous.
But to be honest, this kind of demand not only exists among ordinary people like you and me, but e-commerce companies often have similar needs when making product posters.
With IC-Light, everything seems to have become easier.
Upload the original main image + select the light source position + prompt worddone.
Let’s see the effect——
Such an original image of a Buddha statue, add the prompt words “Buddha statue, detailed face, sci-fi RGB luminescence, cyberpunk”, and then select “Light from the left”.
You will get a brand new finished product:
It is applicable even to everyday scenes.
The final effect looks quite natural to the naked eye:
According to reviews shared by netizens, anime scenes are also applicable…
Technology behind
As mentioned before, IC-Light now provides two types of models, both of which require foreground images as input.
One category isText conditional relighting model.
To put it simply, users can complete the generation by entering prompt words.
For example, if you input “left light”, “moonlight”, etc., the model will use these prompt words and initial latent variables to generate images that meet the requirements and characteristics.
Another category isbackground condition model.
This one is even simpler and does not require complex prompt words. The model combines the background prompt information to change the lighting of the foreground objects in different styles.
The technical principle behind it isThrough the consistency of the latent space, the model output is ensured to be consistent under different light source combinations, so that various lighting effects can be stably synthesized..
details as follows–
In HDR space, the light transmission of all lighting is independent of each other, and the appearance mixing effect of different light sources is mathematically (that is, ideally) consistent with the appearance under the direct effect of multiple light sources.
Taking the lighting stage of the image above as an example, the two images from Appearance Blend and Light Source Blend are consistent (ideally, mathematically equivalent in HDR space).
Therefore, when training the relighting model, the researchers used a multilayer perceptron (MLP) in the latent space to make the combination and transmission of different light sources consistent and guide the generated effects.
The result is a highly consistent relighting effect.
Because the model uses latent diffusion, learning and relighting operations can be implemented within the latent space, resulting in highly consistent results under various lighting conditions.
These results are very consistent – even though the model does not use the normal map data directly when training, the different relightings can be merged into normal maps.
Look at the picture below, from left to right are the input, model output, relighting, split shadow image and merged normal map.
Interested friends can go to the address below to try it out~
GitHub Express:
https://github.com/lllyasviel/IC-Light?tab=readme-ov-file