August 10, 2022
As photographers, we all have situations where we end up with noisy photos, like when we’re shooting in low lighting or shooting fast actions. Even for more traditional photography, we sometimes find noise in shadow areas when adjusted in post-editing. Over the last decade, Topaz Labs has been dedicated to developing the best possible tools of that time to address this problem, along with other common editing hardships.
Removing noise from images is considered a difficult problem because of the variety of noise sources. The atmosphere, optics, sensors, and digital circuits all can contribute to the final capture in different degrees.
To make things worse, the type and contribution vary in each capture and even in the different parts of the same image. There is another aspect that makes the problem evenharder. As a viewer, we perceive different noises in different ways. For example, most of us find chroma noise more irritating than the luminance noise.
Therefore, many clever solutions were devised by many researchers over time to address different types of image noise. Overall, the main idea is to look around the neighboring pixels and only allow a certain type of variation. The main difficulties lay on identifying the variation we do allow and what we don’t. This distinction is very important, as the other type of variation details are very precious to us. Just like any photographer knows, when over-applied, noise removal algorithms are very notorious for removing details and making images very flat.
In the last few years, the scientific community has made a great leap in the Deep Learning field, also known as Artificial Intelligence (AI) in public. Nowadays, there are hardly any days not to see some news about AI. Solutions that were previously beyond most of our imaginations are now available for our daily use. Very rapid development in GPU technology also gives individuals utilizing AI-powered applications which require humongous computational power. With all these advancements, our team at Topaz Labs are able to move beyond traditional image processing to harness the power of AI for the photography community. Developing an AI-based noise-reduction solution is one of our many AI efforts.
Let's say we asked you to picture a loved one – it could be your spouse, a parent, or a close friend. You've seen this person thousands of times in your life, maybe even more than that. If you close your eyes, your brain starts to remember all the occasions you've seen this person and you can start to reconstruct the features – their eyes, their lips, their hair, and so on in your mind.
A deep learning model, the core of the software, works similarly to your own mind. The process of learning is called training a Deep Learning or AI model.
Let’s say the deep learning model is a black box with millions of filters. At the very beginning, we teach the model to learn a specific task based on examples. In the case of developing DeNoise AI, our image noise reduction software, we give the black box a noisy picture and ask it to give us an output as good as the clean version of that image. The black box has to tune its millions of filters to match the target clean picture.
After tuning its filters, the black box gives us its approximated clean picture. Now, we compare the output from the black box with our known clean picture, then give the black box some feedback based on how it performed.
We feed the model another noisy picture and the black box tunes its filters based on the error it made last time and give us another approximate clean picture. We calculate the difference between the original clean picture with the black box’s output. We repeat this process with millions of pictures and over each filter tuning, the black box reduces the error gradually. When training is finished, the model can output a clean picture from your given noisy picture because it saw so many examples and variations.
Now, this is the fun part. As we see the training process of an AI model, we can identify three key elements to play with:
We know it is practically impossible to have a noisy picture and a clean version of it from the camera. So, what we did was we mathematically add some noise on top of a clean image.
When adding this replicated noise, we considered the noise-contributing factors during a camera capture. We randomly adjust those factors to produce millions of noisy images from originally noise-free images.
This noisy and clean image pair generation is a critical part as the AI model learns from these examples.
The AI model has millions of filters and there are many ways to arrange those. We did lots of research on designing these filters and their arrangement. The key idea is to design many models with different setups and see which one is better on reducing the error over time. Currently, we provide a few different models within the DeNoise AI software, a few for faster processing, and a few for better performance.
There are a few ways to calculate the difference between the model output and the original clean image. We can say the error is high if the model removes some details, or makes some mistake in producing the right texture after removing noise. As a critic, when we see the model output and the original clean image we can tell the difference.
In this example, we can see the model removed the noise, but also took some of the detail out of the strands of hair.
Rather than doing this task ourselves, scientists came up with a clever idea. Why don't we create another AI model to do this work? We can call this AI model, a critic model or a detective model.
The original AI will generate an approximate clean image from a noisy image. The critic-AI will see the “true clean image” and the “fake clean image”, and decide which one is the fake clean. In the training phase, the original AI will try to defeat the critic-AI, and the critic-AI will try to distinguish between true clean and fake clean image. Over time, the original AI will produce better and better clean images.
DeNoise AI is trained to remove high ISO noise, sensor noise, thermal noise, banding noise, and noise from some scanned images. Below are a few examples of DeNoise tackling specific noise issues.
In the example above, the original image on the left has very strong noise and banding artifacts. The DeNoise output is clean and some details are generated in the very dark noise area.
This image has very strong banding artifacts at different levels. The AI removed those banding artifacts and generated some convincing details.
The above image was taken at a high ISO setting and contains a heavy pattern of chroma noise. Chroma noise is typically hard to remove completely, and even harder to remove without desaturating the image. The output from DeNoise AI in default settings on the right shows removal of chroma noise to reveal a clean image with crisp details and no noticeable desaturation.
Seeing is believing, so you can put this technology to the test yourself. You can quickly and easily download a free trial of DeNoise AI, no credit card information needed.
Below, we've listed a few images that are all publicly available, so you know with certainty they haven't been manipulated on our end. Download this beautiful wedding shot for example.
‍The biggest challenge with wedding photography is in capturing so many important moments in all sorts of different light, often with restrictions. When no flash photography is possible, what do we do? We bump up our ISO to freeze the moment in low light, even if it means the resulting images will be filled with noise.
Here are the settings we used in DeNoise AI to remove the noise, but keep the bride's facial features and dress details in tact.
You can also learn more about this topic by understanding what causes noise in photography, how to identify the types of noise, and learn how you can reduce it and preserve image details.
