Researchers at the Lassonde School of Engineering have made a breakthrough in tackling image noise, an undesired by-product of digital photography. This research, supported by VISTA and NSERC, could mean a profound enhancement in computer vision applications, which will be of great interest to tech companies.
Image noise in photographs captured by cameras, an important aspect in many computer vision applications, is the variation of brightness or color in images. It can appear as graininess that can seriously affect the quality of the image.
Grad student Abdelrahman (Abdel) Abdelhamed, supervised by Canada Research Chair in Computer Vision and member of Vision: Science to Applications (VISTA) Professor Michael Brown, presented a key paper on this at the 2019 IEEE/CVF International Conference on Computer Vision.
The paper introduces ‘Noise Flow,’ a powerful and accurate noise model where computers, through machine learning, can understand the characteristics of noise and regenerate it. Since this model will lead the way to removing the noise, this new knowledge will capture the attention of many tech companies. It’s a big step forward in computer vision and deep learning.
This research was funded by VISTA and the Natural Sciences and Engineering Research Council of Canada (NSERC). This work was undertaken while Abdelhamed was at Borealis AI in an internship, supported by the Mitacs Accelerate Program.
Abdelhamed and Brown discuss this work and its impact with Brainstorm.
Q: What is image noise and why is it problematic?
AA: Image noise is an undesired effect that we see in digital images. We all use digital cameras but, in some situations – when we’re in a dark room, for example – this negatively affects the quality of the image. You’re measuring the amount of light that gets in, in pixels. If you’re in a low-light environment, you have more noise.
MB: The camera sensors inherently have noise. Now that camera sensors are smaller and smaller – on your phone, for example – so much less light gets in. The chance of having more noise is much higher. For small digital cameras, one of the trade-offs is more inherent noise – a grainer or noisier image.
In this paper, we looked at noise from the perspective of smartphone cameras with very small sensors.
Q: What was the objective of this research?
AA: For computer vision or photography, the objective is to get rid of, or minimize, noise. But we took a different approach: we wanted to model, characterize or understand the noise in order to remove it more efficiently.
MB: Abdel’s method was in reverse; he didn’t try to generate a clean [noise-free] image, he sought to understand the noisy image.
Q: Is this a new area of research?
For small digital cameras, especially in dim environments, one of the trade-offs is a noisier image. The noisy image to the left is a common output of a smartphone camera in a dim environment. The clean image to the right is heavily processed to minimize noise. (Source: Smartphone Image Denoising Dataset (SIDD): https://www.eecs.yorku.ca/~kamel/sidd/)AA: We studied older literature, papers from MIT [Massachusetts Institute of Technology] and Google, and we saw that they had models for this, but the research hadn’t been updated in years – especially important with today’s smart cameras. Ten years ago, they didn’t have these tiny sensors. The technology was so different. We found that the mathematical models used to characterize noise in the literature were outdated – the models were for older, larger sensors, not for the sensors used in smartphones.
Q: How did the Noise Flow model work, and how could this benefit computer vision and deep learning?
AA: The Noise Flow model is a type of machine learning – teaching computers how to solve a problem by providing many examples. In our research, we collected lots of images, noisy images and their corresponding clean images, so the computer model could learn the mapping between the noisy images and the clean images. As a result, the computer could learn the characteristics of noise so it could generate more examples of noisy images.
This is beneficial to computer vision because in computer vision, and deep learning, we need more data to make machine learning algorithms work.
MB: Broadly speaking, we’re trying to mathematically model noise. We wanted to know if we could predict the type of noise that occurs on different sensors. Could the computer generate the type of noise that a particular sensor would produce?
Q: Please describe how you tested this.
MB: Abdel did the work. He sat in the lab and captured 30,000 images. He set the camera on a tripod, very still in a special room, under special lighting, and took these images. Then he fed the information into the computer model and calculated an average. He would have, say, 500 noisy images and one clean image. This is how the computer learned how to generate the noise.
Q: Many tech companies would be very interested in this.
MB: The beauty of this is, if you’re a company that wants to denoise images, you first need information on how to predict noise. Then you could train another model to denoise using this new knowledge, using this new model.
We’ve been approached by many companies; Abdel has been approached by Samsung.
Q: Is this a ‘first?’
MB: Yes. We were the first group to revisit the noise modeling problem and address it with machine learning. This is a big deal. Nobody had revisited this for a very long time.
Marcus Brubaker, another co-author of this paper, is a machine learning expert at Lassonde and a member of VISTA. He was instrumental in helping us to shape this work because he introduced us to the normalizing flow method, a new powerful machine learning method. It was perfect for the type of problem we had before us.
Q: VISTA has supported your work in many ways.
MB: VISTA was instrumental for getting me to Canada. It was the main reason that brought me to York. My CRC was created because of VISTA. VISTA supports our entire lab.
It’s also worth mentioning that if we do anything that’s able to be commercialized or needs a patent, we go through Innovation York. In this case, Borealis filed the patent for our new model, but, generally speaking, Innovation York is our go-to partner.
To read the conference paper, visit the website. To learn more about Brown’s research, visit his Faculty profile page.
To learn more about Research & Innovation at York, follow us at @YUResearch; watch our new video, which profiles current research strengths and areas of opportunity, such as Artificial Intelligence and Indigenous futurities; and see the snapshot infographic, a glimpse of the year’s successes.
By Megan Mueller, senior manager, Research Communications, Office of the Vice-President Research & Innovation, York University, muellerm@yorku.ca