Block
Denoise
- What it does
- It removes most of the grain from the stacked image, before the colours are calibrated.
- What it shows
- A smoother sky and smoother faint nebulae; Inspect shows the image before and after, and the log says by how much the noise went down.
- What can be set
- Method: the default, Pretrained network, suits most nights; for a night without dithering the run advises Pretrained + adapted to the image, which takes minutes instead of seconds.
The Denoise block removes the noise of the stack with a Noise2Noise network: trained to predict one half-stack from the other, whose noise is independent, it learns to estimate the signal both share rather than a clean image it never saw. The network ships with FLOAT and runs in seconds; it can also first learn the noise of the image on its own half-stacks. FLOAT proposes the length of that adaptation from the noise it measures.
- Astrometry
- Gradient
- Deconvolution
- Denoise
- Color
- Stars/background
1Before and after on a real image
The comparison below uses one stack of the Rosette Nebula: 436 exposures of 15 s with a Sony A7S at ISO 4000, on a 130/900 Newtonian. The two images come from the same project, without the Denoise block and with it in its default method, the pretrained network. Everything else is identical, and both are shown with the same stretch: the workflow layer of Post at Sky 0.5 and Faint tones 3.0. The sky’s level is unchanged to about one level out of 255 (at most 1.2, measured as the median of the same faint tiles as Table 1); it only looks lighter on the denoised side because the dark grains are gone. The crops are at 100 %: one image pixel is one screen pixel. Drag the divider, or focus it and use the arrow keys. The block is added in the tutorials.
| Zone | Box in the stack (x, y) | Before | After |
|---|---|---|---|
| 1 Bright nebula and dark globules | 1550–2250, 520–1220 | 6.19 | 2.67 (−57 %) |
| 2 Faint nebula | 1936–2636, 241–941 | 4.72 | 1.39 (−71 %) |
| 3 Star field | 2800–3500, 420–1120 | 4.46 | 0.97 (−78 %) |
| 4 Sky (right of the ring, round stars) | 2750–3450, 1070–1770 | 3.92 | 0.83 (−79 %) |
After
Before
After
Before
After
Before
After
Before The one limit seen in these four zones is in the bright gas of zone 1: its finest texture is a little softer, where the least noise is removed.
2What goes in, what comes out
The block reads the two half-stacks A and B (section 5 of the Stacking page), as the image blocks before it in the graph left them: with the sky gradient removed when the graph has a Gradient block. It also reads the weight map of the stack.
It writes the denoised stack and the network, with the background and the noise of each colour it measured. When a second Denoise block follows in the graph, it also denoises and keeps the two half-stacks for it. FLOAT reports the noise of the green channel before and after, and their ratio, the correlation of the noise between neighbouring pixels (section 5), and the number of adaptation steps. That noise is 1.4826 times the median absolute deviation of the linear green channel minus its Gaussian blur (σ 2 px), over every fifth pixel of the field. The preview shows a central square of the image before and after, side by side.
3Noise2Noise between the half-stacks
The two half-stacks show the same sky; their frame noise is independent. A network trained to predict one from the other cannot predict the noise of its target, which it does not see in its input; what it can learn is the signal common to both.[1] It learns this without any clean image.
During the adaptation (section 5) the pairs are used in both directions: for each tile, A predicts B or B predicts A, chosen at random.
4The network
A U-Net[2] of four levels, with 48, 96, 192 and 320 channels, and two 3 × 3 convolutions with a skip connection in each block. Its input has four channels: the image of the three colours, from which the background of each colour is subtracted and which is divided by its noise and compressed by an arcsinh, and the noise map. Its output is a correction added to the normalised image. The background and the noise of each colour are measured on the half-stack (the median and 1.4826 times the median absolute deviation).
The image is processed in tiles of 512 pixels with a margin of 48 pixels; a graphics card with little free memory gets smaller tiles, which can change the result at the seams of the tiles by a fraction of the noise. The full stack carries less noise than a half-stack: its noise map is divided by √2.
5Pretrained, or adapted to the image
Pretrained. The network that ships with FLOAT, applied as it is. It takes seconds on a graphics card and also runs on a processor. A monochrome stack of fewer than 30 frames gets a network made for monochrome noise.
Pretrained + adapted to the image. The same network first learns the noise of this image. It trains all its weights on random tiles of 192 pixels of the two half-stacks, in batches of 12; each tile is turned by a random multiple of 90 degrees, and the loss is weighted by the noise expected there. The learning rate follows a single cycle, then falls. The adaptation has two lengths:
| Length | Steps | Peak learning rate | Time on a graphics card |
|---|---|---|---|
| Short | 500 | 2 × 10−4 | about 40 s |
| Long | 4000 | 3 × 10−4 | about 5 min |
On a processor, a step takes about 4 s.
Which length. FLOAT measures the correlation of the noise between horizontally neighbouring pixels, on the difference of the two half-stacks (green channel, a central square of 800 pixels, the sky only). It runs from about 0.15 to 0.4 for a dithered session, and near 0 without dithering. Under 0.08, FLOAT proposes the long adaptation, with the reason; otherwise the short one. A session without dithering has noise the pretrained network has not seen, and it needs the long adaptation.
6Four passes, and the sky level
The network leaves a faint pattern on its own grid of 2, 4 and 8 pixels. To spread it, the block runs the network four times, with the image moved by (0, 0), (0, 1), (1, 0) and (1, 1) pixels, and takes the mean. These four passes cancel the period of 2 pixels exactly and take the pattern as a whole to less than half. “Fast pass” runs one pass instead of four, four times faster; the pattern can then show in the sky under a strong stretch.
The network moves the level of the sky by a small amount that differs from one colour to the next, and a strong stretch of a clean image shows it as a colour cast on the sky. The block measures, for each colour, the robust median of the difference between the result and the input over the covered pixels, and takes it off: the sky level of each colour stays what it was, and the objects of the field, which the network keeps, do not count in the measure.
On a monochrome stack, the network moves the three channels slightly apart; the block replaces them by their mean, which keeps the image grey.
7Settings
| Setting | Value | Range | What it does |
|---|---|---|---|
| Method | Pretrained network | Pretrained network, Pretrained + adapted to the image | The network as it is, or first adapted to this image on its half-stacks (section 5). |
| Adaptation | automatic | Short, Long | How long the network learns the image: 500 or 4000 steps. Long for a session without dithering. Proposed from the noise correlation, with the reason. |
| Fast pass | off | on, off | One pass of the network instead of four: four times faster, but its grid pattern can show in the sky under a strong stretch. |
| Random seed | 0 | 0 or more | Seed of the random choice of the tiles of the adaptation, for runs that can be reproduced. |
8Limits, and what was tried
- No dithering: the pretrained network is not enough. With the Pretrained method and a correlation under 0.08, the run warns that the noise is nearly independent between neighbouring pixels, that the network has not seen such noise, and that the Adapted method (long) denoises it much better.
- A longer adaptation is not always better. On one of the test images, the long adaptation did slightly worse than the short one, and on another the pretrained network alone already did as well as either.
- Few monochrome frames. The network for colour stacks turned the noise of monochrome stacks of few frames into dots that look like stars. A stack of fewer than 30 frames therefore gets its own network, which removes them and denoises as much, but brightens the faint stars by 8 to 11 % on the bench: it is for pictures, not for photometry. Only the Pretrained method uses it.
- Training on each image was dropped. A network trained from zero on each image took about 21 minutes on a graphics card and was unusable on a processor. The pretrained network, with or without the short adaptation, replaced it. The settings of that former method, saved in a project, are ignored.
- The grid pattern. It is reduced, not removed: four passes take it to less than half, one pass leaves it whole.
References
- Lehtinen, J. et al. 2018, Noise2Noise: Learning Image Restoration without Clean Data, ICML. arXiv:1803.04189
- Ronneberger, O., Fischer, P. & Brox, T. 2015, U-Net: Convolutional Networks for Biomedical Image Segmentation, MICCAI 2015, LNCS 9351, 234. doi:10.1007/978-3-319-24574-4_28