A Random RGB Cycle Layer is one of two kinds of layer in which Mandelbrot Explorer picks the colour scheme at random for you. The other is the Random HSL Cycle Layer.
To understand how this kind of layer works, you need to grasp that colours can be formed by mixing red, green and blue light. Red and green, when mixed in equal quantities make yellow; all 3, when mixed with maximum brightness, make white; all 3, when mixed with half brightness, make grey, and so on. In this way, every colour can be defined by specifying how much red light, how much green light, and how much blue light is mixed to create the colour. This means that you can pick any colour by selecting a point in a cube, where the 3 axes of the cube represent the amounts of red, green and blue. To put that slightly more technically, you can represent colour-space as a 3-dimensional cube.
Imagine this cube, and then draw an oval (ellipse) somewhere in the cube. As you trace around the outline of that ellipse, the colours move smoothly around. Keep drawing round and round, and the colours cycle through a range of colours. You could do that across a range of dwell values, and you would have a colour scheme that cycles through the same pattern of colours again and again. If you were to draw a different ellipse, you would have a different colour scheme.
The Random RGB Cycle Layer picks a random ellipse in our colour cube, and uses that for the colour scheme.
You can customise how this layer behaves. You will see from the screenshot below that there are 3 coloured rectangles, one for each of the Red, Green and Blue components. These allow you to insist that the ellipse is drawn in a particular part of the cube. You can insist that the red, green or the blue must fall within a particular range of values. Left-click the mouse in the relevant rectangle to set the minimum amount of that colour component that is allowed; right-click the mouse in the relevant rectangle to set the maximum amount. A different shading is applied to mask the areas that are now allowed.
Beneath these three coloured rectangles is another one, this time containing a wave that gets steadily longer. The Random RGB Cycle Layer also determines, at random, how fast the colour scheme will move around the elipse, how fast the colours will cycle through their range. Clicking with the left or right mouse button in this fourth rectangle allows you to restrict how fast or how slow the colours are allowed to cycle.
To the left of this fourth rectangle is a tick-box, that is checked in the example above. When this box is ticked, the colour cycle will follow a strict elipse. That is to say, the red, green and blue components of the colour will cycle at the same rate. If you untick this box, the rates at which the 3 components cycle are randomly picked independently of each other. This allows for more variation in the colour scheme, but may create an image that is less aesthetically pleasing.
To the right of each rectangle is a button marked with a red cross. Click this button to reset that component to its default behaviour, allowing any value to be selected. All masking is removed.
Finally, at the bottom, you will see a preview of the random colour cycle that has been selected. Whenever you change any of the constraints described above, a new random colour cycle is picked. You can also click the Refresh button, to the right of the preview window, to force the selection of new random values manually.