I measured the voltage coming out of these two, and its showing 3.9v, which seems way above the 1.5 - 2.5v your sheet indicates is correct.
It's probably OK, I would have noted those values while working on some working systems, and it's likely there will be some variation between different machines. If you see the datasheet you referenced, you'll see the voltage on COMP is a function of the reference voltage and the 'full scale adjust' input.
And you'll see it is saying the outputs are in the 20-30mV range...
I'm curious about how you're measuring the RGB output signal. It's a complex waveform (see page 7 of your referenced datasheet), not a static DC voltage. It's also important to note that the RGB outputs of the DAC are current sources (see the explanation on page 8), meaning that the voltages observed will depend on the impedance presented to the RGB outputs. What that means is that unless the monitor with your SCART cable terminates each of these three signals with 75 Ohms, the voltage won't be correct, which could explain your issue.
As a point of reference, I've measured one of the RGB outputs of an A500 at the 23-way connector, with the signal driving a 1084S test monitor, correctly displaying the RGB colour bars test screen on Amiga Test Kit at the normal/correct intensity. A correctly working A4000 will show the same waveform. See attached, which shows three complete horizontal lines. Given the large proportion of the white level, the three lines shown likely correspond to the horizontal white bars displayed on the test image. The scope is showing 500mV/division. You'll note the 'blank level' is at 500mV, while the full scale 'white level' is around 1.6V, or 600mV above the 'blank level'.
This might be a good time to check the unknowns around your SCART cable if there's any doubt with that, else you may be going down the rabbit hole of fixing something that isn't broken. Another idea is to connect the A4000 to a known good RGB test monitor such as a 1084S using the standard RGB cable, which will isolate your issue to the A4000, or to your monitor/cable.