I have my wheel set up on my computer desk also and the movement can be a pain. Both my desk and chair are on wheels so I have to be very delicate with my inputs.
Replace the $2 castors with $3 ones that have a lock on them, maybe not that cheap but bunnings is pretty close to that!
Yeah had to go get a chair from the kitchen due to the wheels on my pc chair. The whole setup was uncomfortable, the pedals where almost inline the with wheel due to design of my desk, calves hurt after 20-30 laps! No more of that I hope.