Yes, possibly. If you are put out for back pain, for example, and you then take a job as a day laborer or warehouse worker where you lift things all day, they may find you fit and your rating would be reduced/eliminated. It is not so much the job (you are allowed to work), but what the job may indicate about your condition that matters.