It is imperative that all joints are perfectly clean if an alloy is to flow properly and produce sound joints. It is, however, immaterial as to the cleanliness at room temperature, it is the state at soldering or brazing temperature that is all important.
Excessive attention to joint cleanliness during assembly is un-necessary. No matter how much you clean the components, you are going to create more oxide from the heat source than you have so painstakingly removed.
Oxide removal is the function of the flux!
Certainly all parts should be free of oil and grease. If appropriate de-grease using warm soapy water, solvent, wire wool or a stiff wire brush. This is probably more relevant to soft soldering because the lower temperatures involved may not drive off or burn off grease from handling the components.
Do not use emery cloth or grit based products. These can leave deposits behind that the flux cannot remove leading to porous joints.