I doubt the springs for the plunger can make that. An oval brake drum together with a poor flatspring for the brake shield may make the shield and the plunger rattle enough to touch the wheel...check the wear on the pin for the plunger tube..you don't want that to break off.
Rim offset should be a bit to the left, between 1/2" to 5/8" from the outer edge of the brake drum. And if you take the fork off, it is easier to check if it is bent somehow. Another very sensitive area are the holes for the rockers, making the wheel canted a bit. Anything that can tilt the wheel somehow. Uneven wear in bushings or shafts for the spring or rockers, unequal length of the links to the spring, they can straighten out a bit when hard bottoming out in an accident or big pothole...all that small stuff can make a difference.