First, you have 3 carbs to set the idle mixture with & you're only adjusting one of them. Lewtot's explanation sounds reasonable, so your choices would be to remove the lead plugs & adjust the idle mixture on all 3 carbs OR drill/modify the center carb. I went the remove the lead plugs route.
Brian6pac makes another good point that has always frustrated me with 6-packs & bigger cams. If you have to turn the idle screw on the center carb more than just a little bit off of "completely closed", then the center carb's throttle blades expose the "transfer slots" and idle gets really, really hard to set right since the center carburetor isn't on the idle circuit anymore. I had this problem. When I "fattened up" the mixture on the end carbs I was able to close the center carb enough to cover the transfer slots & then everything was kosher. Before that, my idle mixture screws on the center carb didn't seem to do much at all.
One last possibility..... after setting your idle, did you re-check & re-set the throttle return linkage to the outer carbs? One of the old tricks to get 6-packs to idle was to drill (or enlarge) the holes in the end carburetor throttle plates to allow for more air flow at idle. Some people would loosen the throttle return linkage to the outer carbs about 1 turn which does basically the same thing because when you take your foot off the gas the center carb doesn't pull the outer carbs 100% closed (leaves them cracked open slightly). I tried this approach myself, but found that my idle would do what you describe. "Sometimes" it would return to idle normally (end carbs still cracked open slightly) & other times the idle would drop a lot like you describe (carbs fully closed) & it seemed hit or miss. When you leave that little bit of slop in the throttle return linkage that last little bit of opening/closing of the end carbs is not controlled. The throttle return linkage should be "very slightly snug" in the end carbs & you have to re-adjust that linkage every time you adjust the idle screw on the center carb.