I would echo Aeriallive's experience. It's really a bunch of trial and error. Cover the basics first. Balance the props, make sure the motors are also balanced as well. Once that is done then the real work begins. And unfortunately that is where most of the trial and error occurs as well. Getting the kopter to be as mechanically vibration free is one thing, figuring out how to dampen the residual vibration is yet another bit of experimentation. Mainly you would be looking at how to isolate your camera mount from the kopter but still keep a positive enough connection so the camera doesn't sway too much from the mounts. Just do a search here and you will find a few folks talking about how they are dampening vibration.

I was never able to get my Hexa XL to be as vibration free as I would have liked. I switched out motors, props, and tried all sorts of vibration dampening. Nothing I tried worked to my satisfaction.  Same situation as yours. Smooth air and smooth flying gave me good results. As soon as the wind came up I would get the vibes. I switched to a different frame system and have had better luck in those situations. But there is only so much wind any multi-rotor can put up with.

I can't address your particular settings as I have never flown an OktoXL with your specific load. It's all very individual. Probably a good thing to get an Xbee so changing settings in the field are easier.

Good luck and make sure to report back here when you come up with a good solution.

I agree with all the posts above.  In addition, a bunch of us have posted our parameters on a spreadsheet here
https://docs.google.com/spreadsheet/ccc?key=0ArfrElQnmjucdFFPX3dYZ1d1VnFyLXI2S1FLQUVaemc#gid=9

That is a good place to compare.  Off hand I would say start taking the gyro D up.  I have read what ziggy says
http://www.mikrokopter.us/index.php?topic=2782.0 here and you will probably find that useful, but I have much better luck with my Gyro D at 18 and I have experimented with it as high as 25.  I fly a sony nex 5n which is much lighter than a 5d but I have a c'star6 which is much larger than an mk octo.  I have talked to folks who fly heavier cameras that liked gyro D at 22.  Keep in mind though that everyone's build is at least a little different and it is always suggested to change settings gradually

No worries, Ziggy has the gentLED for Sony IR cameras rite here in his shop.  Very easy to set up and use.

http://www.mikrokopter.us/shop/index.php?route=product/product&product_id=186

Cheers!

Strato snapper is the way to go with the NEx stuff you can trigger Video and pictures very cool.

Ok the thing that really jumped out on me was how low your Gyro D is set..
Maybe I would reset the settings to factory and start over,,
Gyro d is 10 at stock.. i'm not sure about the others, on that page off the top of my head I usually do not touch them. see the link above too, I have some really good notes there.
Moving your Gyro D down makes it slower to come to level. about 8 is as low as I would go with a stock okto XL. At 4 my bet is it doesn't sit in one place very well. Drifting all over.. I run at 8 with the same setup and if windy I bump it up a bit.
this is with .86 firmware.

The strange thing about Gyro D is with some Motors that are not tuned for the MK needs to have a Gyro D way High, for some lower quality motors I've seen as high as in the 20's my guess is it has to do with the frequency of the motors vs the BL's
Or the Kv value is higher than the MK motors and the props spin faster or something happens that need the stability to be faster, I haven't yet quite figured that one out.
if I go much above 10 the kopter jerks really fast to level itself and gives things a rough ride .

If Gyro D is set good with no GPS hold and no wind, you kopter should sit in one place really pretty. That is if your Calibrations are perfect and your load is level..

The Camera swinging from right to left on the Roll can also jerk the MK around.
I recommend turning the Roll compensation way down.. or even off.
if your camera does not sit right in the center of the mount this is really visable.
I do what ever it takes to get the lens in the center so when the mount move
its a rotation and not a swing.. this is hard with the Nex camera because of the offset.

The other thing I see is your Vario Alt,
using the Graupner radio I like it to be set to 127 not auto.. so center is hover.
and for a load I think your Min Gas is to low.. but I'm not sure what props you have but my guess is with that setting your alt hold bounces a bit. bringing that up into the 70's would probably help. But change this slowly it must be just below hover for a smooth ride. to high and it wont come down.. well it will just not with vario on.
and I hate acoustic vario.. the beeps make me think something is wrong with the kopter every-time.. Like it in my earpiece though on the Graupner radio..

hope this helps..
thanks
Ziggy

Please check everything is a much as possible in it's COG. This way the servoos don't need to work and wont overcompensate.