There's no problem specifying an application name with spaces in it. We do it all the time.
My guess is that your -Dappdynamics.agent.applicationName="My Application" is being processed by some other program before it gets to the java command line; that program is not handling the quotes properly.
Try escaping them? -Dappdynamics.agent.applicationName=\"My Application\" ?
Regards,
Brian Wheeldon