We haven't yet looked at using Google Protocol Buffers. DevForce communications are built on a WCF stack, so we'd want to use something like protobuf-net or build some custom serializers and behaviors of our own to use protobuf. DevForce currently uses the DataContractSerializer and a custom message encoder to compress binary data, but we are aware of the size of messages and that there's certainly room for optimization.
Since the WCF stack can be customized in DevForce via configuration or code, you might be able to use protobuf-net today. It would require specifying the binding and behaviors wanted to turn off DevForce defaults, and also possibly additional attribute markup with either Code First or customized T4 templates. It would require some work, but could be worth it.
Thanks for the suggestion. Since we're moving to more fully support a broader array of client types, we are also looking at how to use more lightweight communications.