So in an MP protected OS like, for example, FreeBSD, how do instructions/data get passed between applications and kernel?
Memory protection and a distinction between "kernel" and "user" spaces are not dependent upon one another. You can have one without the other.
Some operating environments / operating systems / virtual machines use invalid opcodes and exception handlers to pass data between user and kernel environments or between subsystems. For example, Virtual PC uses opcode 0Fh,0C7h,0C8h and two bytes of data representing the API function to allow guest systems to communicate with the host environment.
The subsystem/sandbox approach is widely implemented in other operating systems, including consumer products, e.g. NTVDM (Virtual DOS Machine) and WoW (Windows on Windows) on 32-bit Windows NT, WoW64 on 64-bit Windows NT, FX!32 on Windows NT/Alpha, many Unixes which allow binaries from other Unixes to run natively, and Amithlon (I'm grossly oversimplifying, but it could be viewed as an Amiga subsystem on Linux).
The problem you're trying to solve is essentially the same one Microsoft solved to allow DOS and 16-bit Windows applications to run on 32-bit Windows NT. They run in either a shared or stand-alone memory space and communicate with the operating system and hardware through a transparent abstraction layer that converts software interrupts and 16-bit Windows API calls to an appropriate 32-bit Windows NT analog.
It's possible for DOS and 16-bit Windows applications to communicate natively with applications and hardware outside their sandbox by using custom modules that expose interfaces on both sides of the wall.