It has nothing to do with the VC in this case.
Arm cores have the concept of "secure" and "non-secure" memory. This is essentially just one extra bit of physical address that the peripherals can detect. For example, you can have a secure timer and a non-secure timer. Usually, Arm platforms have some static RAM (SRAM) that can only be accessed with secure accesses (secure SRAM). This RAM is internal to the SoC, to make it harder to attack. This can also be expanded to secure DRAM in some ways (TrustZone Controller TZC-400, for example).
Arm cores have secure and non-secure execution modes. There are 4 levels of privilege: EL0 (user), EL1 (OS), EL2 (hypervisor), EL3 (firmware). Non-secure mode has EL0, EL1 and EL2. Secure mode, S-EL0, S-EL1 and EL3. Non-secure execution levels can only access non-secure memory, secure levels can access both secure and non-secure.
The only privilege level that can change the security mode is EL3. This means that if the OS in EL1 wants to request a secure service from a secure OS in S-EL1 (OP-TEE), it has to send the request to EL3, and it will change the security mode to send the request to S-EL1, and then send the reply back to EL1.
This is secure because the non trusted software is limited and can only check non-secure memory (DRAM, mostly). That software physically can't even see whatever is in secure memory. That's why it is secure. Normally, at EL3 you have the Trusted Firmware
(or equivalent), which starts from ROM. This is secure because there is no way to modify it outside of the factory. Through cryptographic signatures the firmware loads the software that runs at S-EL1 into secure SRAM. This can't be accessed by software means or hardware means (you need to decap the chip and probe the conectors inside, which is crazy hard, but it can be done!). Once all secure software is loaded in secure memory and it is initialized, EL3 loads U-Boot or UEFI into non-secure memory and that continues the regular execution that you see (Linux, etc).
The Raspberry Pi 3 could be secure because after the software in the SD card is loaded you could technically block the non-secure world from accessing the SD card, and you would always boot the same software that you trust. However, this is pointless for the following reason.
The Raspberry Pi 3 is not secure because it doesn't have secure memory. There is no way to hide any memory from the normal world. Software at EL3 can see all the memory of the device, but so can EL2, and this is the level at which U-Boot/UEFI runs (Linux can also start running at EL2, it leaves a service handler to deal with hardware virtualization and jumps to EL1). In short, you could load secure software from EL3, but your non-secure software can see it. There is no way to hide it.
The only real option to hide secure software from non-secure software is to use EL2 to hide the memory from EL1. EL2 has hardware support to fake a memory map for EL1. You can map all memory of the Raspberry Pi except for the space used by secure software. You can boot Linux at EL1 so it can't have more privileges, and you can use EL2 to virtualize Linux and hide the "secure memory". This is a real option, but you lose virtualization support. However, I'd be surprised if someone was sad about losing support for virtualization in this device... Also, it's only secure from software, the accesses to DRAM could be intercepted by an external digital analyzer.
PS: The VC can also access all the memory of the device, but I don't think that it can be leaked in any way using the VC. I may be wrong, though.
EDIT: By the way, there is a port of the Trusted Firmware to the RPi3, you can see that in the repository. There's also some effort ongoing to have OP-TEE and U-Boot working at the same time. But again, this is not secure. However, that doesn't mean it's worthless. I think it can be quite useful to see how things work.