Remote rendering collisions Unity Hololens

Hi Jan,
Thank you very, very much for this answer.
I put here a lost image.
So my case is to check if it is possible to get point(s) of collision:

a) Remote Rendered object - let's say it is an umbrella with its attached table - with the 'local' gameobject (the one placed in Unity Scene without ARR - blue table)

b) two Remote Rendered objects - let's say the umbrella and blue table are both ARR

c) Remote Rendered object (umbrella) with real-life geometry (mesh generated with spatial mapping) - i.e. to check if the umbrella will pass through the door, or fit in the room, etc.
I already tried Remote Raycast and it works fine.
I was hoping, that spatial queries let us to make query like: get all points of collision between our local mesh and ARR mesh.
As I understand it will give only results for queries like: get all points of collision between ARR mesh and cuboid box.

Or maybe there is some query, where I can pass as parameter of any mesh (not only box)?

If there is no possibility to get points of collision, maybe there is check for cases above, like: is there any collision between those objects?

Thank you in advance for the answers :)

Unfortunately what you are asking for is a very complicated problem in general and there is no easy solution.

Neither for a), b) nor c) there is a built-in solution that would just work, simply because checking these kinds of overlaps between objects is mathematically challenging and physics engines (the code that's used for simulating physical object interactions in games and such) don't handle these kind of situtations, at all (they do things differently to avoid having to deal with such situations in the first place).

So that's why, if you were to do this only locally, using only Unity physics, you also wouldn't find an out-of-the-box easy solution and would still need to find a more creative solution.

Generally, what you can do both in ARR and also in Unity physics, is "overlap queries". Here you ask the system "does this shape overlap with anything in the scene". In ARR we call these operations "spatial queries". It is common that you can only use very simple shapes for such queries: boxes, spheres and sometimes custom convex shapes (ARR does not support the latter).

Additionally you can do raycasts. Admittedly, this is a very limited set of tools, but if you get a bit creative, many problems can be solved.

The main question is exactly what you intend to achieve. Do you want to use this for placement, so that something properly snaps to the ground? Do you want to know whether something would fit somewhere without intersecting real world objects? In case of the latter, is it sufficient to know THAT something intersects, or do you really need to know WHERE it intersects? And how precisely do you need to know the location of the intersection? Would it be sufficient to know that there is an intersection "somewhere in that area" (10cm radius or so)?

Another question is how much control you have over the meshes. Can you preprocess them? I'd guess so, since you need to convert the assets for ARR anyway.

Note that if you want to do spatial meshes from the HoloLens, you can't dynamically send them to ARR, so you would need a local solution to deal with them either way.

Anyway, a couple of ideas:

You could 'voxelize' one of the meshes. E.g. build a box or sphere approximation of it. So for example you would have 10 small spheres for each leg of the table, and a hundred spheres for the tabletop or so. If you have such a representation, you can ask ARR (or Unity) whether any of those spheres intersects with the rest of the scene. You only get a yes/no answer, but since each sphere represents a small area, you can thus pinpoint where the objects intersect. Note that such a voxelization would also be a "convex decomposition", because it represents your object with lots of convex objects (boxes or spheres). See https://github.com/kmammou/v-hacd for reference (though this generates custom convex meshes, which you won't be able to use with ARR).

Another idea would be to shoot rays downwards. Say in your example you take the bounding box of the blue table, and from its top you shoot rays downwards to the bottom of the table. Assuming the blue table is local (so not seen by ARR) the rays can't hit the blue table, only the white one. If any ray hits something within the bounds of the blue table, you have a potential overlap. You could cross check, by also shooting rays locally (against the blue table) and see if anything hits there. So for example at the rounded corner of the table, you may not hit anything in the blue table, so that area is actually empty, and thus any hits in that area in the remote model can be ignored.

As I said, this topic is very complicated and depending on your use case you can find an easy, good-enough heuristic, or you may need to do very complicated things to find a more precise answer.

Let me know whether that was helpful.

Yes, this is very, very helpful - thank you very much.

I would like just explain purpose of my questions more precisely.

I am making basic research for a client, to check if there is a possibility to detect a collision between ARR and local, between two ARRs, and between ARR and environment (on HL2 device).

I don't know currently what will be final feature to use this collision detection in ARR, just checking capabilities.

Your answers clarify a lot.

I assume from these answers, that there is no possibility to detect any collision during movement (i.e. when we are grabbing ARR model and placing it in another destination and there is some kind of obstacle). Am I right? There is no Physics Simulation API for Remote Rendering as Santhi suggested?

Once again - thank you for your time :)

Nothing like that is built-in, but there are certainly ways to achieve this. You could use Unity's physics engine to do such a collision simulation on the device. Obviously you need a highly optimized mesh for the physics simulation, but that's actually very common for everything that needs to run in real-time. There are definitely ways to achieve this, but it has to be tailored for the specific use case, there is no one-size-fits-all solution that ARR could provide for you.

Please disregard Santhi's comments, I think this account is a bot that just tries to generate credibility. There is no physics simulation API in ARR.

Hi again,

I have, probably the last question.

Is there a way to use SpatialQuerry (not Raycast), to check if there is ARR model inside volume, but not this ARR's bounding box, but mesh?

I mean: SpatialQuery returns Entity - is there a way to check or read from the result if bounds (RemoteRendering.Bounds) passed in this Spatial Query are 'touching' mesh of this ARR model?