Hardware only gets involved in physical memory allocation when realizing backless memory into one-line pages. No other system has one-line pages, nor backless memory, so any memory to be used by the alien core (or chip) must have been already allocated and of full alien size. Such pages are backed, not backless, and so the Mill backless support hardware would not be invoked.
Consequently, if an x86 core tried to access a Mill backless page the x86 would take a page trap, and the software would define a backing page and back the Mill caches with it rather than use the backless mechanism. When the page fault returned the two cores could share cache using the coherence mechanism and share DRAM using normal TLB entries in both. If a Mill core tried to access an x86 page then coherence would permit cache sharing up until a Mill load missed in the LLC or a store was evicted from the LLC. At that point the Mill would take a TLB miss, so the TLB entry would have to be marked as backed, by the right physical page. But that entry is either in DRAM or in coherent cache, so the only requirement is for an x86 allocation to clear an existing backless TLB entry from the Mill TLB.
Consequently I think the only problem in the two-cores-on-a-chip case is coherency. The two-chips-on-a-board problem is easy because the Mill does not extend address space off chip. DRAM allocation thus becomes yet another transaction between independent agents, and the wire protocol has to support that anyway.
I think 🙂