nicolas/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[ORC][ORC-RT] Rewrite the MachO platform to use allocation actions.

Browse Source 
This patch updates the MachO platform (both the ORC MachOPlatform class and the
ORC-Runtime macho_platform.* files) to use allocation actions, rather than EPC
calls, to transfer the initializer information scraped from each linked object.
Interactions between the ORC and ORC-Runtime sides of the platform are
substantially redesigned to accomodate the change.

The high-level changes in this patch are:

1. The MachOPlatform::setupJITDylib method now calls into the runtime to set up
   a dylib name <-> header mapping, and a dylib state object (JITDylibState).

2. The MachOPlatformPlugin builds an allocation action that calls the
   __orc_rt_macho_register_object_platform_sections and
   __orc_rt_macho_deregister_object_platform_sections functions in the runtime
   to register the address ranges for all "interesting" sections in the object
   being allocated (TLS data sections, initializers, language runtime metadata
   sections, etc.).

3. The MachOPlatform::rt_getInitializers method (the entry point in the
   controller for requests from the runtime for initializer information) is
   replaced by MachOPlatform::rt_pushInitializers. The former returned a data
   structure containing the "interesting" section address ranges, but these are
   now handled by __orc_rt_macho_register_object_platform_sections. The new
   rt_pushInitializers method first issues a lookup to trigger materialization
   of the "interesting" sections, then returns the dylib dependence tree rooted
   at the requested dylib for dlopen to consume. (The dylib dependence tree is
   returned by rt_pushInitializers, rather than being handled by some dedicated
   call, because rt_pushInitializers can alter the dependence tree).

The advantage of these changes (beyond the performance advantages of using
allocation actions) is that it moves more information about the materialized
portions of the JITDylib into the executor. This tends to make the runtime
easier to reason about, e.g. the implementation of dlopen in the runtime is now
recursive, rather than relying on recursive calls in the controller to build a
linear data structure for consumption by the runtime. This change can also make
some operations more efficient, e.g. JITDylibs can be dlclosed and then
re-dlopened without having to pull all initializers over from the controller
again.

In addition to the high-level changes, there are some low-level changes to ORC
and the runtime:

* In ORC, at ExecutionSession teardown time JITDylibs are now destroyed in
reverse creation order. This is on the assumption that the ORC runtime will be
loaded into an earlier dylib that will be used by later JITDylibs. This is a
short-term solution to crashes that arose during testing when the runtime was
torn down before its users. Longer term we will likely destroy dylibs in
dependence order.

* toSPSSerializable(Expected<T> E) is updated to explicitly initialize the T
value, allowing it to be used by Ts that have explicit constructors.

* The ORC runtime now (1) attempts to track ref-counts, and (2) distinguishes
not-yet-processed "interesting" sections from previously processed ones. (1)
is necessary for standard dlopen/dlclose emulation. (2) is intended as a step
towards better REPL support -- it should enable future runtime calls that
run only newly registered initializers ("dlopen_more", "dlopen_additions",
...?).
This commit is contained in:
Lang Hames
2022-02-08 16:31:17 +11:00
parent f237ab0dd1
commit f9aef477eb
9 changed files with 1083 additions and 637 deletions
Download Patch File Download Diff File Expand all files Collapse all files

919
compiler-rt/lib/orc/macho_platform.cpp
View File

File diff suppressed because it is too large Load Diff

59
compiler-rt/lib/orc/macho_platform.h
View File

@@ -31,28 +31,6 @@ ORC_RT_INTERFACE void *__orc_rt_macho_jit_dlsym(void *dso_handle,
namespace __orc_rt {
namespace macho {
struct MachOJITDylibInitializers {
using SectionList = std::vector<ExecutorAddrRange>;
MachOJITDylibInitializers() = default;
MachOJITDylibInitializers(std::string Name, ExecutorAddr MachOHeaderAddress)
: Name(std::move(Name)),
MachOHeaderAddress(std::move(MachOHeaderAddress)) {}
std::string Name;
ExecutorAddr MachOHeaderAddress;
ExecutorAddr ObjCImageInfoAddress;
std::unordered_map<std::string, SectionList> InitSections;
};
class MachOJITDylibDeinitializers {};
using MachOJITDylibInitializerSequence = std::vector<MachOJITDylibInitializers>;
using MachOJITDylibDeinitializerSequence =
std::vector<MachOJITDylibDeinitializers>;
enum dlopen_mode : int {
ORC_RT_RTLD_LAZY = 0x1,
ORC_RT_RTLD_NOW = 0x2,
@@ -61,43 +39,6 @@ enum dlopen_mode : int {
};
} // end namespace macho
using SPSNamedExecutorAddrRangeSequenceMap =
SPSSequence<SPSTuple<SPSString, SPSExecutorAddrRangeSequence>>;
using SPSMachOJITDylibInitializers =
SPSTuple<SPSString, SPSExecutorAddr, SPSExecutorAddr,
SPSNamedExecutorAddrRangeSequenceMap>;
using SPSMachOJITDylibInitializerSequence =
SPSSequence<SPSMachOJITDylibInitializers>;
/// Serialization traits for MachOJITDylibInitializers.
template <>
class SPSSerializationTraits<SPSMachOJITDylibInitializers,
macho::MachOJITDylibInitializers> {
public:
static size_t size(const macho::MachOJITDylibInitializers &MOJDIs) {
return SPSMachOJITDylibInitializers::AsArgList::size(
MOJDIs.Name, MOJDIs.MachOHeaderAddress, MOJDIs.ObjCImageInfoAddress,
MOJDIs.InitSections);
}
static bool serialize(SPSOutputBuffer &OB,
const macho::MachOJITDylibInitializers &MOJDIs) {
return SPSMachOJITDylibInitializers::AsArgList::serialize(
OB, MOJDIs.Name, MOJDIs.MachOHeaderAddress, MOJDIs.ObjCImageInfoAddress,
MOJDIs.InitSections);
}
static bool deserialize(SPSInputBuffer &IB,
macho::MachOJITDylibInitializers &MOJDIs) {
return SPSMachOJITDylibInitializers::AsArgList::deserialize(
IB, MOJDIs.Name, MOJDIs.MachOHeaderAddress, MOJDIs.ObjCImageInfoAddress,
MOJDIs.InitSections);
}
};
} // end namespace __orc_rt
#endif // ORC_RT_MACHO_PLATFORM_H

41
compiler-rt/test/orc/TestCases/Darwin/x86-64/Inputs/standalone-ctor-and-cxa-atexit-dtor.S Normal file
View File

@@ -0,0 +1,41 @@
// Contains a static initializer and deinitializer registered with
// ___cxa_atexit. dlopen-ing/dlclose-ing will print "constructor" and
// "destructor" respectively.
.section __TEXT,__text,regular,pure_instructions
.build_version macos, 12, 0 sdk_version 12, 0, 1
.globl _deinitializer
.p2align 4, 0x90
_deinitializer:
pushq %rbp
movq %rsp, %rbp
leaq L_str.2(%rip), %rdi
popq %rbp
jmp _puts
.section __TEXT,__StaticInit,regular,pure_instructions
.p2align 4, 0x90
_initializer:
pushq %rbp
movq %rsp, %rbp
leaq L_str(%rip), %rdi
callq _puts
movq _deinitializer@GOTPCREL(%rip), %rdi
leaq _I(%rip), %rsi
leaq ___dso_handle(%rip), %rdx
popq %rbp
jmp ___cxa_atexit
.globl _I
.zerofill __DATA,__common,_I,1,0
.section __DATA,__mod_init_func,mod_init_funcs
.p2align 3
.quad _initializer
.section __TEXT,__cstring,cstring_literals
L_str:
.asciz "constructor"
L_str.2:
.asciz "destructor"
.subsections_via_symbols

119
compiler-rt/test/orc/TestCases/Darwin/x86-64/trivial-jit-dlopen.S Normal file
View File

@@ -0,0 +1,119 @@
// Test that __orc_rt_macho_jit_dlopen and __orc_rt_macho_jit_dlclose run
// constructors and destructors as expected.
//
// This test calls dlopen and dlclose twice. We expect the inner calls to be
// no-ops.
//
// RUN: %clang -c -o %t.inits.o %p/Inputs/standalone-ctor-and-cxa-atexit-dtor.S
// RUN: %clang -c -o %t.test.o %s
// RUN: %llvm_jitlink \
// RUN: -alias _dlopen=___orc_rt_macho_jit_dlopen \
// RUN: -alias _dlclose=___orc_rt_macho_jit_dlclose \
// RUN: %t.test.o -jd inits %t.inits.o -lmain | FileCheck %s
//
// CHECK: entering main
// CHECK-NEXT: first dlopen
// CHECK-NEXT: constructor
// CHECK-NEXT: second dlopen
// CHECK-NEXT: first dlclose
// CHECK-NEXT: second dlclose
// CHECK-NEXT: destructor
// CHECK-NEXT: leaving main
.section __TEXT,__text,regular,pure_instructions
.build_version macos, 12, 0 sdk_version 13, 0
.globl _main
.p2align 4, 0x90
_main:
pushq %rbp
movq %rsp, %rbp
pushq %r15
pushq %r14
pushq %rbx
pushq %rax
leaq L_str(%rip), %rdi
callq _puts
leaq L_str.9(%rip), %rdi
callq _puts
leaq L_.str.2(%rip), %rdi
movl $1, %esi
callq _dlopen
movl $-1, %r14d
leaq L_str.17(%rip), %r15
testq %rax, %rax
je LBB0_6
movq %rax, %rbx
leaq L_str.11(%rip), %rdi
callq _puts
leaq L_.str.2(%rip), %rdi
movl $1, %esi
callq _dlopen
testq %rax, %rax
je LBB0_6
cmpq %rbx, %rax
je LBB0_4
leaq L_str.18(%rip), %r15
jmp LBB0_6
LBB0_4:
leaq L_str.13(%rip), %rdi
callq _puts
movq %rbx, %rdi
callq _dlclose
cmpl $-1, %eax
je LBB0_6
leaq L_str.14(%rip), %rdi
callq _puts
movq %rbx, %rdi
callq _dlclose
xorl %r14d, %r14d
cmpl $-1, %eax
sete %r14b
leaq L_str.17(%rip), %rax
leaq L_str.15(%rip), %r15
cmoveq %rax, %r15
negl %r14d
LBB0_6:
movq %r15, %rdi
callq _puts
movl %r14d, %eax
addq $8, %rsp
popq %rbx
popq %r14
popq %r15
popq %rbp
retq
.section __TEXT,__cstring,cstring_literals
L_.str.2:
.asciz "inits"
L_str:
.asciz "entering main"
L_str.9:
.asciz "first dlopen"
L_str.11:
.asciz "second dlopen"
L_str.13:
.asciz "first dlclose"
L_str.14:
.asciz "second dlclose"
L_str.15:
.asciz "leaving main"
L_str.17:
.asciz "failed"
L_str.18:
.asciz "handles do not match"
.subsections_via_symbols

3
compiler-rt/test/orc/lit.cfg.py
View File

@@ -29,5 +29,8 @@ config.substitutions.append(
# Default test suffixes.
config.suffixes = ['.c', '.cpp', '.S']
# Exclude Inputs directories.
config.excludes = ['Inputs']
if config.host_os not in ['Darwin', 'FreeBSD', 'Linux']:
config.unsupported = True

146
llvm/include/llvm/ExecutionEngine/Orc/MachOPlatform.h
View File

@@ -26,30 +26,19 @@
namespace llvm {
namespace orc {
struct MachOJITDylibInitializers {
using SectionList = std::vector<ExecutorAddrRange>;
MachOJITDylibInitializers(std::string Name, ExecutorAddr MachOHeaderAddress)
: Name(std::move(Name)),
MachOHeaderAddress(std::move(MachOHeaderAddress)) {}
std::string Name;
ExecutorAddr MachOHeaderAddress;
ExecutorAddr ObjCImageInfoAddress;
StringMap<SectionList> InitSections;
};
class MachOJITDylibDeinitializers {};
using MachOJITDylibInitializerSequence = std::vector<MachOJITDylibInitializers>;
using MachOJITDylibDeinitializerSequence =
std::vector<MachOJITDylibDeinitializers>;
/// Mediates between MachO initialization and ExecutionSession state.
class MachOPlatform : public Platform {
public:
// Used internally by MachOPlatform, but made public to enable serialization.
struct MachOJITDylibDepInfo {
bool Sealed = false;
std::vector<ExecutorAddr> DepHeaders;
};
// Used internally by MachOPlatform, but made public to enable serialization.
using MachOJITDylibDepInfoMap =
std::vector<std::pair<ExecutorAddr, MachOJITDylibDepInfo>>;
/// Try to create a MachOPlatform instance, adding the ORC runtime to the
/// given JITDylib.
///
@@ -161,26 +150,28 @@ private:
Error processObjCImageInfo(jitlink::LinkGraph &G,
MaterializationResponsibility &MR);
Error registerInitSections(jitlink::LinkGraph &G, JITDylib &JD);
Error fixTLVSectionsAndEdges(jitlink::LinkGraph &G, JITDylib &JD);
Error registerEHAndTLVSections(jitlink::LinkGraph &G);
Error registerObjectPlatformSections(jitlink::LinkGraph &G, JITDylib &JD);
Error registerEHSectionsPhase1(jitlink::LinkGraph &G);
std::mutex PluginMutex;
MachOPlatform &MP;
// FIXME: ObjCImageInfos and HeaderAddrs need to be cleared when
// JITDylibs are removed.
DenseMap<JITDylib *, std::pair<uint32_t, uint32_t>> ObjCImageInfos;
DenseMap<JITDylib *, ExecutorAddr> HeaderAddrs;
InitSymbolDepMap InitSymbolDeps;
};
using SendInitializerSequenceFn =
unique_function<void(Expected<MachOJITDylibInitializerSequence>)>;
using SendDeinitializerSequenceFn =
unique_function<void(Expected<MachOJITDylibDeinitializerSequence>)>;
using GetJITDylibHeaderSendResultFn =
unique_function<void(Expected<ExecutorAddr>)>;
using GetJITDylibNameSendResultFn =
unique_function<void(Expected<StringRef>)>;
using PushInitializersSendResultFn =
unique_function<void(Expected<MachOJITDylibDepInfoMap>)>;
using SendSymbolAddressFn = unique_function<void(Expected<ExecutorAddr>)>;
static bool supportedTarget(const Triple &TT);
@@ -193,28 +184,24 @@ private:
// Associate MachOPlatform JIT-side runtime support functions with handlers.
Error associateRuntimeSupportFunctions(JITDylib &PlatformJD);
void getInitializersBuildSequencePhase(SendInitializerSequenceFn SendResult,
JITDylib &JD,
std::vector<JITDylibSP> DFSLinkOrder);
// Implements rt_pushInitializers by making repeat async lookups for
// initializer symbols (each lookup may spawn more initializer symbols if
// it pulls in new materializers, e.g. from objects in a static library).
void pushInitializersLoop(PushInitializersSendResultFn SendResult,
JITDylibSP JD);
void getInitializersLookupPhase(SendInitializerSequenceFn SendResult,
JITDylib &JD);
void rt_getInitializers(SendInitializerSequenceFn SendResult,
StringRef JDName);
void rt_getDeinitializers(SendDeinitializerSequenceFn SendResult,
ExecutorAddr Handle);
// Handle requests from the ORC runtime to push MachO initializer info.
void rt_pushInitializers(PushInitializersSendResultFn SendResult,
ExecutorAddr JDHeaderAddr);
// Handle requests for symbol addresses from the ORC runtime.
void rt_lookupSymbol(SendSymbolAddressFn SendResult, ExecutorAddr Handle,
StringRef SymbolName);
// Records the addresses of runtime symbols used by the platform.
Error bootstrapMachORuntime(JITDylib &PlatformJD);
Error registerInitInfo(JITDylib &JD, ExecutorAddr ObjCImageInfoAddr,
ArrayRef<jitlink::Section *> InitSections);
// Call the ORC runtime to create a pthread key.
Expected<uint64_t> createPThreadKey();
enum PlatformState { BootstrapPhase1, BootstrapPhase2, Initialized };
@@ -229,81 +216,24 @@ private:
ExecutorAddr orc_rt_macho_platform_shutdown;
ExecutorAddr orc_rt_macho_register_ehframe_section;
ExecutorAddr orc_rt_macho_deregister_ehframe_section;
ExecutorAddr orc_rt_macho_register_thread_data_section;
ExecutorAddr orc_rt_macho_deregister_thread_data_section;
ExecutorAddr orc_rt_macho_register_jitdylib;
ExecutorAddr orc_rt_macho_deregister_jitdylib;
ExecutorAddr orc_rt_macho_register_object_platform_sections;
ExecutorAddr orc_rt_macho_deregister_object_platform_sections;
ExecutorAddr orc_rt_macho_create_pthread_key;
DenseMap<JITDylib *, SymbolLookupSet> RegisteredInitSymbols;
// InitSeqs gets its own mutex to avoid locking the whole session when
// aggregating data from the jitlink.
std::mutex PlatformMutex;
DenseMap<JITDylib *, MachOJITDylibInitializers> InitSeqs;
DenseMap<JITDylib *, ExecutorAddr> JITDylibToHeaderAddr;
DenseMap<ExecutorAddr, JITDylib *> HeaderAddrToJITDylib;
DenseMap<JITDylib *, uint64_t> JITDylibToPThreadKey;
};
namespace shared {
using SPSNamedExecutorAddrRangeSequenceMap =
SPSSequence<SPSTuple<SPSString, SPSExecutorAddrRangeSequence>>;
using SPSMachOJITDylibInitializers =
SPSTuple<SPSString, SPSExecutorAddr, SPSExecutorAddr,
SPSNamedExecutorAddrRangeSequenceMap>;
using SPSMachOJITDylibInitializerSequence =
SPSSequence<SPSMachOJITDylibInitializers>;
/// Serialization traits for MachOJITDylibInitializers.
template <>
class SPSSerializationTraits<SPSMachOJITDylibInitializers,
MachOJITDylibInitializers> {
public:
static size_t size(const MachOJITDylibInitializers &MOJDIs) {
return SPSMachOJITDylibInitializers::AsArgList::size(
MOJDIs.Name, MOJDIs.MachOHeaderAddress, MOJDIs.ObjCImageInfoAddress,
MOJDIs.InitSections);
}
static bool serialize(SPSOutputBuffer &OB,
const MachOJITDylibInitializers &MOJDIs) {
return SPSMachOJITDylibInitializers::AsArgList::serialize(
OB, MOJDIs.Name, MOJDIs.MachOHeaderAddress, MOJDIs.ObjCImageInfoAddress,
MOJDIs.InitSections);
}
static bool deserialize(SPSInputBuffer &IB,
MachOJITDylibInitializers &MOJDIs) {
return SPSMachOJITDylibInitializers::AsArgList::deserialize(
IB, MOJDIs.Name, MOJDIs.MachOHeaderAddress, MOJDIs.ObjCImageInfoAddress,
MOJDIs.InitSections);
}
};
using SPSMachOJITDylibDeinitializers = SPSEmpty;
using SPSMachOJITDylibDeinitializerSequence =
SPSSequence<SPSMachOJITDylibDeinitializers>;
template <>
class SPSSerializationTraits<SPSMachOJITDylibDeinitializers,
MachOJITDylibDeinitializers> {
public:
static size_t size(const MachOJITDylibDeinitializers &MOJDDs) { return 0; }
static bool serialize(SPSOutputBuffer &OB,
const MachOJITDylibDeinitializers &MOJDDs) {
return true;
}
static bool deserialize(SPSInputBuffer &IB,
MachOJITDylibDeinitializers &MOJDDs) {
MOJDDs = MachOJITDylibDeinitializers();
return true;
}
};
using SPSNamedExecutorAddrRangeSequence =
SPSSequence<SPSTuple<SPSString, SPSExecutorAddrRange>>;
} // end namespace shared
} // end namespace orc

2
llvm/include/llvm/ExecutionEngine/Orc/Shared/SimplePackedSerialization.h
View File

@@ -586,7 +586,7 @@ SPSSerializableExpected<T> toSPSSerializable(Expected<T> E) {
if (E)
return {true, std::move(*E), {}};
else
return {false, {}, toString(E.takeError())};
return {false, T(), toString(E.takeError())};
}
template <typename T>

2
llvm/lib/ExecutionEngine/Orc/Core.cpp
View File

@@ -1868,7 +1868,7 @@ Error ExecutionSession::endSession() {
// TODO: notifiy platform? run static deinits?
Error Err = Error::success();
for (auto &JD : JITDylibsToClose)
for (auto &JD : reverse(JITDylibsToClose))
Err = joinErrors(std::move(Err), JD->clear());
Err = joinErrors(std::move(Err), EPC->disconnect());

429
llvm/lib/ExecutionEngine/Orc/MachOPlatform.cpp
View File

@@ -22,6 +22,39 @@ using namespace llvm;
using namespace llvm::orc;
using namespace llvm::orc::shared;
namespace llvm {
namespace orc {
namespace shared {
using SPSMachOJITDylibDepInfo = SPSTuple<bool, SPSSequence<SPSExecutorAddr>>;
using SPSMachOJITDylibDepInfoMap =
SPSSequence<SPSTuple<SPSExecutorAddr, SPSMachOJITDylibDepInfo>>;
template <>
class SPSSerializationTraits<SPSMachOJITDylibDepInfo,
MachOPlatform::MachOJITDylibDepInfo> {
public:
static size_t size(const MachOPlatform::MachOJITDylibDepInfo &DDI) {
return SPSMachOJITDylibDepInfo::AsArgList::size(DDI.Sealed, DDI.DepHeaders);
}
static bool serialize(SPSOutputBuffer &OB,
const MachOPlatform::MachOJITDylibDepInfo &DDI) {
return SPSMachOJITDylibDepInfo::AsArgList::serialize(OB, DDI.Sealed,
DDI.DepHeaders);
}
static bool deserialize(SPSInputBuffer &IB,
MachOPlatform::MachOJITDylibDepInfo &DDI) {
return SPSMachOJITDylibDepInfo::AsArgList::deserialize(IB, DDI.Sealed,
DDI.DepHeaders);
}
};
} // namespace shared
} // namespace orc
} // namespace llvm
namespace {
class MachOHeaderMaterializationUnit : public MaterializationUnit {
@@ -199,11 +232,25 @@ MachOPlatform::Create(ExecutionSession &ES, ObjectLinkingLayer &ObjLinkingLayer,
}
Error MachOPlatform::setupJITDylib(JITDylib &JD) {
return JD.define(std::make_unique<MachOHeaderMaterializationUnit>(
*this, MachOHeaderStartSymbol));
if (auto Err = JD.define(std::make_unique<MachOHeaderMaterializationUnit>(
*this, MachOHeaderStartSymbol)))
return Err;
return ES.lookup({&JD}, MachOHeaderStartSymbol).takeError();
}
Error MachOPlatform::teardownJITDylib(JITDylib &JD) { return Error::success(); }
Error MachOPlatform::teardownJITDylib(JITDylib &JD) {
std::lock_guard<std::mutex> Lock(PlatformMutex);
auto I = JITDylibToHeaderAddr.find(&JD);
if (I != JITDylibToHeaderAddr.end()) {
assert(HeaderAddrToJITDylib.count(I->second) &&
"HeaderAddrToJITDylib missing entry");
HeaderAddrToJITDylib.erase(I->second);
JITDylibToHeaderAddr.erase(I);
}
JITDylibToPThreadKey.erase(&JD);
return Error::success();
}
Error MachOPlatform::notifyAdding(ResourceTracker &RT,
const MaterializationUnit &MU) {
@@ -305,16 +352,6 @@ MachOPlatform::MachOPlatform(
State = BootstrapPhase2;
// PlatformJD hasn't been 'set-up' by the platform yet (since we're creating
// the platform now), so set it up.
if (auto E2 = setupJITDylib(PlatformJD)) {
Err = std::move(E2);
return;
}
RegisteredInitSymbols[&PlatformJD].add(
MachOHeaderStartSymbol, SymbolLookupFlags::WeaklyReferencedSymbol);
// Associate wrapper function tags with JIT-side function implementations.
if (auto E2 = associateRuntimeSupportFunctions(PlatformJD)) {
Err = std::move(E2);
@@ -329,23 +366,24 @@ MachOPlatform::MachOPlatform(
return;
}
// PlatformJD hasn't been set up by the platform yet (since we're creating
// the platform now), so set it up.
if (auto E2 = setupJITDylib(PlatformJD)) {
Err = std::move(E2);
return;
}
State = Initialized;
}
Error MachOPlatform::associateRuntimeSupportFunctions(JITDylib &PlatformJD) {
ExecutionSession::JITDispatchHandlerAssociationMap WFs;
using GetInitializersSPSSig =
SPSExpected<SPSMachOJITDylibInitializerSequence>(SPSString);
WFs[ES.intern("___orc_rt_macho_get_initializers_tag")] =
ES.wrapAsyncWithSPS<GetInitializersSPSSig>(
this, &MachOPlatform::rt_getInitializers);
using GetDeinitializersSPSSig =
SPSExpected<SPSMachOJITDylibDeinitializerSequence>(SPSExecutorAddr);
WFs[ES.intern("___orc_rt_macho_get_deinitializers_tag")] =
ES.wrapAsyncWithSPS<GetDeinitializersSPSSig>(
this, &MachOPlatform::rt_getDeinitializers);
using PushInitializersSPSSig =
SPSExpected<SPSMachOJITDylibDepInfoMap>(SPSExecutorAddr);
WFs[ES.intern("___orc_rt_macho_push_initializers_tag")] =
ES.wrapAsyncWithSPS<PushInitializersSPSSig>(
this, &MachOPlatform::rt_pushInitializers);
using LookupSymbolSPSSig =
SPSExpected<SPSExecutorAddr>(SPSExecutorAddr, SPSString);
@@ -356,53 +394,83 @@ Error MachOPlatform::associateRuntimeSupportFunctions(JITDylib &PlatformJD) {
return ES.registerJITDispatchHandlers(PlatformJD, std::move(WFs));
}
void MachOPlatform::getInitializersBuildSequencePhase(
SendInitializerSequenceFn SendResult, JITDylib &JD,
std::vector<JITDylibSP> DFSLinkOrder) {
MachOJITDylibInitializerSequence FullInitSeq;
{
std::lock_guard<std::mutex> Lock(PlatformMutex);
for (auto &InitJD : reverse(DFSLinkOrder)) {
LLVM_DEBUG({
dbgs() << "MachOPlatform: Appending inits for \"" << InitJD->getName()
<< "\" to sequence\n";
});
auto ISItr = InitSeqs.find(InitJD.get());
if (ISItr != InitSeqs.end()) {
FullInitSeq.emplace_back(std::move(ISItr->second));
InitSeqs.erase(ISItr);
}
}
}
SendResult(std::move(FullInitSeq));
}
void MachOPlatform::getInitializersLookupPhase(
SendInitializerSequenceFn SendResult, JITDylib &JD) {
auto DFSLinkOrder = JD.getDFSLinkOrder();
if (!DFSLinkOrder) {
SendResult(DFSLinkOrder.takeError());
return;
}
void MachOPlatform::pushInitializersLoop(
PushInitializersSendResultFn SendResult, JITDylibSP JD) {
DenseMap<JITDylib *, SymbolLookupSet> NewInitSymbols;
DenseMap<JITDylib *, SmallVector<JITDylib *>> JDDepMap;
SmallVector<JITDylib *, 16> Worklist({JD.get()});
ES.runSessionLocked([&]() {
for (auto &InitJD : *DFSLinkOrder) {
auto RISItr = RegisteredInitSymbols.find(InitJD.get());
while (!Worklist.empty()) {
// FIXME: Check for defunct dylibs.
auto DepJD = Worklist.back();
Worklist.pop_back();
// If we've already visited this JITDylib on this iteration then continue.
if (JDDepMap.count(DepJD))
continue;
// Add dep info.
auto &DM = JDDepMap[DepJD];
DepJD->withLinkOrderDo([&](const JITDylibSearchOrder &O) {
for (auto &KV : O) {
if (KV.first == DepJD)
continue;
DM.push_back(KV.first);
Worklist.push_back(KV.first);
}
});
// Add any registered init symbols.
auto RISItr = RegisteredInitSymbols.find(DepJD);
if (RISItr != RegisteredInitSymbols.end()) {
NewInitSymbols[InitJD.get()] = std::move(RISItr->second);
NewInitSymbols[DepJD] = std::move(RISItr->second);
RegisteredInitSymbols.erase(RISItr);
}
}
});
// If there are no further init symbols to look up then move on to the next
// phase.
// If there are no further init symbols to look up then send the link order
// (as a list of header addresses) to the caller.
if (NewInitSymbols.empty()) {
getInitializersBuildSequencePhase(std::move(SendResult), JD,
std::move(*DFSLinkOrder));
// To make the list intelligible to the runtime we need to convert all
// JITDylib pointers to their header addresses.
DenseMap<JITDylib *, ExecutorAddr> HeaderAddrs;
HeaderAddrs.reserve(JDDepMap.size());
{
std::lock_guard<std::mutex> Lock(PlatformMutex);
for (auto &KV : JDDepMap) {
auto I = JITDylibToHeaderAddr.find(KV.first);
if (I == JITDylibToHeaderAddr.end()) {
// The header address should have been materialized by the previous
// round, but we need to handle the pathalogical case where someone
// removes the symbol on another thread while we're running.
SendResult(
make_error<StringError>("JITDylib " + KV.first->getName() +
" has no registered header address",
inconvertibleErrorCode()));
return;
}
HeaderAddrs[KV.first] = I->second;
}
}
// Build the dep info map to return.
MachOJITDylibDepInfoMap DIM;
DIM.reserve(JDDepMap.size());
for (auto &KV : JDDepMap) {
assert(HeaderAddrs.count(KV.first) && "Missing header addr");
auto H = HeaderAddrs[KV.first];
MachOJITDylibDepInfo DepInfo;
for (auto &Dep : KV.second) {
assert(HeaderAddrs.count(Dep) && "Missing header addr");
DepInfo.DepHeaders.push_back(HeaderAddrs[Dep]);
}
DIM.push_back(std::make_pair(H, std::move(DepInfo)));
}
SendResult(DIM);
return;
}
@@ -412,58 +480,38 @@ void MachOPlatform::getInitializersLookupPhase(
if (Err)
SendResult(std::move(Err));
else
getInitializersLookupPhase(std::move(SendResult), JD);
pushInitializersLoop(std::move(SendResult), JD);
},
ES, std::move(NewInitSymbols));
}
void MachOPlatform::rt_getInitializers(SendInitializerSequenceFn SendResult,
StringRef JDName) {
LLVM_DEBUG({
dbgs() << "MachOPlatform::rt_getInitializers(\"" << JDName << "\")\n";
});
JITDylib *JD = ES.getJITDylibByName(JDName);
if (!JD) {
LLVM_DEBUG({
dbgs() << " No such JITDylib \"" << JDName << "\". Sending error.\n";
});
SendResult(make_error<StringError>("No JITDylib named " + JDName,
inconvertibleErrorCode()));
return;
}
getInitializersLookupPhase(std::move(SendResult), *JD);
}
void MachOPlatform::rt_getDeinitializers(SendDeinitializerSequenceFn SendResult,
ExecutorAddr Handle) {
LLVM_DEBUG({
dbgs() << "MachOPlatform::rt_getDeinitializers(\""
<< formatv("{0:x}", Handle.getValue()) << "\")\n";
});
JITDylib *JD = nullptr;
void MachOPlatform::rt_pushInitializers(PushInitializersSendResultFn SendResult,
ExecutorAddr JDHeaderAddr) {
JITDylibSP JD;
{
std::lock_guard<std::mutex> Lock(PlatformMutex);
auto I = HeaderAddrToJITDylib.find(Handle);
auto I = HeaderAddrToJITDylib.find(JDHeaderAddr);
if (I != HeaderAddrToJITDylib.end())
JD = I->second;
}
LLVM_DEBUG({
dbgs() << "MachOPlatform::rt_pushInitializers(" << JDHeaderAddr << ") ";
if (JD)
dbgs() << "pushing initializers for " << JD->getName() << "\n";
else
dbgs() << "No JITDylib for header address.\n";
});
if (!JD) {
LLVM_DEBUG({
dbgs() << " No JITDylib for handle "
<< formatv("{0:x}", Handle.getValue()) << "\n";
});
SendResult(make_error<StringError>("No JITDylib associated with handle " +
formatv("{0:x}", Handle.getValue()),
inconvertibleErrorCode()));
SendResult(
make_error<StringError>("No JITDylib with header addr " +
formatv("{0:x}", JDHeaderAddr.getValue()),
inconvertibleErrorCode()));
return;
}
SendResult(MachOJITDylibDeinitializerSequence());
pushInitializersLoop(std::move(SendResult), JD);
}
void MachOPlatform::rt_lookupSymbol(SendSymbolAddressFn SendResult,
@@ -526,10 +574,14 @@ Error MachOPlatform::bootstrapMachORuntime(JITDylib &PlatformJD) {
&orc_rt_macho_platform_bootstrap},
{ES.intern("___orc_rt_macho_platform_shutdown"),
&orc_rt_macho_platform_shutdown},
{ES.intern("___orc_rt_macho_register_thread_data_section"),
&orc_rt_macho_register_thread_data_section},
{ES.intern("___orc_rt_macho_deregister_thread_data_section"),
&orc_rt_macho_deregister_thread_data_section},
{ES.intern("___orc_rt_macho_register_jitdylib"),
&orc_rt_macho_register_jitdylib},
{ES.intern("___orc_rt_macho_deregister_jitdylib"),
&orc_rt_macho_deregister_jitdylib},
{ES.intern("___orc_rt_macho_register_object_platform_sections"),
&orc_rt_macho_register_object_platform_sections},
{ES.intern("___orc_rt_macho_deregister_object_platform_sections"),
&orc_rt_macho_deregister_object_platform_sections},
{ES.intern("___orc_rt_macho_create_pthread_key"),
&orc_rt_macho_create_pthread_key}}))
return Err;
@@ -537,45 +589,6 @@ Error MachOPlatform::bootstrapMachORuntime(JITDylib &PlatformJD) {
return ES.callSPSWrapper<void()>(orc_rt_macho_platform_bootstrap);
}
Error MachOPlatform::registerInitInfo(
JITDylib &JD, ExecutorAddr ObjCImageInfoAddr,
ArrayRef<jitlink::Section *> InitSections) {
std::unique_lock<std::mutex> Lock(PlatformMutex);
MachOJITDylibInitializers *InitSeq = nullptr;
{
auto I = InitSeqs.find(&JD);
if (I == InitSeqs.end()) {
// If there's no init sequence entry yet then we need to look up the
// header symbol to force creation of one.
Lock.unlock();
auto SearchOrder =
JD.withLinkOrderDo([](const JITDylibSearchOrder &SO) { return SO; });
if (auto Err = ES.lookup(SearchOrder, MachOHeaderStartSymbol).takeError())
return Err;
Lock.lock();
I = InitSeqs.find(&JD);
assert(I != InitSeqs.end() &&
"Entry missing after header symbol lookup?");
}
InitSeq = &I->second;
}
InitSeq->ObjCImageInfoAddress = ObjCImageInfoAddr;
for (auto *Sec : InitSections) {
// FIXME: Avoid copy here.
jitlink::SectionRange R(*Sec);
InitSeq->InitSections[Sec->getName()].push_back(
{ExecutorAddr(R.getStart()), ExecutorAddr(R.getEnd())});
}
return Error::success();
}
Expected<uint64_t> MachOPlatform::createPThreadKey() {
if (!orc_rt_macho_create_pthread_key)
return make_error<StringError>(
@@ -617,11 +630,6 @@ void MachOPlatform::MachOPlatformPlugin::modifyPassConfig(
return Err;
return processObjCImageInfo(G, MR);
});
Config.PostFixupPasses.push_back(
[this, &JD = MR.getTargetJITDylib()](jitlink::LinkGraph &G) {
return registerInitSections(G, JD);
});
}
// --- Add passes for eh-frame and TLV support ---
@@ -639,10 +647,12 @@ void MachOPlatform::MachOPlatformPlugin::modifyPassConfig(
return fixTLVSectionsAndEdges(G, JD);
});
// Add a pass to register the final addresses of the eh-frame and TLV sections
// with the runtime.
Config.PostFixupPasses.push_back(
[this](jitlink::LinkGraph &G) { return registerEHAndTLVSections(G); });
// Add a pass to register the final addresses of any special sections in the
// object with the runtime.
Config.PostAllocationPasses.push_back(
[this, &JD = MR.getTargetJITDylib()](jitlink::LinkGraph &G) {
return registerObjectPlatformSections(G, JD);
});
}
ObjectLinkingLayer::Plugin::SyntheticSymbolDependenciesMap
@@ -661,7 +671,6 @@ MachOPlatform::MachOPlatformPlugin::getSyntheticSymbolDependencies(
Error MachOPlatform::MachOPlatformPlugin::associateJITDylibHeaderSymbol(
jitlink::LinkGraph &G, MaterializationResponsibility &MR) {
auto I = llvm::find_if(G.defined_symbols(), [this](jitlink::Symbol *Sym) {
return Sym->getName() == *MP.MachOHeaderStartSymbol;
});
@@ -670,10 +679,14 @@ Error MachOPlatform::MachOPlatformPlugin::associateJITDylibHeaderSymbol(
auto &JD = MR.getTargetJITDylib();
std::lock_guard<std::mutex> Lock(MP.PlatformMutex);
auto HeaderAddr = (*I)->getAddress();
MP.JITDylibToHeaderAddr[&JD] = HeaderAddr;
MP.HeaderAddrToJITDylib[HeaderAddr] = &JD;
assert(!MP.InitSeqs.count(&JD) && "InitSeq entry for JD already exists");
MP.InitSeqs.insert(
std::make_pair(&JD, MachOJITDylibInitializers(JD.getName(), HeaderAddr)));
G.allocActions().push_back(
{cantFail(
WrapperFunctionCall::Create<SPSArgList<SPSString, SPSExecutorAddr>>(
MP.orc_rt_macho_register_jitdylib, JD.getName(), HeaderAddr)),
cantFail(WrapperFunctionCall::Create<SPSArgList<SPSExecutorAddr>>(
MP.orc_rt_macho_deregister_jitdylib, HeaderAddr))});
return Error::success();
}
@@ -792,37 +805,6 @@ Error MachOPlatform::MachOPlatformPlugin::processObjCImageInfo(
return Error::success();
}
Error MachOPlatform::MachOPlatformPlugin::registerInitSections(
jitlink::LinkGraph &G, JITDylib &JD) {
ExecutorAddr ObjCImageInfoAddr;
SmallVector<jitlink::Section *> InitSections;
if (auto *ObjCImageInfoSec = G.findSectionByName(ObjCImageInfoSectionName)) {
if (auto Addr = jitlink::SectionRange(*ObjCImageInfoSec).getStart())
ObjCImageInfoAddr = Addr;
}
for (auto InitSectionName : InitSectionNames)
if (auto *Sec = G.findSectionByName(InitSectionName))
InitSections.push_back(Sec);
// Dump the scraped inits.
LLVM_DEBUG({
dbgs() << "MachOPlatform: Scraped " << G.getName() << " init sections:\n";
if (ObjCImageInfoAddr)
dbgs() << " " << ObjCImageInfoSectionName << ": "
<< formatv("{0:x}", ObjCImageInfoAddr.getValue()) << "\n";
for (auto *Sec : InitSections) {
jitlink::SectionRange R(*Sec);
dbgs() << " " << Sec->getName() << ": "
<< formatv("[ {0:x} -- {1:x} ]", R.getStart(), R.getEnd()) << "\n";
}
});
return MP.registerInitInfo(JD, ObjCImageInfoAddr, InitSections);
}
Error MachOPlatform::MachOPlatformPlugin::fixTLVSectionsAndEdges(
jitlink::LinkGraph &G, JITDylib &JD) {
@@ -879,11 +861,10 @@ Error MachOPlatform::MachOPlatformPlugin::fixTLVSectionsAndEdges(
return Error::success();
}
Error MachOPlatform::MachOPlatformPlugin::registerEHAndTLVSections(
jitlink::LinkGraph &G) {
Error MachOPlatform::MachOPlatformPlugin::registerObjectPlatformSections(
jitlink::LinkGraph &G, JITDylib &JD) {
// Add a pass to register the final addresses of the eh-frame and TLV sections
// with the runtime.
// Add an action to register the eh-frame.
if (auto *EHFrameSection = G.findSectionByName(EHFrameSectionName)) {
jitlink::SectionRange R(*EHFrameSection);
if (!R.empty())
@@ -912,6 +893,8 @@ Error MachOPlatform::MachOPlatformPlugin::registerEHAndTLVSections(
ThreadDataSection = ThreadBSSSection;
}
SmallVector<std::pair<StringRef, ExecutorAddrRange>, 8> MachOPlatformSecs;
// Having merged thread BSS (if present) and thread data (if present),
// record the resulting section range.
if (ThreadDataSection) {
@@ -922,16 +905,64 @@ Error MachOPlatform::MachOPlatformPlugin::registerEHAndTLVSections(
"MachOPlatform has not finished booting",
inconvertibleErrorCode());
G.allocActions().push_back(
{cantFail(
WrapperFunctionCall::Create<SPSArgList<SPSExecutorAddrRange>>(
MP.orc_rt_macho_register_thread_data_section, R.getRange())),
cantFail(
WrapperFunctionCall::Create<SPSArgList<SPSExecutorAddrRange>>(
MP.orc_rt_macho_deregister_thread_data_section,
R.getRange()))});
MachOPlatformSecs.push_back({ThreadDataSectionName, R.getRange()});
}
}
// If any platform sections were found then add an allocation action to call
// the registration function.
StringRef PlatformSections[] = {
ModInitFuncSectionName, ObjCClassListSectionName,
ObjCImageInfoSectionName, ObjCSelRefsSectionName,
Swift5ProtoSectionName, Swift5ProtosSectionName,
Swift5TypesSectionName,
};
for (auto &SecName : PlatformSections) {
auto *Sec = G.findSectionByName(SecName);
if (!Sec)
continue;
jitlink::SectionRange R(*Sec);
if (R.empty())
continue;
MachOPlatformSecs.push_back({SecName, R.getRange()});
}
if (!MachOPlatformSecs.empty()) {
Optional<ExecutorAddr> HeaderAddr;
{
std::lock_guard<std::mutex> Lock(MP.PlatformMutex);
auto I = MP.JITDylibToHeaderAddr.find(&JD);
if (I != MP.JITDylibToHeaderAddr.end())
HeaderAddr = I->second;
}
if (!HeaderAddr)
return make_error<StringError>("Missing header for " + JD.getName(),
inconvertibleErrorCode());
// Dump the scraped inits.
LLVM_DEBUG({
dbgs() << "MachOPlatform: Scraped " << G.getName() << " init sections:\n";
for (auto &KV : MachOPlatformSecs)
dbgs() << " " << KV.first << ": " << KV.second << "\n";
});
using SPSRegisterObjectPlatformSectionsArgs =
SPSArgList<SPSExecutorAddr,
SPSSequence<SPSTuple<SPSString, SPSExecutorAddrRange>>>;
G.allocActions().push_back(
{cantFail(
WrapperFunctionCall::Create<SPSRegisterObjectPlatformSectionsArgs>(
MP.orc_rt_macho_register_object_platform_sections, *HeaderAddr,
MachOPlatformSecs)),
cantFail(
WrapperFunctionCall::Create<SPSRegisterObjectPlatformSectionsArgs>(
MP.orc_rt_macho_deregister_object_platform_sections,
*HeaderAddr, MachOPlatformSecs))});
}
return Error::success();
}