nicolas/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Flush bitcode incrementally for LTO output

Browse Source 
Bitcode writer does not flush buffer until the end by default. This is
fine to small bitcode files. When -flto,--plugin-opt=emit-llvm,-gmlt are
used, the final bitcode file is large, for example, >8G. Keeping all
data in memory consumes a lot of memory.

This change allows bitcode writer flush data to disk early when buffered
data size is above some threshold. This is only enabled when lld emits
LLVM bitcode.

One issue to address is backpatching bitcode: subblock length, function
body indexes, meta data indexes need to backfill. If buffer can be
flushed partially, we introduced raw_fd_stream that supports
read/seek/write, and enables backpatching bitcode flushed in disk.

Reviewed-by: tejohnson, MaskRay

Differential Revision: https://reviews.llvm.org/D86905
This commit is contained in:
Jianzhou Zhao
2020-09-12 19:35:17 +00:00
parent 0dd4d70ec2
commit 11201315d5
4 changed files with 115 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
View File

@@ -86,6 +86,9 @@ static cl::opt<unsigned>
IndexThreshold("bitcode-mdindex-threshold", cl::Hidden, cl::init(25),
cl::desc("Number of metadatas above which we emit an index "
"to enable lazy-loading"));
static cl::opt<uint32_t> FlushThreshold(
"bitcode-flush-threshold", cl::Hidden, cl::init(512),
cl::desc("The threshold (unit M) for flushing LLVM bitcode."));
static cl::opt<bool> WriteRelBFToSummary(
"write-relbf-to-summary", cl::Hidden, cl::init(false),
@@ -4453,8 +4456,8 @@ static void writeBitcodeHeader(BitstreamWriter &Stream) {
Stream.Emit(0xD, 4);
}
BitcodeWriter::BitcodeWriter(SmallVectorImpl<char> &Buffer)
: Buffer(Buffer), Stream(new BitstreamWriter(Buffer)) {
BitcodeWriter::BitcodeWriter(SmallVectorImpl<char> &Buffer, raw_fd_stream *FS)
: Buffer(Buffer), Stream(new BitstreamWriter(Buffer, FS, FlushThreshold)) {
writeBitcodeHeader(*Stream);
}
@@ -4565,7 +4568,7 @@ void llvm::WriteBitcodeToFile(const Module &M, raw_ostream &Out,
if (TT.isOSDarwin() || TT.isOSBinFormatMachO())
Buffer.insert(Buffer.begin(), BWH_HeaderSize, 0);
BitcodeWriter Writer(Buffer);
BitcodeWriter Writer(Buffer, dyn_cast<raw_fd_stream>(&Out));
Writer.writeModule(M, ShouldPreserveUseListOrder, Index, GenerateHash,
ModHash);
Writer.writeSymtab();
@@ -4575,7 +4578,8 @@ void llvm::WriteBitcodeToFile(const Module &M, raw_ostream &Out,
emitDarwinBCHeaderAndTrailer(Buffer, TT);
// Write the generated bitstream to "Out".
Out.write((char*)&Buffer.front(), Buffer.size());
if (!Buffer.empty())
Out.write((char *)&Buffer.front(), Buffer.size());
}
void IndexBitcodeWriter::write() {