Architecture

Synced from the repository

This page is generated from docs/ARCHITECTURE.md at every build.

PdfNetStack is a rewriting optimizer, not a PDF library. It understands enough of the file format to rebuild the object graph exactly, and nothing about rendering. That narrow scope is what keeps it small, fast and provable.

Pipeline

input bytes
   |
   v
PdfFile.Open ............ header, xref chain (tables, streams, hybrid), object streams,
   |                      reconstruction for damaged files; stream data sliced, not copied
   v
Safety screen ........... encrypted, signed, incremental, damaged, too large -> return input
   |
   v
ObjectGraph.Collect ..... breadth first from /Root and /Info; everything else is garbage;
   |                      marks identity sensitive objects; optional stripping
   v
ContentAnalysis ......... reads all content once: names each resource dictionary is looked
   |                      up with, which streams are content only, glyphs each font shows
   v
ResourcePruner .......... drops resource entries no content can name
FontProgramMerger ....... replaces subsets of one TrueType font by their union
InheritedAttributes ..... (Linearize) moves inherited page attributes onto the pages
   |
   v
StreamRecompressor ...... parallel; decode compression layers, compact content streams,
   |                      try candidates, keep smallest; hash what each stream decodes to
   v
Deduplicator ............ canonical serialization + SHA-256, streams by decoded content,
   |                      merged to a fixpoint
   v
PdfDocumentWriter ....... renumber, pack object streams, write xref stream
LinearizedWriter ........ (Linearize) Annex F layout, two xref sections, hint stream
   |
   v
EquivalenceVerifier ..... re-open source and output, walk both graphs, compare decoded streams
   |
   v
output = verified rewrite, or input bytes

Source layout

Folder	Responsibility
Objects/	Object model. Numbers keep canonical text so reals never pass through floating point. Dictionaries keep insertion order for deterministic output.
Parsing/	Tokenizer, object parser, PdfSource (a buffer in memory or a seekable stream), PdfFile (xref loading, lazy object resolution, object streams, reconstruction).
Filters/	Flate, LZW, ASCIIHex, ASCII85, RunLength decoders; PNG and TIFF predictors; pooled, size capped buffers; ExhaustiveDeflate, the opt-in Zopfli style encoder.
Imaging/	JPEG at the coefficient level: marker parser, Huffman decoder for baseline, extended and progressive frames, Huffman code optimizer, baseline writer, requantization, and the coefficient equivalence check. On the read side, JpegDecoder (inverse DCT, fancy upsampling and color conversion as libjpeg does them), CcittDecoder (Group 3 and 4 fax), Jpeg2000Decoder (JP2 boxes, codestream, tier 2 packets, EBCOT with its MQ coder, wavelets, color transforms), ImageSamples (1 to 16 bit samples, Decode arrays, color spaces) and a PNG reader and writer. For image input, GIF, BMP and TIFF readers, and WebP: Vp8Decoder (lossy key frames, RFC 6386) and Vp8LDecoder (lossless, RFC 9649), both producing a DecodedImage that PdfImage embeds. For the opt-in lossy image options: JpegEncoder (forward DCT, libjpeg tables), progressive output in JpegWriter, Resampler (Lanczos), Ssim and ImageClassifier. Without those options the optimizer never decodes pixels.
Content/	Content streams on the read side: ContentReader (operators, operands, inline images, tolerant of damage) and ContentInterpreter (graphics and text state, forms, marked content, bounded work), which image and text extraction build on. ContentCompactor rewrites a content stream as the same tokens in fewer bytes and checks two streams token by token; InlineImageBounds finds where inline image data ends, and says when readers could disagree.
Text/Extraction/	Text extraction: TextExtractor (glyph placement from the text state, ActualText), TextFont (codes, Unicode and widths of simple, Type 3 and composite fonts), CMap (code spaces, CID and ToUnicode maps, usecmap), PredefinedCMaps (Adobe’s CJK CMaps and CID to Unicode tables, Brotli packed resources generated by tools/make_cjk_cmaps.py, read lazily), GlyphList (Adobe Glyph List, generated by tools/make_glyph_list.py), FontEncodings (the predefined encodings) and TextLayout (lines, words, right to left order).
Fonts/	Public font types, PdfFontCollection, and FontMatcher: the CSS Fonts 4 matching of width, style and weight over faces with ranges, and the variation values to apply, shared by collections and @font-face in the HTML package.
Fonts/OpenType/	OpenType fonts: OpenTypeFile (sfnt, collections, WOFF, WOFF2), CffFont and TrueTypeSubsetter (embedded subsets), Shaper with ShaperNormalization, IndicShaper, UseShaper, ThaiShaper, KhmerShaper, MyanmarShaper, HangulShaper and SyllableGrammar (GSUB and GPOS, normalization, the script shapers and their syllable grammars), and FontVariations, FontInstancer and Cff2Converter (static instances of variable fonts).
Optimization/	Reachability, content analysis, resource pruning, font program merging, recompression, deduplication, verification; ImagePlacement (largest drawn size of each image) and LossyImages (downsampling, JPEG conversion, the quality gate and its proof). ReferenceIndex records where every reference sits, so a pass can prove an object is reached only in the role it understands.
Documents/	The editing model. ObjectStore holds a document’s objects, or reads them from its file on demand.
Writing/	Token serializer with minimal separators; document writer for compact and classic layouts, whole in memory or as a sequence of chunks; linearized writer; incremental update writer.
PdfOptimizer.cs	Public entry point and orchestration. Every exception path returns the input. PdfOptimizer.Streaming.cs runs the same passes with the source on demand and the rewrite in a temporary file.

Design decisions

The optimizer reads everything into memory, by default. PDFs are random access by design and every pass touches most of the file. A single buffer with ReadOnlyMemory<byte> slices is the simplest correct model and the fastest one. Stream data is never copied until it is re-encoded.

Streaming mode keeps the passes and the proof, not the data. With PdfOptimizerOptions.Streaming the source is a PdfFile on a stream, so only object dictionaries and the cross-reference data stay in memory. Pruning, font merging and compaction change dictionaries and a few font programs, as in memory. Recompression is planned for every stream first (StreamRecompressor.Prepare) and done while writing: the writer asks for each stream in output order, and the streams from that one on are recompressed side by side up to 32 MiB of source data, handed over once and forgotten. Deduplication must be decided before the first object is written, so it keys each planned stream by a hash of what it decodes to through its general filters, plus what its plan may do to it (compact it, requantize it, resample it to which size); two streams with equal keys and equal dictionaries recompress to the same content. That costs one extra decode per stream and no encoding. The rewrite goes to a temporary file, and the proof is EquivalenceVerifier on a fresh reader over the source (PdfFile.Fork) and a reader over that file, both on demand: the same walk and the same stream comparisons as in memory, with each stream decoded while it is compared and then dropped.

Saving writes as it goes. PdfDocumentWriter produces the file as a sequence of chunks: a 64 KB buffer of serialized objects, or stream data of that size or more passed on as it is, with offsets counted as the bytes go out and the cross-reference data last. In memory the same code keeps everything in one buffer, so ToArray and a progressive Save cannot drift apart. SaveAsync awaits each chunk’s write. A save with the optimizer still builds the file in memory first, because the optimizer proves its rewrite against all of it.

Documents can be read on demand. PdfLoadMode.OnDemand puts PdfFile on a seekable stream instead of a buffer, through the same PdfSource abstraction, with 64 bit offsets: the caller’s stream, a file the document opened by path, or a Spool, the document’s own copy of a stream that cannot seek (in memory up to a threshold, else a temporary file deleted on dispose). Disposing the document closes the source, so any later read throws ObjectDisposedException. Opening reads the header, the cross-reference data (checking each in-file entry’s object header, 4 KB at a time) and the page tree. Any other object is parsed when first resolved and kept, so it stays the same instance; dictionaries are parsed from a window that grows only while the parse ends too close to its edge. Stream data is not read with the dictionary: each access to PdfStream.Data reads it from the stream again, and nothing keeps it, until new data is assigned. Decoded object streams share the usual cache, capped at one decode limit. Memory is therefore the objects the application has resolved plus the data it is holding, whatever the file size: reading the text of one page of a large scan reads that page’s content and fonts, not its images. SaveIncremental writes only what changed after the original bytes, copying them a megabyte at a time, or appends to the source stream in place. A full save without the optimizer is written progressively, its stream data read from the source as it goes; ToArray and a save with the optimizer build the whole output in memory. On a synthetic 400 MB file of 100 image pages, opening on demand and extracting one page’s text took 0.24 s and added 9 MB to the working set; the same with the file read into memory took 0.63 s and 387 MB. Parsed objects stay until ReleaseUnusedObjects drops those that still read as the file has them (each is parsed again and compared) and that nothing else references. The second test is a full garbage collection: the candidates are held only weakly across it, and any that survive are referenced from somewhere and return to the cache, so no instance anyone holds is ever replaced and no edit is lost.

Incremental saves compare, they do not track. The document keeps no change log. On SaveIncremental a second reader parses the original objects afresh, and every object the document has resolved is serialized and compared with its original; stream data counts as changed once it was assigned. Inherited attributes copied onto pages when the document was opened are not a change. New objects are written when a written object, or the trailer, reaches them.

Content is analysed once, before anything changes. Pruning, compaction and font merging all rest on one ContentAnalysis of the unmodified file. It errs one way only: a name anywhere in the bytes of a content stream counts as used, content it cannot decode keeps all of its resources, and a budget overrun turns pruning and font merging off. The verifier repeats the analysis on its own parse of the source rather than trusting the optimizer’s.

Linearize the way qpdf checks it. Annex F leaves room in how objects are grouped. The linearized writer assigns them to parts by who uses them exactly as qpdf does, so qpdf --check-linearization computes the same layout from the output. Hint table offsets are those of the file without the hint stream, so the hint stream is built after everything else is laid out and then inserted; the numbers in the linearization dictionary and first trailer are padded to a fixed width so their own size never moves an offset.

Numbers as canonical text. 0.50 and .5 are the same number, but 0.1 + 0.2 in floating point is not 0.3. Keeping the digits avoids any chance of drift and still lets the writer emit the shortest form.

Recompression decodes only compression layers. Predictor transforms stay in place under the new Flate layer, so a predicted image or xref stream is recompressed without reimplementing the predictor on the encode side. Candidate encodings are compared by total cost, data plus the /Filter and /DecodeParms entries they need.

JPEG stays in the frequency domain. A JPEG is quantized DCT coefficients plus Huffman coding. Re-encoding the coding alone is lossless by definition, and requantizing the coefficients is the same loss a JPEG encoder applies at a lower quality, applied once instead of after a decode and a re-encode. Skipping pixels also means color conversion, chroma upsampling and the color model never enter the picture, so CMYK, YCCK, untransformed RGB and gray images keep their meaning without special cases, and the result is integer exact on every platform.

Fast trial, slow winner. For images, plain and PNG predicted data are both compressed at the fastest zlib level, and only the smaller one is compressed at the configured level. Streams are scheduled largest first so one big image never runs alone at the end.

Identity before equality. Two equal annotation dictionaries are still two annotations. The graph walk marks objects reached through identity keys (/Parent, /Kids, /Annots, /P, /Pg, /Obj, outline links, form fields and more), and the deduplicator also skips types such as pages, annotations and structure elements. Everything else is merged only when its canonical serialization, with references mapped through the merge table, is identical.

Pooled buffers for decoded data. Decoders write into ArrayPool backed buffers that are handed out as leases and returned once consumed. On a 72 MiB drawing catalogue this took peak working set from 1.8 GB to under 700 MB.

Verification is a separate parse. The optimizer mutates its object model as it works, so the verifier opens the original bytes again and compares two independent parses. A writer bug and a reader bug would have to cancel out exactly to slip through, and the independent PDFium render check in tools/ covers that case.

Threading

Parsing and graph work run on one thread. Stream recompression and stream comparison run in parallel, after every object they need has been resolved, so workers never call back into the parser. PdfFile.GetObject is still locked as a guard. MaxDegreeOfParallelism caps both phases and is the main lever on peak memory. A stream source serializes its reads, so a document opened on demand can be read from several threads, though each read waits for the last.

Static state is shared by every document in the process and kept bounded for long running hosts. Lookup tables are built once, lazily where only some inputs need them (the CJK CMaps, the syllable grammars of the complex script shapers), under locks or LazyInitializer, and their size depends only on data shipped with the library. Caches that grow with the input belong to a document or a conversion and go with it. Variable font instances are held weakly by the font that made them, so a font shared across requests keeps only the instances still in use.