# Architecture

> How the PdfNetStack optimizer is built: the pipeline from parsing to verification, the source layout, design decisions and threading.

> **Synced from the repository:** This page is generated from [docs/ARCHITECTURE.md](https://github.com/jams4code/pdf.netstack/blob/main/docs/ARCHITECTURE.md) at every build.
PdfNetStack is a rewriting optimizer, not a PDF library. It understands enough of the file
format to rebuild the object graph exactly, and nothing about rendering. That narrow scope is
what keeps it small, fast and provable.

## Pipeline

```
input bytes
   |
   v
PdfFile.Open ............ header, xref chain (tables, streams, hybrid), object streams,
   |                      reconstruction for damaged files; stream data sliced, not copied
   v
Safety screen ........... encrypted, signed, incremental, damaged, too large -> return input
   |
   v
ObjectGraph.Collect ..... breadth first from /Root and /Info; everything else is garbage;
   |                      marks identity sensitive objects; optional stripping
   v
ContentAnalysis ......... reads all content once: names each resource dictionary is looked
   |                      up with, which streams are content only, glyphs each font shows
   v
ResourcePruner .......... drops resource entries no content can name
FontProgramMerger ....... replaces subsets of one TrueType font by their union
InheritedAttributes ..... (Linearize) moves inherited page attributes onto the pages
   |
   v
StreamRecompressor ...... parallel; decode compression layers, compact content streams,
   |                      try candidates, keep smallest; hash what each stream decodes to
   v
Deduplicator ............ canonical serialization + SHA-256, streams by decoded content,
   |                      merged to a fixpoint
   v
PdfDocumentWriter ....... renumber, pack object streams, write xref stream
LinearizedWriter ........ (Linearize) Annex F layout, two xref sections, hint stream
   |
   v
EquivalenceVerifier ..... re-open source and output, walk both graphs, compare decoded streams
   |
   v
output = verified rewrite, or input bytes
```

## Source layout

| Folder | Responsibility |
|---|---|
| `Objects/` | Object model. Numbers keep canonical text so reals never pass through floating point. Dictionaries keep insertion order for deterministic output. |
| `Parsing/` | Tokenizer, object parser, `PdfSource` (a buffer in memory or a seekable stream), `PdfFile` (xref loading, lazy object resolution, object streams, reconstruction). |
| `Filters/` | Flate, LZW, ASCIIHex, ASCII85, RunLength decoders; PNG and TIFF predictors; pooled, size capped buffers; `ExhaustiveDeflate`, the opt-in Zopfli style encoder. |
| `Imaging/` | JPEG at the coefficient level: marker parser, Huffman decoder for baseline, extended and progressive frames, Huffman code optimizer, baseline writer, requantization, and the coefficient equivalence check. On the read side, `JpegDecoder` (inverse DCT, fancy upsampling and color conversion as libjpeg does them), `CcittDecoder` (Group 3 and 4 fax), `Jpeg2000Decoder` (JP2 boxes, codestream, tier 2 packets, EBCOT with its MQ coder, wavelets, color transforms), `ImageSamples` (1 to 16 bit samples, Decode arrays, color spaces) and a PNG reader and writer. For image input, GIF, BMP and TIFF readers, and WebP: `Vp8Decoder` (lossy key frames, RFC 6386) and `Vp8LDecoder` (lossless, RFC 9649), both producing a `DecodedImage` that `PdfImage` embeds. For the opt-in lossy image options: `JpegEncoder` (forward DCT, libjpeg tables), progressive output in `JpegWriter`, `Resampler` (Lanczos), `Ssim` and `ImageClassifier`. Without those options the optimizer never decodes pixels. |
| `Content/` | Content streams on the read side: `ContentReader` (operators, operands, inline images, tolerant of damage) and `ContentInterpreter` (graphics and text state, forms, marked content, bounded work), which image and text extraction build on. `ContentCompactor` rewrites a content stream as the same tokens in fewer bytes and checks two streams token by token; `InlineImageBounds` finds where inline image data ends, and says when readers could disagree. |
| `Text/Extraction/` | Text extraction: `TextExtractor` (glyph placement from the text state, ActualText), `TextFont` (codes, Unicode and widths of simple, Type 3 and composite fonts), `CMap` (code spaces, CID and ToUnicode maps, usecmap), `PredefinedCMaps` (Adobe's CJK CMaps and CID to Unicode tables, Brotli packed resources generated by `tools/make_cjk_cmaps.py`, read lazily), `GlyphList` (Adobe Glyph List, generated by `tools/make_glyph_list.py`), `FontEncodings` (the predefined encodings) and `TextLayout` (lines, words, right to left order). |
| `Fonts/` | Public font types, `PdfFontCollection`, and `FontMatcher`: the CSS Fonts 4 matching of width, style and weight over faces with ranges, and the variation values to apply, shared by collections and `@font-face` in the HTML package. |
| `Fonts/OpenType/` | OpenType fonts: `OpenTypeFile` (sfnt, collections, WOFF, WOFF2), `CffFont` and `TrueTypeSubsetter` (embedded subsets), `Shaper` with `ShaperNormalization`, `IndicShaper`, `UseShaper`, `ThaiShaper`, `KhmerShaper`, `MyanmarShaper`, `HangulShaper` and `SyllableGrammar` (GSUB and GPOS, normalization, the script shapers and their syllable grammars), and `FontVariations`, `FontInstancer` and `Cff2Converter` (static instances of variable fonts). |
| `Optimization/` | Reachability, content analysis, resource pruning, font program merging, recompression, deduplication, verification; `ImagePlacement` (largest drawn size of each image) and `LossyImages` (downsampling, JPEG conversion, the quality gate and its proof). `ReferenceIndex` records where every reference sits, so a pass can prove an object is reached only in the role it understands. |
| `Documents/` | The editing model. `ObjectStore` holds a document's objects, or reads them from its file on demand. |
| `Writing/` | Token serializer with minimal separators; document writer for compact and classic layouts, whole in memory or as a sequence of chunks; linearized writer; incremental update writer. |
| `PdfOptimizer.cs` | Public entry point and orchestration. Every exception path returns the input. `PdfOptimizer.Streaming.cs` runs the same passes with the source on demand and the rewrite in a temporary file. |

## Design decisions

**The optimizer reads everything into memory, by default.** PDFs are random access by design and
every pass touches most of the file. A single buffer with `ReadOnlyMemory<byte>` slices is the
simplest correct model and the fastest one. Stream data is never copied until it is re-encoded.

**Streaming mode keeps the passes and the proof, not the data.** With
`PdfOptimizerOptions.Streaming` the source is a `PdfFile` on a stream, so only object
dictionaries and the cross-reference data stay in memory. Pruning, font merging and compaction
change dictionaries and a few font programs, as in memory. Recompression is planned for every
stream first (`StreamRecompressor.Prepare`) and done while writing: the writer asks for each
stream in output order, and the streams from that one on are recompressed side by side up to
32 MiB of source data, handed over once and forgotten. Deduplication must be decided before the
first object is written, so it keys each planned stream by a hash of what it decodes to through
its general filters, plus what its plan may do to it (compact it, requantize it, resample it to
which size); two streams with equal keys and equal dictionaries recompress to the same content.
That costs one extra decode per stream and no encoding. The rewrite goes to a temporary file,
and the proof is `EquivalenceVerifier` on a fresh reader over the source (`PdfFile.Fork`) and a
reader over that file, both on demand: the same walk and the same stream comparisons as in
memory, with each stream decoded while it is compared and then dropped.

**Saving writes as it goes.** `PdfDocumentWriter` produces the file as a sequence of chunks: a
64 KB buffer of serialized objects, or stream data of that size or more passed on as it is, with
offsets counted as the bytes go out and the cross-reference data last. In memory the same code
keeps everything in one buffer, so `ToArray` and a progressive `Save` cannot drift apart.
`SaveAsync` awaits each chunk's write. A save with the optimizer still builds the file in
memory first, because the optimizer proves its rewrite against all of it.

**Documents can be read on demand.** `PdfLoadMode.OnDemand` puts `PdfFile` on a seekable
stream instead of a buffer, through the same `PdfSource` abstraction, with 64 bit offsets: the
caller's stream, a file the document opened by path, or a `Spool`, the document's own copy of a
stream that cannot seek (in memory up to a threshold, else a temporary file deleted on dispose).
Disposing the document closes the source, so any later read throws `ObjectDisposedException`.
Opening reads the header, the cross-reference data (checking each in-file entry's object header,
4 KB at a time) and the page tree. Any other object is parsed when first resolved and kept, so
it stays the same instance; dictionaries are parsed from a window that grows only while the
parse ends too close to its edge. Stream data is not read with the dictionary: each access to
`PdfStream.Data` reads it from the stream again, and nothing keeps it, until new data is
assigned. Decoded object streams share the usual cache, capped at one decode limit. Memory is
therefore the objects the application has resolved plus the data it is holding, whatever the
file size: reading the text of one page of a large scan reads that page's content and fonts,
not its images. `SaveIncremental` writes only what changed after the original bytes, copying
them a megabyte at a time, or appends to the source stream in place. A full save without the
optimizer is written progressively, its stream data read from the source as it goes; `ToArray`
and a save with the optimizer build the whole output in memory. On a synthetic 400 MB
file of 100 image pages, opening on demand and extracting one page's text took 0.24 s and added
9 MB to the working set; the same with the file read into memory took 0.63 s and 387 MB.
Parsed objects stay until `ReleaseUnusedObjects` drops those that still read as the file has
them (each is parsed again and compared) and that nothing else references. The second test is a
full garbage collection: the candidates are held only weakly across it, and any that survive
are referenced from somewhere and return to the cache, so no instance anyone holds is ever
replaced and no edit is lost.

**Incremental saves compare, they do not track.** The document keeps no change log. On
`SaveIncremental` a second reader parses the original objects afresh, and every object the
document has resolved is serialized and compared with its original; stream data counts as
changed once it was assigned. Inherited attributes copied onto pages when the document was
opened are not a change. New objects are written when a written object, or the trailer,
reaches them.

**Content is analysed once, before anything changes.** Pruning, compaction and font merging all
rest on one `ContentAnalysis` of the unmodified file. It errs one way only: a name anywhere in
the bytes of a content stream counts as used, content it cannot decode keeps all of its
resources, and a budget overrun turns pruning and font merging off. The verifier repeats the
analysis on its own parse of the source rather than trusting the optimizer's.

**Linearize the way qpdf checks it.** Annex F leaves room in how objects are grouped. The
linearized writer assigns them to parts by who uses them exactly as qpdf does, so `qpdf
--check-linearization` computes the same layout from the output. Hint table offsets are those of
the file without the hint stream, so the hint stream is built after everything else is laid out
and then inserted; the numbers in the linearization dictionary and first trailer are padded to a
fixed width so their own size never moves an offset.

**Numbers as canonical text.** `0.50` and `.5` are the same number, but `0.1 + 0.2` in floating
point is not `0.3`. Keeping the digits avoids any chance of drift and still lets the writer emit
the shortest form.

**Recompression decodes only compression layers.** Predictor transforms stay in place under the
new Flate layer, so a predicted image or xref stream is recompressed without reimplementing the
predictor on the encode side. Candidate encodings are compared by total cost, data plus the
`/Filter` and `/DecodeParms` entries they need.

**JPEG stays in the frequency domain.** A JPEG is quantized DCT coefficients plus Huffman
coding. Re-encoding the coding alone is lossless by definition, and requantizing the coefficients
is the same loss a JPEG encoder applies at a lower quality, applied once instead of after a
decode and a re-encode. Skipping pixels also means color conversion, chroma upsampling and the
color model never enter the picture, so CMYK, YCCK, untransformed RGB and gray images keep
their meaning without special cases, and the result is integer exact on every platform.

**Fast trial, slow winner.** For images, plain and PNG predicted data are both compressed at the
fastest zlib level, and only the smaller one is compressed at the configured level. Streams are
scheduled largest first so one big image never runs alone at the end.

**Identity before equality.** Two equal annotation dictionaries are still two annotations. The
graph walk marks objects reached through identity keys (`/Parent`, `/Kids`, `/Annots`, `/P`,
`/Pg`, `/Obj`, outline links, form fields and more), and the deduplicator also skips types such
as pages, annotations and structure elements. Everything else is merged only when its canonical
serialization, with references mapped through the merge table, is identical.

**Pooled buffers for decoded data.** Decoders write into `ArrayPool` backed buffers that are
handed out as leases and returned once consumed. On a 72 MiB drawing catalogue this took peak
working set from 1.8 GB to under 700 MB.

**Verification is a separate parse.** The optimizer mutates its object model as it works, so the
verifier opens the original bytes again and compares two independent parses. A writer bug and a
reader bug would have to cancel out exactly to slip through, and the independent PDFium render
check in `tools/` covers that case.

## Threading

Parsing and graph work run on one thread. Stream recompression and stream comparison run in
parallel, after every object they need has been resolved, so workers never call back into the
parser. `PdfFile.GetObject` is still locked as a guard. `MaxDegreeOfParallelism` caps both
phases and is the main lever on peak memory. A stream source serializes its reads, so a document
opened on demand can be read from several threads, though each read waits for the last.

Static state is shared by every document in the process and kept bounded for long running hosts.
Lookup tables are built once, lazily where only some inputs need them (the CJK CMaps, the
syllable grammars of the complex script shapers), under locks or `LazyInitializer`, and their size
depends only on data shipped with the library. Caches that grow with the input belong to a
document or a conversion and go with it. Variable font instances are held weakly by the font that
made them, so a font shared across requests keeps only the instances still in use.

Source: https://www.jadev-corp.com/pdfnetstack/concepts/architecture/