Dev Docs Caching Architecture

Caching Architecture

Audited against Netatalk d63155e13 on 2026-10-10. This describes the current source checkout; older releases may differ.

Parent page: Architecture Overview

Each afpd worker child maintains its own private, per-process cache hierarchy. There is no shared memory between workers — cross-process coherence is handled by the IPC Cache Hint Relay described below.

Implementation Files:

Cache Tier Overview

%%{ init: { 'themeVariables': { 'fontSize': '14px' }, 'flowchart': { 'nodeSpacing': 15, 'rankSpacing': 40 } } }%%
graph TB
    classDef blue fill:#74b9ff,stroke:#0984e3,color:#2d3436,rx:10,ry:10
    classDef green fill:#55efc4,stroke:#00b894,color:#2d3436,rx:10,ry:10
    classDef yellow fill:#ffeaa7,stroke:#fdcb6e,color:#2d3436,rx:10,ry:10
    classDef purple fill:#a29bfe,stroke:#6c5ce7,color:#2d3436,rx:10,ry:10
    classDef grey fill:#dfe6e9,stroke:#b2bec3,color:#2d3436,rx:10,ry:10
    classDef cream fill:#f8f4e8,stroke:#d4c5a9,color:#2d3436,rx:10,ry:10
    classDef salmon fill:#fab1a0,stroke:#e17055,color:#2d3436,rx:10,ry:10

    REQ["AFP command<br/>(e.g. FPGetFileDirParms)"]:::blue

    subgraph "Tier 1: struct dir (ARC/LRU dircache)"
        STAT["Cached stat fields<br/>dcache_ctime, dcache_ino,<br/>dcache_mode, dcache_size"]:::green
        AD["Cached AD metadata<br/>dcache_finderinfo[32]<br/>dcache_filedatesi[16]<br/>dcache_afpfilei[4]<br/>dcache_rlen"]:::green
    end

    subgraph "Tier 2: rfork content (byte-budget LRU)"
        RFORK["dcache_rfork_buf<br/>malloc'd resource fork data<br/>budget: rfork_cache_budget bytes"]:::yellow
    end

    DISK_STAT["stat() syscall"]:::grey
    DISK_AD["ad_metadata() → getxattr()"]:::grey
    DISK_RF["ad_read() → read/pread()"]:::grey

    REQ --> STAT
    STAT -->|"hit"| AD
    STAT -->|"miss / validate"| DISK_STAT --> STAT
    AD -->|"hit (rlen ≥ 0)"| RFORK
    AD -->|"miss (rlen = -1)"| DISK_AD --> AD
    AD -->|"no AD (rlen = -2)"| REQ
    RFORK -->|"hit (buf ≠ NULL)"| REQ
    RFORK -->|"miss (buf = NULL)"| DISK_RF --> RFORK

Tier 1: ARC Directory Cache

The directory cache (etc/afpd/dircache.c) implements the Adaptive Replacement Cache (ARC) algorithm from Megiddo & Modha (IBM, 2004). ARC automatically balances between recency and frequency by maintaining four lists:

The adaptation parameter p (target size for T1) shifts automatically: - B1 ghost hit → increase p (favor recency): arc_case_ii_adapt_and_replace() - B2 ghost hit → decrease p (favor frequency): arc_case_iii_adapt_and_replace()

%%{ init: { 'themeVariables': { 'fontSize': '14px' }, 'flowchart': { 'nodeSpacing': 15, 'rankSpacing': 40 } } }%%
graph LR
    classDef blue fill:#74b9ff,stroke:#0984e3,color:#2d3436,rx:10,ry:10
    classDef green fill:#55efc4,stroke:#00b894,color:#2d3436,rx:10,ry:10
    classDef yellow fill:#ffeaa7,stroke:#fdcb6e,color:#2d3436,rx:10,ry:10
    classDef purple fill:#a29bfe,stroke:#6c5ce7,color:#2d3436,rx:10,ry:10
    classDef grey fill:#dfe6e9,stroke:#b2bec3,color:#2d3436,rx:10,ry:10
    classDef cream fill:#f8f4e8,stroke:#d4c5a9,color:#2d3436,rx:10,ry:10
    classDef salmon fill:#fab1a0,stroke:#e17055,color:#2d3436,rx:10,ry:10

    subgraph "ARC Cache (size c)"
        T1["T1: Recent<br/>(target p entries)<br/>LRU → MRU"]:::blue
        T2["T2: Frequent<br/>(c − p entries)<br/>LRU → MRU"]:::green
    end

    subgraph "Ghost Directory (size c)"
        B1["B1: Ghost/Recent<br/>evicted from T1<br/>DIRF_ARC_GHOST"]:::grey
        B2["B2: Ghost/Frequent<br/>evicted from T2<br/>DIRF_ARC_GHOST"]:::grey
    end

    MISS["Cache miss<br/>arc_case_iv()"]:::cream
    HIT["Cache hit<br/>arc_case_i()"]:::cream

    MISS -->|"new entry"| T1
    T1 -->|"2nd access<br/>promote"| T2
    T1 -->|"evict LRU"| B1
    T2 -->|"evict LRU"| B2
    B1 -->|"ghost hit<br/>p increases<br/>Case II"| T2
    B2 -->|"ghost hit<br/>p decreases<br/>Case III"| T2
    HIT -->|"T1 hit"| T2
    HIT -->|"T2 hit → MRU"| T2

ARC Implementation Details

Configuration (bounds in etc/afpd/dircache.h, default in include/atalk/globals.h):

Hash indexes — the dircache maintains two parallel hash tables: - CNID index (hash_vid_did): lookup by volume ID + directory/file CNID — FNV-1a hash - DID/name index (hash_didname): lookup by parent DID + Unix filename — SuperFastHash

In ARC mode, hash tables are sized to 2c (c cached entries + c ghost entries per the ARC paper). In LRU mode, sized to c. See dircache_init() in dircache.c.

Ghost entries retain ALL data (full struct dir with DIRF_ARC_GHOST flag set). This differs from the IBM paper’s lightweight ghosts but allows zero-allocation promotion back to T2 via queue_move_to_tail_of(). Ghost entries stay in both hash tables for O(1) lookup.

ARC cases (as implemented in dircache.c):

LRU fallback: configurable via afp.conf dircache mode option. Uses a single queue with chunk eviction (DIRCACHE_FREE_QUANTUM = 256 entries at a time).

Cache Validation

Cache entries are validated against the filesystem using sampled validation to reduce stat() syscalls:

• should_validate_cache_entry() in dircache.c: validates every Nth access (configurable via dircache validation freq in afp.conf)
• Default: DEFAULT_DIRCACHE_VALIDATION_FREQ = 100 (sample every 100th eligible lookup); accepted range 1–100. multi protocol = yes defaults to 1 unless explicitly configured
• Validation checks: st_ino (inode) and st_ctime (metadata change time)
• On mismatch: dir_modify(DCMOD_STAT) refreshes stat fields, CNID (on inode change), and invalidates AD cache
• Internal AFP operations use explicit invalidation via dir_remove(), so frequent validation is mainly needed for external filesystem changes

dircache_report_invalid_entry() tracks entries that passed validation but were found stale on actual use — this metric helps tune the validation frequency.

Tier 1b: AD Metadata Cache

AppleDouble metadata is cached inline in each struct dir entry (defined in include/atalk/directory.h):

dcache_rlen State Machine

Current sentinels are defined in include/atalk/directory.h:

ad_rlen_meta_absent() recognizes -2 and -3. It must not collapse a fork-only file into a file with no resource fork. ad_metadata_cached() can cache a negative metadata result while preserving that fork distinction.

Unified AD Access: ad_metadata_cached()

ad_metadata_cached() (in etc/afpd/ad_cache.c) is the single entry point for read-only AD metadata:

• strict=false (default): cache-first path for enumerate, FPGetFileDirParms
• If dcache_rlen ≥ 0: calls ad_rebuild_from_cache() → instant return
• If dcache_rlen == -2 or -3: returns ENOENT without an AD metadata read; -3 preserves the distinction that a resource fork exists
• If dcache_rlen == -1: falls through to disk read
• strict=true: validates ctime/inode before serving (used by moveandrename, deletecurdir)
• On ctime mismatch: calls dir_modify(DCMOD_STAT) for coherent update, then falls through to disk
• On match with dcache_rlen ≥ 0: serves from cache

A non-strict call supplied with recent_st also checks that stat result against cached ctime/inode before serving; non-strict does not mean ignoring known changes.

On disk read (cache miss): ad_metadata() → ad_store_to_cache() saves FinderInfo, FileDatesI, AFPFileI, and rlen into the struct dir. The ad_close() is called internally — callers needing a writable fd must call ad_open() separately.

Pre-computed mdate: ad_store_to_cache() stores max(ad_mdate, st_mtime) in dcache_filedatesi + AD_DATE_MODIFY. This is the served FILPBIT_MDATE value — avoids recomputation on every enumerate hit.

AD Cache Statistics

Logged by log_dircache_stat() in dircache.c on session close:

Tier 2: Resource Fork Content Cache

For files with small resource forks (icons, custom metadata), the actual resource fork content is cached in a malloc()‘d buffer attached to struct dir:

%%{ init: { 'themeVariables': { 'fontSize': '14px' }, 'flowchart': { 'nodeSpacing': 15, 'rankSpacing': 40 } } }%%
graph TB
    classDef blue fill:#74b9ff,stroke:#0984e3,color:#2d3436,rx:10,ry:10
    classDef green fill:#55efc4,stroke:#00b894,color:#2d3436,rx:10,ry:10
    classDef yellow fill:#ffeaa7,stroke:#fdcb6e,color:#2d3436,rx:10,ry:10
    classDef grey fill:#dfe6e9,stroke:#b2bec3,color:#2d3436,rx:10,ry:10
    classDef cream fill:#f8f4e8,stroke:#d4c5a9,color:#2d3436,rx:10,ry:10
    classDef salmon fill:#fab1a0,stroke:#e17055,color:#2d3436,rx:10,ry:10

    READ["FPRead (resource fork)"]:::blue
    CHECK{"dcache_rfork_buf<br/>!= NULL?"}:::cream

    subgraph "Rfork Cache Hit"
        HIT["memcpy from<br/>dcache_rfork_buf"]:::green
    end

    subgraph "Rfork Cache Miss"
        AD_CHECK{"dcache_rlen > 0<br/>AND ≤ max_entry?"}:::cream
        DISK["ad_read(adp, eid,<br/>0, buf, rlen)"]:::grey
        BUDGET{"rfork_cache_used<br/>+ rlen ≤ budget?"}:::cream
        EVICT["rfork_cache_evict_to_budget()<br/>LRU head → free oldest"]:::salmon
        STORE["rfork_cache_store_from_fd()<br/>malloc, ad_read, enqueue LRU"]:::yellow
        SKIP["Serve from fd<br/>(not cached)"]:::grey
    end

    READ --> CHECK
    CHECK -->|"hit"| HIT
    CHECK -->|"miss"| AD_CHECK
    AD_CHECK -->|"yes"| BUDGET
    AD_CHECK -->|"too large or rlen ≤ 0"| SKIP
    BUDGET -->|"fits"| STORE
    BUDGET -->|"over budget"| EVICT --> STORE
    DISK -.->|"ad_read inside"| STORE

Invariant: dcache_rfork_buf != NULL implies dcache_rlen >= 0 (Tier 1 AD must be loaded first).

Budget Management

Globals defined in etc/afpd/dircache.c, declared in etc/afpd/dircache.h:

API Functions (in etc/afpd/ad_cache.c)

Rfork Cache Statistics

Cross-Process Cache Invalidation (IPC Hint System)

Since each worker child has its own private dircache, a cross-process hint relay system propagates invalidation signals when one child modifies the filesystem.

Architecture

%%{ init: { 'themeVariables': { 'fontSize': '14px' }, 'flowchart': { 'nodeSpacing': 15, 'rankSpacing': 40 } } }%%
sequenceDiagram
    participant W1 as Worker Child 1<br/>(modifying)
    participant P as afpd Parent<br/>(hint relay)
    participant W2 as Worker Child 2
    participant W3 as Worker Child 3

    Note over W1: AFP command modifies file<br/>(e.g., FPWrite, FPDelete)
    W1->>P: ipc_send_cache_hint()<br/>via IPC socketpair<br/>{REFRESH, vid, cnid}

    Note over P: ipc_server_read() →<br/>ipc_relay_cache_hint()<br/>appends to hint_buf[]

    Note over P: poll() returns (50ms timeout<br/>or hint_buf full)
    P->>P: hint_flush_pending()<br/>priority sort, serialize

    P->>W2: write(hint_pipe)<br/>PIPE_BUF-safe chunks
    P->>W3: write(hint_pipe)<br/>(excludes source W1)

    Note over W2: poll() on hint_pipe →<br/>process_cache_hints()
    W2->>W2: hash_lookup(cnid) →<br/>dir_modify() or dir_remove()

    Note over W3: poll() on hint_pipe →<br/>process_cache_hints()
    W3->>W3: hash_lookup(cnid) →<br/>dir_modify() or dir_remove()

Communication Channels

Each worker child has two channels to the parent (created in dsi_getsession() in libatalk/dsi/dsi_getsess.c):

Why separate channels? The hint pipe is separated from the IPC socketpair to avoid interfering with send_fd()/recv_fd() + SIGURG signaling used for AFP session transfer (primary reconnect). Bounded pipe writes ≤ the platform’s PIPE_BUF do not interleave. Reads can still return partial messages; the receiver maintains a carry buffer. Atomic writes do not guarantee delivery when a nonblocking channel is full.

Wire Format

IPC header (IPC_HEADERLEN = 14 bytes):

[command:2][child_pid:4][uid:4][len:4]

Cache hint payload (struct ipc_cache_hint_payload in include/atalk/server_ipc.h, 8 bytes, __attribute__((packed))):

Total message size: 14 + 8 = 22 bytes per hint.

Hint Types

Sender: ipc_send_cache_hint()

Called directly from AFP command handlers (e.g., afp_write(), afp_delete(), afp_moveandrename()). Writes to the child’s IPC socketpair fd (obj->ipc_fd). Independent of the external FCE system — always active when IPC is available.

Parent Relay: Batching

%%{ init: { 'themeVariables': { 'fontSize': '14px' }, 'flowchart': { 'nodeSpacing': 15, 'rankSpacing': 40 } } }%%
graph TB
    classDef purple fill:#a29bfe,stroke:#6c5ce7,color:#2d3436,rx:10,ry:10
    classDef yellow fill:#ffeaa7,stroke:#fdcb6e,color:#2d3436,rx:10,ry:10
    classDef green fill:#55efc4,stroke:#00b894,color:#2d3436,rx:10,ry:10
    classDef cream fill:#f8f4e8,stroke:#d4c5a9,color:#2d3436,rx:10,ry:10
    classDef cyan fill:#81ecec,stroke:#00cec9,color:#2d3436,rx:10,ry:10
    classDef grey fill:#dfe6e9,stroke:#b2bec3,color:#2d3436,rx:10,ry:10

    IPC["ipc_server_read()<br/>receives hint from child"]:::purple
    RELAY["ipc_relay_cache_hint()<br/>rate check (1000/sec)<br/>append to hint_buf[]"]:::purple
    BUF["hint_buf[128]<br/>accumulating"]:::yellow

    POLL{"poll() returns?"}:::cream
    TIMEOUT["50ms timeout<br/>(ret == 0)"]:::grey
    FULL["buffer full<br/>(count ≥ 128)"]:::grey

    FLUSH["hint_flush_pending()"]:::green
    SORT["Priority sort:<br/>VOLUME_RESET → REFRESH →<br/>DELETE → DELETE_CHILDREN"]:::green
    WRITE["Per-child: serialize<br/>skip source_pid<br/>PIPE_BUF chunked write"]:::cyan

    IPC --> RELAY --> BUF
    BUF --> POLL
    POLL -->|"50ms timeout"| TIMEOUT --> FLUSH
    POLL -->|"buffer full"| FULL --> FLUSH
    POLL -->|"fd event, not full"| BUF
    FLUSH --> SORT --> WRITE

Safety Mechanisms

Receiver: process_cache_hints()

process_cache_hints() (in etc/afpd/dircache.c) runs in the worker child, called from the afp_over_dsi() event loop when the hint pipe has data:

1. Bulk-reads up to MAX_HINTS_PER_CYCLE (32) × 22 bytes in a single read()
2. Parses in 22-byte increments (IPC header + payload)
3. For a reset, matches an open volume by UUID tag and requests session termination; ordinary hints resolve volume via getvolbyvid(hint.vid)
4. Uses direct hash_lookup() on the CNID hash table (not dircache_search_by_did() which would remove file entries)
5. Dispatches on hint.event:
6. CACHE_HINT_REFRESH: ostat() + dir_modify(DCMOD_STAT | DCMOD_AD_INV | DCMOD_NO_PROMOTE)
7. CACHE_HINT_DELETE: dir_remove(vol, entry, 0)
8. CACHE_HINT_DELETE_CHILDREN: dircache_remove_children_defer() + parent ostat/remove

Handles partial reads via a static carry buffer for message boundary alignment.

Hint Statistics

Logged by log_dircache_stat() in dircache.c on session close:

Sender side (per child): - hints_sent: total hints written to IPC socketpair

Parent side (aggregate): - hints_batched: total hints processed through buffer - hints_rate_dropped: hints dropped by rate limiter - flush_count: number of flush cycles

Receiver side (per child): - hints_received: total hints read from pipe - hints_acted_on: hints that matched a local cache entry and triggered invalidation - hints_no_match: hints for entries not in local cache (no cache mutation)

Deferred Cleanup (Idle Worker)

Large dircache_remove_children() operations (e.g., after directory rename) can be deferred to an idle worker thread that runs during poll() idle periods:

• dircache_remove_children_defer(): O(1) enqueue to deferred_queue[] (capacity MAX_DEFERRED_CLEANUPS = 256)
• dircache_process_deferred_chain(): processes one hash chain per call, collecting up to DEFERRED_CHAIN_BATCH (16) matching children per iteration
• dircache_flush_deferred_for_vol(): synchronous flush for a specific volume (called during afp_closevol())
• Falls back to synchronous dircache_remove_children() if idle worker is not active or queue is full

struct dir Cache Fields

All cache data is stored inline in struct dir (defined in include/atalk/directory.h) for cache locality. Fields grouped by alignment:

Pointer Fields (8-byte)

Time/Size Fields (8-byte)

ID/Flag Fields (4-byte)

Byte Fields (1-byte + arrays)

Current Defaults and Multi-Protocol Access

Global defaults from libatalk/util/netatalk_conf.c are 65,536 live entries, ARC mode, validation frequency 100, a 32,768 KiB resource-fork budget, and a 1,024 KiB per-fork limit. Runtime resource-fork globals are in bytes.

multi protocol = yes applies process-wide strict-locking and per-access-validation defaults unless explicitly set. Its volume is excluded from resource-fork caching unless the Global budget was explicit. ea = samba chooses metadata format independently. These rules are implemented by apply_multiprotocol_defaults() and the per-volume admission gate in etc/afpd/fork.c.

Delivery Limits and CNID Resets

Ordinary hints are best effort: full sender socket buffers, relay rate limits, or full sibling pipes can drop hints. They supplement filesystem validation; they do not provide lossless or immediate cross-process coherence. The parent uses a 50 ms poll timeout when buffering, but continuous fd activity can postpone a timeout flush. Buffer-full and reset handling can force flushing.

CACHE_HINT_VOLUME_RESET invalidates persistent identity, not just cache content. The payload’s cnid field carries cnid_volume_tag(), and the receiver matches an open volume by UUID-derived tag rather than process-local vid. It raises SIGTERM for the affected session. Reset hints precede ordinary REFRESH/DELETE/DELETE_CHILDREN hints, have separate rate limiting/deduplication, and use bounded waits for delivery. Delivery failure is logged; even this path is not an unconditional delivery guarantee. See libatalk/util/server_ipc.c and libatalk/util/cnid.c.

Parent-Directory Descriptor Cache

etc/afpd/pfd_cache.c maintains 16 MRU-ordered descriptor slots keyed by volume ID and parent DID. pfd_ostat() accelerates validation with a parent fd and relative stat, falling back to full-path ostat() when necessary. Its interface is etc/afpd/pfd_cache.h.

The cache observes parent identity changes learned by dircache and re-grounds against the absolute path every PFD_REGROUND_INTERVAL (256) slot uses. Under concurrent parent rename/replacement there can be a convergence window; validation frequency 1 alone does not eliminate this fd-cache behavior. Removal, volume purge, and session close retire descriptors. Stats include hits, opens, refreshes, re-grounds, fallbacks, purges, and path repairs.

pfd_ostat() is main-thread-only; purge hooks can run on the idle worker while the main thread has granted exclusive temporal access. No concurrent main/worker access is allowed. See test/afpd/subtests_pfd.c for divergence and fallback coverage.

Idle-Worker Handshake

The current API uses iw_* names. Producers call iw_note_work(), main grants before poll() and revokes on every return, and the worker publishes/rechecks its grant before accessing queues. It wakes every 10 ms through nanosleep(). Queue discovery occurs only inside a validated worker window. See AFP Daemon: Idle Worker.

Footnotes

Generated from the Netatalk GitHub Wiki

Last updated 2026-10-21