netatalk.io

search netatalk.io

powered by DuckDuckGo

Dev Docs Data Flow

Data Flow Architecture

System Data Flow Overview

This document describes how data flows through the Netatalk system from client requests to filesystem operations, including the interaction between different components and services.

Implementation Files:

High-Level Data Flow

%%{ init: { 'themeVariables': { 'fontSize': '14px' }, 'flowchart': { 'nodeSpacing': 20, 'rankSpacing': 45, 'subGraphTitleMargin': { 'top': 10, 'bottom': 5 } } } }%%
graph TB
    subgraph clients["Client Side"]
        A["AFP Client Application"]:::client
        B["macOS Finder / Apps"]:::client
    end

    subgraph network["Network Layer"]
        C["AFP over TCP/DSI"]:::network
        D["AFP over AppleTalk/ASP"]:::network
    end

    subgraph master["Netatalk Master Process"]
        E["netatalk daemon"]:::master
        F["Configuration Manager"]:::master
        G["Service Coordinator"]:::master
    end

    subgraph afpd["AFP Service Layer (afpd)"]
        HP["afpd Parent Process<br/>Listener + IPC Relay"]:::afpparent
        H["afpd Worker Process<br/>(forked per connection)"]:::worker
        I["Authentication Handler"]:::worker
        J["Volume Manager"]:::worker
        K["File Operation Handler"]:::worker
    end

    subgraph cnid["CNID Metadata"]
        L["cnid_metad"]:::cnid
        M["cnid_dbd"]:::cnid
        N["Berkeley DB / SQLite / MySQL"]:::cnid
    end

    subgraph storage["Storage Layer"]
        O["Unix Filesystem"]:::storage
        P["Extended Attributes"]:::storage
        Q["AppleDouble Files"]:::storage
    end

    A & B --> C & D
    C & D --> HP
    E --> F & G
    G --> HP & L
    HP -->|"fork()"| H
    H --> I & J & K
    K --> L
    L --> M --> N
    K --> O & P & Q

    classDef client fill:#74b9ff,stroke:#0984e3,color:#fff,rx:10,ry:10
    classDef network fill:#a29bfe,stroke:#6c5ce7,color:#fff,rx:10,ry:10
    classDef master fill:#fd79a8,stroke:#e84393,color:#fff,rx:10,ry:10
    classDef afpparent fill:#ffeaa7,stroke:#fdcb6e,color:#333,rx:10,ry:10
    classDef worker fill:#55efc4,stroke:#00b894,color:#333,rx:10,ry:10
    classDef cnid fill:#fab1a0,stroke:#e17055,color:#333,rx:10,ry:10
    classDef storage fill:#dfe6e9,stroke:#636e72,color:#333,rx:10,ry:10

AppleDouble Metadata Structure

Netatalk uses AppleDouble format for storing Mac metadata on Unix filesystems. The structure provides comprehensive metadata storage:

Implementation Files:

%%{ init: { 'themeVariables': { 'fontSize': '14px' }, 'flowchart': { 'nodeSpacing': 15, 'rankSpacing': 40, 'subGraphTitleMargin': { 'top': 10, 'bottom': 5 } } } }%%
graph TB
    subgraph adfile["AppleDouble File Structure"]
        Header["AppleDouble Header<br/>26 bytes"]:::header
        Header --> NumEntries["Number of Entries<br/>2 bytes"]:::header
        NumEntries --> EntryDesc["Entry Descriptors<br/>12 bytes each"]:::header
        EntryDesc --> DataFork["Data Fork (ID 1)"]:::entry
        EntryDesc --> ResAttrs["Resource Fork (ID 2)"]:::entry
        EntryDesc --> FileName["Real Name (ID 3)"]:::entry
        EntryDesc --> Comment["Comment (ID 4)"]:::entry
        EntryDesc --> FinderInfo["Finder Info (ID 9)<br/>32 bytes"]:::entry
        EntryDesc --> FileDates["File Dates (ID 8)<br/>16 bytes"]:::entry
        EntryDesc --> AFPInfo["AFP File Info (ID 14)"]:::entry
    end

    subgraph private["Netatalk Private Entries"]
        P1["PRIVDEV (ID 16)"]:::private
        P2["PRIVINO (ID 17)"]:::private
        P3["PRIVSYN (ID 18)"]:::private
        P4["PRIVID (ID 19)"]:::private
    end

    classDef header fill:#a29bfe,stroke:#6c5ce7,color:#fff,rx:10,ry:10
    classDef entry fill:#55efc4,stroke:#00b894,color:#333,rx:10,ry:10
    classDef private fill:#fab1a0,stroke:#e17055,color:#333,rx:10,ry:10

AppleDouble Header Structure

// AppleDouble header format (from include/atalk/adouble.h)
// AD_HEADER_LEN = 26 bytes: Magic(4) + Version(4) + Filler(16) + NumEntries(2)
// AD_ENTRY_LEN  = 12 bytes per entry descriptor

// In-memory entry representation
struct ad_entry {
    off_t     ade_off;   // Offset to entry data
    ssize_t   ade_len;   // Length of entry data
};

Character Conversion Pipeline

Netatalk implements character set conversion between Mac and Unix filesystems:

Implementation Files:

%%{ init: { 'themeVariables': { 'fontSize': '14px' }, 'flowchart': { 'nodeSpacing': 15, 'rankSpacing': 40, 'subGraphTitleMargin': { 'top': 10, 'bottom': 5 } } } }%%
flowchart TD
    MacClient["AFP Client<br/>Mac Roman / UTF-8"]:::client

    subgraph pipeline["Character Conversion Pipeline"]
        Input["Input Character Data"]:::input
        DetectEnc["Detect Encoding<br/>Mac Roman vs UTF-8"]:::process

        subgraph m2u["Mac → Unix"]
            MacToUTF8["Convert to UTF-8"]:::convert
            ApplyCaseM2U["Case Folding<br/>AFPVOL_MTOU* flags"]:::convert
            HandleIllegal["Handle Illegal Sequences<br/>AFPVOL_EILSEQ"]:::convert
        end

        subgraph u2m["Unix → Mac"]
            UTF8ToMac["Convert from UTF-8"]:::convert
            ApplyCaseU2M["Case Folding<br/>AFPVOL_UTOM* flags"]:::convert
            EncodeResult["Encode for AFP Client"]:::convert
        end

        FilesystemPath["Unix Filesystem Path"]:::storage
        MacPath["AFP Client Path"]:::client
    end

    MacClient --> Input --> DetectEnc --> MacToUTF8 --> ApplyCaseM2U --> HandleIllegal --> FilesystemPath
    FilesystemPath --> UTF8ToMac --> ApplyCaseU2M --> EncodeResult --> MacPath --> MacClient

    classDef client fill:#74b9ff,stroke:#0984e3,color:#fff,rx:10,ry:10
    classDef input fill:#dfe6e9,stroke:#636e72,color:#333,rx:10,ry:10
    classDef process fill:#a29bfe,stroke:#6c5ce7,color:#fff,rx:10,ry:10
    classDef convert fill:#55efc4,stroke:#00b894,color:#333,rx:10,ry:10
    classDef storage fill:#dfe6e9,stroke:#636e72,color:#333,rx:10,ry:10

Character Set Implementation

// Character conversion control (from include/atalk/unicode.h)
struct charset_functions {
    const char *name;
    const long kTextEncoding;
    size_t (*pull)(void *, char **inbuf, size_t *inbytesleft,
                   char **outbuf, size_t *outbytesleft);
    size_t (*push)(void *, char **inbuf, size_t *inbytesleft,
                   char **outbuf, size_t *outbytesleft);
    uint32_t flags;
    const char *iname;
    struct charset_functions *prev, *next;
};

// Volume character conversion flags (from include/atalk/volume.h)
#define AFPVOL_MTOUUPPER    (1 << 0)  // Mac→Unix uppercase
#define AFPVOL_MTOULOWER    (1 << 1)  // Mac→Unix lowercase
#define AFPVOL_UTOMUPPER    (1 << 2)  // Unix→Mac uppercase
#define AFPVOL_UTOMLOWER    (1 << 3)  // Unix→Mac lowercase
#define AFPVOL_EILSEQ       (1 << 20) // Preserve illegal sequences

Detailed Request Processing Flow

1. Client Connection Establishment

Implementation Files:

sequenceDiagram
    participant Client as AFP Client
    participant Parent as afpd Parent
    participant Worker as afpd Worker (forked)
    participant Auth as UAM Module

    Client->>Parent: TCP Connect (port 548)
    Parent->>Parent: poll() detects LISTEN_FD
    Parent->>Worker: fork() via dsi_getsession()

    Client->>Worker: DSI OpenSession
    Worker-->>Client: Session parameters

    Client->>Worker: FPLogin (username/password)
    Worker->>Auth: Authenticate user
    Auth-->>Worker: Authentication result
    Worker-->>Client: Login response

    Note over Client,Worker: Session established

2. Volume Mount Process

Implementation Files:

sequenceDiagram
    participant Client as AFP Client
    participant AFP as afpd Worker
    participant Vol as Volume Manager
    participant CNID as CNID System
    participant FS as Filesystem

    Client->>AFP: FPOpenVol (volume name)
    AFP->>Vol: Locate volume config
    Vol->>FS: Check volume path access
    Vol->>CNID: Initialize CNID database
    CNID-->>Vol: Database ready
    Vol-->>AFP: Volume opened
    AFP-->>Client: Volume parameters

3. File Operation Processing

Read Operation Flow

Implementation Files:

sequenceDiagram
    participant Client as AFP Client
    participant AFP as afpd Worker
    participant CNID as CNID System
    participant FS as Unix Filesystem

    Client->>AFP: FPOpenFork (file path)
    AFP->>CNID: Resolve path to CNID
    CNID-->>AFP: File CNID + metadata
    AFP->>FS: Open file descriptor
    AFP-->>Client: Fork reference number

    Client->>AFP: FPRead (offset, length)
    AFP->>FS: Read from filesystem
    FS-->>AFP: File data
    AFP-->>Client: File data via DSI stream

Write Operation Flow

Implementation Files:

sequenceDiagram
    participant Client as AFP Client
    participant AFP as afpd Worker
    participant FS as Unix Filesystem
    participant AD as AppleDouble

    Client->>AFP: FPWrite (data, offset, length)
    AFP->>FS: write() to file
    FS-->>AFP: Write confirmation
    AFP->>AD: Update modification time
    AD->>FS: Sync metadata (xattr or ._ file)
    AFP-->>Client: Write confirmation

4. Directory Management Architecture

Core Directory Structures

Netatalk implements directory management through two primary data structures (from include/atalk/directory.h):

Implementation Files:

// Core directory object (from include/atalk/directory.h)
struct dir {
    bstring d_fullpath;          // Complete Unix path to directory (or file)
    bstring d_m_name;            // Mac name representation
    bstring d_u_name;            // Unix name (may share storage with d_m_name)
    ucs2_t *d_m_name_ucs2;      // Mac name as UCS2 Unicode

    // Queue and cache management
    qnode_t *qidx_node;          // Position in directory cache queue
    time_t d_ctime;              // Inode ctime for reenumeration detection
    time_t dcache_ctime;         // Directory cache ctime
    ino_t dcache_ino;            // Inode number for change detection

    // Directory metadata and state
    int d_flags;                 // Directory flags (filesystem type, cache state)
    cnid_t d_pdid;               // Parent directory CNID
    cnid_t d_did;                // Directory CNID
    uint32_t d_offcnt;           // Offspring count (when valid)
    uint16_t d_vid;              // Volume ID (for multi-volume cache)
    uint32_t d_rights_cache;     // Cached access rights (mode + ACL)
};

// Path resolution structure
struct path {
    int m_type;                  // Mac name type (long name, Unicode)
    char *m_name;                // Mac name pointer
    char *u_name;                // Unix name pointer
    cnid_t id;                   // File/directory CNID (for getmetadata)
    struct dir *d_dir;           // Associated directory object

    // Filesystem state caching
    int st_valid;                // Whether stat information is valid
    int st_errno;                // Last stat() error code
    struct stat st;              // Cached stat() results
};

// Directory flags and constants
#define DIRF_FSMASK    (3<<0)    // Filesystem type mask
#define DIRF_UFS       (1<<1)    // UFS filesystem
#define DIRF_ISFILE    (1<<3)    // Cached file, not directory
#define DIRF_OFFCNT    (1<<4)    // Offspring count is valid
#define DIRF_CNID      (1<<5)    // Requires CNID renumeration

// Reserved directory CNIDs
#define DIRDID_ROOT_PARENT  htonl(1)  // Parent of root directory
#define DIRDID_ROOT         htonl(2)  // Root directory CNID

Directory Enumeration

sequenceDiagram
    participant Client as AFP Client
    participant AFP as afpd Worker
    participant DirCache as Directory Cache
    participant CNID as CNID System
    participant FS as Unix Filesystem

    Client->>AFP: FPOpenDir (directory path)
    AFP->>AFP: Parse path into struct path
    AFP->>DirCache: Lookup struct dir in cache

    alt Directory cached and valid
        DirCache-->>AFP: Return cached struct dir
        Note over AFP: Check d_ctime vs dcache_ctime
    else Directory not cached/invalid
        AFP->>FS: stat() directory for metadata
        AFP->>CNID: Get/create directory CNID
        AFP->>DirCache: Create new struct dir object
    end

    AFP-->>Client: Directory reference (d_did)

    Client->>AFP: FPEnumerate (count, start index)
    AFP->>FS: readdir() directory contents
    FS-->>AFP: Directory entries

    loop For each directory entry
        AFP->>CNID: Resolve u_name to CNID + metadata
        CNID-->>AFP: File/directory metadata
        AFP->>DirCache: Update or create child struct dir
    end

    AFP-->>Client: Directory listing with CNIDs

Directory Cache Management

The directory cache uses a hash-based system with LRU eviction and change detection:

// Cache invalidation triggers:
// 1. d_ctime != dcache_ctime   (inode ctime changed)
// 2. dcache_ino != current_ino (inode number changed)
// 3. DIRF_CNID flag set        (requires renumeration)
// 4. Volume remount            (d_vid mismatch)

5. Parent Main Event Loop (Poll-Driven Hint Batching)

The parent afpd process uses a single-threaded poll()-based event loop (etc/afpd/main.c) to handle client connections, IPC cache hints from child workers, and hint batching.

The poll timeout is dynamic: 50ms when hints are buffered (flush every 50ms), or infinite (-1) when no hints are pending. Signals are blocked during event processing and unblocked during poll(), ensuring the child process table is stable when iterating during hint flush.

Key ordering invariant: SIGCHLD is always processed first (step 1), to remove closed children from the table before hint flush or fd events access it, to eliminate writes to broken pipes and enable direct table iteration.

%%{ init: { 'themeVariables': { 'fontSize': '14px' }, 'flowchart': { 'nodeSpacing': 15, 'rankSpacing': 25 } } }%%
flowchart TD
    A([Main Event Loop]):::loop --> SIG[Unblock signals]:::signal
    SIG --> B{"poll()<br/>with dynamic timeout"}:::poll
    B --> MASK[Block signals]:::signal
    MASK --> S["1 · child_handler()<br/>Remove dead children"]:::sigchld
    S --> E{"2 · ret < 0 ?"}:::decision

    E -->|EINTR| A
    E -->|Error| X([break — exit loop]):::error

    E -->|No error| R{"3 · reloadconfig ?"}:::decision
    R -->|Yes| RC["Reload config<br/>Reset sockets<br/>SIGHUP children"]:::config
    RC --> A

    R -->|No| FD{"4 · ret > 0 ?"}:::decision

    FD -->|"Yes: fd events"| LOOP["for each ready fd"]:::fdloop
    LOOP --> LFD{"fd type?"}:::decision
    LFD -->|LISTEN_FD| K["dsi_start()<br/>fork new child"]:::listen
    LFD -->|IPC_FD| C["ipc_server_read()"]:::ipc
    C --> D["ipc_relay_cache_hint()<br/>Append to hint_buf"]:::ipc
    K --> FL
    D --> FL

    FD -->|"No: timeout"| FL

    FL{"5 · Flush?<br/>count > 0 AND<br/>timeout OR buf full"}:::decision
    FL -->|Yes| G["hint_flush_pending()<br/>Sort → serialize → write<br/>to child hint pipes"]:::flush
    FL -->|No| A
    G --> A

    B -.- NOTE["timeout = 50ms when<br/>hint_buf count > 0<br/>timeout = ∞ when empty"]:::note

    classDef loop fill:#4a90d9,stroke:#2c5f8a,color:#fff,rx:20,ry:20
    classDef signal fill:#e8d44d,stroke:#b8a730,color:#333,rx:10,ry:10
    classDef poll fill:#f5f5f5,stroke:#999,color:#333,rx:15,ry:15
    classDef sigchld fill:#ff9f43,stroke:#cc7a2e,color:#fff,rx:10,ry:10
    classDef decision fill:#f8f4e8,stroke:#c4a63b,color:#333
    classDef config fill:#a8d8ea,stroke:#5fa8c8,color:#333,rx:10,ry:10
    classDef error fill:#ee5a5a,stroke:#b83030,color:#fff,rx:20,ry:20
    classDef fdloop fill:#dfe6e9,stroke:#636e72,color:#333,rx:10,ry:10
    classDef listen fill:#81ecec,stroke:#00b894,color:#333,rx:10,ry:10
    classDef ipc fill:#a29bfe,stroke:#6c5ce7,color:#fff,rx:10,ry:10
    classDef flush fill:#55efc4,stroke:#00b894,color:#333,rx:10,ry:10
    classDef note fill:#ffeaa7,stroke:#fdcb6e,color:#666,rx:5,ry:5,stroke-dasharray:5 5

Hint Data Flow

When a child afpd worker modifies a file or directory, it sends an IPC_CACHE_HINT message through its IPC socketpair to the parent. The parent buffers these hints (max 50ms) and relays them to all sibling workers in batched, priority-sorted writes.

While hint_flush_pending() writes to child pipes, new IPC messages from children accumulate in the kernel socket buffer and are read on the next poll() iteration. No data is lost.

%%{ init: { 'themeVariables': { 'fontSize': '14px' }, 'flowchart': { 'nodeSpacing': 15, 'rankSpacing': 40, 'subGraphTitleMargin': { 'top': 10, 'bottom': 5 } } } }%%
flowchart LR
    subgraph children["Child Workers"]
        CW["Child afpd<br/>modifies file/dir"]:::child
    end

    CW -->|"ipc_child_write()<br/>IPC_CACHE_HINT"| KB["Kernel<br/>socket buffer"]:::kernel

    subgraph parent["Parent Process"]
        direction TB
        KB -->|"poll() POLLIN"| ISR["ipc_server_read()"]:::ipc
        ISR --> RCH["ipc_relay_cache_hint()"]:::ipc
        RCH --> BUF["hint_buf<br/>(max 128 entries)"]:::buf
        BUF --> TRIG{"Flush trigger:<br/>50ms timeout<br/>OR buf full"}:::decision
        TRIG --> SORT["Sort by priority:<br/>REFRESH → DELETE<br/>→ DELETE_CHILDREN"]:::flush
        SORT --> SER["Serialize + write<br/>PIPE_BUF-safe chunks<br/>non-blocking"]:::flush
    end

    SER -->|"hint pipe<br/>(skip source)"| C1["child₁"]:::child
    SER -->|"hint pipe<br/>(skip source)"| C2["child₂"]:::child
    SER -->|"hint pipe<br/>(skip source)"| C3["child₃"]:::child

    classDef child fill:#81ecec,stroke:#00b894,color:#333,rx:10,ry:10
    classDef kernel fill:#dfe6e9,stroke:#636e72,color:#333,rx:10,ry:10
    classDef ipc fill:#a29bfe,stroke:#6c5ce7,color:#fff,rx:10,ry:10
    classDef buf fill:#ffeaa7,stroke:#fdcb6e,color:#333,rx:10,ry:10
    classDef decision fill:#f8f4e8,stroke:#c4a63b,color:#333
    classDef flush fill:#55efc4,stroke:#00b894,color:#333,rx:10,ry:10

CNID Database Operations

CNID Resolution Process

Netatalk supports multiple CNID backends: Berkeley DB (via cnid_dbd), SQLite, and MySQL.

Implementation Files:

%%{ init: { 'themeVariables': { 'fontSize': '14px' }, 'flowchart': { 'nodeSpacing': 15, 'rankSpacing': 25 } } }%%
graph LR
    A["File Path"]:::input --> B["Path Parser"]:::process
    B --> C["Component Lookup"]:::process
    C --> D{"CNID Cache"}:::decision
    D -->|Hit| E["Return CNID"]:::output
    D -->|Miss| F{"Backend"}:::decision
    F -->|BDB| G["cnid_dbd"]:::bdb
    F -->|SQLite| H["SQLite DB"]:::sqlite
    F -->|MySQL| I["MySQL DB"]:::mysql
    G & H & I --> J["CNID Record"]:::output
    J --> K["Update Cache"]:::process
    K --> E

    classDef input fill:#dfe6e9,stroke:#636e72,color:#333,rx:10,ry:10
    classDef process fill:#a29bfe,stroke:#6c5ce7,color:#fff,rx:10,ry:10
    classDef decision fill:#f8f4e8,stroke:#c4a63b,color:#333
    classDef output fill:#55efc4,stroke:#00b894,color:#333,rx:10,ry:10
    classDef bdb fill:#fab1a0,stroke:#e17055,color:#333,rx:10,ry:10
    classDef sqlite fill:#fab1a0,stroke:#e17055,color:#333,rx:10,ry:10
    classDef mysql fill:#fab1a0,stroke:#e17055,color:#333,rx:10,ry:10

Database Transaction Flow (BDB Backend)

Implementation Files:

sequenceDiagram
    participant AFP as afpd Worker
    participant Meta as cnid_metad
    participant DBD as cnid_dbd
    participant BDB as Berkeley DB

    AFP->>Meta: CNID request
    Meta->>DBD: Forward request
    DBD->>BDB: Begin transaction

    DBD->>BDB: Database operation
    BDB-->>DBD: Operation result

    alt Success
        DBD->>BDB: Commit transaction
        BDB-->>DBD: Commit success
    else Failure
        DBD->>BDB: Rollback transaction
        BDB-->>DBD: Rollback complete
    end

    DBD-->>Meta: Response
    Meta-->>AFP: CNID result

AppleDouble Metadata Architecture

Core AppleDouble Structure

Netatalk implements AppleDouble metadata handling through a comprehensive structure (from include/atalk/adouble.h):

Implementation Files:

// Core AppleDouble object (from include/atalk/adouble.h)
struct adouble {
    uint32_t ad_magic;              // Magic number (0x00051607)
    uint32_t ad_version;            // AD_VERSION2=0x00020000 or AD_VERSION_EA=0x00020002
    char ad_filler[16];
    struct ad_entry ad_eid[ADEID_MAX]; // Entry table (19 entries max)

    struct ad_fd ad_data_fork;      // Data fork fd
    struct ad_fd ad_resource_fork;  // Resource fork fd
    struct ad_fd *ad_rfp;           // Resource fork pointer
    struct ad_fd *ad_mdp;           // Metadata pointer

    int ad_vers;                    // Version info
    int ad_adflags;                 // Open flags
    uint32_t ad_inited;             // Init state (AD_INITED = 0xad494e54)
    int ad_options;                 // Volume options

    int ad_refcount;                // Overall reference count
    int ad_data_refcount;           // Data fork references
    int ad_meta_refcount;           // Metadata references
    int ad_reso_refcount;           // Resource fork references

    off_t ad_rlen;                  // Resource fork length
    char *ad_name;                  // Mac name
    struct adouble_fops *ad_ops;    // Function operations pointer
    uint16_t ad_open_forks;         // Open forks count
    size_t valid_data_len;          // Bytes read into ad_data
    char ad_data[AD_DATASZ_MAX];    // Metadata buffer (1024 bytes)
};

struct ad_fd {
    int adf_fd;                     // File descriptor (-1: invalid)
    char *adf_syml;                 // Symlink target
    int adf_flags;                  // Open flags
    adf_lock_t *adf_lock;          // Lock array
    int adf_refcount, adf_lockcount, adf_lockmax;
};

AppleDouble Format Versions

// Format version data sizes (compile-time validated)
#define AD_DATASZ2      741   // AppleDouble v2 format size
#define AD_DATASZ_EA    402   // Extended attributes format size
#define AD_DATASZ_OSX   82    // Mac OS X format size
#define AD_DATASZ_MAX   1024  // Maximum buffer size

Metadata Read Process

sequenceDiagram
    participant Client as AFP Client
    participant AFP as afpd Worker
    participant AD as AppleDouble Handler
    participant FS as Filesystem
    participant EA as Extended Attributes

    Client->>AFP: Request file metadata
    AFP->>AD: ad_open(path, ADFLAGS_HF)
    AD->>AD: ad_init() — set version, ops

    alt Extended Attributes mode (AD_VERSION_EA)
        AD->>FS: open() data file for metadata fd
        AD->>EA: fgetxattr() for each entry type
    else AppleDouble v2 mode (AD_VERSION2)
        AD->>FS: open() ._ AppleDouble file
        AD->>FS: read() header + entries
    end

    AD->>AD: ad_init_offsets()
    AD-->>AFP: struct adouble with metadata
    AFP-->>Client: File information

Metadata Write Process with Locking

sequenceDiagram
    participant Client as AFP Client
    participant AFP as afpd Worker
    participant AD as AppleDouble Handler
    participant FS as Filesystem
    participant EA as Extended Attributes

    Client->>AFP: Set file metadata
    AFP->>AD: ad_open(ADFLAGS_HF | ADFLAGS_RDWR)
    AD->>AD: ad_lock(ADLOCK_WR) — exclusive

    AFP->>AD: ad_setattr() or ad_setdate()

    alt Extended Attributes mode
        AD->>EA: fsetxattr() for modified entries
    else AppleDouble v2 mode
        AD->>AD: ad_rebuild_header()
        AD->>FS: write() complete file
    end

    AD->>AD: ad_flush()
    AD->>AD: ad_unlock()
    AFP-->>Client: Success response

File Locking Architecture

// Lock types (from include/atalk/adouble.h)
#define ADLOCK_RD           (1<<0)    // Read lock
#define ADLOCK_WR           (1<<1)    // Write lock
#define ADLOCK_FILELOCK     (1<<3)    // File-level lock

// Lock ranges (using high offset values to avoid data range conflicts)
#define AD_FILELOCK_OPEN_WR      (AD_FILELOCK_BASE + 0)  // Data fork write
#define AD_FILELOCK_OPEN_RD      (AD_FILELOCK_BASE + 1)  // Data fork read
#define AD_FILELOCK_RSRC_OPEN_WR (AD_FILELOCK_BASE + 2)  // Resource fork write
#define AD_FILELOCK_RSRC_OPEN_RD (AD_FILELOCK_BASE + 3)  // Resource fork read

Spotlight Search Integration

Search Request Flow

Spotlight search is implemented using GNOME Tracker as the indexing backend with SPARQL queries:

%%{ init: { 'themeVariables': { 'fontSize': '14px' }, 'flowchart': { 'nodeSpacing': 15, 'rankSpacing': 25 } } }%%
sequenceDiagram
    participant Client as AFP Client
    participant AFP as afpd Worker
    participant SL as Spotlight Module
    participant Tracker as GNOME Tracker
    participant Index as Search Index

    Client->>AFP: Spotlight search query
    AFP->>SL: Parse search criteria
    SL->>Tracker: SPARQL query
    Tracker->>Index: Search index
    Index-->>Tracker: Matching files
    Tracker-->>SL: Result set
    SL->>SL: Filter by volume/permissions
    SL-->>AFP: Filtered results
    AFP-->>Client: Search results

Error Handling and Recovery

Connection Error Recovery

%%{ init: { 'themeVariables': { 'fontSize': '14px' }, 'flowchart': { 'nodeSpacing': 15, 'rankSpacing': 25 } } }%%
graph TD
    A["Connection Error"]:::error --> B{"Error Type"}:::decision
    B -->|Network| C["Reconnect with backoff"]:::recovery
    B -->|Protocol| D["Reset session state"]:::recovery
    B -->|Authentication| E["Re-authenticate user"]:::recovery
    B -->|Resource| F["Free resources and retry"]:::recovery

    C & D & E & F --> G["Resume operations"]:::success

    classDef error fill:#ee5a5a,stroke:#b83030,color:#fff,rx:10,ry:10
    classDef decision fill:#f8f4e8,stroke:#c4a63b,color:#333
    classDef recovery fill:#a29bfe,stroke:#6c5ce7,color:#fff,rx:10,ry:10
    classDef success fill:#55efc4,stroke:#00b894,color:#333,rx:10,ry:10

Database Error Recovery

%%{ init: { 'themeVariables': { 'fontSize': '14px' }, 'flowchart': { 'nodeSpacing': 15, 'rankSpacing': 25 } } }%%
graph TD
    A["Database Error"]:::error --> B{"Error Type"}:::decision
    B -->|Corruption| C["Database repair"]:::recovery
    B -->|Lock timeout| D["Retry with backoff"]:::recovery
    B -->|Disk full| E["Alert and cleanup"]:::recovery
    B -->|Permission| F["Check filesystem access"]:::recovery

    C & D & F --> G["Resume operations"]:::success
    E --> H["Graceful degradation"]:::warning

    classDef error fill:#ee5a5a,stroke:#b83030,color:#fff,rx:10,ry:10
    classDef decision fill:#f8f4e8,stroke:#c4a63b,color:#333
    classDef recovery fill:#a29bfe,stroke:#6c5ce7,color:#fff,rx:10,ry:10
    classDef success fill:#55efc4,stroke:#00b894,color:#333,rx:10,ry:10
    classDef warning fill:#ffeaa7,stroke:#fdcb6e,color:#333,rx:10,ry:10

Implementation Files Reference

Component File Description
Main event loop etc/afpd/main.c Poll loop, signal handling, fd dispatch
Hint buffering & flush libatalk/util/server_ipc.c ipc_relay_cache_hint(), hint_flush_pending()
Hint API include/atalk/server_ipc.h HINT_BUF_SIZE, HINT_FLUSH_INTERVAL_MS, public API
Child table libatalk/util/server_child.c server_child_t, child add/remove/kill
Hint receiver etc/afpd/dircache.c Child-side hint processing, dircache invalidation
CNID interface libatalk/cnid/cnid.c Backend-agnostic CNID operations
CNID BDB backend libatalk/cnid/dbd/ Berkeley DB client library
CNID SQLite backend libatalk/cnid/sqlite/ SQLite embedded backend
CNID MySQL backend libatalk/cnid/mysql/ MySQL backend
CNID daemon etc/cnid_dbd/main.c BDB-based CNID service
AppleDouble libatalk/adouble/ad_open.c File handling and structure management
Locking libatalk/adouble/ad_lock.c File locking mechanisms
Character conversion libatalk/unicode/charcnv.c Core conversion engine
Spotlight etc/afpd/spotlight.c Search indexing with GNOME Localsearch

This data flow documentation provides a comprehensive view of how information moves through the Netatalk system, enabling better understanding of performance characteristics, optimization opportunities, and failure modes.

Footnotes

This is a mirror of the Netatalk GitHub Wiki

Last updated 2026-04-06

Valid HTML5 Valid CSS

Netatalk and the Netatalk website are licensed under the GNU General Public License 2.0.