Skip to main content

Data Flow — Write Operation

This page traces the complete lifecycle of a single UPLOAD operation through the system. Every read and write follows a similar pattern, but writes are the most complex because they involve the TO-Multicast protocol.

Sequence Diagram​

Phase-by-Phase Breakdown​

Phase 1 — Login (Authentication)​

  1. The client sends UserCredentials (username + plaintext password) over mTLS to the AuthService
  2. The AuthService hashes the password with the user's stored salt using PBKDF2 (260,000 iterations)
  3. If the hash matches, the AuthService generates a UUID session token and returns it
  4. The client stores this token locally and sends it with every subsequent operation

Phase 2 — Write Operation (Client → Node)​

  1. The client constructs a FileOperation using the static factory: FileOperation.upload(username, filename, data)
  2. Two critical security fields are auto-populated:
    • timestamp: System.currentTimeMillis() — for replay window enforcement
    • nonce: UUID.randomUUID().toString() — for replay uniqueness
  3. The client calls node.handleClientOperation(op, token) over mTLS RMI

Phase 3 — Authentication Check (Node → Auth)​

  1. Before touching the file system, the node calls authService.validateToken(token)
  2. The AuthService looks up the token in its internal map
  3. If valid, returns the username associated with the token
  4. If invalid or expired, the node immediately returns an OperationResult with success=false

This check happens before any file I/O — an attacker can't even list files without a valid token.

Phase 4 — Totally Ordered Multicast​

This is the heart of the distributed consistency guarantee. Here's what happens:

  1. Node 0 (the receiving node) calls clock.tick() to increment its Lamport clock. The new timestamp ts=1 is attached to the ClockMessage.

  2. Node 0 calls peer.receiveMulticast(msg) on every node, including itself.

  3. Each receiving node:

    • Calls clock.update(msg.timestamp) → clock = max(local, msg.ts) + 1
    • Enqueues the message in a PriorityQueue sorted by (timestamp, senderId)
    • Broadcasts sendAck(fromNodeId, messageId, newClockValue) to all nodes

  4. The ACKs serve two purposes:

    • They tell the originator "I received this message"
    • Their timestamps prove "I have nothing earlier in flight"

Phase 5 — Delivery​

A background thread (the delivery loop, running every 50 ms) polls the queue:

  1. Look at the head of the priority queue
  2. Has this message received ACKs from all nodes?
  3. Is every ACK's timestamp strictly greater than the message's timestamp?
  4. If yes: dequeue the message, execute the file operation, complete the CompletableFuture
  5. If no: wait for the next loop iteration

:::tip Why "strictly greater"? If an ACK arrives with timestamp = msg.timestamp, that means the sending node's clock hasn't advanced past this message — it might still be processing an earlier message from a different sender. The strict inequality guarantees that no earlier-timestamped message can still be in transit from that node. :::

Phase 6 — Return to Client​

The originating node's handleClientOperation was blocked on a CompletableFuture.get(10, SECONDS). When the delivery loop executes the message, it completes the future, and the result is serialized back to the client over mTLS.

Read Operations​

Read operations (DOWNLOAD, SEARCH, LIST) are much simpler:

  1. Client calls handleClientOperation(readOp, token)
  2. Node validates the token with AuthService
  3. Node reads from the local file system or performs the search
  4. Node returns the result immediately — no multicast needed

Reads are never broadcast. They go to a single node. For linearisable reads, the client is directed to the current Raft leader.

Delete and Rename​

Both DELETE and RENAME follow the same TO-Multicast flow as UPLOAD. They are write operations because they modify the file system state, and that state must be consistent across all replicas.