> ## Documentation Index
> Fetch the complete documentation index at: https://docs.labtrace.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Upload Process

> Technical details of how files are processed and stored in LabTrace

# Upload Process

The file upload process in LabTrace is a sophisticated multi-step operation that ensures file integrity, immutability, and verifiability through blockchain technology and distributed storage.

## Process Overview

```mermaid theme={null}
flowchart TD
    A[File Upload] --> B[File Validation]
    B --> C[Generate CID Hash]
    C --> L[Store in AWS S3 if Public File] --> D[Create JSON file]
    C --> D[Create JSON file]
    D --> E[Store JSON file in IPFS]
    E --> F[Record on Blockchain]
    F --> G[Update Database]
    G --> H[Generate Certificates]
    H --> I[Upload Complete]
```

## Step-by-Step Technical Process

### 1. File Validation & Processing

<Steps>
  <Step title="File Reception">
    The system receives the uploaded file through the web interface or API
  </Step>

  <Step title="Security Checks">
    File is scanned for malware and validated against allowed file types
  </Step>

  <Step title="Size Validation">
    File size is checked against project and user limits
  </Step>

  <Step title="Content Analysis">
    Basic file metadata is extracted (size, type, timestamps)
  </Step>
</Steps>

### 2. CID Hash Generation

The **Content Identifier (CID)** is generated using IPFS's content-addressing system:

```javascript theme={null}
// Pseudocode for CID generation
const fileBuffer = readFileContent(uploadedFile);
const hash = await ipfs.add(fileBuffer, {
  onlyHash: true,  // Generate hash without storing
  cidVersion: 1,   // Use CIDv1 format
  hashAlg: 'sha2-256'  // SHA-256 hashing algorithm
});
const cidHash = hash.cid.toString();
```

**Key Properties of CID Hash:**

* **Deterministic**: Same content always produces same CID
* **Unique**: Different content produces different CIDs
* **Verifiable**: Anyone can verify file integrity using the CID
* **Immutable**: Content cannot be changed without changing the CID

### 3. JSON file Creation

A JSON file object is created with different content based on file type:

**For Public Files:**

```json theme={null}
{
  "FileHash": "QmX1234567890abcdef...",
  "FileLinkS3": "https://bucket.s3.amazonaws.com/filename.pdf"
}
```

**For Private Files:**

```json theme={null}
{
  "FileHash": "QmX1234567890abcdef..."
}
```

<Note>
  **Important**: Private files do NOT include `FileLinkS3` in their metadata because they are not stored in AWS S3 - only their metadata is stored on IPFS.
</Note>

**Key Differences:**

* **Public Files**: Stored in AWS S3 + metadata on IPFS
* **Private Files**: Stored on IPFS only + metadata on IPFS
* **Both Types**: JSON file is stored on IPFS and operations recorded on blockchain

### 4. Storage Process

<CardGroup cols={2}>
  <Card title="Public Files" icon="globe">
    * **File Content**: Stored in AWS S3 (publicly accessible)
    * **JSON file**: Stored on IPFS
    * **Contains**: FileHash + FileLinkS3
  </Card>

  <Card title="Private Files" icon="lock">
    * **File Content**: Stored nowhere
    * **JSON file**: Stored on IPFS
    * **Contains**: FileHash
  </Card>

  <Card title="Pinning Service" icon="thumbtack">
    JSON file files are pinned to ensure persistence in the IPFS network
  </Card>

  <Card title="Replication" icon="copy">
    IPFS content is replicated across multiple nodes for redundancy
  </Card>
</CardGroup>

**Storage Benefits:**

* **Content Addressing**: Files are accessed by their hash, not location
* **Deduplication**: Identical files share the same storage space
* **Hybrid Storage**: Public files get S3 performance + IPFS verification
* **Privacy Control**: Private files remain fully decentralized
* **Permanent**: Pinned metadata remains accessible indefinitely

### 5. Blockchain Transaction Recording

{/* ```python
# Pseudocode for blockchain transaction
transaction = {
  "type": "application_call",
  "sender": user_blockchain_address,
  "app_id": labtrace_smart_contract_id,
  "note": json.dumps({
      "operation": "file_upload",
      "file_cid": cidHash,
      "metadata_cid": metadataCid,
      "project_id": projectId,
      "timestamp": current_timestamp,
      "user_id": userId
  })
}
``` */}

**Blockchain Record Contains:**

* **Operation Type**: file\_upload, file\_delete, etc.
* **JSON file CID**: Content identifier for the JSON file
* **Project Association**: Which project the file belongs to according to the project smart contract asset
* **User Information**: The user address that performed the operation
* **Timestamp**: When the operation occurred

### 6. Database Updates

<AccordionGroup>
  <Accordion title="File Records">
    <ul>
      <li>File metadata stored in PostgreSQL database</li>
      <li>Relationship to projects and users established</li>
      <li>Search indexes created for efficient queries</li>
      <li>Access permissions configured</li>
    </ul>
  </Accordion>

  <Accordion title="Blockchain References">
    <ul>
      <li>Transaction hash stored for verification</li>
      <li>Block number and timestamp recorded</li>
      <li>Smart contract interaction details saved</li>
      <li>Verification status tracked</li>
    </ul>
  </Accordion>

  <Accordion title="Audit Trail">
    <ul>
      <li>Complete operation history logged</li>
      <li>User actions tracked with timestamps</li>
      <li>System events recorded for debugging</li>
      <li>Performance metrics collected</li>
    </ul>
  </Accordion>
</AccordionGroup>

## Primary and Secondary Files

Primary files are the main files that are uploaded to the project.

Secondary files are files that are linked to a primary file, or many primary files, in the same project.

In addition, for secondary files, the user has to provide a link to the primary file or a description of the procedure.

This link from a secondary file\*\* \*\*to a primary file is also added into the note field of the blockchain transaction.

## Error Handling & Recovery

<CardGroup cols={2}>
  <Card title="Upload Failures" icon="exclamation-triangle">
    Robust retry mechanisms and partial upload recovery
  </Card>

  <Card title="IPFS Issues" icon="network-wired">
    Automatic failover to backup IPFS nodes
  </Card>

  <Card title="Blockchain Delays" icon="clock">
    Graceful handling of network congestion
  </Card>

  <Card title="Storage Redundancy" icon="shield">
    Multiple storage layers ensure data safety
  </Card>
</CardGroup>

{/* ## Security Measures

<Warning>
All uploaded files are encrypted before storage and only decrypted when accessed by authorized users.
</Warning>

- **End-to-End Encryption**: Files encrypted with user-specific keys
- **Access Control**: Role-based permissions enforced
- **Audit Logging**: All operations logged for security review
- **Integrity Checks**: Continuous verification of stored files  */}