Skip to main content

S3 Plugin

AWS S3 integration package for cloud storage and CDN functionality in the Token Ring AI ecosystem.

Overview​

The @tokenring-ai/s3 package provides comprehensive AWS S3 integration for the Token Ring AI system. It implements both CDN (Content Delivery Network) and File System providers for seamless cloud storage and content delivery. The package integrates with the Token Ring application framework through plugin architecture and supports configuration-based provider registration.

Key Features​

  • CDN Provider: S3-based Content Delivery Network integration with configurable providers
  • File System Provider: S3-backed file system implementation with full CRUD operations
  • Configuration-driven: Providers registered automatically from application configuration
  • Service Integration: Seamless integration with Token Ring's CDN and FileSystemService
  • Type-safe Configuration: Zod schemas for validation and type safety
  • Plugin Architecture: Automatic service registration through TokenRingApp

Providers​

The package implements two main provider types:

S3CDNProvider​

AWS S3 integration for CDN services, providing content delivery capabilities.

Configuration Schema: S3CDNProviderOptionsSchema

  • bucket: S3 bucket name
  • region: AWS region (e.g., 'us-east-1')
  • accessKeyId: AWS access key ID
  • secretAccessKey: AWS secret access key
  • baseUrl: Custom base URL (optional, defaults to https://{bucket}.s3.amazonaws.com)

S3FileSystemProvider​

S3-backed file system provider with complete file operations.

Configuration Schema: S3FileSystemProviderOptionsSchema

  • bucketName: S3 bucket name
  • clientConfig: AWS SDK client configuration (optional)

Core Components​

CDN Integration​

The S3CDNProvider registers with the CDNService and provides:

interface S3CDNProvider {
  upload(data: Buffer, options?: UploadOptions): Promise<UploadResult>;
  // Uploads content to S3 and returns upload result with URL
  
  delete(url: string): Promise<DeleteResult>;
  // Deletes content from S3 bucket
  
  exists(url: string): Promise<boolean>;
  // Checks if content exists in S3
}

File System Integration​

The S3FileSystemProvider registers with FileSystemService and provides:

interface S3FileSystemProvider {
  writeFile(fsPath: string, content: string | Buffer): Promise<boolean>;
  // Writes file content to S3
  
  appendFile(filePath: string, content: string | Buffer): Promise<boolean>;
  // Appends content to an existing file or creates it if not exists
  
  readFile(fsPath: string, encoding?: BufferEncoding | "buffer"): Promise<any>;
  // Reads file content from S3
  
  deleteFile(fsPath: string): Promise<boolean>;
  // Deletes file from S3
  
  exists(fsPath: string): Promise<boolean>;
  // Checks if file exists in S3
  
  stat(fsPath: string): Promise<StatLike>;
  // Gets file statistics
  
  createDirectory(fsPath: string, options?: { recursive?: boolean }): Promise<boolean>;
  // Creates directory (simulated using prefixes)
  
  getDirectoryTree(fsPath: string, params?: DirectoryTreeOptions): AsyncGenerator<string>;
  // Lists directory contents (prefix-based)
  
  copy(sourceFsPath: string, destinationFsPath: string, options?: { overwrite?: boolean }): Promise<boolean>;
  // Copies file within S3
  
  rename(oldPath: string, newPath: string): Promise<boolean>;
  // Renames/moves file within S3
}

Usage Examples​

Configuration-driven Integration​

The plugin automatically registers providers based on application configuration:

// In your Token Ring application configuration
{
  cdn: {
    providers: {
      s3-public: {
        type: "s3",
        bucket: "your-public-bucket",
        region: "us-east-1",
        accessKeyId: process.env.AWS_ACCESS_KEY_ID,
        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
        baseUrl: "https://cdn.example.com"
      }
    }
  },
  filesystem: {
    providers: {
      s3-data: {
        type: "s3",
        bucketName: "your-data-bucket",
        clientConfig: {
          region: "us-east-1"
        }
      }
    }
  }
}

Programmatic Provider Registration​

import { TokenRingApp } from '@tokenring-ai/app';
import { CDNService } from '@tokenring-ai/cdn';
import { FileSystemService } from '@tokenring-ai/filesystem';
import { S3CDNProvider, S3FileSystemProvider } from '@tokenring-ai/s3';

const app = new TokenRingApp();

// Register CDN provider
const cdnService = app.requireServiceByType(CDNService);
cdnService.registerProvider('s3-cdn', new S3CDNProvider({
  bucket: 'my-cdn-bucket',
  region: 'us-east-1',
  accessKeyId: 'your-access-key',
  secretAccessKey: 'your-secret-key',
  baseUrl: 'https://cdn.example.com'
}));

// Register File System provider  
const fileSystemService = app.requireServiceByType(FileSystemService);
fileSystemService.registerFileSystemProvider('s3-fs', new S3FileSystemProvider({
  bucketName: 'my-data-bucket',
  clientConfig: {
    region: 'us-east-1'
  }
}));

Using CDN Service​

import { CDNService } from '@tokenring-ai/cdn';

const cdnService = app.requireServiceByType(CDNService);
const s3Provider = cdnService.getResource('s3-public');

// Upload content
const result = await s3Provider.upload(Buffer.from('Hello from S3!'), {
  filename: 'welcome.txt',
  contentType: 'text/plain',
  metadata: { author: 'User' }
});

console.log('Uploaded to:', result.url);
console.log('File ID:', result.id);

// Check if file exists
const exists = await s3Provider.exists(result.url);
console.log('File exists:', exists);

// Delete the file
const deleteResult = await s3Provider.delete(result.url);
console.log('Delete success:', deleteResult.success);

Using File System Service​

import { FileSystemService } from '@tokenring-ai/filesystem';

const fileSystemService = app.requireServiceByType(FileSystemService);
const s3Fs = fileSystemService.getFileSystem('s3-data');

// Write file
await s3Fs.writeFile('documents/README.md', '# My Documents');

// Read file
const content = await s3Fs.readFile('documents/README.md', 'utf8');
console.log(content);

// Check file existence
const exists = await s3Fs.exists('documents/README.md');
console.log('File exists:', exists);

// Append to file
await s3Fs.appendFile('documents/README.md', '\n## Additional Content');
const updatedContent = await s3Fs.readFile('documents/README.md', 'utf8');
console.log(updatedContent);

// Get file statistics
const stats = await s3Fs.stat('documents/README.md');
console.log('Size:', stats.size);
console.log('Modified:', stats.modified);

// List directory contents
console.log('Directory contents:');
for await (const path of s3Fs.getDirectoryTree('documents')) {
  console.log(path);
}

// Create directory
await s3Fs.createDirectory('new-folder');

// Copy file
await s3Fs.copy('documents/README.md', 'new-folder/README-copy.md');

// Rename file
await s3Fs.rename('new-folder/README-copy.md', 'new-folder/README-new.md');

Configuration Options​

CDN Configuration​

interface S3CDNProviderOptions {
  bucket: string;           // S3 bucket name
  region: string;           // AWS region (e.g., 'us-east-1')
  accessKeyId: string;      // AWS access key ID
  secretAccessKey: string;  // AWS secret access key
  baseUrl?: string;         // Custom base URL
}

interface CDNConfig {
  providers: {
    [providerName: string]: {
      type: 's3';
      bucket: string;
      region: string;
      accessKeyId: string;
      secretAccessKey: string;
      baseUrl?: string;
    };
  };
}

File System Configuration​

interface S3FileSystemProviderOptions {
  bucketName: string;       // S3 bucket name
  clientConfig?: object;    // AWS SDK client configuration
}

interface FilesystemConfig {
  providers: {
    [providerName: string]: {
      type: 's3';
      bucketName: string;
      clientConfig?: object;
    };
  };
}

Integration with Token Ring Ecosystem​

Plugin Integration​

The S3 plugin automatically integrates with Token Ring applications:

export default {
  name: '@tokenring-ai/s3',
  version: '0.2.0',
  install(app: TokenRingApp) {
    // CDN provider registration
    const cdnConfig = app.getConfigSlice('cdn', CDNConfigSchema);
    if (cdnConfig) {
      app.waitForService(CDNService, cdnService => {
        for (const name in cdnConfig.providers) {
          const provider = cdnConfig.providers[name];
          if (provider.type === 's3') {
            cdnService.registerProvider(name, new S3CDNProvider(S3CDNProviderOptionsSchema.parse(provider)));
          }
        }
      });
    }

    // Filesystem provider registration
    const filesystemConfig = app.getConfigSlice('filesystem', FileSystemConfigSchema);
    if (filesystemConfig) {
      app.waitForService(FileSystemService, fileSystemService => {
        for (const name in filesystemConfig.providers) {
          const provider = filesystemConfig.providers[name];
          if (provider.type === 's3') {
            fileSystemService.registerFileSystemProvider(name, new S3FileSystemProvider(S3FileSystemProviderOptionsSchema.parse(provider)));
          }
        }
      });
    }
  },
}

Service Dependencies​

The S3 plugin requires these services to be available:

  1. CDNService: For CDN provider registration and management
  2. FileSystemService: For file system provider registration
  3. TokenRingApp: For configuration access and service coordination

Environment Variables​

Ensure these environment variables are set:

export AWS_ACCESS_KEY_ID='your-access-key-id'
export AWS_SECRET_ACCESS_KEY='your-secret-access-key'
export AWS_REGION='us-east-1'  # Optional, can be configured per provider

API Reference​

S3CDNProvider Methods​

upload(data, options?)​

Uploads content to S3 bucket.

Parameters:

  • data: Buffer - Content to upload
  • options: UploadOptions (optional) - Upload options
    • filename: string - File name
    • contentType: string - MIME type
    • metadata: object - Custom metadata

Returns: Promise<UploadResult> - Upload result containing URL and metadata

  • url: string - Public URL of uploaded content
  • id: string - File ID/key in S3
  • metadata: object - Uploaded metadata

delete(url)​

Deletes content from S3 bucket.

Parameters:

  • url: string - URL of content to delete

Returns: Promise<DeleteResult> - Delete result

  • success: boolean - Whether deletion was successful
  • message: string - Result message

exists(url)​

Checks if content exists in S3.

Parameters:

  • url: string - URL of content to check

Returns: Promise<boolean> - True if content exists

S3FileSystemProvider Methods​

writeFile(fsPath, content)​

Writes file content to S3.

Parameters:

  • fsPath: string - Destination path
  • content: string | Buffer - Content to write

Returns: Promise<boolean> - True if write was successful

appendFile(filePath, content)​

Appends content to an existing file or creates it if not exists.

Parameters:

  • filePath: string - Path of file to append to
  • content: string | Buffer - Content to append

Returns: Promise<boolean> - True if operation was successful

readFile(fsPath, encoding?)​

Reads file content from S3.

Parameters:

  • fsPath: string - Path of file to read
  • encoding: BufferEncoding | "buffer" (optional) - Encoding or "buffer"

Returns: Promise<string | Buffer> - File content

deleteFile(fsPath)​

Deletes file from S3.

Parameters:

  • fsPath: string - Path of file to delete

Returns: Promise<boolean> - True if deletion was successful

exists(fsPath)​

Checks if file exists in S3.

Parameters:

  • fsPath: string - Path to check

Returns: Promise<boolean> - True if file exists

stat(fsPath)​

Gets file statistics.

Parameters:

  • fsPath: string - Path to check

Returns: Promise<StatLike> - File statistics object

  • path: string - Original path
  • absolutePath: string - Absolute S3 path
  • isFile: boolean - Whether it's a file
  • isDirectory: boolean - Whether it's a directory
  • size: number - File size in bytes
  • modified: Date - Last modified date
  • created: Date - Creation date
  • accessed: Date - Last accessed date

createDirectory(fsPath, options?)​

Creates directory (simulated using prefixes).

Parameters:

  • fsPath: string - Directory path to create
  • options: object (optional) - Options
    • recursive: boolean - Create parent directories

Returns: Promise<boolean> - True if creation was successful

getDirectoryTree(fsPath, params?)​

Lists directory contents (prefix-based).

Parameters:

  • fsPath: string - Directory path to list
  • params: DirectoryTreeOptions (optional) - Options
    • ignoreFilter: function - Filter function to ignore paths
    • recursive: boolean - List recursively

Returns: AsyncGenerator<string> - Generator yielding file paths

copy(sourceFsPath, destinationFsPath, options?)​

Copies file within S3.

Parameters:

  • sourceFsPath: string - Source file path
  • destinationFsPath: string - Destination file path
  • options: object (optional) - Options
    • overwrite: boolean - Overwrite if destination exists

Returns: Promise<boolean> - True if copy was successful

rename(oldPath, newPath)​

Renames/moves file within S3.

Parameters:

  • oldPath: string - Original file path
  • newPath: string - New file path

Returns: Promise<boolean> - True if rename was successful

Package Structure​

pkg/s3/
├── index.ts              # Main entry point and exports
├── plugin.ts             # Plugin integration logic
├── S3CDNProvider.ts      # CDN provider implementation
├── S3FileSystemProvider.ts # File system provider implementation
├── package.json         # Package configuration and dependencies
├── README.md            # Package documentation
└── LICENSE              # MIT license

Testing​

The package includes Vitest configuration for testing:

// vitest.config.ts
import {defineConfig} from "vitest/config";

export default defineConfig({
  test: {
    include: ["**/*.test.ts"],
    environment: "node", 
    globals: true,
    isolate: true,
  },
});

Error Handling​

The S3 plugin provides comprehensive error handling:

  1. Configuration Validation: Zod schemas validate all configuration inputs
  2. Service Dependencies: Graceful handling when required services aren't available
  3. AWS API Errors: Proper error handling for S3 API failures
  4. Permission Errors: Handling of permission and access errors
  5. Network Errors: Proper handling of network connectivity issues
  6. Path Validation: Prevents directory traversal above bucket root

Performance Considerations​

  • Connection Pooling: AWS SDK handles connection pooling
  • Region Optimization: Uses specified AWS region for optimal performance
  • Content Delivery: Leverages S3's global CDN infrastructure
  • Prefix-based Operations: Efficient directory listing using S3 prefixes
  • Buffer Handling: Optimized buffer handling for file operations

Limitations​

  • Filesystem: No real-time file watching, shell execution, or advanced filesystem features
  • CDN: No automatic URL signing or CDN-specific caching controls
  • Directories: S3 directories are simulated using prefixes; true directory operations are limited
  • Advanced Features: Methods like chmod(), watch(), executeCommand(), glob(), and grep() are not supported

License​

MIT License - see LICENSE for details.