@tokenring-ai/s3

The @tokenring-ai/s3 package provides comprehensive AWS S3 integration for the Token Ring AI ecosystem. It implements both CDN (Content Delivery Network) and File System providers for seamless cloud storage and content delivery. The package integrates with the Token Ring application framework through plugin architecture and supports configuration-based provider registration.

Overview

The @tokenring-ai/s3 package provides AWS S3 integration for cloud storage and content delivery in the Token Ring AI system. It implements both CDN and File System providers, handling S3-specific details like path normalization, error handling, and directory simulation using S3 prefixes.

Key Features

CDN Provider: S3-based Content Delivery Network integration with configurable providers
File System Provider: S3-backed file system implementation with full CRUD operations
Configuration-driven: Providers registered automatically from application configuration
Service Integration: Seamless integration with Token Ring's CDN and FileSystemService
Type-safe Configuration: Zod schemas for validation and type safety
Plugin Architecture: Automatic service registration through TokenRingApp
Path Normalization: Automatic path conversion between relative and absolute S3 paths
Directory Simulation: S3 prefix-based directory operations

Core Components

S3FileSystemProvider

A filesystem provider that maps S3 buckets to a virtual filesystem interface.

Constructor

new S3FileSystemProvider(options: S3FileSystemProviderOptions)

Options:

bucketName: Name of the S3 bucket (required)
clientConfig: AWS SDK client configuration object (optional)

Path Conversion Methods

// Convert a path to absolute S3 URI format
const absolutePath = provider.relativeOrAbsolutePathToAbsolutePath('path/to/file.txt')
// Returns: 's3://bucket-name/path/to/file.txt'

// Convert a path to bucket-relative format (without s3:// prefix)
const relativePath = provider.relativeOrAbsolutePathToRelativePath('s3://bucket-name/path/to/file.txt')
// Returns: 'path/to/file.txt'

Key Methods

File Operations

// Write content to a file
await provider.writeFile('path/to/file.txt', 'content or buffer')

// Read file content
const content = await provider.readFile('path/to/file.txt', 'utf8')

// Append to a file
await provider.appendFile('path/to/file.txt', 'additional content')

// Delete a file
await provider.deleteFile('path/to/file.txt')

File Information

// Check if file/directory exists
const exists = await provider.exists('path/to/file.txt')

// Get file/directory statistics
const stats = await provider.stat('path/to/file.txt')
// Returns: { path, absolutePath, isFile, isDirectory, isSymbolicLink, size, modified, created, accessed }

Directory Operations

// Create a directory
await provider.createDirectory('path/to/directory')

// Get directory tree listing
for await (const path of provider.getDirectoryTree('path/to/directory')) {
  console.log(path)
}

// Copy files
await provider.copy('source.txt', 'destination.txt', { overwrite: true })

// Rename files (copy + delete)
await provider.rename('old-name.txt', 'new-name.txt')

Path Handling

Supports both relative paths (file.txt) and absolute S3 paths (s3://bucket/file.txt)
Automatically normalizes paths and prevents directory traversal above bucket root
Simulates directories using S3 prefixes (objects ending with /)
Handles both forward slashes and backslashes for cross-platform compatibility

Limitations

chmod(), watch(), glob(), and grep() are not supported
S3 is object storage, not a true filesystem, so some filesystem features are limited

S3CDNProvider

A CDN provider for uploading and managing content in S3 buckets. Extends the base CDNProvider class.

Constructor

new S3CDNProvider(options: S3CDNProviderOptions)

Options:

bucket: S3 bucket name (required)
region: AWS region (required)
accessKeyId: AWS access key ID (required)
secretAccessKey: AWS secret access key (required)
baseUrl: Custom base URL for CDN (optional, defaults to https://{bucket}.s3.amazonaws.com)

Key Methods

// Upload data with options
const result = await provider.upload(buffer, {
  filename: 'image.png',
  contentType: 'image/png',
  metadata: { author: 'User', category: 'images' }
})
// Returns: { url, id, metadata }

// Delete by URL
const deleteResult = await provider.delete('https://bucket.s3.amazonaws.com/file.txt')
// Returns: { success: boolean, message: string }

// Check if resource exists
const exists = await provider.exists('https://bucket.s3.amazonaws.com/file.txt')

Services

CDN Service

The package registers S3CDNProvider instances with CDNService:

Registration:

Automatically registers with CDNService when CDN configuration is provided via plugin
Uses CDNConfigSchema for configuration validation
Providers are registered with the name specified in the configuration

Provider Interface:

interface CDNProvider {
  upload(data: Buffer, options?: UploadOptions): Promise<UploadResult>;
  delete(url: string): Promise<DeleteResult>;
  exists(url: string): Promise<boolean>;
}

File System Service

The package registers S3FileSystemProvider instances with FileSystemService:

Registration:

Automatically registers with FileSystemService when filesystem configuration is provided via plugin
Uses FileSystemConfigSchema for configuration validation
Providers are registered with the name specified in the configuration

Provider Interface:

interface FileSystemProvider {
  writeFile(fsPath: string, content: string | Buffer): Promise<boolean>;
  appendFile(filePath: string, content: string | Buffer): Promise<boolean>;
  readFile(fsPath: string, encoding?: BufferEncoding | "buffer"): Promise<any>;
  deleteFile(fsPath: string): Promise<boolean>;
  exists(fsPath: string): Promise<boolean>;
  stat(fsPath: string): Promise<StatLike>;
  createDirectory(fsPath: string, options?: { recursive?: boolean }): Promise<boolean>;
  getDirectoryTree(fsPath: string, params?: DirectoryTreeOptions): AsyncGenerator<string>;
  copy(sourceFsPath: string, destinationFsPath: string, options?: { overwrite?: boolean }): Promise<boolean>;
  rename(oldPath: string, newPath: string): Promise<boolean>;
}

Provider Documentation

S3CDNProvider

AWS S3 integration for CDN services, providing content delivery capabilities.

Configuration Schema: S3CDNProviderOptionsSchema

const S3CDNProviderOptionsSchema = z.object({
  bucket: z.string(),
  region: z.string().optional(),
  accessKeyId: z.string().optional(),
  secretAccessKey: z.string().optional(),
  baseUrl: z.string().optional(),
});

Provider Interface:

interface S3CDNProviderOptions {
  bucket: string;
  region: string;
  accessKeyId: string;
  secretAccessKey: string;
  baseUrl?: string;
}

Key Methods:

upload(data: Buffer, options?: UploadOptions): Promise<UploadResult> - Upload data to S3
delete(url: string): Promise<DeleteResult> - Delete an object by URL
exists(url: string): Promise<boolean> - Check if an object exists

S3FileSystemProvider

S3-backed file system provider with complete file operations.

Configuration Schema: S3FileSystemProviderOptionsSchema

const S3FileSystemProviderOptionsSchema = z.object({
  bucketName: z.string(),
  clientConfig: z.any().optional(),
});

Provider Interface:

interface S3FileSystemProviderOptions {
  bucketName: string;
  clientConfig?: Record<string, unknown>;
}

Key Methods:

writeFile(fsPath: string, content: string | Buffer): Promise<boolean> - Write file content
readFile(fsPath: string, encoding?: BufferEncoding): Promise<any> - Read file content
deleteFile(fsPath: string): Promise<boolean> - Delete a file
exists(fsPath: string): Promise<boolean> - Check if file/directory exists
stat(fsPath: string): Promise<StatLike> - Get file/directory statistics
createDirectory(fsPath: string, options?: { recursive?: boolean }): Promise<boolean> - Create directory
getDirectoryTree(fsPath: string, params?: DirectoryTreeOptions): AsyncGenerator<string> - List directory contents
copy(sourceFsPath: string, destinationFsPath: string, options?: { overwrite?: boolean }): Promise<boolean> - Copy files
rename(oldPath: string, newPath: string): Promise<boolean> - Rename files
appendFile(filePath: string, content: string | Buffer): Promise<boolean> - Append to file

RPC Endpoints

This package does not define any RPC endpoints.

Chat Commands

This package does not register any chat commands.

Configuration

Plugin Configuration Schema

The plugin uses Zod schemas for configuration validation:

const packageConfigSchema = z.object({
  cdn: CDNConfigSchema.optional(),
  filesystem: FileSystemConfigSchema.optional()
});

CDN Configuration

interface CDNConfig {
  providers: {
    [providerName: string]: {
      type: 's3';
      bucket: string;
      region: string;
      accessKeyId: string;
      secretAccessKey: string;
      baseUrl?: string;
    };
  };
}

Filesystem Configuration

interface FilesystemConfig {
  providers: {
    [providerName: string]: {
      type: 's3';
      bucketName: string;
      clientConfig?: Record<string, unknown>;
    };
  };
}

Configuration Example

import TokenRingApp from '@tokenring-ai/app'
import s3Plugin from '@tokenring-ai/s3'

const app = new TokenRingApp({
  config: {
    cdn: {
      providers: {
        mainCDN: {
          type: 's3',
          bucket: 'my-cdn-bucket',
          region: 'us-east-1',
          accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
          secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
          baseUrl: 'https://cdn.example.com'
        }
      }
    },
    filesystem: {
      providers: {
        s3Storage: {
          type: 's3',
          bucketName: 'my-files-bucket',
          clientConfig: { region: 'us-east-1' }
        }
      }
    }
  }
})

app.registerPlugin(s3Plugin)
await app.start()

Integration

Plugin Registration

The S3 plugin automatically integrates with Token Ring applications:

export default {
  name: '@tokenring-ai/s3',
  version: '0.2.0',
  install(app: TokenRingApp, config: PackageConfig) {
    // CDN provider registration
    if (config.cdn) {
      app.waitForService(CDNService, cdnService => {
        for (const name in config.cdn.providers) {
          const provider = config.cdn.providers[name];
          if (provider.type === "s3") {
            cdnService.registerProvider(name, new S3CDNProvider(S3CDNProviderOptionsSchema.parse(provider)));
          }
        }
      });
    }

    // Filesystem provider registration
    if (config.filesystem) {
      app.waitForService(FileSystemService, fileSystemService => {
        for (const name in config.filesystem.providers) {
          const provider = config.filesystem.providers[name];
          if (provider.type === "s3") {
            fileSystemService.registerFileSystemProvider(name, new S3FileSystemProvider(S3FileSystemProviderOptionsSchema.parse(provider)));
          }
        }
      });
    }
  },
  config: packageConfigSchema
}

Service Dependencies

The S3 plugin requires these services to be available:

CDNService: For CDN provider registration and management
FileSystemService: For file system provider registration
TokenRingApp: For configuration access and service coordination

Programmatic Provider Registration

import { TokenRingApp } from '@tokenring-ai/app';
import { CDNService } from '@tokenring-ai/cdn';
import { FileSystemService } from '@tokenring-ai/filesystem';
import { S3CDNProvider, S3FileSystemProvider } from '@tokenring-ai/s3';

const app = new TokenRingApp();

// Register CDN provider
const cdnService = app.requireServiceByType(CDNService);
cdnService.registerProvider('s3-cdn', new S3CDNProvider({
  bucket: 'my-cdn-bucket',
  region: 'us-east-1',
  accessKeyId: 'your-access-key',
  secretAccessKey: 'your-secret-key',
  baseUrl: 'https://cdn.example.com'
}));

// Register File System provider  
const fileSystemService = app.requireServiceByType(FileSystemService);
fileSystemService.registerFileSystemProvider('s3-fs', new S3FileSystemProvider({
  bucketName: 'my-data-bucket',
  clientConfig: {
    region: 'us-east-1'
  }
}));

Usage Examples

Basic Filesystem Usage

import { S3FileSystemProvider } from '@tokenring-ai/s3'

const provider = new S3FileSystemProvider({
  bucketName: 'my-bucket',
  clientConfig: { region: 'us-east-1' }
})

// Write a file
await provider.writeFile('hello.txt', 'Hello, S3!')

// Read it back
const content = await provider.readFile('hello.txt', 'utf8')
console.log(content) // "Hello, S3!"

// Check if file exists
const exists = await provider.exists('hello.txt')
console.log(exists) // true

// Get file statistics
const stats = await provider.stat('hello.txt')
console.log(`Size: ${stats.size} bytes`)
console.log(`Modified: ${stats.modified}`)

// List directory contents
console.log('Directory contents:')
for await (const path of provider.getDirectoryTree('.')) {
  console.log(path)
}

// Path conversion
const absolute = provider.relativeOrAbsolutePathToAbsolutePath('docs/readme.md')
console.log(absolute) // s3://my-bucket/docs/readme.md

// Create directory
await provider.createDirectory('docs')

// Copy file
await provider.copy('hello.txt', 'docs/hello-copy.txt')

// Rename file
await provider.rename('hello.txt', 'greeting.txt')

CDN Usage

import { S3CDNProvider } from '@tokenring-ai/s3'

const cdn = new S3CDNProvider({
  bucket: 'my-cdn-bucket',
  region: 'us-east-1',
  accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
  secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
  baseUrl: 'https://cdn.example.com'
})

// Upload an image
const imageBuffer = Buffer.from('image data...')
const uploadResult = await cdn.upload(imageBuffer, {
  filename: 'profile.png',
  contentType: 'image/png',
  metadata: {
    author: 'John Doe',
    tags: ['avatar', 'profile']
  }
})

console.log(`Uploaded to: ${uploadResult.url}`)
console.log(`File ID: ${uploadResult.id}`)

// Check if file exists
const exists = await cdn.exists(uploadResult.url)
console.log(`File exists: ${exists}`)

// Delete the file
const deleteResult = await cdn.delete(uploadResult.url)
console.log(`Delete success: ${deleteResult.success}`)

Plugin-Based Configuration

import TokenRingApp from '@tokenring-ai/app'
import s3Plugin from '@tokenring-ai/s3'

const app = new TokenRingApp({
  config: {
    cdn: {
      providers: {
        s3CDN: {
          type: 's3',
          bucket: 'cdn-bucket',
          region: 'us-west-2',
          accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
          secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!
        }
      }
    },
    filesystem: {
      providers: {
        s3Files: {
          type: 's3',
          bucketName: 'files-bucket',
          clientConfig: { region: 'us-west-2' }
        }
      }
    }
  }
})

// Register the plugin
app.registerPlugin(s3Plugin)

await app.start()

// Access the registered providers
const cdnService = await app.getService(CDNService)
const cdnProvider = cdnService.getProvider('s3CDN')

const fileSystemService = await app.getService(FileSystemService)
const fsProvider = fileSystemService.getFileSystemProvider('s3Files')

Best Practices

Security

Use IAM roles with least-privilege access when possible
Store credentials securely (environment variables or AWS Secrets Manager)
Enable S3 bucket policies for proper access control
Consider using signed URLs for temporary access when appropriate
Enable S3 server-side encryption for sensitive data
Never expose credentials in client-side code

Performance

Consider batch operations for large file transfers
Use appropriate content types for better CDN caching
Implement retry logic for transient network errors
Use S3 Transfer Acceleration for geographically distributed access

Error Handling

Always wrap S3 operations in try-catch blocks
Handle specific S3 error codes (NoSuchKey, AccessDenied, etc.)
Implement exponential backoff for rate limiting
Log errors for debugging and monitoring

Testing and Development

Testing Setup

This package uses vitest for unit testing.

# Run tests
bun test

# Run tests in watch mode
bun test:watch

# Run tests with coverage
bun test:coverage

# Type check
bun build

Development

Clone the repository
Install dependencies: bun install
Make changes to the source files
Run tests: bun test
Type check: bun build

Vitest Configuration

// vitest.config.ts
import {defineConfig} from "vitest/config";

export default defineConfig({
  test: {
    include: ["**/*.test.ts"],
    environment: "node", 
    globals: true,
    isolate: true,
  },
});

Dependencies

Production Dependencies

@aws-sdk/client-s3: ^3.1000.0
@tokenring-ai/app: 0.2.0
@tokenring-ai/agent: 0.2.0
zod: ^4.3.6

Peer Dependencies

@tokenring-ai/cdn: 0.2.0
@tokenring-ai/filesystem: 0.2.0

Development Dependencies

vitest: ^4.0.18
typescript: ^5.9.3

@tokenring-ai/cdn - Core CDN service and provider interface
@tokenring-ai/filesystem - Core filesystem service and provider interface
@tokenring-ai/app - Base application framework with plugin system
@tokenring-ai/agent - Agent orchestration system

Package Structure

pkg/s3/
├── index.ts              # Main entry point and exports
├── plugin.ts             # Plugin integration logic
├── S3CDNProvider.ts      # CDN provider implementation
├── S3FileSystemProvider.ts # File system provider implementation
├── package.json         # Package configuration and dependencies
├── vitest.config.ts     # Testing configuration
└── README.md            # Package documentation

Limitations

Filesystem: No real-time file watching, shell execution, or advanced filesystem features
CDN: No automatic URL signing or CDN-specific caching controls
Performance: S3 operations have network latency; consider batch operations for large files
Directories: S3 directories are simulated using prefixes; true directory operations are limited
Consistency: S3 offers eventual consistency for some operations
Unsupported Methods: chmod(), watch(), glob(), and grep() are not supported

State Management

This package does not manage state directly. It relies on the Token Ring state management system through services.

License

MIT License - see LICENSE for details.

Overview​

Key Features​

Core Components​

S3FileSystemProvider​

Constructor​

Path Conversion Methods​

Key Methods​

File Operations​

File Information​

Directory Operations​

Path Handling​

Limitations​

S3CDNProvider​

Constructor​

Key Methods​

Services​

CDN Service​

File System Service​

Provider Documentation​

S3CDNProvider​

S3FileSystemProvider​

RPC Endpoints​

Chat Commands​

Configuration​

Plugin Configuration Schema​

CDN Configuration​

Filesystem Configuration​

Configuration Example​

Integration​

Plugin Registration​

Service Dependencies​

Programmatic Provider Registration​

Usage Examples​

Basic Filesystem Usage​

CDN Usage​

Plugin-Based Configuration​

Best Practices​

Security​

Performance​

Error Handling​

Testing and Development​

Testing Setup​

Development​

Vitest Configuration​

Dependencies​

Production Dependencies​

Peer Dependencies​

Development Dependencies​

Related Components​

Package Structure​

Limitations​

State Management​

License​

Overview

Key Features

Core Components

S3FileSystemProvider

Constructor

Path Conversion Methods

Key Methods

File Operations

File Information

Directory Operations

Path Handling

Limitations

S3CDNProvider

Constructor

Key Methods

Services

CDN Service

File System Service

Provider Documentation

S3CDNProvider

S3FileSystemProvider

RPC Endpoints

Chat Commands

Configuration

Plugin Configuration Schema

CDN Configuration

Filesystem Configuration

Configuration Example

Integration

Plugin Registration

Service Dependencies

Programmatic Provider Registration

Usage Examples

Basic Filesystem Usage

CDN Usage

Plugin-Based Configuration

Best Practices

Security

Performance

Error Handling

Testing and Development

Testing Setup

Development

Vitest Configuration

Dependencies

Production Dependencies

Peer Dependencies

Development Dependencies

Related Components

Package Structure

Limitations

State Management

License