Skip to main content

@tokenring-ai/s3

The @tokenring-ai/s3 package provides comprehensive AWS S3 integration for the Token Ring AI ecosystem. It implements both CDN (Content Delivery Network) and File System providers for seamless cloud storage and content delivery. The package integrates with the Token Ring application framework through plugin architecture and supports configuration-based provider registration.

Overview

The @tokenring-ai/s3 package provides AWS S3 integration for cloud storage and content delivery in the Token Ring AI system. It implements both CDN and File System providers, handling S3-specific details like path normalization, error handling, and directory simulation using S3 prefixes.

Key Features

  • CDN Provider: S3-based Content Delivery Network integration with configurable providers
  • File System Provider: S3-backed file system implementation with full CRUD operations
  • Configuration-driven: Providers registered automatically from application configuration
  • Service Integration: Seamless integration with Token Ring's CDN and FileSystemService
  • Type-safe Configuration: Zod schemas for validation and type safety
  • Plugin Architecture: Automatic service registration through TokenRingApp
  • Path Normalization: Automatic path conversion between relative and absolute S3 paths
  • Directory Simulation: S3 prefix-based directory operations

Core Components

S3FileSystemProvider

A filesystem provider that maps S3 buckets to a virtual filesystem interface.

Constructor

new S3FileSystemProvider(options: S3FileSystemProviderOptions)

Options:

  • bucketName: Name of the S3 bucket (required)
  • clientConfig: AWS SDK client configuration object (optional)

Path Conversion Methods

// Convert a path to absolute S3 URI format
const absolutePath = provider.relativeOrAbsolutePathToAbsolutePath('path/to/file.txt')
// Returns: 's3://bucket-name/path/to/file.txt'

// Convert a path to bucket-relative format (without s3:// prefix)
const relativePath = provider.relativeOrAbsolutePathToRelativePath('s3://bucket-name/path/to/file.txt')
// Returns: 'path/to/file.txt'

Key Methods

File Operations
// Write content to a file (overwrites existing)
await provider.writeFile('path/to/file.txt', 'content or buffer')

// Read file content
const content = await provider.readFile('path/to/file.txt', 'utf8')
// Supports 'utf8', 'buffer', or other BufferEncoding

// Append to a file (creates file if it doesn't exist)
await provider.appendFile('path/to/file.txt', 'additional content')
// If file exists: reads existing content, appends new content, writes back
// If file doesn't exist: creates file with new content

// Delete a file
await provider.deleteFile('path/to/file.txt')
File Information
// Check if file/directory exists
const exists = await provider.exists('path/to/file.txt')

// Get file/directory statistics
const stats = await provider.stat('path/to/file.txt')
// Returns: StatLike {
// exists: boolean;
// path: string;
// absolutePath: string;
// isFile: boolean;
// isDirectory: boolean;
// isSymbolicLink: boolean;
// size: number;
// modified: Date | undefined;
// created: Date | undefined;
// accessed: Date | undefined
// }
// Note: S3 doesn't track creation/access time separately, so these may be undefined or same as modified

Stat Behavior

The stat method handles both files and directories:

  • File: Returns metadata from S3 object (ContentLength, LastModified)
  • Directory: Checks if any objects exist with the path as a prefix
  • Non-existent: Returns { exists: false, path: string }
// Check if a path is a directory
const dirStat = await provider.stat('path/to/directory/')
console.log(dirStat.isDirectory) // true if objects exist with that prefix

// Check if a path is a file
const fileStat = await provider.stat('path/to/file.txt')
console.log(fileStat.isFile) // true if object exists
Directory Operations
// Create a directory (S3 uses prefixes, so this is virtual)
// Creates a trailing slash prefix to represent a directory
await provider.createDirectory('path/to/directory')
// Returns: true
// Note: Directories are created automatically when files are written with nested paths

// Get directory tree listing (async generator with pagination)
for await (const path of provider.getDirectoryTree('path/to/directory')) {
console.log(path)
}
// Yields: string (full S3 key path)
// Supports pagination automatically via S3 ContinuationToken
// Optional params: { ignoreFilter?: (key: string) => boolean, recursive?: boolean }

// Non-recursive listing (only immediate children)
for await (const path of provider.getDirectoryTree('path/to/directory', { recursive: false })) {
console.log(path)
}
// Only yields paths at the immediate level

// Copy files (throws if destination exists unless overwrite is true)
await provider.copy('source.txt', 'destination.txt', { overwrite: true })
// Options: { overwrite?: boolean }
// Default: overwrite is false (throws if destination exists)

// Rename files (implemented as copy + delete)
await provider.rename('old-name.txt', 'new-name.txt')
// Note: If rename fails after copy, the original file remains

// Append to file (creates file if doesn't exist)
await provider.appendFile('path/to/file.txt', 'additional content')

Path Handling

  • Supports both relative paths (file.txt) and absolute S3 paths (s3://bucket/file.txt)
  • Automatically normalizes paths and prevents directory traversal above bucket root
  • Simulates directories using S3 prefixes (objects ending with /)
  • Handles both forward slashes and backslashes for cross-platform compatibility
  • Path traversal with .. is validated to prevent escaping bucket root
  • Path normalization removes redundant separators and . references
  • Empty paths are treated as the bucket root

Path Normalization Examples

// Forward slashes are normalized
provider.relativeOrAbsolutePathToAbsolutePath('path//to///file.txt')
// Returns: 's3://bucket-name/path/to/file.txt'

// Backslashes are converted to forward slashes
provider.relativeOrAbsolutePathToAbsolutePath('path\\to\\file.txt')
// Returns: 's3://bucket-name/path/to/file.txt'

// Dot references are resolved
provider.relativeOrAbsolutePathToAbsolutePath('path/to/../file.txt')
// Returns: 's3://bucket-name/path/file.txt'

// Attempting to traverse above bucket root throws
try {
provider.relativeOrAbsolutePathToAbsolutePath('../../file.txt')
} catch (error) {
console.error(error.message)
// "Invalid path: ../../file.txt attempts to traverse above bucket root."
}

Limitations

  • chmod(): S3 does not support Unix-style file permissions
  • watch(): S3 does not support file system watching
  • glob(): S3 only supports prefix-based listing via getDirectoryTree()
  • grep(): S3 does not support content search; use S3 Select or download files for local search
  • S3 is object storage, not a true filesystem, so some filesystem features are limited

S3CDNProvider

A CDN provider for uploading and managing content in S3 buckets. Extends the base CDNProvider class.

Constructor

new S3CDNProvider(options: S3CDNProviderOptions)

Options:

  • bucket: S3 bucket name (required)
  • region: AWS region (required)
  • accessKeyId: AWS access key ID (required)
  • secretAccessKey: AWS secret access key (required)
  • baseUrl: Custom base URL for CDN (optional, defaults to https://{bucket}.s3.amazonaws.com)

Note: While the Zod schema marks region, accessKeyId, secretAccessKey, and baseUrl as optional, the constructor validates that bucket, region, accessKeyId, and secretAccessKey are provided and throws errors if they are missing.

Key Methods

// Upload data with options
const result = await provider.upload(buffer, {
filename: 'image.png',
contentType: 'image/png',
metadata: { author: 'User', category: 'images' }
})
// Returns: UploadResult {
// url: string;
// id: string;
// metadata?: Record<string, string>;
// }
// Note: If filename is not provided, generates 'timestamp-randomstring'

// Delete by URL
const deleteResult = await provider.delete('https://bucket.s3.amazonaws.com/file.txt')
// Returns: DeleteResult {
// success: boolean;
// message: string;
// }

// Check if resource exists
const exists = await provider.exists('https://bucket.s3.amazonaws.com/file.txt')
// Returns: boolean

URL Handling

  • Automatically extracts S3 keys from various URL formats
  • Supports custom baseUrl for CDN domains
  • Falls back to standard S3 URL format if no custom base is provided
// Extract key from custom baseUrl
const key = provider.extractKeyFromUrl('https://cdn.example.com/path/to/file.png')
// Returns: 'path/to/file.png'

// Extract key from standard S3 URL
const key = provider.extractKeyFromUrl('https://bucket.s3.amazonaws.com/path/to/file.png')
// Returns: 'path/to/file.png'

Services

CDN Service

The package registers S3CDNProvider instances with CDNService:

Registration:

  • Automatically registers with CDNService when CDN configuration is provided via plugin
  • Uses CDNConfigSchema for configuration validation
  • Providers are registered with the name specified in the configuration

Provider Interface:

interface CDNProvider {
upload(data: Buffer, options?: UploadOptions): Promise<UploadResult>;
delete(url: string): Promise<DeleteResult>;
exists(url: string): Promise<boolean>;
}

File System Service

The package registers S3FileSystemProvider instances with FileSystemService:

Registration:

  • Automatically registers with FileSystemService when filesystem configuration is provided via plugin
  • Uses FileSystemConfigSchema for configuration validation
  • Providers are registered with the name specified in the configuration

Provider Interface:

interface FileSystemProvider {
writeFile(fsPath: string, content: string | Buffer): Promise<boolean>;
appendFile(filePath: string, content: string | Buffer): Promise<boolean>;
readFile(fsPath: string, encoding?: BufferEncoding | "buffer"): Promise<any>;
deleteFile(fsPath: string): Promise<boolean>;
exists(fsPath: string): Promise<boolean>;
stat(fsPath: string): Promise<StatLike>;
createDirectory(fsPath: string, options?: { recursive?: boolean }): Promise<boolean>;
getDirectoryTree(fsPath: string, params?: DirectoryTreeOptions): AsyncGenerator<string>;
copy(sourceFsPath: string, destinationFsPath: string, options?: { overwrite?: boolean }): Promise<boolean>;
rename(oldPath: string, newPath: string): Promise<boolean>;
}

Provider Documentation

S3CDNProvider

AWS S3 integration for CDN services, providing content delivery capabilities.

Configuration Schema: S3CDNProviderOptionsSchema

const S3CDNProviderOptionsSchema = z.object({
bucket: z.string(),
region: z.string().optional(),
accessKeyId: z.string().optional(),
secretAccessKey: z.string().optional(),
baseUrl: z.string().optional(),
});

Provider Interface:

interface S3CDNProviderOptions {
bucket: string;
region: string;
accessKeyId: string;
secretAccessKey: string;
baseUrl?: string;
}

Key Methods:

  • upload(data: Buffer, options?: UploadOptions): Promise<UploadResult> - Upload data to S3
  • delete(url: string): Promise<DeleteResult> - Delete an object by URL
  • exists(url: string): Promise<boolean> - Check if an object exists

S3FileSystemProvider

S3-backed file system provider with complete file operations.

Configuration Schema: S3FileSystemProviderOptionsSchema

const S3FileSystemProviderOptionsSchema = z.object({
bucketName: z.string(),
clientConfig: z.any().optional(),
});

Provider Interface:

interface S3FileSystemProviderOptions {
bucketName: string;
clientConfig?: Record<string, unknown>;
}

Key Methods:

  • writeFile(fsPath: string, content: string | Buffer): Promise<boolean> - Write file content (overwrites existing)
  • readFile(fsPath: string, encoding?: BufferEncoding | "buffer"): Promise<any> - Read file content (supports string or buffer)
  • deleteFile(fsPath: string): Promise<boolean> - Delete a file
  • exists(fsPath: string): Promise<boolean> - Check if file/directory exists
  • stat(fsPath: string): Promise<StatLike> - Get file/directory statistics
  • createDirectory(fsPath: string, options?: { recursive?: boolean }): Promise<boolean> - Create directory (virtual via S3 prefix)
  • getDirectoryTree(fsPath: string, params?: DirectoryTreeOptions): AsyncGenerator<string> - List directory contents (async generator with pagination)
  • copy(sourceFsPath: string, destinationFsPath: string, options?: { overwrite?: boolean }): Promise<boolean> - Copy files
  • rename(oldPath: string, newPath: string): Promise<boolean> - Rename files (implemented as copy + delete)
  • relativeOrAbsolutePathToAbsolutePath(p: string): string - Convert path to absolute S3 URI
  • relativeOrAbsolutePathToRelativePath(p: string): string - Convert path to relative format
  • appendFile(filePath: string, content: string | Buffer): Promise<boolean> - Append to file (creates file if doesn't exist)

RPC Endpoints

This package does not define any RPC endpoints.

Chat Commands

This package does not register any chat commands.

Configuration

Plugin Configuration Schema

The plugin uses Zod schemas for configuration validation:

const packageConfigSchema = z.object({
cdn: CDNConfigSchema.optional(),
filesystem: FileSystemConfigSchema.optional()
});

CDN Configuration

interface CDNConfig {
providers: {
[providerName: string]: {
type: 's3';
bucket: string;
region: string;
accessKeyId: string;
secretAccessKey: string;
baseUrl?: string;
};
};
}

Filesystem Configuration

interface FilesystemConfig {
providers: {
[providerName: string]: {
type: 's3';
bucketName: string;
clientConfig?: Record<string, unknown>;
};
};
}

Configuration Example

import TokenRingApp from '@tokenring-ai/app'
import s3Plugin from '@tokenring-ai/s3'

const app = new TokenRingApp({
config: {
cdn: {
providers: {
mainCDN: {
type: 's3',
bucket: 'my-cdn-bucket',
region: 'us-east-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
baseUrl: 'https://cdn.example.com'
}
}
},
filesystem: {
providers: {
s3Storage: {
type: 's3',
bucketName: 'my-files-bucket',
clientConfig: { region: 'us-east-1' }
}
}
}
}
})

app.registerPlugin(s3Plugin)
await app.start()

Integration

Plugin Registration

The S3 plugin automatically integrates with Token Ring applications:

export default {
name: '@tokenring-ai/s3',
version: '0.2.0',
/**
* Plugin installation logic
* @param app - TokenRingApp instance
* @param config - Parsed configuration from packageConfigSchema
*
* The plugin uses waitForService to ensure services are available
* before registering providers. Providers are only registered if
* their respective configuration sections (cdn or filesystem) are present.
*/
install(app: TokenRingApp, config: PackageConfig) {
// CDN provider registration
if (config.cdn) {
app.waitForService(CDNService, cdnService => {
for (const name in config.cdn.providers) {
const provider = config.cdn.providers[name];
if (provider.type === "s3") {
cdnService.registerProvider(name, new S3CDNProvider(S3CDNProviderOptionsSchema.parse(provider)));
}
}
});
}

// Filesystem provider registration
if (config.filesystem) {
app.waitForService(FileSystemService, fileSystemService => {
for (const name in config.filesystem.providers) {
const provider = config.filesystem.providers[name];
if (provider.type === "s3") {
fileSystemService.registerFileSystemProvider(name, new S3FileSystemProvider(S3FileSystemProviderOptionsSchema.parse(provider)));
}
}
});
}
},
config: packageConfigSchema
}

Service Dependencies

The S3 plugin requires these services to be available:

  1. CDNService: For CDN provider registration and management
  2. FileSystemService: For file system provider registration
  3. TokenRingApp: For configuration access and service coordination

Programmatic Provider Registration

import { TokenRingApp } from '@tokenring-ai/app';
import { CDNService } from '@tokenring-ai/cdn';
import { FileSystemService } from '@tokenring-ai/filesystem';
import { S3CDNProvider, S3FileSystemProvider } from '@tokenring-ai/s3';

const app = new TokenRingApp();

// Register CDN provider
const cdnService = app.requireServiceByType(CDNService);
cdnService.registerProvider('s3-cdn', new S3CDNProvider({
bucket: 'my-cdn-bucket',
region: 'us-east-1',
accessKeyId: 'your-access-key',
secretAccessKey: 'your-secret-key',
baseUrl: 'https://cdn.example.com'
}));

// Register File System provider
const fileSystemService = app.requireServiceByType(FileSystemService);
fileSystemService.registerFileSystemProvider('s3-fs', new S3FileSystemProvider({
bucketName: 'my-data-bucket',
clientConfig: {
region: 'us-east-1'
}
}));

Usage Examples

Basic Filesystem Usage

import { S3FileSystemProvider } from '@tokenring-ai/s3'

const provider = new S3FileSystemProvider({
bucketName: 'my-bucket',
clientConfig: { region: 'us-east-1' }
})

// Write a file
await provider.writeFile('hello.txt', 'Hello, S3!')

// Read it back
const content = await provider.readFile('hello.txt', 'utf8')
console.log(content) // "Hello, S3!"

// Check if file exists
const exists = await provider.exists('hello.txt')
console.log(exists) // true

// Get file statistics
const stats = await provider.stat('hello.txt')
console.log(`Size: ${stats.size} bytes`)
console.log(`Modified: ${stats.modified}`)

// List directory contents
console.log('Directory contents:')
for await (const path of provider.getDirectoryTree('.')) {
console.log(path)
}

// Path conversion
const absolute = provider.relativeOrAbsolutePathToAbsolutePath('docs/readme.md')
console.log(absolute) // s3://my-bucket/docs/readme.md

// Create directory
await provider.createDirectory('docs')

// Copy file
await provider.copy('hello.txt', 'docs/hello-copy.txt')

// Rename file
await provider.rename('hello.txt', 'greeting.txt')

CDN Usage

import { S3CDNProvider } from '@tokenring-ai/s3'

const cdn = new S3CDNProvider({
bucket: 'my-cdn-bucket',
region: 'us-east-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
baseUrl: 'https://cdn.example.com'
})

// Upload an image
const imageBuffer = Buffer.from('image data...')
const uploadResult = await cdn.upload(imageBuffer, {
filename: 'profile.png',
contentType: 'image/png',
metadata: {
author: 'John Doe',
tags: ['avatar', 'profile']
}
})

console.log(`Uploaded to: ${uploadResult.url}`)
console.log(`File ID: ${uploadResult.id}`)

// Check if file exists
const exists = await cdn.exists(uploadResult.url)
console.log(`File exists: ${exists}`)

// Delete the file
const deleteResult = await cdn.delete(uploadResult.url)
console.log(`Delete success: ${deleteResult.success}`)

Plugin-Based Configuration

import TokenRingApp from '@tokenring-ai/app'
import s3Plugin from '@tokenring-ai/s3'

const app = new TokenRingApp({
config: {
cdn: {
providers: {
s3CDN: {
type: 's3',
bucket: 'cdn-bucket',
region: 'us-west-2',
accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!
}
}
},
filesystem: {
providers: {
s3Files: {
type: 's3',
bucketName: 'files-bucket',
clientConfig: { region: 'us-west-2' }
}
}
}
}
})

// Register the plugin
app.registerPlugin(s3Plugin)

await app.start()

// Access the registered providers
const cdnService = await app.getService(CDNService)
const cdnProvider = cdnService.getProvider('s3CDN')

const fileSystemService = await app.getService(FileSystemService)
const fsProvider = fileSystemService.getFileSystemProvider('s3Files')

Best Practices

Security

  • Use IAM roles with least-privilege access when possible
  • Store credentials securely (environment variables or AWS Secrets Manager)
  • Enable S3 bucket policies for proper access control
  • Consider using signed URLs for temporary access when appropriate
  • Enable S3 server-side encryption for sensitive data
  • Never expose credentials in client-side code

Performance

  • Consider batch operations for large file transfers
  • Use appropriate content types for better CDN caching
  • Implement retry logic for transient network errors
  • Use S3 Transfer Acceleration for geographically distributed access

Error Handling

  • Always wrap S3 operations in try-catch blocks
  • Handle specific S3 error codes (NoSuchKey, AccessDenied, etc.)
  • Implement exponential backoff for rate limiting
  • Log errors for debugging and monitoring

Error Types

The following error types may be encountered:

  • NoSuchKey or NotFound: File or object does not exist
  • AccessDenied: Insufficient permissions
  • 404: HTTP status code for not found
  • 403: HTTP status code for forbidden
  • Network errors: Connection timeouts, DNS failures

Example error handling:

try {
const content = await provider.readFile('nonexistent.txt', 'utf8')
} catch (error: any) {
if (error.name === 'NoSuchKey' || error.name === 'NotFound' || error.$metadata?.httpStatusCode === 404) {
console.log('File not found')
} else {
console.error('S3 error:', error.message)
}
}

Testing and Development

Testing Setup

This package uses vitest for unit testing.

# Run tests
bun test

# Run tests in watch mode
bun test:watch

# Run tests with coverage
bun test:coverage

# Type check
bun build

Development

  1. Clone the repository
  2. Install dependencies: bun install
  3. Make changes to the source files
  4. Run tests: bun test
  5. Type check: bun build

Vitest Configuration

// vitest.config.ts
import {defineConfig} from "vitest/config";

export default defineConfig({
test: {
include: ["**/*.test.ts"],
environment: "node",
globals: true,
isolate: true,
},
});

Dependencies

Production Dependencies

  • @aws-sdk/client-s3: ^3.1017.0
  • @tokenring-ai/app: 0.2.0
  • @tokenring-ai/agent: 0.2.0
  • zod: ^4.3.6

Peer Dependencies

  • @tokenring-ai/cdn: 0.2.0
  • @tokenring-ai/filesystem: 0.2.0

Development Dependencies

  • vitest: ^4.1.1
  • typescript: ^6.0.2

Package Structure

pkg/s3/
├── index.ts # Main entry point and exports
├── plugin.ts # Plugin integration logic
├── S3CDNProvider.ts # CDN provider implementation
├── S3FileSystemProvider.ts # File system provider implementation
├── package.json # Package configuration and dependencies
├── vitest.config.ts # Testing configuration
└── README.md # Package documentation

Limitations

  • Filesystem: No real-time file watching, shell execution, or advanced filesystem features
  • CDN: No automatic URL signing or CDN-specific caching controls
  • Performance: S3 operations have network latency; consider batch operations for large files
  • Directories: S3 directories are simulated using prefixes; true directory operations are limited
  • Consistency: S3 offers eventual consistency for some operations
  • Unsupported Methods:
    • chmod(): S3 does not support Unix-style file permissions
    • watch(): S3 does not support file system watching
    • glob(): S3 only supports prefix-based listing via getDirectoryTree()
    • grep(): S3 does not support content search; use S3 Select or download files for local search
  • Atomicity: Operations like rename are not atomic (implemented as copy + delete)
  • Append: appendFile reads entire file before appending; not efficient for large files
  • Concurrency: No built-in locking; concurrent writes may overwrite each other

State Management

This package does not manage state directly. It relies on the Token Ring state management system through services.

License

MIT License - see LICENSE for details.