Lyo.FileStorage 2.0.0

Lyo.FileStorage

Save, stream-save, read, delete, and file metadata. Optional compression (Lyo.Compression), two-key encryption (Lyo.Encryption), duplicate hashing, access policies, audit hooks, multipart uploads (via IMultipartUploadService), and presigned/direct-upload/copy on cloud-capable backends.

With GenerateDocumentationFile set in Directory.Build.props, IntelliSense shows the same summaries as this README for documented members.

Examples

Store files on disk

using Lyo.FileStorage;
using Lyo.FileStorage.Models;

var options = new DiskFileStorageOptions
{
    RootDirectoryPath = "/path/to/storage",
    EnableDuplicateDetection = true,
    DuplicateStrategy = DuplicateHandlingStrategy.ReturnExisting
};

var service = new LocalFileStorageService(options);

// Save a file
var data = File.ReadAllBytes("document.pdf");
var result = await service.SaveFileAsync(
    data,
    originalFileName: "document.pdf",
    compress: true,
    encrypt: true,
    keyId: "my-encryption-key");

// Retrieve a file
var retrievedData = await service.GetFileAsync(result.Id);

// Delete a file
await service.DeleteFileAsync(result.Id);

Store files on S3-compatible backends

using Lyo.FileStorage.S3;
using Lyo.FileStorage.Models;

var options = new S3FileStorageOptions
{
    BucketName = "my-bucket",
    Region = "us-east-1",
    AccessKeyId = "your-access-key",
    SecretAccessKey = "your-secret-key"
};

var metadataStore = new YourMetadataStore(); // IFileMetadataStore
var service = new S3FileStorageService(options, metadataStore);

// Same IFileStorageService contract as LocalFileStorageService

Where the docs live

For multipart session stores and Postgres metadata, follow the references from your host registration (e.g. Lyo.FileMetadataStore.Postgres).

What each backend can do

Plaintext direct uploads (BeginDirectUpload) skip encryption and compression on the edge PUT. Finalized metadata still runs through the usual content policy and availability flags.

CancellationToken vs cloud signing

ct honours cooperative cancellation during most async I/O. Presigned URL generation, some AWS signing, and synchronous SDK paths may still finish after cancellation is requested. See GetPreSignedReadUrlAsync and related remarks on IFileStorageService.

IFileStorageService

• Writes. SaveFileAsync, SaveFromStreamAsync
• Reads. GetFileAsync, GetFileStreamAsync. Compressed payloads are decompressed via ICompressionService.Resolver using stored CompressionAlgorithm metadata. Optional per-call compressionAlgorithmOverride or FileStorageServiceBaseOptions.DecompressionAlgorithmOverride. Bounded by MaxDecompressedFileSize through MaxBytesWriteStream.
• Zip archive. IFileStorageArchiveService.CreateArchiveAsync spools files through Lyo.IO.Temp, zips on disk (nested ZipPath, caller fileName), and returns a stream that deletes the temp session on dispose. Caps: FileStorageArchiveOptions.MaxFileCount / MaxTotalUncompressedBytes. DI: AddFileStorageArchiveServiceFromConfiguration + AddFileStorageArchiveServiceKeyed.
• Metadata / delete. GetMetadataAsync, DeleteFileAsync(Guid, FileDeletionMode, CancellationToken). Deletes the backing object then, depending on FileDeletionMode, either tombstones metadata (RemoveObjectAndTombstoneMetadata, default) or purges it via IFileMetadataStore.PurgeMetadataAsync (RemoveObjectAndPurgeMetadata). Operator/governance flows only. Never accept this mode from end-user input.
• Temporary links. GetPreSignedReadUrlAsync (with optional PreSignedReadUrlOptions overrides). Both overloads accept CancellationToken, defaulting to default.
• Direct single PUT. BeginDirectUploadAsync, CompleteDirectUploadAsync. S3/Blob fill RequiredPutHeaders for SSE + signed Content-Type parity.
• Copy. CopyFileAsync (new file id)
• Move / rename. MoveFileAsync (same file id, relocate by PathPrefix). RenameFileAsync changes metadata OriginalFileName only.
• Key ops. MigrateDeksAsync, RotateDeksAsync. Results split Skipped vs Failed. Missing blobs and short encryption headers fail fast.
• Events. FileSaved, FileRetrieved, FileDeleted, FileMoved, FileRenamed, FileMetadataRetrieved, FileAuditOccurred. Payloads carry a redacted FileStoreSnapshot (wrapped DEK and KEK salt omitted).
• Health. CheckHealthAsync via IHealth. Depth follows HealthCheckMode.

IFileStorageDiagnosticsService

Optional ListStorageKeysAsync is implemented by backends that can enumerate object/path keys (Local, S3, Blob). All implementations normalize the caller-supplied prefix through Lyo.Exceptions.FileHelpers.NormalizeAndValidatePathPrefix, the same helper that backs FileStorageServiceBase.ValidatePathPrefix. It strips wrapping slashes and rejects .. segments, doubled separators, and embedded \0 with ArgumentException (HTTP 400 in ASP.NET).

IMultipartUploadService

Large-file client-part uploads. Register multipart services together with a keyed IFileStorageService (Extensions.AddLocalMultipartUploadService or the S3/Blob equivalents).

Types this assembly ships

Extension points

Remote protocol backends (Remote/)

FTP and SFTP storage differ only in the protocol client they drive, so the storage logic lives here once and each protocol package ships a thin adapter:

• IRemoteFileTransport. The POSIX-path file operations a remote backend needs: HealthPingAsync, FileExistsAsync, CreateDirectoryAsync, DeleteFileAsync, GetLengthAsync, OpenReadAsync, OpenCreateAsync, DownloadBytesAsync, UploadAsync, RenameAsync, and CopyFileAsync, plus ProtocolName and RootRemoteDirectory for logs and path building. Implementations adapt a protocol library without Lyo.FileStorage depending on it.
• RemoteFileStorageServiceBase. A FileStorageServiceBase that implements every storage override (output streams, reads, deletes, size, encryption-header extraction and rewrite, partial-file cleanup, copy, move, health) against an IRemoteFileTransport.

Paths handed to the transport are already absolute and jailed under RootRemoteDirectory, so adapters do not repeat that check. Adding a protocol means implementing the transport interface and a DI extension — not another copy of the storage service.

Multipart, direct upload, DEK operations

• FileStorageDekOperations. Implements MigrateDeksAsync and RotateDeksAsync the same way across backends, including short-encryption-header detection and per-file failure isolation.
• FileStorageStreamingPipelines. Composes compression / encryption / hash / max-size guards over streamed save and read paths.
• PlainDirectUploadCoordinator. Finalizes plaintext direct-upload PUTs into a normal SaveFileAsync outcome (policy, availability) and runs only when the caller used BeginDirectUploadAsync without compression/encryption hints.

DiskFileStorageOptions

DiskFileStorageOptions adds:

Compression resolver

Duplicate detection

When EnableDuplicateDetection is true, saves hash plaintext and call IFileMetadataStore.FindByHashAsync before persisting transformed bytes.

Reads stay the same: GetFileAsync / GetFileStreamAsync decode according to stored metadata, not per-request compress/encrypt flags.

Legacy appsettings LocalFileStorage vs DiskFileStorage binder details are documented on DiskFileStorageOptions.LegacySectionName and DiskFileStorageConfigurationBinder. The internal BindDiskFileStorage(IServiceProvider, string? preferredSection) is invoked by the keyed-disk AddFileStorageServiceKeyed overloads that accept a configSectionName. It tries the preferred section, then DiskFileStorageOptions.SectionName (DiskFileStorage), then DiskFileStorageOptions.LegacySectionName (LocalFileStorage) and logs a warning when the legacy section is matched.

Keyed registration overload matrix

AddFileStorageServiceKeyed ships in seven shapes so the same key can wire (a) an existing keyed file store, (b) a fresh LocalFileStorageService from options/section, and either reuse or define the encryption key:

For cloud backends, use the package-specific keyed entry points: AddS3FileStorageServiceKeyed(keyName) in Lyo.FileStorage.S3 (fluent builder) and the non-keyed AddAzureBlobFileStorageService(...) in Lyo.FileStorage.AzureBlob (keyed Blob is an open item. See that README).

DTOs

For disk, dependency injection is usually Extensions.AddFileStorageServiceKeyed overloads keyed with your tenant/service key, alongside IFileMetadataStore registration.

Features

• Multiple storage backends. Local disk lives in this package. Cloud lives in Lyo.FileStorage.S3 and Lyo.FileStorage.AzureBlob.
• Compression and encryption. Optional ICompressionService (exposes Resolver and ResolveForCompress; policy via AddCompressionPolicySelector) and ITwoKeyEncryptionService.
• Metadata. IFileMetadataStore (FileStoreResult).
• Duplicate detection. Configurable hashing strategies via DuplicateHandlingStrategy.
• Streaming. SaveFromStreamAsync, pipeline reads via GetFileStreamAsync.
• Lifetimes. Honour lifetimes on IFileMetadataStore and keystores. Overlapping async calls on one service instance are supported.
• Cleanup. Partial file cleanup helpers on FileStorageServiceBase.
• Metrics and logging. Hooks into IMetrics / ILoggerFactory.

Error handling

Errors cover missing optional services, invalid prefixes, traversal attempts (diagnostics/listing paths), FileNotAvailableException for availability-aware reads, FilePolicyRejectedException for content-policy rejections, and FileNotFoundException when configured to throw.

Security

Path prefixes are normalised on cloud/local paths via shared helpers: Lyo.Exceptions.FileHelpers.NormalizeAndValidatePathPrefix for both listing prefixes and save/direct-upload entry points, and CloudObjectKeyBuilder for object/blob key shape (Build, FromMetadata from SourceFileName + path prefix, InferTrailingSuffixAfterFileId). Save paths also apply EnsureUnderRoot before writing to disk. HashVerifyingReadStream uses a fixed-time compare and only verifies on EOF. Pre-signed reads fall back to the metadata-recorded PathPrefix so SAS/GET URLs work even when the caller cannot supply the original prefix. DirectUploadReceiveBaseUri trusts the named host. Use only inside controlled Test API topologies.

Concurrency

LocalFileStorageService and FileStorageServiceBase accept overlapping async calls. Honour the lifetimes of IFileMetadataStore and keystores.

Health checks

Call await fileStorage.CheckHealthAsync(ct). Backends pick lightweight vs deeper modes via HealthCheckMode.

Tests

Cloud backends use a DispatchProxy-based lightweight stub for IAmazonS3. Deeper end-to-end coverage would need LocalStack/Azurite.

Dependencies

Generated from ProjectReference / PackageReference (same model as docs/Lyo.ProjectGraph.html).

• Lyo.Common.Metadata (direct, lyo)
• Lyo.Compression (direct, lyo)
• Lyo.Configuration (direct, lyo)
• Lyo.Encryption (direct, lyo)
• Lyo.Exceptions (direct, lyo)
• Lyo.FileMetadataStore (direct, lyo)
• Lyo.Hashing (direct, lyo)
• Lyo.Health (direct, lyo)
• Lyo.IO.Temp (direct, lyo)
• Lyo.Metrics (direct, lyo)
• Lyo.Streams (direct, lyo)
• Microsoft.Bcl.AsyncInterfaces 10.0.5 (direct, microsoft, netstandard2.0)
• Microsoft.Extensions.Hosting.Abstractions 10.0.5 (direct, microsoft)
• System.Text.Json 10.0.5 (direct, microsoft, netstandard2.0)
• Lyo.Common.Core (transitive, lyo)
• Lyo.KeyStore (transitive, lyo)
• Lyo.Result (transitive, lyo)
• BouncyCastle.Cryptography 2.6.2 (transitive, third-party, netstandard2.0)
• EasyCompressor 2.1.0 (transitive, third-party)
• Konscious.Security.Cryptography.Argon2 1.3.1 (transitive, third-party)
• Microsoft.Extensions.Configuration.Binder 10.0.5 (transitive, microsoft)
• Microsoft.Extensions.DependencyInjection.Abstractions 10.0.5 (transitive, microsoft, net10.0, netstandard2.0)
• Microsoft.Extensions.Logging.Abstractions 10.0.5 (transitive, microsoft)
• Microsoft.Extensions.Options.ConfigurationExtensions 10.0.5 (transitive, microsoft)
• Microsoft.Extensions.Options.DataAnnotations 10.0.5 (transitive, microsoft)
• System.Buffers 4.6.1 (transitive, microsoft, netstandard2.0)
• System.IO.Hashing 10.0.5 (transitive, microsoft, net10.0)
• System.Memory 4.6.3 (transitive, microsoft, netstandard2.0)
• System.Threading.Tasks.Extensions 4.6.3 (transitive, microsoft, netstandard2.0)