magic.data.cql 10.0.20

NoSQL IO and logging adapters for Hyperlambda

This data adapter contains alternative NoSQL file system services, implementing IFileService, IFolderService, and IStreamService, allowing you to use as an interchangeable "virtual file system" for cases where you want to have 100% stateless magic instances, which is important if you're using Magic in a Kubernetes cluster or something similar, load balancing invocations, virtually resolving files and folders towards a virtual file system. If you take this path you'll have to configure your "appsettings.json" file such as illustrated further down in this document. The project also contains an ILogger implementation service you can use that will create log entries in a NoSQL based storage of your choice. See below for details about how to configure this.

Configuration

The primary configuration for the project to apply for your "appsettings.json" file can be found below. Notice, the IO and log services requires you to use generic as your cluster name, and you cannot change this.

{
  "magic": {
    "cql": {
      "generic": {
        "host": "127.0.0.1"
      }
    }
  }
}

The above configures the adapter to use 127.0.0.1 as the host for your contact point or cluster. To configure the adapter to store files and folders inside of its CQL based database, you can alternatively add something such as follows to your "appsettings.json" file.

{
  "magic": {
    "io": {
      "file-service": "magic.data.cql.io.CqlFileService",
      "folder-service": "magic.data.cql.io.CqlFolderService",
      "stream-service": "magic.data.cql.io.CqlStreamService"
    }
  }
}

If you want to use a CQL based log implementation, you'll have to configure Magic to use the NoSQL ILogger service such as follows.

{
  "magic": {
    "logging": {
      "service": "magic.data.cql.logging.Logger"
    }
  }
}

Schema

To use the alternative CQL based file storage system you'll have to create your "magic_files" keyspace and its "files" table as follows.

create keyspace if not exists magic_files with replication = { 'class': 'NetworkTopologyStrategy', 'replication_factor': 5 };

use magic_files;

create table if not exists files(
   tenant text,
   cloudlet text,
   folder text,
   filename text,
   content blob,
   primary key((tenant, cloudlet), folder, filename));

To use the alternative CQL based log implementation you'll have to create your "magic_log" keyspace and its "log" table as follows.

create keyspace if not exists magic_log with replication = { 'class': 'NetworkTopologyStrategy', 'replication_factor': 3 };

use magic_log;

create table if not exists log(
   tenant text,
   cloudlet text,
   created timeuuid,
   type text,
   content text,
   exception text,
   meta frozen<map<text, text>>,
   primary key((tenant, cloudlet), created)) with clustering order by (created desc);

alter table log with default_time_to_live = 1209600;

Notice - The above setting for TTL implies log items will be automatically deleted after 14 days, since 1,209,600 seconds implies 14 days. Depending upon your needs you might want to increase or decrease this setting.

Adding existing files into keyspace

The following Hyperlambda will insert all your existing files and folders into your cluster keyspace, allowing you to play around with an existing CQL file system implementation. Notice, you'll have to change the [.tenant] and [.cloudlet] values to resemble the absolute root folder for your Magic backend. The values in the file below is obviously just an example of how it might look like if you've got the files on a Mac within your "Documents" folder. The "tenant" and the "cloudlet" parts are resolved by doing a string split operation on your magic:io:root-files root folder upon the last (/) found in your folder - Implying if you use the Docker images with the default configuration the "tenant" part would become "magic" and the "cloudlet" part would become "files", since the Docker images stores files within the "/magic/files/" folder.

/*
 * Inserts all dynamic files and folders into the magic CQL database.
 */
cql.connect:[generic|magic_files]

   /*
    * The root folder where your Magic backend is running.
    */
   .tenant:Users
   .cloudlet:"thomashansen/Documents/projects/magic/magic/backend"

   /*
    * Inserting root folder.
    */
   cql.execute:"insert into files (tenant, cloudlet, folder, filename) values (:tenant, :cloudlet, '/files/', '')"
      tenant:x:@.tenant
      cloudlet:x:@.cloudlet

   /*
    * Inserting appsettings.json and its folder.
    */
   config.load
   convert:x:-
      type:bytes
   cql.execute:"insert into files (tenant, cloudlet, folder, filename, content) values (:tenant, :cloudlet, '/config/', 'appsettings.json', :config)"
      tenant:x:@.tenant
      cloudlet:x:@.cloudlet
      config:x:@convert
   cql.execute:"insert into files (tenant, cloudlet, folder, filename) values (:tenant, :cloudlet, '/config/', '')"
      tenant:x:@.tenant
      cloudlet:x:@.cloudlet

   /*
    * Inserting folders.
    */
   io.folder.list-recursively:/
      display-hidden:true
   for-each:x:-/*

      strings.concat
         .:/files
         get-value:x:@.dp/#
      
      cql.execute:"insert into files (tenant, cloudlet, folder, filename) values (:tenant, :cloudlet, :folder, '')"
         tenant:x:@.tenant
         cloudlet:x:@.cloudlet
         folder:x:@strings.concat

   /*
    * Inserting files.
    */
   io.file.list-recursively:/
      display-hidden:true
   for-each:x:-/*

      io.file.load.binary:x:@.dp/#
   
      strings.split:x:@.dp/#
         .:/
      unwrap:x:+
      .filename:x:@strings.split/0/-
      remove-nodes:x:@strings.split/0/-
      strings.join:x:@strings.split/*
         .:/
      strings.concat
         .:/files/
         get-value:x:@strings.join
         .:/
      strings.replace:x:-
         .://
         .:/
      cql.execute:"insert into files (tenant, cloudlet, folder, filename, content) values (:tenant, :cloudlet, :folder, :filename, :content)"
         tenant:x:@.tenant
         cloudlet:x:@.cloudlet
         folder:x:@strings.replace
         filename:x:@.filename
         content:x:@io.file.load.binary

remove-nodes:x:../**/io.folder.list-recursively/*
remove-nodes:x:../**/io.file.list-recursively/*

Project website

The source code for this repository can be found at github.com/polterguy/magic.data.cql, and you can provide feedback, provide bug reports, etc at the same place.

Quality gates