MRIIOT.DIME 1.2.0

Data In Motion Enterprise

Reshape and move data from enterprise and industrial sources to message queues, databases, and other sinks.

<i>Data integration made easy!</i>™

Videos

• Quick Introduction

How to Run

Windows

Download the latest release and run DIME.exe. Alternatively, DIME.exe install will install the DIME Windows Service.

Docker

cd ~
mkdir -p volumes/dime/configs
mkdir -p volumes/dime/lua
mkdir -p volumes/dime/logs
git clone https://github.com/ladder99/DIME
cp DIME/DIME/nlog.config volumes/dime/nlog.config
cp DIME/DIME/Configs/* volumes/dime/configs
cp DIME/DIME/Lua/* volumes/dime/lua

docker run \
   -p 8080:8080 \
   -p 8081:8081 \
   -v ~/volumes/dime/nlog.config:/app/nlog.config \
   -v ~/volumes/dime/configs:/app/Configs \
   -v ~/volumes/dime/lua:/app/Lua \
   -v ~/volumes/dime/logs:/app/Logs \
   ladder99/dime:latest

Configuration Example

Below configuration moves data from a Rockwell PLC and an MQTT broker to an MQTT broker.

sinks:
  - name: mqttSink1
    connector: MQTT
    address: wss.sharc.tech
    port: !!int 1883
    base_topic: ids
sources:
  - name: plcSource1
    connector: EthernetIP
    type: !!int 5
    address: 192.168.111.20
    path: 1,0
    items:
      - name: boolTag1
        type: bool
        address: B3:0/2
      - name: boolTag2
        type: bool
        address: B3:0/3
  - name: mqttSource1
    connector: MQTT
    address: wss.sharc.tech
    port: !!int 1883
    items:
      - name: subscribe1
        enabled: !!bool true
        address: sharc/+/evt/#

Connectors

<table><tr><td valign="top">

</td><td valign="top">

</td></tr></table>

ASC CPC

Source Example

  - name: ascCpcSource1
    connector: AscCPC
    address: 192.168.111.12
    port: !!int 9999
    bypass_ping: !!bool true
    init_script: ~
    items:
      - name: Temperature
        address: .Autoclave.Inputs.AIRTC\Value

Ethernet/IP

Source Example

  - name: plcSource1
    connector: EthernetIP
    type: !!int 5
    address: 192.168.111.20
    path: 1,0
    log: !!int 0
    timeout: !!int 1000
    bypass_ping: !!bool true
    items:
      - name: boolTag1
        type: bool
        address: B3:0/2
      - name: intTag2
        type: int
        address: N7:1

Haas SHDR

Source Example

  - name: haasSource1
    connector: HaasSHDR
    itemized_read: !!bool false
    address: 192.168.111.221
    port: !!int 9998
    timeout: !!int 1000
    heartbeat_interval: !!int 4000
    retry_interval: !!int 10000
    items:
      - name: CPU
        enabled: !!bool true
        address: CPU
        script: |
          if tonumber(result) > 0.5 then
            return 'HIGH';
          else
            return 'LOW';
          end

HTTP Server

Sink Example

  - name: httpServerSink1
    connector: HttpServer
    uri: http://localhost:8080/

Source Example

  - httpServerSource1: &httpServerSource1
    name: httpServerSource1
    connector: HTTPServer
    uri: http://localhost:8081/
    init_script: |
      json = require('json');
    items:
      - name: postData
        address: post/data
        script: |
          return json.decode(result).hello;

Influx LP

Sink Example

  - name: influxLpSink1
    connector: InfluxLP
    address: 172.24.56.104
    port: !!int 8086
    token: abc123
    bucket_name: bucket1
    org_id: org1

Modbus TCP

Source Example

  - name: modbusSource1
    connector: ModbusTCP
    address: 192.168.111.20
    port: !!int 502
    slave: !!int 1
    timeout: !!int 1000
    init_script: |
      -- https://github.com/iryont/lua-struct
       struct = require('struct')
    items:
      - name: coilTags
        type: !!int 1
        address: !!int 1
        count: !!int 10
      - name: holdingTags
        type: !!int 3
        address: !!int 24
        count: !!int 2
        script: |
           -- https://www.scadacore.com/tools/programming-calculators/online-hex-converter/
           return struct.unpack('<I', struct.pack('<HH', result[0], result[1]));

MQTT

Sink Example

  - name: mqttSink1
    connector: MQTT
    address: wss.sharc.tech
    port: !!int 1883
    base_topic: ids
    qos: !!int 0
    retain: !!bool true

Source Example

  - name: mqttSource1
    connector: MQTT
    address: wss.sharc.tech
    port: !!int 1883
    items:
      - name: subscribe1
        address: sharc/+/evt/#

MTConnect Agent

Sink Example

  - name: mtConnectSink1
    connector: MTConnectAgent
    port: !!int 5000
    device_uuid: 123
    device_id: device1
    device_name: device1
    device_manufacturer: acme
    device_model: 123
    device_serial_number: 123

Source Example

  - name: mtConnectSource1
    connector: MTConnectAgent
    address: mtconnect.mazakcorp.com
    port: !!int 5719
    interval: !!int 100
    items:
      - name: PathPositionSample
        address: pathpos
        script: |
          return result[0].Value;

MTConnect SHDR

Sink Example

  - name: shdrSink1
    connector: MTConnectSHDR
    port: !!int 7878
    device_key: ~
    heartbeat_interval: !!int 10000
    filter_duplicates: !!bool true

Redis

Sink Example

  - name: redisSink1
    connector: Redis
    address: 172.24.56.104
    port: !!int 6379
    database: !!int 0

Source Example

  - name: redisSink1
    connector: Redis
    address: 172.24.56.104
    port: !!int 6379
    database: !!int 0
    items:
      - name: plcGoodPartCount
        address: eipSource1/GoodPartCount

Script

Source Example

  - name: scriptSource1
    connector: Script
    init_script: ~
    deinit_script: ~
    enter_script: ~
    exit_script: ~
    items:
      - name: Temperature
        script: |
          return math.random(100);

SNMP

Source Example

  - name: snmpSource1
    connector: SNMP
    address: 192.168.150.143
    port: !!int 161
    community: public
    timeout: !!int 1000
    items:
      - name: Temperature
        address: 1.3.6.1.4.1.6574.1.2.0
      - name: Model
        address: 1.3.6.1.4.1.6574.1.5.1.0
      - name: SerialNumber
        address: 1.3.6.1.4.1.6574.1.5.2.0

SparkplugB

Sink Example

  - name: sparkplugBSink1
    connector: SparkplugB
    address: localhost
    port: !!int 1883
    username: admin
    password: admin
    host_id: dime
    group_id: dime
    node_id: dime
    device_id: dime
    reconnect_interval: !!int 15000
    birth_delay: !!int 10000

Timebase Websocket

Source Example

   - name: timebaseWsSource1
     connector: TimebaseWS
     address: localhost
     port: 4511
     items:
       - name: plcExecution
         group: MQTT Data
         address: dime/eipSource1/Execution/Data

Wintriss SmartPac

Source Example

  - name: smartpacSource1
    connector: SmartPac
    address: 172.16.200.18
    port: !!int 1007
    items:
      - name: PressType
        enabled: !!bool true
        script: return result[0];
      - name: PressName
        enabled: !!bool true
        script: return result[1];

Scripting

Each connector configuration allows for Lua script execution. The init_script property is executed on startup and is used to import additional .NET or Lua libraries. The deinit_script property is executed on shutdown. The enter_script and exit_script properties are executed before and after reading all items, respectively. Within each item script, the primary cache can be accessed using the cache(path, defaultValue), cache_ts(path, defaultValue) function calls. The path refers to the item's unique path which is a combination of the connector's and item's name (e.g. eipSource1/boolTag2, mqttSource1/ffe4Sensor). Within the connector's execution context, the connector name can be omitted and replaced with a period, ./boolTag2. A secondary cache can be accessed using the get(key, defaultValue) and set(key, value) function calls. This user-defined cache is scoped to the individual connector.

mqttSink1: &mqttSink1
   name: mqttSink1
   connector: MQTT
   address: wss.sharc.tech
   port: !!int 1883
   base_topic: ids

eipSource1: &eipSource1
   name: eipSource1
   connector: EthernetIP
   type: !!int 5
   address: 192.168.111.20
   path: 1,0
   log: !!int 0
   timeout: !!int 1000
   init_script: ~
   deinit_script: ~
   enter_script: ~
   exit_script: ~
   items:
      - name: boolSetUserCacheOnly
        enabled: !!bool true
        type: bool
        address: B3:0/2
        script: |
           set('boolTag', result);
           return nil;
      - name: boolGetUserCache
        enabled: !!bool true
        script: |
           return get('boolTag', false);
      - name: Execution
        enabled: !!bool true
        type: bool
        address: B3:0/3
        script: |
           local m = { [0]='Ready', [1]='Active' };
           return m[result and 1 or 0];
        
mqttSource1: &mqttSource1
   name: mqttSource1
   connector: MQTT
   itemized_read: !!bool true
   address: wss.sharc.tech
   port: !!int 1883
   init_script: |
      -- https://github.com/rxi/json.lua
      json = require('json');
   items:
      - name: subscribe1
        enabled: !!bool false
        address: sharc/+/evt/#
      - name: ffe4Sensor
        rbe: !!bool false
        address: sharc/08d1f953ffe4/evt/io/s1
        script: |
           return json.decode(result).v.s1.v;
      - name: ffe4SensorAndDelta
        rbe: !!bool false
        address: sharc/08d1f953ffe4/evt/io/s1
        script: |
           return json.decode(result).v.s1.v, json.decode(result).v.s1.d;

scriptSource1: &scriptSource1
   name: scriptSource1
   connector: Script
   init_script: |
      luanet.load_assembly("System")
      CLR = {
        env = luanet.import_type("System.Environment")
      };
      -- https://github.com/rxi/json.lua
      json = require('json');
      -- https://github.com/Yonaba/Moses
      moses = require('moses');
      pcArray = {}
   items:
      - name: machineNameDiscrete
        rbe: !!bool false
        script: |
           return CLR.env.MachineName;
      - name: machineNameByRefRbe
        script: |
           return cache('./machineNameDiscrete', nil);
      - name: dateTime
        script: |
           return os.date("%Y-%m-%d %H:%M:%S");
      - name: randomUserCacheOnly
        script: |
           set('random', math.random(500));
           return nil;
      - name: randomFromUserCache
        script: |
           return get('random', -1);
      - name: mqttSensorReading
        script: |
           return cache('mqttSource1/ffe4Sensor', nil);
      - name: mqttSensorReadingMedian
        script: |
           table.insert(pcArray, cache('mqttSource1/ffe4Sensor', 0));
           pcArray = moses.last(pcArray, 100);
           return moses.median(pcArray);
      - name: AcmeCorp/ChicagoPlant/AssemblyArea/Line1/PartCount
        rbe: !!bool false
        script: |
           return cache('mqttSource1/ffe4Sensor', nil)
      - name: AcmeCorp/ChicagoPlant/AssemblyArea/Line1/ReportPeriod
        rbe: !!bool false
        script: |
           return cache('mqttSource1/ffe4SensorAndDelta', {0, 0})[1]
      - name: AcmeCorp/ChicagoPlant/AssemblyArea/Line1/Execution
        rbe: !!bool true
        script: |
           return cache('eipSource1/Execution', nil)
      - name: OverallAvailabilityArrayNonRbe
        rbe: !!bool false
        script: |
           local n = cache('eipSource1/$SYSTEM/IsConnected', nil);
           return n, n==true;
      - name: OverallAvailabilityArrayRbe
        script: |
           local n = cache('eipSource1/$SYSTEM/IsConnected', nil);
           return n, n==true;
      - name: OverallAvailabilityRbe
        script: |
           local n = cache('eipSource1/$SYSTEM/IsConnected', nil);
           return n==true and 'Available' or 'Unavailable';

sinks:
   - *mqttSink1
sources:
   - *eipSource1
   - *mqttSource1
   - *scriptSource1

Docker

cd ~
git clone https://github.com/ladder99/DIME
cd DIME/DIME
docker build -f Dockerfile --tag=ladder99/dime:1.0.0 --tag=ladder99/dime:latest .
docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock wagoodman/dive:latest ladder99/dime:latest
docker login
docker push ladder99/dime:1.0.0
docker push ladder99/dime:latest
docker logout

cd ~
mkdir -p volumes/dime/configs
mkdir -p volumes/dime/lua
mkdir -p volumes/dime/logs
cp DIME/DIME/nlog.config volumes/dime/nlog.config
cp DIME/DIME/Configs/* volumes/dime/configs
cp DIME/DIME/Lua/* volumes/dime/lua

docker run \
   -p 8080:8080 \
   -p 8081:8081 \
   -v ~/volumes/dime/nlog.config:/app/nlog.config \
   -v ~/volumes/dime/configs:/app/Configs \
   -v ~/volumes/dime/lua:/app/Lua \
   -v ~/volumes/dime/logs:/app/Logs \
   ladder99/dime:latest

Creating a New Connector

1. Add configuration mapper classes in Configuration.{new_connector} folder.
   1. ConnectorConfiguration.cs - inherits from IDS.Transporter.Configuration.ConnectorConfiguration.
   2. ConnectorItem.cs - inherits from IDS.Transporter.Configuration.ConnectorItem.
2. Add a configurator factory for the new connector in Configurator.{new_connector} folder.
   1. Source.cs - static class in IDS.Transporter.Configurator.{new_connector} folder.
   2. Sink.cs - static class in IDS.Transporter.Configurator.{new_connector} folder.
   3. Update SourceConnectorFactory.cs or SourceConnectorFactory.cs.
3. Add connector implementation in Connectors.{new_connector} folder.
   1. Source.cs - inherits from IDS.Connectors.SourceConnector<IDS.Transporter.Configuration.{new_connector}.ConnectorConfiguration, IDS.Transporter.Configuration.{new_connector}.ConnectorItem>.
   2. Source.cs - inherits from IDS.Connectors.SourceConnector<IDS.Transporter.Configuration.{new_connector}.ConnectorConfiguration, IDS.Transporter.Configuration.{new_connector}.ConnectorItem>.

solution
 |
 |- Configuration (1)
 |    |- NewConnector
 |         |- ConnectorConfiguration.cs
 |         |- ConnectorItem.cs
 |- Configurator (2)
 |    |- NewConnector
 |    |    |- Source.cs
 |    |    |- Sink.cs
 |    |- SourceConnectorFactory.cs
 |    |- SinkConnectorFactory.cs   
 |- Connectors (3)
      |- NewConnector
           |- Source.cs
           |- Sink.cs