Application Archiving

Application Archive

Application Archiving is a way to save a copy of an Application’s Device data, Events, and/or Data Tables on either Amazon S3 or Google Cloud Storage. The archive configuration allows you to specify which Devices’, Events’, and Data Tables’ data should be backed up. After your archive configuration has been set you will also have the options to backfill your device data archive and fully archive Events and Data Tables.

Configuration

Amazon S3: In order to configure archiving for Amazon S3, Bucket, Region, Access Key ID and Secret Access Key are required.

Google Cloud Storage: In order to configure archiving from Google Cloud Storage, Bucket, Project ID, and Account Key (JSON) are required.

Both have one optional field, Directory Inside the Bucket, which specifies a directory for archival files to go; if left unset, the files will be appended to the top-level directory.

Permissions

Before setting up your configuration to archive, make sure that your 3rd-party user has the correct permissions.

For Google Cloud Services, create a service account with ObjectAdmin permissions, which are the minimal permissions.

For Amazon S3 archival, set the user’s IAM Policy to the following guidelines, such that bucketname is replaced with the actual bucket name:

{
  "Version": "YYYY-MM-DD",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:getBucketAcl"],
      "Resource": ["arn:aws:s3:::bucketname"]
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": ["arn:aws:s3:::bucketname/*"]
    }
  ]
}

Device Data

Application Archive Device Data

If Enable Device Data Archiving is selected Devices matching your Archive Mode will be archived when the data becomes older than 30 days, the application will create a directory for the archived date and a CSV for each device within that directory.

There is also an Archive Backfill button which will immediately enqueue a job to archive data from 31 days ago.

The reason we wait for the data to be older than 30 days is because device state data can be overwritten until the data is older than 30 days.

Archive Modes include:

  • Archive all devices: All Devices within the application will be selected for data to archive.
  • Archive all EXCEPT these devices…: A blacklist of Devices and/or Device Tags.
  • Archive ONLY these devices…: A whitelist of Devices and/or Device Tags.

Device Data Generated CSV

Each generated CSV file will be placed within a directory named by the ISO timestamp of the archived date. The files themselves will be named by the applicationId, deviceId, and the start and end time of the data contained in the file. An example directory and file would be:

losant-bucket/2017-12-03T00:00:00.000Z/568beedeb436ab01007be53d-568bf74a1ff37b0100f5123e-1512259200000-1512345599999.csv

Each Device Data CSV will have an “ID” column (for the device ID), a “Timestamp” column (where the timestamp will be represented as a Unix timestamp in milliseconds), an ”ISO Date” column (where the time is represented in human-readable form), as well as a column for each attribute on your device. The following is an example of a CSV:

"ID","Timestamp","ISO Date","Current","On","Inuse"
"568bf74a1ff37b0100f5123e",1512259242342,"2017-12-03T00:00:42.342Z",21.666666666666668,1,1
"568bf74a1ff37b0100f5123e",1512259302424,"2017-12-03T00:01:42.424Z",21.733333333333334,1,1
"568bf74a1ff37b0100f5123e",1512259362505,"2017-12-03T00:02:42.505Z",21.8,1,1
"568bf74a1ff37b0100f5123e",1512259422586,"2017-12-03T00:03:42.586Z",21.866666666666667,1,1
"568bf74a1ff37b0100f5123e",1512259482697,"2017-12-03T00:04:42.697Z",21.933333333333334,1,1

Events

Application Archive Events

If Enable Events Archiving is selected all Events will be archived daily that have been created or modified the day before, the application will create a directory for the archived date and a CSV containing all Events and updates.

There is also a Full Archive button which will immediately enqueue a job to archive all accessible Events in the current application.

Events Generated CSV

Each generated CSV file will be placed within a directory named by the ISO Date of the archived day. The files themselves will be named by the applicationId, the word “events”, and the archived time of the event data contained in the file. An example directory and file would be:

losant-bucket/2017-12-03T00:00:00.000Z/568beedeb436ab01007be53d-events-1512345599999.csv

The Events CSV will have the following columns: “ID” (for the event ID),“Creation Date” (ISO Date), “Last Updated” (ISO Date), “Source ID” (where the event originated), “Source Type”, “Level”, “State”, “Subject”, “Message”, “Device ID”, “Data” (in the form of a JSON), and an “Update Number” (0 for original/parent event). The following is an example of a CSV:

"ID","Creation Date","Last Updated","Source ID","Source Type","Level","State","Subject","Message","Device ID","Data","Update Number"
"5d24b2fc84cd730006e40cbe","2019-07-09T15:30:04.040Z","2019-07-09T15:30:04.042Z","5b031db82f4fa100050fdf35","user","critical","new","broken device",,,,0
"5d24b2e62f909b0006e2a145","2019-07-09T15:29:42.612Z","2019-07-09T15:29:42.619Z","5b031db82f4fa100050fdf35","user","critical","new","lost device",,,,0
"5c54cbfd10f81d000bac06aa","2019-02-01T22:45:17.629Z","2019-05-08T16:24:57.204Z","5c54c7ff1ef216000626fc38","flow","info","acknowledged","Checking","this is an event",,,0
"5c54cbfd10f81d000bac06aa","2019-05-08T14:57:24.085Z",,"5b0db05e3a4350000933f993","user",,"acknowledged",,"this is an update",,,1

Data Tables

Application Archive Tables

If Enable Data Tables Archiving is selected all Data Tables matching your Archive Mode will be archived daily, the application will create a directory for the archived date and a CSV for each data table within that directory.

There is also a Full Archive button which will immediately enqueue a job to archive all Data Tables connected to the application.

Archive Modes include:

  • Archive all data tables: All Data Tables within the application will be selected for data to archive.
  • Archive all EXCEPT these data tables…: A blacklist of Data Tables.
  • Archive ONLY these data tables…: A whitelist of Data Tables.

Data Tables Generated CSV

Each generated CSV file will be placed within a directory provided or named by the ISO timestamp of the archived date. The files themselves will be named by the applicationId, the word “data-table-data”, and the data table ID. An example directory and file would be:

losant-bucket/2017-12-03T00:00:00.000Z/568beedeb436ab01007be53d-data-table-data-5d24b2fc84cd730006e40cbt.csv

Or if you provide a directory in the configuration:

losant-bucket/2017-12-03T00:00:00.000Z/your-directory/568beedeb436ab01007be53d-data-table-data-5d24b2fc84cd730006e40cbt.csv

Each Data Table CSV will have an “id” column (for the row ID), your data columns, an “updatedAt” column (ISO Date), a “createdAt” column (ISO Date). The following is an example of a CSV:

"id","Hour","PriceScale","updatedAt","createdAt"
"5d2d45183c4fad0006e657b0",0,0.8199146941389697,"2019-07-16T03:31:36.593Z","2019-07-16T03:31:36.593Z"
"5d2d45183c4fad0006e657b1",1,0.7345838500288444,"2019-07-16T03:31:36.593Z","2019-07-16T03:31:36.593Z"
"5d2d45183c4fad0006e657b2",2,0.5752394968861998,"2019-07-16T03:31:36.593Z","2019-07-16T03:31:36.593Z"
"5d2d45183c4fad0006e657b3",3,0.44334167226038557,"2019-07-16T03:31:36.593Z","2019-07-16T03:31:36.593Z"