Application Archiving

Application Archiving is a way to save a copy of an Application’s Device data, Events, and/or Data Tables on either Amazon S3, Google Cloud Storage, Microsoft Azure.

Application Archive

The archive configuration allows you to specify which Devices’, Events’, and Data Tables’ data should be backed up. After your archive configuration has been set, you have the option to backfill your device data archive and fully archive Events and Data Tables.

Configuration

Each storage service provider has its own configuration:

Amazon S3: In order to configure archiving for Amazon S3, “Bucket”, “Region”, “Access Key ID” and “Secret Access Key” are required.

Google Cloud Storage: In order to configure archiving from Google Cloud Storage, “Bucket”, “Project ID”, and “Account Key (JSON)” are required.

Azure Storage: In order to configure archiving from Google Cloud Storage, “Container”, “Account Name”, and “Account Key” are required.

All three have the optional field “Directory Inside the Bucket” (or “Container” for Azure), which specifies a directory for archival files to go; if left unset, the files will be appended to the top-level directory.

Templating

All archive configurations are templatable via your application globals.

Application Archive Templating

Note: If a template is provided for a configuration field, it must resolve to a valid value in order to save the archive configuration.

Permissions

Before setting up your configuration to archive, make sure that your 3rd-party user has the correct permissions.

For Amazon S3, set the user’s IAM Policy to the following guidelines, such that bucketname is replaced with the actual bucket name:

{
  "Version": "YYYY-MM-DD",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:getBucketAcl"],
      "Resource": ["arn:aws:s3:::bucketname"]
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": ["arn:aws:s3:::bucketname/*"]
    }
  ]
}

For Google Cloud Services, create a service account with ObjectAdmin permissions, which are the minimal permissions. More information on permissions within Google Cloud Services can be found in Google Cloud’s Documentation.

For Azure Storage, ensure that the accessing account has at least the role of Storage Blob Data Contributor assigned to it. More information on role assignment in Azure can be found here.

Device Data

If “Enable Device Data Archiving” is selected, Devices matching your Archive Mode will be archived.

Application Archive Device Data

Device Data Archive Frequency

When device data becomes older than 30 days, Losant creates a directory in the configured storage service that represents the archived date. The directory will contain a CSV for each device.

The reason Losant waits for the data to be older than 30 days is because device state data can be overwritten until the data is older than 30 days.

Device Data Archive Modes

When specifying the “Archive Modes,” the following options are available:

  • Archive all devices: All Devices within the application will be selected for data to archive.
  • Archive all EXCEPT these devices…: A blacklist of Devices and/or Device Tags.
  • Archive ONLY these devices…: A whitelist of Devices and/or Device Tags.

There is also an “Archive Backfill” button which will immediately enqueue a job to archive data from 31 days ago.

Device Data Generated CSV

Each generated CSV file is placed within a directory named by the ISO timestamp of the archived date. The files themselves will be named by the applicationId, deviceId, and the start and end time of the data contained in the file. An example directory and file would be:

losant-bucket/2017-12-03T00:00:00.000Z/568beedeb436ab01007be53d-568bf74a1ff37b0100f5123e-1512259200000-1512345599999.csv

Each Device Data CSV will have an “ID” column (for the device ID), a “Timestamp” column (where the timestamp will be represented as a Unix timestamp in milliseconds), an ”ISO Date” column (where the time is represented in human-readable form), as well as a column for each attribute on your device. The following is an example of a CSV:

"ID","Timestamp","ISO Date","Current","On","Inuse"
"568bf74a1ff37b0100f5123e",1512259242342,"2017-12-03T00:00:42.342Z",21.666666666666668,1,1
"568bf74a1ff37b0100f5123e",1512259302424,"2017-12-03T00:01:42.424Z",21.733333333333334,1,1
"568bf74a1ff37b0100f5123e",1512259362505,"2017-12-03T00:02:42.505Z",21.8,1,1
"568bf74a1ff37b0100f5123e",1512259422586,"2017-12-03T00:03:42.586Z",21.866666666666667,1,1
"568bf74a1ff37b0100f5123e",1512259482697,"2017-12-03T00:04:42.697Z",21.933333333333334,1,1

Blob Data

If any of your devices have attributes that are of type Blob, the archive will contain the original base64 strings of data within the CSV.

Archives will never contain the Blob Signed URLs because they expire.

Events

If “Enable Events Archiving” is selected, all Events will be archived.

Application Archive Events

There is also a “Full Archive” button which will immediately enqueue a job to archive all accessible Events in the current application.

Event Archive Frequency

Events are archived daily. For each day all the Events that have been created or modified the day before, Losant will create a directory in the configured storage service that represents the archived date. The directory will contain a CSV containing all Events and updates.

Events Generated CSV

Each generated CSV file is placed within a directory named by the ISO Date of the archived day. The files themselves will be named by the applicationId, the word “events”, and the archived time of the event data contained in the file. An example directory and file would be:

losant-bucket/2017-12-03T00:00:00.000Z/568beedeb436ab01007be53d-events-1512345599999.csv

The Events CSV will have the following columns: “ID” (for the event ID),“Creation Date” (ISO Date), “Last Updated” (ISO Date), “Source ID” (where the event originated), “Source Type”, “Level”, “State”, “Subject”, “Message”, “Device ID”, “Data” (in the form of a JSON), and an “Update Number” (0 for original/parent event). The following is an example of a CSV:

"ID","Creation Date","Last Updated","Source ID","Source Type","Level","State","Subject","Message","Device ID","Data","Update Number"
"5d24b2fc84cd730006e40cbe","2019-07-09T15:30:04.040Z","2019-07-09T15:30:04.042Z","5b031db82f4fa100050fdf35","user","critical","new","broken device",,,,0
"5d24b2e62f909b0006e2a145","2019-07-09T15:29:42.612Z","2019-07-09T15:29:42.619Z","5b031db82f4fa100050fdf35","user","critical","new","lost device",,,,0
"5c54cbfd10f81d000bac06aa","2019-02-01T22:45:17.629Z","2019-05-08T16:24:57.204Z","5c54c7ff1ef216000626fc38","flow","info","acknowledged","Checking","this is an event",,,0
"5c54cbfd10f81d000bac06aa","2019-05-08T14:57:24.085Z",,"5b0db05e3a4350000933f993","user",,"acknowledged",,"this is an update",,,1

Data Tables

If “Enable Data Tables Archiving” is selected all Data Tables matching your “Archive Mode” will be archived.

Application Archive Tables

Data Tables Frequency

Data Tables are archived daily. Each day, Losant will create a directory in the configured storage service that represents the archived date. The directory will contain a CSV for each Data Table.

Data Tables Archive Modes

When specifying the “Archive Modes,” the following options are available:

  • Archive all data tables: All Data Tables within the application will be selected for data to archive.
  • Archive all EXCEPT these data tables…: A blacklist of Data Tables.
  • Archive ONLY these data tables…: A whitelist of Data Tables.

There is also a “Full Archive” button which will immediately enqueue a job to archive all Data Tables connected to the application.

Data Tables Generated CSV

Each generated CSV file is placed within a directory provided or named by the ISO timestamp of the archived date. The files themselves will be named by the applicationId, the word “data-table-data”, and the Data Table’s ID. An example directory and file would be:

losant-bucket/2017-12-03T00:00:00.000Z/568beedeb436ab01007be53d-data-table-data-5d24b2fc84cd730006e40cbt.csv

Or if you provide a directory in the configuration:

losant-bucket/2017-12-03T00:00:00.000Z/your-directory/568beedeb436ab01007be53d-data-table-data-5d24b2fc84cd730006e40cbt.csv

Each Data Table CSV will have an “id” column (for the row ID), your data columns, an “updatedAt” column (ISO Date), a “createdAt” column (ISO Date). The following is an example of a CSV:

"id","Hour","PriceScale","updatedAt","createdAt"
"5d2d45183c4fad0006e657b0",0,0.8199146941389697,"2019-07-16T03:31:36.593Z","2019-07-16T03:31:36.593Z"
"5d2d45183c4fad0006e657b1",1,0.7345838500288444,"2019-07-16T03:31:36.593Z","2019-07-16T03:31:36.593Z"
"5d2d45183c4fad0006e657b2",2,0.5752394968861998,"2019-07-16T03:31:36.593Z","2019-07-16T03:31:36.593Z"
"5d2d45183c4fad0006e657b3",3,0.44334167226038557,"2019-07-16T03:31:36.593Z","2019-07-16T03:31:36.593Z"