Instrument Testing for Remote Deployment

The following testing procedure aims to setup and verify the end-to-end test for a new instrument deployment. You will need access to the equipment, knowledge of the Station Commands, and have already completed the Instrument Provisioning for Remote Access guide.

This procedure will use the Python script provided by AirPhoton to quickly send commands to the API server. More information on this tool below.

Before we start testing, review the overview section below for a common setup, connectivity explanation, and walk-through of the testing process.

End-to-End Overview

The end-to-end testing aims to connect all hardware as it will be deployed, communicate using the entire technology stack, and test remote control even if the unit is sitting in the lab next to you or in the field.

There are multiple technical steps in this process, but you need to be aware of the following key points in the test.

  1. End-user Interface to API Server

  2. Optional, Cellular Modem

  3. Remote Communication Module (COM100) to API Server via Internet (WiFi)

  4. Remote Instrument commands via COM100 relay.

End-user Interface for API Server

See API Server for sending commands to the server, or use the provided Python scripts to help manage the commands.

This interface is you using a computer to only queue up or access information on the server. On a modern Internet connection, these commands from your computer to the server a nearly instant, but once on the server, there is no guarantee on the time to reach the instrument.

Your commands to the server take two forms. You are either asking for a ControlResponse (information) or submitting a Control (command to instrument).

When sending Controls, they will get queued until the instrument next connects to the server. This delay is usually very long (5+ minutes), but can be short for in-lab testing. Just know, that you can send many Control commands, but the instrument may never look, if it is not connected correctly.

Optional, Cellular Modem

If your equipment was purchased with or you have a cellular modem, you should test it in this end-to-end test. It is not a requirement, but is highly recommended.

The modem is simply an Internet connection. Turn it on, and it is a WiFi router to the Internet. The instruments and COM100 do not know about the cellular modems as they simple see the WiFi and utilize the connection.

As long as the COM100 can communicate on the Internet, it and the instrument will connect to the API Server.

Warning

For an end-to-end test, AirPhoton recommends testing every step of the technology stack. This could include your cellular modem. If you choose to test everything excluding the modem, please keep WiFi credentials organized and be prepared for connectivity issues due to mis-configuration during deployment.

Finally, just because a cellular modem works in one location (ie. with North American cellular providers), it is very possible your deployment site could have connectivity or provider issues. Please plan ahead.

As of 2020, AirPhoton provided modems use Hologram.io.

Remote Communication Module

The COM100 equipment simply creates a secure connection to the AirPhoton API server and relays messages (Controls and ControlResposnes) to and from the instruments connected.

As of COM100 firmware v0.2.0, the COM100 persists very few settings in memory. The Internet credentials and server credentials are stored in the instrument. The COM100 knows its check-in interval and stores in-the-air data until reboot or erasure.

This property make COM100 interchangeable.

Warning

As of COM100 v0.2.0, only one instrument can be connected at a time.

Remote Instrument Commands

When a COM100 checks in with the API server, it processes the Controls queue for the locally connected instrument. This could be zero, one, or many commands at one check-in.

Learning how Controls and Station Commands work may take time, but the following end-to-end testing is setup to walk through a typical process. We will use the Python script to manage one command at a time. While this may be slow and frustrating, it allows for a simple process for testing.

Note

With use, you will understand how each station command behaves. Some are stateless where they are executed and the COM100 and instrument never consider the command again while other commands (ie. executescript) change the instrument state.

Warning

If you queue up multiple stateful commands, you will likely not get the results expected.

Planning

The end-to-end test aims at tested key deployment commands. Before continuing make sure you have the following ready.

  1. Make sure you have your API credentials and access to a computer with Python installed.

  2. Verify you can use the Python script provided by AirPhoton

  3. Make sure you have completed Instrument Provisioning for Remote Access

  4. The first test will repeat the sendcommand echo example from the provisioning guide

The following test aim to verify the following:

  1. echo test to confirm COM100 connectivity

  2. sync date and time to UTC

  3. echo test to confirm instrument connectivity

  4. stage an example station script on the COM100

  5. clear space on instrument storage

  6. transfer the script to the instrument

  7. stage an example erroneous script on the COM100

  8. transfer the erroneous script to the instrument

  9. syntax check both script files

  10. set the active script file

  11. check standby status

  12. execute the script

  13. verify operating status mid-script

  14. show datafiles being created

  15. verify end-of-script

End-to-End Testing

Each step of the test will be broken into sub-sections. Use the Python script and recommended command line arguments from 00 - Test Setup to communicate with the instrument. Type each command from the test and verify the output.

00 - Test Setup

This procedure will use the Python script to manage communication with the API server and automatically retrieve ControlResponses.

Follow the readme from the script repository. After you have the script functioning, you can check out command line arguments using python apitool.py --help.

We recommend using the tool with the following options:

  • --user <username>

  • --hostname api

  • --instrument <serialnumber>

  • --quiet

Use the following full command to start the tool. It waits for new commands, and you need to type exit to stop the program.

python apitool.py --username <username> --hostname api --instrument <serialno>

Important

Remember, for each command, you will get a new Control UUID nearly instantly, but it could take roughly 300 seconds for the ControlRepsonse.

If after 300 seconds, a command still does not response, consider checking instrument connectivity.

Note

If 300 seconds is excessive for on-site testing, see Remote Command: setinterval but realize this can lead to excessive cost while using metered Intern (ie. cell modems).

01 - Echo Check

Before any real tests, we need to make sure the communication modules is on-line and responsive. This command is useful for any new site setup or when connectivity needs to be verified.

We will use the following command.

echo E2E Check

In the control response, verify:

  • no error message

  • a response containing “echo E2E Check\r\n”

02 - Set Date and Time

Note

For end-to-end testing, we will use the settime COM100 command, but consider carefully, how you want to handle date and time in a deployment. This command sets the date and time based on the API server’s data packet. The server uses UTC.

Send the following command to update the date and time to UTC.

settime

In the control response, verify:

  • no error message

  • a response containing “time set to: <DATETIME>\n\r\ndate set to: <DATETIME>\n\r\n” response

Important

As of COM100 v0.2.0, there is a known bug in settime for the month. The following test uses sendcommand setdate to verify and sendcommand setdate <yymmdd> to fix, if needed.

Please, avoid using this work-around near UTC midnight.

Skip this step if you are using COM100 v0.3.0 or newer.

# verify
sendcommand setdate

# if 'setdate' returns the month off by one...
# replace the YY, MM, DD to set
sendcommand setdate YY-MM-DD

In the control response, verify:

  • no error message

  • a response containing “Instrument:systemdate=YYYY-MM-DD” response

  • for v0.2.0, verify the month is correct.

03 - Instrument Echo Check

While the instrument and COM100 work in tandem, the following echo check is directed at the instrument. In many cases, you will want to simply echo check the instrument since if the instrument is operational remotely so is the COM100.

We will use the following command.

sendcommand echo E2E Instrument Check

In the control response, verify:

  • no error message

  • a response containing “Instrument:echo E2E Instrument Check\r\n”

04 - Stage Script

Now that the COM100 and instrument are on-line and working, we will stage a script files. For testing we will use the simple-eight file from Remote Command: downloadfile.

Stage the test script with the following command.

# download good script file
downloadfile https://www.airphoton.io/_downloads/d9f683385e77db0c392924ed251a01e2/simple-eight.aps /script.aps

In the control response, verify:

  • no error message

  • a response containing “downloading file succeeded\r\n”

05 - Clear Storage Space

Before transmitting the file to instrument memory, lets be sure to free up the filenames script.aps and error.aps for testing. The error.aps file will be a step later.

Run each command below, and inspect each ControlResponse:

sendcommand rmfile Bscript.aps
sendcommand rmfile Berrors.aps

In each control response, verify:

  • no error message

  • a response containing “Instrument:Done\r\n”

06 - Transfer Script

With space and filenames freed, transfer the script.asp file to instrument memory.

Note

This step requires the Backup memory card to be present in the instrument.

Run the command below, and inspect the ControlResponse:

transmitfile /script.aps script.aps

In the control response, verify:

  • no error message

  • a response containing “file transferred successfully\r\n”

Warning

If you get the response: “file transfer failed\r\n”, return to 05 - Clear Storage Space

07 - Stage Error Script

Now that a valid test script is in instrument memory, lets stage (and transfer) an example script with errors. This will only be used for testing the checkscript command.

Stage the error script with the following command.

# download an erroneous script file
downloadfile https://www.airphoton.io/_downloads/f8df051da9bab2cca3a830911a5c0c8d/errors.aps /errors.aps

In the control response, verify:

  • no error message

  • a response containing “downloading file succeeded\r\n”

08 - Transfer Error Script

Transfer the error script to instrument memory.

Note

This step requires the Backup memory card to be present in the instrument.

Run each command below, and inspect each ControlResponse:

transmitfile /errors.aps errors.aps

In each control response, verify:

  • no error message

  • a response containing “file transferred successfully\r\n”

Warning

If you get the response: “file transfer failed\r\n”, return to 05 - Clear Storage Space

09 - Syntax Check Script

Now that both files are on the instrument, we will perform syntax checks on each file. The script.aps file should have a successful check while the error.aps file should flag errors.

sendmlcommand EOC checkscript Bscript.aps EOC
sendmlcommand EOC checkscript Berrors.aps EOC

Verify the script.aps file has

  • no error message

  • a response containing “Instrument:Line 68 FINISHED SUCCESS\n\r\nOutcome:1\r\nENDCHECK\r\n”

Verify the error.aps file has

  • no error message

  • a response containing “Instrument:Line 7 UNKNOWN COMMAND\nLine 10 UNKNOWN COMMAND\nLine 10 UNKNOWN COMMAND\nLine 10 UNKNOWN COMMAND\nLine 10 UNKNOWN COMMAND\nLine 10 UNKNOWN COMMAND\nLine 10 UNKNOWN COMMAND\nLine 12 FINISHED FAIL\n\r\nOutcome:3\r\n\r\n”

10 - Set Active Script

We are nearly ready to operate the station. Use the following command to set the script.aps file as the active script:

sendcommand setscript script.aps

In the control response, verify:

  • no error message

  • a response containing “Instrument:scriptname=script.aps\r\n”

11 - Check Standby Status

Before running the script, verify operation status:

sendmlcommand ENDSTATUS status

In the control response, verify:

  • no error message

  • a response containing expected values

Note

For now you may want to check the status response for:

  • serialnumber

  • date / time

  • mode

  • storage devices (SD Cards)

  • scriptname

  • scriptstate

Warning

If mode is not mode=script, you will need to run the following command to enable script mode.

sendcommand setmode script

12 - Execute Script

We are now ready to run the script and operate the sampling station.

The following command will start the script.aps script. It will …

sendcommand startscript strict

This command is more complex. It has actively changed the operation state. Iniitally check ControlResponse, but know there is more to check.

Verify the ControlResponse has:

  • no error message

  • a response containing “*”Instrument:Started script, no errors\r\n*”

Additionally, at this point the station is operating. Use the Step 13 - Check Operating Status and Step 14 - Check Data Files to verify operation. If you are physicaly on-site with the instrument, you should hear the pump operate, hear periodic valve changes, and see the information on-screen update in real time.

13 - Check Operating Status

While the script is operating, execute the following status command, and verify the response:

sendmlcommand ENDSTATUS status

In the control response, verify:

  • no error message

  • a response containing expected values

Note

The system status has already been checked, but now that a script is operating, the following fields should be check for logical values:

  • scriptname

  • scripterror

  • scriptline

  • activefilter

  • dataname

  • pump_a or pump_b

  • flow_a and flow_b

  • vacuum_a and vacuum_b

Note

While the script is running, run this status command multiple times to verify changes.

14 - Check Data Files

During or after script operation, check for new data files. Since data files are added to a sub-directory using the instrument’s serialnumber, the listfiles response can get cluttered.

Run the following command:

sendmlcommand EOC listfiles B EOC

15 - Check Status

After all sampling is completed. Reverify standby status:

sendmlcommand ENDSTATUS status

In the control response, verify:

  • no error message

  • a response containing expected values