:orphan: Instrument Testing for Remote Deployment ======================================== The following testing procedure aims to setup and verify the *end-to-end* test for a new instrument deployment. You will need access to the equipment, knowledge of the :ref:`Station Commands`, and have already completed the :ref:`Instrument Provisioning for Remote Access` guide. This procedure will use the `Python script`_ provided by AirPhoton to quickly send commands to the API server. More information on this tool below. Before we start testing, review the overview section below for a common setup, connectivity explanation, and walk-through of the testing process. .. _`Python Script`: https://gitlab.com/airphoton/ap-io-api-scripts End-to-End Overview ------------------- The end-to-end testing aims to connect all hardware as it will be deployed, communicate using the entire technology stack, and test remote control even if the unit is sitting in the lab next to you or in the field. There are multiple technical steps in this process, but you need to be aware of the following key points in the test. 1. End-user Interface to API Server 2. Optional, Cellular Modem 3. Remote Communication Module (COM100) to API Server via Internet (WiFi) 4. Remote Instrument commands via COM100 relay. End-user Interface for API Server ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ See :ref:`API Server` for sending commands to the server, or use the provided Python scripts to help manage the commands. This interface is you using a computer to only queue up or access information on the server. On a modern Internet connection, these commands from your computer to the server a nearly instant, but once on the server, there is no guarantee on the time to reach the instrument. Your commands to the server take two forms. You are either asking for a *ControlResponse* (information) or submitting a *Control* (command to instrument). When sending *Controls*, they will get queued until the instrument next connects to the server. This delay is usually very long (5+ minutes), but can be short for in-lab testing. Just know, that you can send many *Control* commands, but the instrument may never look, if it is not connected correctly. Optional, Cellular Modem ~~~~~~~~~~~~~~~~~~~~~~~~ If your equipment was purchased with or you have a cellular modem, you should test it in this end-to-end test. It is not a requirement, but is highly recommended. The modem is simply an Internet connection. Turn it on, and it is a WiFi router to the Internet. The instruments and COM100 *do not* know about the cellular modems as they simple see the WiFi and utilize the connection. As long as the COM100 can communicate on the Internet, it and the instrument will connect to the API Server. .. WARNING :: For an end-to-end test, AirPhoton recommends testing every step of the technology stack. This could include your cellular modem. If you choose to test everything *excluding* the modem, please keep WiFi credentials organized and be prepared for connectivity issues due to mis-configuration during deployment. Finally, just because a cellular modem works in one location (ie. with North American cellular providers), it is very possible your deployment site could have connectivity or provider issues. Please plan ahead. As of 2020, AirPhoton provided modems use Hologram.io_. .. _Hologram.io: https://www.hologram.io/ Remote Communication Module ~~~~~~~~~~~~~~~~~~~~~~~~~~~ The COM100 equipment simply creates a secure connection to the AirPhoton API server and relays messages (*Controls* and *ControlResposnes*) to and from the instruments connected. As of COM100 firmware v0.2.0, the COM100 persists very few settings in memory. The Internet credentials and server credentials are stored in the instrument. The COM100 knows its check-in interval and stores in-the-air data until reboot or erasure. This property make COM100 interchangeable. .. WARNING :: As of COM100 v0.2.0, only one instrument can be connected at a time. Remote Instrument Commands ~~~~~~~~~~~~~~~~~~~~~~~~~~ When a COM100 checks in with the API server, it processes the *Controls* queue for the locally connected instrument. This could be zero, one, or many commands at one check-in. Learning how *Controls* and :ref:`Station Commands` work may take time, but the following end-to-end testing is setup to walk through a typical process. We will use the Python script to manage one command at a time. While this may be slow and frustrating, it allows for a simple process for testing. .. NOTE :: With use, you will understand how each *station command* behaves. Some are stateless where they are executed and the COM100 and instrument never consider the command again while other commands (ie. *executescript*) change the instrument state. .. WARNING :: If you queue up multiple stateful commands, you will likely not get the results expected. Planning -------- The end-to-end test aims at tested key deployment commands. Before continuing make sure you have the following ready. 1. Make sure you have your API credentials and access to a computer with Python installed. 2. Verify you can use the Python script provided by AirPhoton 3. Make sure you have completed :ref:`Instrument Provisioning for Remote Access` 4. The first test will repeat the ``sendcommand echo`` example from the provisioning guide The following test aim to verify the following: 1. echo test to confirm COM100 connectivity 2. sync date and time to UTC 3. echo test to confirm instrument connectivity 4. stage an example station script on the COM100 5. clear space on instrument storage 6. transfer the script to the instrument 7. stage an example erroneous script on the COM100 8. transfer the erroneous script to the instrument 9. syntax check both script files 10. set the active script file 11. check standby status 12. execute the script 13. verify operating status mid-script 14. show datafiles being created 15. verify end-of-script End-to-End Testing ------------------ Each step of the test will be broken into sub-sections. Use the Python script and recommended command line arguments from :ref:`Test Setup` to communicate with the instrument. Type each command from the test and verify the output. .. _`test setup`: 00 - Test Setup ~~~~~~~~~~~~~~~ This procedure will use the `Python script`_ to manage communication with the API server and automatically retrieve *ControlResponses*. Follow the ``readme`` from the script repository. After you have the script functioning, you can check out command line arguments using ``python apitool.py --help``. We recommend using the tool with the following options: * ``--user `` * ``--hostname api`` * ``--instrument `` * ``--quiet`` Use the following full command to start the tool. It waits for new commands, and you need to type ``exit`` to stop the program. .. code-block:: bash python apitool.py --username --hostname api --instrument .. IMPORTANT:: Remember, for each command, you will get a new *Control UUID* nearly instantly, but it could take roughly 300 seconds for the *ControlRepsonse*. If after 300 seconds, a command still does not response, consider checking instrument connectivity. .. NOTE:: If 300 seconds is excessive for on-site testing, see :ref:`setinterval` but realize this can lead to excessive cost while using metered Intern (ie. cell modems). .. _`Python Script`: https://gitlab.com/airphoton/ap-io-api-scripts 01 - Echo Check ~~~~~~~~~~~~~~~ Before any real tests, we need to make sure the communication modules is on-line and responsive. This command is useful for any new site setup or when connectivity needs to be verified. We will use the following command. .. code-block:: bash echo E2E Check In the control response, verify: * no *error* message * a *response* containing "*echo E2E Check\\r\\n*" 02 - Set Date and Time ~~~~~~~~~~~~~~~~~~~~~~ .. NOTE :: For end-to-end testing, we will use the ``settime`` COM100 command, but consider carefully, how you want to handle *date* and *time* in a deployment. This command sets the date and time based on the API server's data packet. The server uses UTC. Send the following command to update the date and time to UTC. .. code-block:: bash settime In the control response, verify: * no *error* message * a *response* containing "*time set to: \\n\\r\\ndate set to: \\n\\r\\n*" response .. IMPORTANT :: As of COM100 v0.2.0, there is a known bug in ``settime`` for the *month*. The following test uses ``sendcommand setdate`` to verify and ``sendcommand setdate `` to fix, if needed. Please, avoid using this work-around near UTC midnight. **Skip this step if you are using COM100 v0.3.0 or newer.** .. code-block:: bash # verify sendcommand setdate # if 'setdate' returns the month off by one... # replace the YY, MM, DD to set sendcommand setdate YY-MM-DD In the control response, verify: * no *error* message * a *response* containing "*Instrument:systemdate=YYYY-MM-DD*" response * for v0.2.0, verify the month is correct. 03 - Instrument Echo Check ~~~~~~~~~~~~~~~~~~~~~~~~~~ While the instrument and COM100 work in tandem, the following echo check is directed at the instrument. In many cases, you will want to simply echo check the instrument since if the instrument is operational remotely so is the COM100. We will use the following command. .. code-block:: bash sendcommand echo E2E Instrument Check In the control response, verify: * no *error* message * a *response* containing "*Instrument:echo E2E Instrument Check\\r\\n*" 04 - Stage Script ~~~~~~~~~~~~~~~~~ Now that the COM100 and instrument are on-line and working, we will stage a script files. For testing we will use the :download:`simple-eight <./media/simple-eight.aps>` file from :ref:`downloadfile`. Stage the test script with the following command. .. code-block:: bash # download good script file downloadfile https://www.airphoton.io/_downloads/d9f683385e77db0c392924ed251a01e2/simple-eight.aps /script.aps In the control response, verify: * no *error* message * a *response* containing "*downloading file succeeded\\r\\n*" .. _`clearstoragespace`: 05 - Clear Storage Space ~~~~~~~~~~~~~~~~~~~~~~~~ Before transmitting the file to instrument memory, lets be sure to free up the filenames `script.aps` and `error.aps` for testing. The *error.aps* file will be a step later. Run each command below, and inspect each *ControlResponse*: .. code-block:: bash sendcommand rmfile Bscript.aps sendcommand rmfile Berrors.aps In each control response, verify: * no *error* message * a *response* containing "*Instrument:Done\\r\\n*" 06 - Transfer Script ~~~~~~~~~~~~~~~~~~~~ With space and filenames freed, transfer the *script.asp* file to instrument memory. .. NOTE :: This step requires the *Backup* memory card to be present in the instrument. Run the command below, and inspect the *ControlResponse*: .. code-block:: bash transmitfile /script.aps script.aps In the control response, verify: * no *error* message * a *response* containing "*file transferred successfully\\r\\n*" .. WARNING :: If you get the *response*: "*file transfer failed\\r\\n*", return to :ref:`clearstoragespace` 07 - Stage Error Script ~~~~~~~~~~~~~~~~~~~~~~~ Now that a valid test script is in instrument memory, lets stage (and transfer) an example script :download:`with errors <./media/errors.aps>`. This will only be used for testing the `checkscript` command. Stage the error script with the following command. .. code-block:: bash # download an erroneous script file downloadfile https://www.airphoton.io/_downloads/f8df051da9bab2cca3a830911a5c0c8d/errors.aps /errors.aps In the control response, verify: * no *error* message * a *response* containing "*downloading file succeeded\\r\\n*" 08 - Transfer Error Script ~~~~~~~~~~~~~~~~~~~~~~~~~~ Transfer the error script to instrument memory. .. NOTE :: This step requires the *Backup* memory card to be present in the instrument. Run each command below, and inspect each *ControlResponse*: .. code-block:: bash transmitfile /errors.aps errors.aps In each control response, verify: * no *error* message * a *response* containing "*file transferred successfully\\r\\n*" .. WARNING :: If you get the *response*: "*file transfer failed\\r\\n*", return to :ref:`clearstoragespace` 09 - Syntax Check Script ~~~~~~~~~~~~~~~~~~~~~~~~ Now that both files are on the instrument, we will perform syntax checks on each file. The `script.aps` file should have a successful check while the `error.aps` file should flag errors. .. code-block:: bash sendmlcommand EOC checkscript Bscript.aps EOC sendmlcommand EOC checkscript Berrors.aps EOC Verify the `script.aps` file has * no *error* message * a *response* containing "*Instrument:Line 68 FINISHED SUCCESS\\n\\r\\nOutcome:1\\r\\nENDCHECK\\r\\n*" Verify the `error.aps` file has * no *error* message * a *response* containing "*Instrument:Line 7 UNKNOWN COMMAND\\nLine 10 UNKNOWN COMMAND\\nLine 10 UNKNOWN COMMAND\\nLine 10 UNKNOWN COMMAND\\nLine 10 UNKNOWN COMMAND\\nLine 10 UNKNOWN COMMAND\\nLine 10 UNKNOWN COMMAND\\nLine 12 FINISHED FAIL\\n\\r\\nOutcome:3\\r\\n\\r\\n*" 10 - Set Active Script ~~~~~~~~~~~~~~~~~~~~~~ We are nearly ready to operate the station. Use the following command to set the *script.aps* file as the active script: .. code-block:: bash sendcommand setscript script.aps In the control response, verify: * no *error* message * a *response* containing "*Instrument:scriptname=script.aps\\r\\n*" 11 - Check Standby Status ~~~~~~~~~~~~~~~~~~~~~~~~~ Before running the script, verify operation status: .. code-block:: bash sendmlcommand ENDSTATUS status In the control response, verify: * no *error* message * a *response* containing expected values .. NOTE:: For now you may want to check the status response for: * serialnumber * date / time * mode * storage devices (SD Cards) * scriptname * scriptstate .. WARNING:: If `mode` is not *mode=script*, you will need to run the following command to enable script mode. ``sendcommand setmode script`` 12 - Execute Script ~~~~~~~~~~~~~~~~~~~ We are now ready to run the script and operate the sampling station. The following command will start the *script.aps* script. It will ... .. code-block:: bash sendcommand startscript strict This command is more complex. It has actively changed the operation state. Iniitally check *ControlResponse*, but know there is more to check. Verify the *ControlResponse* has: * no *error* message * a *response* containing "*"Instrument:Started script, no errors\\r\\n*" Additionally, at this point the station is operating. Use the Step 13 - Check Operating Status and Step 14 - Check Data Files to verify operation. If you are physicaly on-site with the instrument, you should hear the pump operate, hear periodic valve changes, and see the information on-screen update in real time. 13 - Check Operating Status ~~~~~~~~~~~~~~~~~~~~~~~~~~~ While the script is operating, execute the following status command, and verify the response: .. code-block:: bash sendmlcommand ENDSTATUS status In the control response, verify: * no *error* message * a *response* containing expected values .. NOTE:: The system status has already been checked, but now that a script is operating, the following fields should be check for logical values: * scriptname * scripterror * scriptline * activefilter * dataname * pump_a or pump_b * flow_a and flow_b * vacuum_a and vacuum_b .. NOTE:: While the script is running, run this `status` command multiple times to verify changes. 14 - Check Data Files ~~~~~~~~~~~~~~~~~~~~~ During or after script operation, check for new data files. Since data files are added to a sub-directory using the instrument's serialnumber, the `listfiles` response can get cluttered. Run the following command: .. code-block:: bash sendmlcommand EOC listfiles B EOC 15 - Check Status ~~~~~~~~~~~~~~~~~ After all sampling is completed. Reverify standby status: .. code-block:: bash sendmlcommand ENDSTATUS status In the control response, verify: * no *error* message * a *response* containing expected values