Step 3 - Parametrizing notebooks

At this point, we are able to perform our first complete study in our application scenario.

We aim to verify the impact of different \(\lambda\) on the response time of our system. For that, we vary the sleep interval of our load-injector and fix all other parameters (duration, M3 firmware, number of nodes, etc).

The experiment is divided in 3 phases:

  • Description: describe the experiments that will be done and the order of their execution

  • Running: run the experiments using EnOSlib and G5K/FIT/IoT-LAB, collect and save results

  • Analysis: perform the data analysis of the results

Objective

  • Introduction to experiments workflow

  • Parametrize notebooks using papermill

  • Run 1 experiment

  • Handle variablity

  • Analyze results

Notebooks

  • analysis/04-load-injection/04-load-injection.ipynb: base notebook for single experiment

  • analysis/04-load-injection/2021-09-07-load-exp2: example

  • analysis/04-load-injection/04-2-load-injection_Ntimes.ipynb: base notebook for multiple executions

  • analysis/04-load-injection/2021-09-07-load-3x-exp1/: example

What to do?

  1. Open notebook: analysis/04-load-injection/04-load-injection.ipynb

  2. Inspect the code

  1. Identify the 3 phases. How are they interconnected?

  2. How and which the parameters are passed to the 03-g5k-iotlab notebook?

  1. Run the experiment (see Running 1 experiment)

  1. Where are the output artefacts saved? How the experiments are identified?

  2. Give a look to the different files generated at each phase (CSV files, notebooks)

  3. Can you stop the experiment and restart it later? If so, how does it work?

  4. Did you have some problem executing it? If so, analyze the logs and try again

The execution takes some time (~ 30m), meanwhile you can open our example and analyze it.

  1. See files in folder: analysis/04-load-injection/2021-09-07-load-exp2/

  2. You can redo the analysis part:

  • Run the first cell to adjust the current path

  • Run and modify (if you want) the cells below the Analysis section

  1. Analysis

    1. Are the curves as expected? How is the system load?

    2. Notice the difference between \(\lambda\) expected and observed. Can you explain why it happens?

    3. What do you think can impact the results (network, sensors, firmware, location, etc)?

    4. Any point would you like to dig in?

    5. Compare your analysis with that in the example (analysis/04-load-injection/2021-09-07-load-exp2/)

  2. Variability study

    1. See section below Running several experiments in a row

    2. Open notebook: analysis/04-load-injection/04-2-load-injection_Ntimes.ipynb

    3. Inspect and run it

    1. How many times each test is executed? Are they randomized?

    2. What about the variability? Is it stable or it depends on the sleep interval?

    3. What would you like to study further? Increase the load? Test different sleep intervals? More executions? Different sensors/locations? More sensors?

    4. Compare your analysis with that in the provided example (analysis/04-load-injection/2021-09-07-load-3x-exp1/)

    1. In the same way, you can open and analyze our example in: analysis/04-load-injection/2021-09-07-load-3x-exp1/

Running 1 experiment

The notebook in analysis/04-load-injection/04-load-injection.ipynb is able to run the 03-g5k-iotlab.ipynb notebook N times, changing their parameters and comparing different load injections in the system.

During your research, you may want to execute this notebook many times. For that, we suggest you to do the following:

  1. Create a directory under analysis/04-load-injection to save the new experiment. Create a folder hierarchy that you are comfortable with: e.g. 2021-10-03-load-exp1/.

mkdir analysys/04-load-injection/2021-10-03-load-exp1/

  1. Copy the notebook to the recently created directory

cp analysys/04-load-injection/04-load-injection.ipynb analysys/04-load-injection/2021-10-03-exp1/

  1. Adjust (if necessary) the notebook so it starts executing from the root directory (cd command at the beginning of the notebook)

Running several experiments in a row

In 04-load-injection.ipynb, we run each configuration (each line in the csv description) only once. In the real world however, you would like to run each configuration several times to cope with the possible inter-run variability (even in this extremely controlled environment).

The notebook analysis/04-load-injection/04-2-load-injection_Ntimes.ipynb shows how to run the same kind of study, repeating each experiment N times. It’s quite similar to the first one, the only difference is the collect and analysis which must consider the N executions.