Replication Package for: Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud (Q6711263)

Replication Package for: Benchmarking scalability of stream processing frameworks deployed asmicroservices in the cloudThis is our replication package for our study on Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud.All scalability experiments are performed with the scalability benchmarking framework Theodoliteat Kiel University's Software Performance Engineering Lab (SPEL) or Google Cloud.With this replication package, we provide:Benchmark execution filesin executions,our benchmark (raw) results in results, andanalysis script for our results in analysis.Repeating Benchmark ExecutionsAll our Theodolite executions are tailored to either the SPEL cluster or the Google Cloud.Kiel University's Software Performance Engineering Lab (SPEL)The SPEL clusterconsists of 5 nodes, named kube1-1to kube1-5and labeled with env=dev. To run them in your local cluster, make sure to provide the same infrastructure or rename node selectors in the execution files accordingly.To install Theodolite, run:helm install theodolite theodolite/theodolite --version 0.8.6 -f https://raw.githubusercontent.com/cau-se/theodolite/main/helm/preconfigs/extended-metrics.yaml -f se-cluster-dev.yamlor for the vertical scalability experiment:helm install theodolite theodolite/theodolite --version 0.8.6 -f https://raw.githubusercontent.com/cau-se/theodolite/main/helm/preconfigs/extended-metrics.yaml -f se-cluster-dev.yaml -f se-cluster-dev-vertical.yamlSee Theodolite's documentationfor further usage instructions.Google CloudIn the public cloud baseline experiments, the cluster consists of 5 e2-standard-32 nodes.To install Theodolite, run:helm install theodolite theodolite/theodolite --version 0.8.6 -f https://raw.githubusercontent.com/cau-se/theodolite/main/helm/preconfigs/extended-metrics.yaml -f gcp-cluster-dev.yamlFor the experiments testing higher load intensities, the cluster consists of 4 e2-standard-16 nodes labeled with type=infraand 4 or 8 e2-standard-16 nodes with label type=sut. To install Theodolite in this cluster, run:helm install theodolite theodolite/theodolite --version 0.8.6 -f https://raw.githubusercontent.com/cau-se/theodolite/main/helm/preconfigs/extended-metrics.yaml -f gcp-cluster-stress.yamlIn both cases, change the maximum load generated per load generator instance:# Generate max. 100000 rec/sec per load generator instance export MAX_RECORDS_PER_INSTANCE=100000 kubectl patch benchmarks uc1-beam-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc1-beam-samza --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc1-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc1-hazelcastjet --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc1-kstreams --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc2-beam-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc2-beam-samza --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc2-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc2-hazelcastjet --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc2-kstreams --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc3-beam-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc3-beam-samza --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc3-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc3-hazelcastjet --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc3-kstreams --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc4-beam-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc4-beam-samza --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc4-flink --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc4-hazelcastjet --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]" kubectl patch benchmarks uc4-kstreams --type json --patch "[{op: replace, path: /spec/loadTypes/0/patchers/1/properties/loadGenMaxRecords, value: $MAX_RECORDS_PER_INSTANCE}]"See Theodolite's documentationfor further usage instructions.Repeating Results AnalysisTo inspect, repeat, or extend our results analysis, see resultsor run the corresponding notebooks in analysis.For analyzing and visualizing benchmark results, either Docker or a Jupyter installation with Python 3.7 or 3.8 is required (e.g., in a virtual environment). Moreover, we require some Python libraries, which can be installed by:python3.8 -m venv .venv # source .venv/bin/activate pip install -r analysis/requirements.txt