Skip to content

Commit

Permalink
routing + new_pki part 2
Browse files Browse the repository at this point in the history
  • Loading branch information
j9plante committed Aug 19, 2022
1 parent 210bc4a commit 284c831
Show file tree
Hide file tree
Showing 62 changed files with 16,208 additions and 1,411 deletions.
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
!**/*.svg
!**/*.sh
!**/
!./test/*
!**/.vscode/*

!/.gitignore
!/go.mod
Expand Down
26 changes: 26 additions & 0 deletions .vscode/launch.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
// Utilisez IntelliSense pour en savoir plus sur les attributs possibles.
// Pointez pour afficher la description des attributs existants.
// Pour plus d'informations, visitez : https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Attach to Process",
"type": "go",
"request": "attach",
"mode": "local",
"processId": "${command:pickGoProcess}"
},
{
"name": "Launch Package",
"type": "go",
"preLaunchTask": "DeleteFolder",
"request": "launch",
"mode": "auto",
"program": "./test",
"args": [
"4",
]
}
]
}
84 changes: 84 additions & 0 deletions .vscode/tasks.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
{
// See https://go.microsoft.com/fwlink/?LinkId=733558
// for the documentation about the tasks.json format
"version": "2.0.0",
"tasks": [
{
"label": "MultiProcesses_prep",
"type": "shell",
"command": "./test/multiProcesses_prep.sh",
"problemMatcher": []
},
{
"label": "DeleteFolder",
"type": "shell",
"command": "rm -rf ~/.shoset",
"problemMatcher": []
},
{
"label": "RunMultiProcesses",
"runOptions": {},
"dependsOrder": "sequence",
"dependsOn": [
"MultiProcesses_prep",
"RunMultiProcesses_run"
],
"problemMatcher": []
},
{
"label": "RunMultiProcesses_run",
"runOptions": {},
"dependsOrder": "parallel",
"dependsOn": [
"RunMultiProcesses_E",
"RunMultiProcesses_D",
"RunMultiProcesses_C",
"RunMultiProcesses_B",
"RunMultiProcesses_A"
],
"problemMatcher": [
"$go"
]
},
{
"label": "RunMultiProcesses_A",
"type": "shell",
"command": "./test/multiProcesses_A.sh",
"presentation": {
"group": "test"
}
},
{
"label": "RunMultiProcesses_B",
"type": "shell",
"command": "./test/multiProcesses_B.sh",
"presentation": {
"group": "test"
}
},
{
"label": "RunMultiProcesses_C",
"type": "shell",
"command": "./test/multiProcesses_C.sh",
"presentation": {
"group": "test"
}
},
{
"label": "RunMultiProcesses_D",
"type": "shell",
"command": "./test/multiProcesses_D.sh",
"presentation": {
"group": "test"
}
},
{
"label": "RunMultiProcesses_E",
"type": "shell",
"command": "./test/multiProcesses_E.sh",
"presentation": {
"group": "test"
}
}
]
}
41 changes: 37 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,48 @@ A smart multi-ends socket library.

## Documentation

Missing. -- Find a usage example [in the gandalf project](https://github.com/ditrit/gandalf-core/blob/master/aggregator/aggregator.go).
See in `guide.md`. </br>
Find a usage example [in the gandalf project](https://github.com/ditrit/gandalf-core/blob/master/aggregator/aggregator.go).

## Design principles

For now, over to the [team's taiga](https://taiga.orness.com/project/xavier-namt/wiki/shoset)
For now, over to the [team's taiga](https://taiga.orness.com/project/xavier-namt/wiki/shoset).

## Running tests

You can run multiple tests in Shoset. There are three types of tests; unit test, script test or functional tests.

### Unit tests
To run unit tests, you need to go in your editor (I did it on VSCode) in the file with a name corresponding to `*_test.go`, then you can simply click on `run test` or `debug test` button. There is also a dedicated tab in the navigation bar with a flask as icon to run each test one after the other.

### Script tests (Not working)
To run script tests, you can find two scripts in the folder `shoset/script/` which are `cert_checker.sh` and `shoset_checker.sh`. These tests can be run in a Linux terminal as follows :

```txt
./script/cert_checker
./script/shoset_checker number_of_files_expected
```

#### cert_checker.sh
`cert_checker.sh` is a script done to check the validity of certificates generated for each Shoset after those have run a functional test. It will run a server in background and then run a client who will connect to this server by showing its certificates. If the certificates are valid, then you must see in the file `~/.shoset/cert_checker.txt` the word `ACCEPTED` for each Shoset.

#### shoset_checker.sh
`shoset_checker.sh` is a script done to launch multiple functional tests one after the other to check if there are no errors occurring when multiples test are run. You need to add the argument `number_of_files_expected` which is an integer representing the number of files you want to be obtained after each test. For example, if you have a network composed of 4 Shoset (1 PKI and 3 classic Shoset), then you will obtain 17 files (4 for the PKI, 3 for each classic Shoset and 1 config file for each Shoset - depending on your network complexity, there is not always a config file). To precisely know the number of file you need, you can simply run a functional test one time and see how many files you have by running this command :
```txt
git clone https://github.com/ditrit/shoset
go run test/test.go
tree ~/.shoset/
```
After the script is run, you can watch if errors occurred in the file `shoset/log_error.txt`, if this file is empty, then it means that you are good to go !

### Functional tests

You can run multiple functional tests in the file `shoset/test/test.go` with this command :
```shell
go run test/test.go arg
```
The arg argument corresponds to the test you want to run. Take a look at the test file and adapt the test if you want. Don't forget to remove existing certificates and config files in `~/.shoset/folder`.

You can set an alias to run tests easily : `alias shoset='rm -rf ~/.shoset && timeout 30s go run -race test/test.go 4 > log.txt 2>&1'`. It removes existing files, timeouts the test at 30 seconds (it shouldn't last more than 15 seconds except if the network is large), runs the test with -race argument to detect data races and print the output in a log.txt file that you can find in the `shoset/` folder.

If you need to kill a Shoset at running time for testing. Then I advise you to launch two terminals. In the first one you will run your program with 2 as arg which will run the simpleCluster() function that simply creates a PKI Shoset. In the other terminal, run your program with 4 as arg which will run the testJoin4() function that creates multiple classic Shosets. Then you can `CTRL+C` in one of the two terminal to kill the Shoset(s). Of course, you need to adapt the call to the function corresponding to your arg in the main() function of the test file.

If you want to create your own test, don't forget that the network needs that the first Shoset is initialized as a PKI with InitPki() function and the other with the Protocol() protocol - most of the function are deprecated because they do not use InitPKI() nor Protocol(), please don't use them.
106 changes: 106 additions & 0 deletions concurent_data/concurentSlice.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
package concurentData

import (
"errors"
"fmt"
"sync"
"time"

eventBus "github.com/ditrit/shoset/event_bus"
)

// This is a thread safe slice that allows you to wait for the next change or for it to be empty.
type ConcurentSlice struct {
sliceValues []string
eventBus eventBus.EventBus // topics : change, empty (value not used)
m sync.RWMutex
}

func NewConcurentSlice() ConcurentSlice {
return ConcurentSlice{eventBus: eventBus.NewEventBus()}
}

// Appends data to the slice and publishes the change event.
func (cSlice *ConcurentSlice) AppendToConcurentSlice(data string) {
cSlice.m.Lock()
defer cSlice.m.Unlock()

cSlice.sliceValues = append(cSlice.sliceValues, data)
cSlice.eventBus.Publish("change", true)
}

// Deletes data from the slice and publishes the change (and empty) event.
func (cSlice *ConcurentSlice) DeleteFromConcurentSlice(data string) {
cSlice.m.Lock()
defer cSlice.m.Unlock()

for i, a := range cSlice.sliceValues {
if a == data {
// Deletes data from the slice in an efficient way (avoids moving remaining data).
cSlice.sliceValues[i] = cSlice.sliceValues[len(cSlice.sliceValues)-1]
cSlice.sliceValues = cSlice.sliceValues[:len(cSlice.sliceValues)-1]

cSlice.eventBus.Publish("change", true)

if len(cSlice.sliceValues) == 0 {
cSlice.eventBus.Publish("empty", true)
}
return
}
}
}

// Waits for the Slice to be empty
func (cSlice *ConcurentSlice) WaitForEmpty(timeout int) error {
cSlice.m.RLock()
if len(cSlice.sliceValues) != 0 {
// Subscribes a channel to the empty topic :
chEmpty := make(chan interface{})
cSlice.eventBus.Subscribe("empty", chEmpty)
defer cSlice.eventBus.UnSubscribe("empty", chEmpty)

cSlice.m.RUnlock()
select {
case <-chEmpty:
return nil
case <-time.After(time.Duration(timeout) * time.Second):
return errors.New("the list is no empty (timeout)")
}
}
cSlice.m.RUnlock()
return nil
}

// Wait for the Slice to change
func (cSlice *ConcurentSlice) WaitForChange(timeout int) error {
cSlice.m.RLock()

// Subscribes a channel to the change topic :
chChange := make(chan interface{})
cSlice.eventBus.Subscribe("change", chChange)
defer cSlice.eventBus.UnSubscribe("change", chChange)

cSlice.m.RUnlock()
select {
case <-chChange:
return nil
case <-time.After(time.Duration(timeout) * time.Second):
return errors.New("the list did not change (timeout)")
}
}

func (cSlice *ConcurentSlice) Contains(data string) bool {
// contains range through a slice to search for a particular string
for _, v := range cSlice.sliceValues {
if v == data {
return true
}
}
return false
}

func (cSlice *ConcurentSlice) String() string {
cSlice.m.RLock()
defer cSlice.m.RUnlock()
return fmt.Sprint(cSlice.sliceValues)
}
84 changes: 84 additions & 0 deletions concurent_data/concurentSlice_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
package concurentData

import (
"fmt"
"sync"
"testing"
"time"
)

func Test_concurentSlice(t *testing.T) {
cSlice := NewConcurentSlice()

number := 10 //Changing this requires updating the strings the result is compared to.

go func() {
for {
cSlice.WaitForChange(5)
fmt.Println("Change received.")
}
}()

time.Sleep(10 * time.Millisecond)

for i := 0; i < number; i++ {
cSlice.AppendToConcurentSlice("Test " + fmt.Sprint(i))
}
if !cSlice.Contains("Test 0") {
t.Errorf("Wrong content after appending.")
}
if cSlice.String() != "[Test 0 Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 Test 8 Test 9]" {
t.Errorf("Wrong content after appending.")
}
fmt.Println(cSlice.String())

for i := 0; i < number-(number/2); i++ {
cSlice.DeleteFromConcurentSlice("Test " + fmt.Sprint(i))
}
if cSlice.String() != "[Test 9 Test 8 Test 7 Test 6 Test 5]" {
t.Errorf("Wrong content after Clearing.")
}
fmt.Println(cSlice.String())

var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
for i := 0; i < number-(number/2); i++ {
cSlice.AppendToConcurentSlice("Test " + fmt.Sprint(i))
}
}()
wg.Add(1)
go func() {
defer wg.Done()
for i := 5; i < number; i++ {
cSlice.DeleteFromConcurentSlice("Test " + fmt.Sprint(i))
}
}()

wg.Wait()

go func() {
for i := 0; i < number-(number/2); i++ {
cSlice.DeleteFromConcurentSlice("Test " + fmt.Sprint(i))
fmt.Println(cSlice.String())
time.Sleep(10 * time.Millisecond)
}
}()

cSlice.WaitForEmpty(5)

if cSlice.String() != "[]" {
t.Errorf("Wrong content after Clearing.")
}

fmt.Println(cSlice.String())

cSlice.AppendToConcurentSlice("TEST")

err := cSlice.WaitForEmpty(5)

if err == nil {
t.Errorf("Timeout error failed.")
}
}
Loading

1 comment on commit 284c831

@j9plante
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changelog :

  • For the rest of the changes, see branch new_pki.

  • Refacto of the test folder to have multiples files in multiple packages :

    • Topology system : Store data about the shoset’s configuration
    • utils_for_test : Functions to easily deploy a topology or a single shoset from a topology.
    • Example of use of the project.
    • Script (run_shoset.sh) to avoid using aliases to clear the .shoset folder before testing.
    • Test multiprocesses :
      • VScode task to launch many scripts in many dedicated terminals at once (need an IDE agnostic way of doing it).
  • Data race elimination :

    • Add missing Lock on IsEmpty() and way to lock it from the outside in queue.go.
    • Add RWMutex to Shoset and ShosetConn.
  • Acknowledge on join and link to know when the connection is ready. (Move storage of the connection after the acknowledge is received.

  • Add protocol field to shosetConn.

  • Change in the behaviour of the configuration storage :

    • Configuration folder for a shoset is in a folder named after the Lname (for readability).
    • Every connection established is stored in the config folder (not only out).
    • When a connection is established, any prior connection to the same Lname on the same IP address is deleted.
    • When an OEF is received, the connection is deleted.
    • Connections can switch from “in” to “out” if a shoset is relaunched.
    • Update script cert_checker.sh (not fully working).
  • mapsyncmap : Generic accessor for append, get and delete.

  • config : Generic accessor for append and delete.

  • Generic function for receiving (Wait) and sending (Send) a message of any type (for which the speceific fonctions are defined).

  • Event bus :

    • Event based Wait function for increased speed and efficiency (not benchmarked) (only for simpleMessage and event).
    • Used in concurentSlice, allows to wait for changes or the slice to be empty. (Used to know when every connection of the shoset is ready.)
  • Routing system :

    • See guide for general principle.
    • Logic to forward receive and send forwardable messages.
    • Message type routingEvent to advertise Lnames in the network.
    • Message type simpleMessage to test message forwarding.
    • Message type forwardAck to confirm the reception of the forwarded message.
  • const.go :

    Various list of message types to define which can do what. (See guide on how to create a new message type.)

  • Script to generate class diagram and call diagram. (see guide)

  • CPU/memory profiler and tracer.

Please sign in to comment.