mark issueshttps://gitlab.cylab.be/cylab/mark/-/issues2021-10-04T13:05:46Zhttps://gitlab.cylab.be/cylab/mark/-/issues/101check mongodb indeces2021-10-04T13:05:46ZTibocheck mongodb indeces![image](/uploads/f914c33ca525138aa3f7aba7071166b6/image.png)![image](/uploads/f914c33ca525138aa3f7aba7071166b6/image.png)https://gitlab.cylab.be/cylab/mark/-/issues/100Handle detector exceptions nicely2021-09-15T19:49:40ZTiboHandle detector exceptions nicelyCurrently the whole stack is printed. For example with WOWA...
![mark-bug-detector](/uploads/697bc76183791e23f77af63fb4b8c373/mark-bug-detector.png)Currently the whole stack is printed. For example with WOWA...
![mark-bug-detector](/uploads/697bc76183791e23f77af63fb4b8c373/mark-bug-detector.png)https://gitlab.cylab.be/cylab/mark/-/issues/99Error parsing config.yml file when "data_directory" parameter is present2021-09-13T11:22:13ZGeorgiError parsing config.yml file when "data_directory" parameter is presentWhen starting Mark with a config.yml file, where the "data_directory" parameter has been set (while testing it was set to "./data"), an error in the yaml parsing occured as shown in the screenshot attached.
![error_masfad_config_file](...When starting Mark with a config.yml file, where the "data_directory" parameter has been set (while testing it was set to "./data"), an error in the yaml parsing occured as shown in the screenshot attached.
![error_masfad_config_file](/uploads/250eece8492088d28181870ff27fff35/error_masfad_config_file.png)https://gitlab.cylab.be/cylab/mark/-/issues/98Incorrect figures added to Frequency detector agent report2021-09-13T08:40:11ZGeorgiIncorrect figures added to Frequency detector agent report![frequency_agent_output_error](/uploads/64a0e5ac200b70859aacf9b141349189/frequency_agent_output_error.png)![frequency_agent_output_error](/uploads/64a0e5ac200b70859aacf9b141349189/frequency_agent_output_error.png)https://gitlab.cylab.be/cylab/mark/-/issues/97Review code Wowaggregation2021-09-01T08:32:28Za.croixReview code WowaggregationHave to review code. Should be optimized and may be correct an error.
https://gitlab.cylab.be/cylab/mark/-/blob/master/server/src/main/java/be/cylab/mark/detection/WOWAgregation.java#L71
```java
if (checkDefaultParameters(w_weigh...Have to review code. Should be optimized and may be correct an error.
https://gitlab.cylab.be/cylab/mark/-/blob/master/server/src/main/java/be/cylab/mark/detection/WOWAgregation.java#L71
```java
if (checkDefaultParameters(w_weights_string)) {
w_weights = generateDefaultWVector();
//w_weights = normalizeVector(w_weights);
} else {
w_weights = parseDoubleArray(w_weights_string);
//w_weights = normalizeVector(w_weights);
}
if (checkDefaultParameters(p_weights_string)) {
p_weights = generateDefaultPVector();
//p_weights = normalizeVector(p_weights);
} else {
p_weights = parseDoubleArray(p_weights_string);
//p_weights = normalizeVector(p_weights);
}
```
https://gitlab.cylab.be/cylab/mark/-/blob/master/server/src/main/java/be/cylab/mark/detection/WOWAgregation.java#L112
```java
this.initParams(profile);
w_weights = normalizeVector(w_weights);
p_weights = normalizeVector(p_weights);
WOWA aggregator = new WOWA(w_weights, p_weights);
```a.croixa.croixhttps://gitlab.cylab.be/cylab/mark/-/issues/96Warnings generated when no valid data directory found by ActivationController2021-08-31T09:09:58ZGeorgiWarnings generated when no valid data directory found by ActivationController- too many warnings generated when launching detectors, when no valid data directory is present to save generated figures to.
![mark_valid_data_directory](/uploads/1f7feec28e6ef5e67ea215f23a982f1b/mark_valid_data_directory.png)
- possi...- too many warnings generated when launching detectors, when no valid data directory is present to save generated figures to.
![mark_valid_data_directory](/uploads/1f7feec28e6ef5e67ea215f23a982f1b/mark_valid_data_directory.png)
- possible solutions:
- add one time warning at the start to signal no valid data directory
- if no data directory present, generate onehttps://gitlab.cylab.be/cylab/mark/-/issues/95In FrequencyAgent (and possibly others), use StringTemplate to produce the re...2021-08-30T14:45:38ZTiboIn FrequencyAgent (and possibly others), use StringTemplate to produce the reporthttps://www.stringtemplate.org/index.htmlhttps://www.stringtemplate.org/index.htmlhttps://gitlab.cylab.be/cylab/mark/-/issues/94Add shared PersistentVolume to kubernetes config, so we can support agents th...2021-08-28T15:58:11ZTiboAdd shared PersistentVolume to kubernetes config, so we can support agents that produce images etc.https://gitlab.cylab.be/cylab/mark/-/issues/93What if only several wowa input have value?2021-08-26T13:18:43Za.croixWhat if only several wowa input have value?It could be possible several wowa detectors are present in a single project. We have to implement a system to select which wowa detector will be used for the training and to switch between these detector.It could be possible several wowa detectors are present in a single project. We have to implement a system to select which wowa detector will be used for the training and to switch between these detector.a.croixa.croixhttps://gitlab.cylab.be/cylab/mark/-/issues/92FileSource should support string timestamps2021-07-15T11:35:52ZTiboFileSource should support string timestamps- add optional parameter time_format
- use Java SimpleDateFormat https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html to parse the line- add optional parameter time_format
- use Java SimpleDateFormat https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html to parse the linehttps://gitlab.cylab.be/cylab/mark/-/issues/91FileSource should support milliseconds timestamp2021-07-15T11:34:13ZTiboFileSource should support milliseconds timestampCurrent version assumes that timestamps in the input file are expressed in seconds (like in the example https://cylab.be/blog/115/mark-use-built-in-file-data-source)Current version assumes that timestamps in the input file are expressed in seconds (like in the example https://cylab.be/blog/115/mark-use-built-in-file-data-source)https://gitlab.cylab.be/cylab/mark/-/issues/90create and publish a default docker-compose file to easily run the latest ver...2021-07-15T08:49:03ZTibocreate and publish a default docker-compose file to easily run the latest version- use a template
- publish to some artifacts repository (download.cylab.be ?)- use a template
- publish to some artifacts repository (download.cylab.be ?)https://gitlab.cylab.be/cylab/mark/-/issues/89Update java-aggregation version2020-11-27T14:26:36Za.croixUpdate java-aggregation versionUpdate java aggregation library version in pom.xml and update potential errors in testsUpdate java aggregation library version in pom.xml and update potential errors in testsa.croixa.croixhttps://gitlab.cylab.be/cylab/mark/-/issues/87Implement an Elastic Search sink2020-11-22T08:32:00ZTiboImplement an Elastic Search sinkdetection agent that copies evidences to elastic search ...
https://www.baeldung.com/elasticsearch-javadetection agent that copies evidences to elastic search ...
https://www.baeldung.com/elasticsearch-javahttps://gitlab.cylab.be/cylab/mark/-/issues/86Implement SocketListener data agent2020-11-22T08:31:22ZTiboImplement SocketListener data agentData agent that opens a socket, listen for incoming connection, and uses a named group regex to extract the subject.Data agent that opens a socket, listen for incoming connection, and uses a named group regex to extract the subject.https://gitlab.cylab.be/cylab/mark/-/issues/85findEvidenceSince must have additional parameter till (like findData)2020-11-19T14:12:45ZTibofindEvidenceSince must have additional parameter till (like findData)https://gitlab.cylab.be/cylab/mark/-/blob/master/core/src/main/java/be/cylab/mark/core/ServerInterface.java#L98https://gitlab.cylab.be/cylab/mark/-/blob/master/core/src/main/java/be/cylab/mark/core/ServerInterface.java#L98https://gitlab.cylab.be/cylab/mark/-/issues/84rename findLastEvidences2020-11-19T14:11:28ZTiborename findLastEvidenceshttps://gitlab.cylab.be/cylab/mark/-/blob/master/core/src/main/java/be/cylab/mark/core/ServerInterface.java#L154
should be renamedo to findLastEvidencesByPattern() to avoid confusion !https://gitlab.cylab.be/cylab/mark/-/blob/master/core/src/main/java/be/cylab/mark/core/ServerInterface.java#L154
should be renamedo to findLastEvidencesByPattern() to avoid confusion !https://gitlab.cylab.be/cylab/mark/-/issues/79Timestamps are milliseconds !2020-10-28T10:21:23ZTiboTimestamps are milliseconds !* Clearly states in the documetation of RawData and Evidence that timestamps are milliseconds
* check detectors and fix accordingly
* update documentation of datastore client accordingly* Clearly states in the documetation of RawData and Evidence that timestamps are milliseconds
* check detectors and fix accordingly
* update documentation of datastore client accordinglyhttps://gitlab.cylab.be/cylab/mark/-/issues/78Optimize trigger_label matching2020-10-28T09:49:35ZTiboOptimize trigger_label matchingCurrently, the ```trigger_label``` (which is a regex) is compiled each time we test if a detector should be triggered:
https://gitlab.cylab.be/cylab/mark/-/blob/master/server/src/main/java/be/cylab/mark/activation/ActivationController.j...Currently, the ```trigger_label``` (which is a regex) is compiled each time we test if a detector should be triggered:
https://gitlab.cylab.be/cylab/mark/-/blob/master/server/src/main/java/be/cylab/mark/activation/ActivationController.java#L207
This is actually quite heavy => the regexes should be compiled only once!https://gitlab.cylab.be/cylab/mark/-/issues/76Implement count detector2020-10-28T09:02:25ZTiboImplement count detectorimplement a count detector that:
* counts the number of data records in the ```training_window```
* counts the number of data records in the ```test_window```
* trigger an alert if ```count_test >> count_training```
Allows to detect a ...implement a count detector that:
* counts the number of data records in the ```training_window```
* counts the number of data records in the ```test_window```
* trigger an alert if ```count_test >> count_training```
Allows to detect a sudden volume increase.