### Trigger Logic and Regex Overlapping

In our last posts, we have talked about that we are interested in if the possible output is able to trigger the input condition.

The expected output and input condition are key-value pairs connected with logical operators (&&, ||, !). We define the output expression as A, and input as B, and we are wondering of their is an implication between them (A->B).

On a high perspective view, we have 4 possibilities between A and B

Complete Trigger. If A happens, B must happen (A->B)

Possible Trigger. If a set of certain cases of A happens (A can be A1||A2||A3…), B must happen in any environment (A-?>B)

Partial Trigger. If a set of certain cases of A happens, B can happen under a certain environment (A–>B), which means A is not enough to trigger B alone but it does contribute.

Irrelevant. A and B are completely irrelevant. (A-!>B)

To prove A->B, Automated Theorem Proving is applied. However, a simple deduction (A->B) can only help distinguish the complete trigger from the rest. To differentiate the possible trigger from the last two, we have to transform the output expression to a Disjunctive Normal Form (A1|A2|…|Ai), if any Ai -> B, then it satisfies the definition of a possible trigger. To tell apart a partial trigger from irrelevant needs even more computation, my method is to transfer the input condition into a Conjunctive Normal Form (B1&B2…&Bj), if any Ai->Bj, we know it is a partial trigger. Otherwise the output and input are irrelevant.

In order to create Axioms for theorem proving, we have to compare the values belongs to the same keys, the values can be either a constant string or a regular expression. And the challenge is to find out the overlaps between two regular expressions.

As a solution, we can transfer regular expressions to Deterministic Finite Automatons (DFAs) and derive their intersections.

A DFA is a five tuple (Q, Σ, ð, q0, F)

Q is a finite set called the state

Σ is a finite set called the alphabet

ð is the transition function

q0 is the start state and

F is the set of accept state

Here is a concrete example explains how it works.

regex 1 : [ch]at, regex 2: [^b]at

Obviously there are only two words matches both expression, they are hat and cat.

If we transfer the two regex to DFA, they will become

DFA 1 :

initial state: 3
state 0 [reject]:
t -> 1
state 1 [accept]:
state 2 [reject]:
a -> 0
state 3 [reject]:
c -> 2
h -> 2

DFA 2 :

initial state: 3
state 0 [accept]:
state 1 [reject]:
t -> 0
state 2 [reject]:
a -> 1
state 3 [reject]:
\u0000-a -> 2
c-\uffff -> 2

We can see that DFA 1 is enclosed by DFA 2. So the intersection is DFA 1.

### UI, Programming Model Design

UI

The latest choice to create an interactive UI is DOT Language + Grappa Applet + JSP + Jersey Framework. However, Grappa doesn’t support automatically layout a DOT file, the graph formatting relies on the external invoking of GraphViz tool, which could be a potential issue for our cloud server. In the worst case we have to revert back to a static/non-interactive UI. We keep it as simple as possible temporarily.

Graph design can be more specific, e.g. the node shape and line style should follow the workflow description model.

Programming Model Design (3 layers)

Layer 1:

Storlet Code, provides the basic functionality/template of the storlet, which is created by programmer. As a user, the hard coded part is a black box and immutable.

Layer 2:

Trigger Evaluator, which specifies the trigger condition in a special logical operation. This part defines both the input and output logic expression. It is not required for a end-user to understand the logic, but a user can always add extra conditions for strengthen the constraints.

Layer 3:

Variable, which expose the interfaces from layer 2. A common user just need to know what is this storlet used for, and simply specify those variables. Basically, the variables are inputkey, inputvalue, outputkey, outputvalue.

Somehow, there might be multiple outputkey and outputvalue for a single storlet, in that case, we assume the output can be any combination of these keys and values.

Next Step:

Create  well designed use cases (cover most cases), and plot them with my UI

Study workflow description model, and apply it to my UI.

### Visualization Tool, Graph Design & User Interaction

DOT Language Specification

DOT supports subgraphs clustering, all nodes in the same subgraph could have the same node and edge attributes. additionally they can be clustered in one group. These two features facilitates the representing of multiple handlers belongs to a single storlet. I will explain it in the following graph design part.

DOT language also support HTML like attribute, and flexible syntax.

Grappa

Since the original web service framework is in Java, I switched to Grappa, a java graph visualization framework, which support DOT language and user interaction.

Graph Design

Each node are created as a single handler of a storlet, it is good for cycle detection, because cycles are happened in handler level but not storlet. The annotations of a node may contain the information of storlet and handler. Edges between the handler may represent the satisfied condition of a trigger condition.

User Interaction

To design the UI and programming model, first step is to analyze the possible user interaction.

1. Visualize the storlets/handlers relations of a container

2. Add a storlet, specify the handler, specify the parameter, then the service will check if it connects to the target storlet hander

3. After a new storlet is created and just before injection, check if there is any possible cycles, shortcut duplication, overlapping

4. Take snapshot of a workflow and migrate to other containers

Currently investigating on Jersey web service framework, reading the existing APIs of the system, which is used to access to storlets information of a container.

### Graph Description Language and Programming Package

Currently we decided to use DOT Language as default. Two more formats, GML and GraphML are also popular languages. It is not a big problem to translate between other languages, since our graph is small in each container (30 nodes in general)

DOT

``` graph graphname { // This attribute applies to the graph itself size="1,1"; // The label attribute can be used to change the label of a node a [label="Foo"]; // Here, the node shape is changed. b [shape=box]; // These edges both have different line properties a -- b -- c [color=blue]; b -- d [style=dotted]; } ```

• Attributed and direct graphs
• The language is straight forward, users could have a brief impression of the graph without plotting it out.
• It is widely supported in graph programming packages

GML

``` graph [ comment "This is a sample graph" directed 1 id 42 label "Hello, I am a graph" node [ id 1 label "node 1" thisIsASampleAttribute 42 ] node [ id 2 label "node 2" thisIsASampleAttribute 43 ] edge [ source 1 target 2 label "Edge from node 1 to node 2" ] ] ```

• ASCII for simplicity and portability
• Attributed and direct graphs
• Simple structure
• Extensibility and flexibility
• Representation of graphs

GraphML

```<graphml xmlns="http://graphml.graphdrawing.org/xmlns"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://graphml.graphdrawing.org/xmlns
http://graphml.graphdrawing.org/xmlns/1.0/graphml.xsd">
<graph id="G" edgedefault="undirected">
<node id="n0"/>
<node id="n1"/>
<edge id="e1" source="n0" target="n1"/>
</graph>
</graphml>```
• XML format, good for transforming
• Attributed, direct, and hierarchical graphs
• Graphical representation
• Application specified attribute data

NetworkX is elected as the preliminary graph programming package, besides pygraph and graph-tool are also very powerful libraries. Network is chosen because it support all three languages mentioned above, it provides powerful graph statistic analysis algorithms, especially the cycle detection api greatly facilitate our work. Instead, pygraph only returns the first cycle it detects and graph-tool only support DOT language, which is a serious limit. NetworkX usually associates with mabplotlib or pygraphviz for plotting graphs, both of them provides python APIs. However, the interface of the existing system are generally in java. It would be great to have a java plotting tool supplies more graph annotation features.

A quick Demo of cycle detection and graph plotting here

Brain Storm

Keep workflow of storlets information as objects in VISION Cloud, for migration to other container and restoration from corrupt workflows.

However, I think if we can have all required annotations stored in the graph, migration and restoring can be done with the existing and snapshots of graphs.

Next Step:

• Have a deeper look at LOG Language (some syntax are not very clear in the presentation 😦 )
• Look for visualization tool satisfies:
• Java API
• Support Cycle Detection
• Support interactive visualization
• Support DOT and more language (at least Dot)
• Good Annotation Functionality
• Consider what kind of annotations to store
• What to do with the graph

### Master Thesis Kick-off

It is my 3rd day at SICS to do my master thesis.

Let’s start from a description of my tasks and possible approaches.

My thesis is a part of the VISION Cloud project of SICS.

VISION Cloud is a powerful information and communication technology (ICT)
infrastructure, which efficiently supports data-intensive storage service. Additionally, it
facilitates the convergence of ICT, and supplies the on-demand storage service with a
competitive cost while the QoS and security are guaranteed.

My task is related to the “storlet”, which is the execution unit of the computational storage in VISION Cloud. These computations are typically data-centric, i.e., analyzing or transcoding newly uploaded or updated data objects in the cloud.

A storlet is created by the tenant and stored in the storlet library as a wrapped component. Whenever a user performs computations in the cloud, he fetches the appropriate storlets, specifies the trigger conditions and injects them into the cloud storage. The trigger event could be any state change of the storage state, which is reflected by the update of data object metadata. A storlet might remain in passive state until it is triggered by certain event, then it becomes active and starts operation. Eventually, when the operation is completed, it turns to passive again.

storlet life cycle

There are many use cases relies on the storlets mentioned above, e.g. media transcoding, Telco and health care. Most of the use case requirements expect an interface allowing to inject a chain/workflow of storlets automatically. However, the current implementation of this computational storage doesn’t support this feature. The only chance to connect two storlets is to predict the output of the first storlet manually and set an appropriate trigger condition for the second. Additionally, even connected storlets might form cycles, overlaps or starving.

Before discussing about the approaches, it is better to have a brief view of the logical storlet runtime

storlet runtime

Whenever a new storlet is injected into the cloud, its trigger information is extracted and registered in the Notification Service (NS). When a state change of a data object happens, the Object Service (ObS) sends the event to the NS. The NS then check its registration, if some trigger conditions match the event, the registered storlet is triggered to become active.

Since the output of a storlet might trigger another one, we treat the storlets distributed as a graph. However, the triggers between storlets are indirect, some mechanism is required to generate the graph.

Static Method:

Produce an algorithm to extract and predict the input and output information based on the storlet programming model (a new model with the properties is by design right now), analyze all possible connections and definitely not possible connections, then generate a potential graph.

• All potential issues would be discovered before new storlets injected.
• Definitely not possible connection information is useful when isolating storlets.
• Helps to exclude the possibility of cycling for failure detection.

• Big overhead, analyze possible input and output is expensive.
• Change the programming model is a challenge.

Dynamic Method:

Use a machine learning algorithm to generate graph dynamically with trigger event logs.

• Efficient problem detector, able to kill storlets while cycling is noticed.
• policies of injecting storlets can be learned heuristically by cycles are observed under certain situations.
• Verification for static analysis

• Rare completely secure injection of storlet.
• Not friendly to user when storlets killed automatically.

Comment:

Two approaches are complementary, it is possible to have both. However, it depends on the discussion result of new storlet programming model next Monday. :p

TO BE CONTINUED…

### FTS project update – Estimate RRT

The round trip time is estimated roughly in this version of my program.

I use GeoIP APIs to tell if the source and destination pair are from the same country, same continent or different continent.

And set a specific RRT to each case.

Globus APIs doesn’t return RRT for each underlying command, and only RETR command is interacted between source and destination servers, the rest are all about client and servers.

From the experiment, I tested this approach with several different rough RRT, and they all converge to a good result eventuality.

Since in half to one year, the RRT of third party transfer monitoring mechanism will be implemented in FTS, we can obtain more accurate optimal result at that time.

### FTS Optimization Update

Most time spent on looking for a possible source-destination pair accept third party transfervia globus-url-copy, and support a big number of parallel TCP streams. Finally it is solved 🙂

Here comes to the update of my program

1.

Last version: Default input parameters are Bandwidth and RRT (Round Trip Time)

New version: Default input parameters are RRT, TCP Socket Buffer Size, Number of streams.

Reason: It is more intuitive since bandwidth is not easy to estimate at very beginning without any information. Choose a more proper default configuration avoids exceeding.

2.

Last version: New estimated bandwidth is derived by current goodput * number of streams

New version: Bandwidth equals current goodput (unit changed)

Reason: It was a mistake, the goodput is already a sum of all streams.

Expectation of the algorithm, the derived number of streams and buffer size converge to an optimal domain, regardless of the first guess of bandwidth, but a relatively closed RRT is required. (ex. If the best number of streams is 50, it drifting between 40 – 60).

Furthermore, to experiment with multiple default RRT and choose the best one is a good approach to me, because the current infrastructure doesn’t provide RRT in a 3rd party transfer. But if the default RRT is too far from the real RRT, the algorithm makes result converge to another domain because of the inaccurate RRT.

In terms of simultaneous transfer, it highly depands on the memory management of the system. My task is to figure out if the optimal number of streams (connections) is 100, what is the best combination of simultaneous transfers and number of streams.

simultaneous transfers * number of streams = 100

Now, have a nice weekend to Lausanne !!!!

### FTS3 (File Transfer Service) Optimization @ CERN

It is my third week here on the boundary of Switzerland and France, work at Mayrin site of CERN (n), live in hostel of Saint-Genis-Pouilly.

Nice sightseeing and perfect weather, but just tooooo expensive :p

Let’s come back to the project.

FTS3 is  a transfer job scheduler for scientific experiments producing, analyzing and storing high amount of data, used mainly in CERN/LHC experiments, and applied GridFTP as transfer protocol.

FTS3 provides the following properties mainly benefit from GridFTP

• Produce and analyze lots of data, in petabyte scale
• Replicate data between different sites
• Data storage and network infrastructure is heterogeneous

GridFTP is a high-performance, secure, reliable data transfer protocol optimized for high-bandwidth wide-area networks, and it is based on FTP.

As shown in the link, GridFTP is supported by Globus toolkit, therefore I use globus-url-copy in data transfer experiment.

In each single transfer of GridFTP, there are three basic input parameters to be specified, and they seriously affect the performance of transfer. They are Number of TCP streams in one transfer, TCP buffer size and file block size.

Relatively, the file block size is the least sensitive factor, we leave it aside first.

Our goal is to maximum the goodput (achieved bandwidth), and the optimal number of streams and TCP buffer size are highly rely on the other two variables, link bottleneck bandwidth and round trip time (RRT) .

There are pretty much existing literature demonstrate the derivation of those optimal values by formulas. But there are still two obstacles remains.

One is that only 3rd-party transfer is considered in this optimization project, and RRT between the remote source and destination pair is unlikely to be obtained currently. Fortunately a new mechanism to get this RRT will be implemented in FTS3 in a near future.

The other obstacle is that it would not be known of the current bottleneck bandwidth. Even globus-url-copy provides a method to check the disk and network bandwidth for each transfer (-nlb), but the Globus Netlogger need to be configured on each server side, therefore this approach is not suitable for our case.

Here it comes a little algorithm to tuning the number of streams and TCP buffer size. The explanation of those capital letters are on the right bottom corner of this graph.

Due to the estimation of Round Trip Time and Bottleneck Bandwidth, this algorithm might not reach to the best solution, but it keeps converge to an optimal configuration. With low cost and overhead, when the result enters a critical interval and becomes generally stable, it is time to terminate the algorithm.

### Confine, a project for the future internet

Let the story starts with an introduction of a large-scale, low cost and self-management project, “Confine“.

Confine Project is an integrated project constructs a new experimental testbed for researching in Community Networks (CN). The testbed provides support for any stakeholders interested in developing and testing systems with network infrastructures as fixed and ad-hoc routing schemes, management schemes, etc. It is also a platform to discover the bottlenecks and obstacles in those combined CNs, to keep sustainability of CNs as a model for the future internet.

The following graph illustrates the architecture

## Characteristics

• More than 20,000 nodes and links
• Researchers select a set of resources, deploy, run, monitor and experiment with services and protocol
• Wide variety of wired and wireless links, nodes, routing, applications and users.
• A model of self-provisioned, dynamic and self-organized network using unlicensed and public radio spectrum and optical links.

## Principles

• Federation: A physical or logical interconnection of independent testbeds.
• Virtualization: Share resources, isolate concurrent experiments, prevent experimental services disrupting, and provide greater controllability of experiments.
• Decentralization: A demand of moving the decision-making governance closer to the elements of concern.
• Openness: Open platform for development, deployment, monitoring and usage of services and protocols.
• Usability: Unified, user-friendly access for web-based and service-based tools that supports the experiment life cycle.

## Examples

• Cooperation and comparison between nodes using diverse mesh routing protocol (OLSR, Batman, Babel)
• Self-managing application protocols that adapt to the dynamic conditions of nodes, links and routes in the networks
• Network self-management and decentralized management
• Adaption of services as VoIP or live radio streaming to the conditions of community networks.

## References

http://cordis.europa.eu/fp7/ict/fire/docs/fp7-factsheets/confine_en.pdf

http://wiki.confine-project.eu/_media/pub:confine-guifi.pdf

http://confine-project.eu/