Causeway

From Erights

(Difference between revisions)
Jump to: navigation, search
(Menu Commands)
 
(5 intermediate revisions not shown)
Line 1: Line 1:
-
== Causeway ==
+
== Causeway: A message-oriented distributed debugger ==
-
Causeway, an open source distributed debugger written in '''''E''''', lets you browse the causal graph of events in a distributed computation.
+
Causeway is an open source postmortem distributed debugger for examining the behavior of distributed programs built as communicating event loops. Its message-oriented approach follows the flow of messages across process and machine boundaries.
-
Causeway provides a post-mortem view, gathered from trace files written by the processes you wish to debug. (We explain below how to enable this tracing.) In the upper left pane, each tab represents the full order of events recorded by each process. This gives a "''follow the process''" view common to conventional distributed debuggers. In the upper right pane, we see an alternative "''follow the conversation''" outline view, in which each event expands to show the events it causes. Causeway assigns a unique color per vat. As we see, causality flows back and forth between the vats.
+
== Getting Started ==
-
<!-- images/cw-screenshot.png goes here -->
+
The simplest way to get started is to launch Causeway from a command line shell and then open one of the examples.
 +
<pre>
 +
$ cd e/src/esrc/scripts
 +
$ rune causeway.e-swt
 +
</pre>
-
Causeway presents several different views of the causal relations. The views are coordinated such that, selecting an item in one view causes corresponding selections in other views.
+
From the Welcome view select an example program from the Help menu.
-
* <u>Causality Tree View</u> This tree view contains the full graph built from the trace files.
 
-
* <u>Pruned Causality Tree View</u> This tree view is built by walking the full graph and applying a relevance function to each causal relation. The relations are either clipped, skipped, or kept, depending on how relevant, or interesting, they seem. Tree pruning is guided by relevance functions and filters. Currently, default options are selected, but we expect to support user-defined functions and filters in a future release.
+
[[Image:Welcome-help.png]]
-
* <u>Full-order View</u> The view to the left of the tree view lists the events from a single trace file, in full order. This is equivalent to the full order of events sent within a single process.
 
-
* <u>Stack Frame View</u> This view is not quite what you would expect. For a selected message in the tree view, the stack shows the state at the time the message was sent, not received. It's the stack of the cause, not the effect. Selecting an item in the stack view shows the corresponding source code.
+
Optionally, the sources and trace logs can be specified on the command line.
-
* <u>Source Code View</u> This view shows the filename and '''''E''''' source code, if available.
+
<pre>
 +
$ rune -Dsrc=<srcRootDir> causeway.e-swt <logs>...
 +
</pre>
-
=== Generating causality traces ===
+
Java's default memory settings are sufficient for the examples but larger programs need more stack and heap space.
 +
Use the -Xss (stack) and -Xmx (heap) options to increase Java default memory sizes. Follow the amount with m for Mb or k for Kb.
 +
Notice the format does not follow the name=value convention. The J option tells rune to pass the option to Java.
-
'''''E''''' supports tracing and there are many options for turning on various kinds of traces, generated at different levels of detail. Tracing can be turned on using command line properties, or by explicit actions within the programs. The example programs &quot;seller.e&quot; and &quot;buyer.e&quot; turn causality tracing on and off in order to bracket just that portion of their computation that would be interesting to view in Causeway. We recommend this technique to avoid being swamped in irrelevant detail.
+
<pre>
 +
$ rune -J-Xss128m -J-Xmx128m causeway.e-swt
 +
</pre>
-
In the code below, the trace begins with the eventual send to <code>x</code> and ends eventually, sometime after <code>answer</code> is resolved. The trace includes the message sends (effects) caused by <code>x &lt;- question()</code> (the cause) and the message sends caused by these effects, and so on.
+
== Waterken Example (Ajax-style) ==
-
<pre>
+
This Java program ran on the Waterken server instrumented to generate Causeway's trace log format. The program is a distributed implementation of a procedure for handling new purchase orders.
-
# import tracing support
+
-
def tcr := &lt;unsafe:org.erights.e.develop.trace.TraceController&gt;
+
-
# ...
+
Before an order is placed, certain conditions must be met: the item is in stock and available, the customer’s account is in good standing, and the delivery options are up to date. An object residing in the “buyer” process (or vat) has remote references to objects
 +
residing in the “product” and “accounts” processes. The buyer queries the remote objects with asynchronous message sends. This example uses Ajax-style continuation-passing: a request carries a callback argument to which a response should be sent.
-
# turn causality tracing on
+
The screenshot below shows the principal views from Causeway's postmortem display for this example.
-
tcr.setProperty(&quot;TraceLog_causality&quot;, &quot;debug&quot;)
+
-
def answer := x &lt;- question()
+
== Causeway Viewer ==
-
when (answer) -&gt; done(_) :any {
+
-
    # ...
+
-
} catch problem {
+
-
    # ...
+
-
} finally {
+
-
    # turn causality tracing off
+
-
    tcr.setProperty(&quot;TraceLog_causality&quot;, &quot;warning&quot;)
+
-
}
+
-
</pre>
+
-
So that the examples below will be more reusable, first set a shell variable EHOME to the location where you installed '''''E'''''. On MSWindows in the cygwin bash shell, this would typically look like
 
-
<pre>
+
[[Image:viewer.png|alt=Causeway Viewer|Causeway Viewer]]
-
$ EHOME=&quot;c:/Program Files/erights.org&quot;
+
-
</pre>
+
-
In a bash shell on a Unixoid system, this would typically be
 
-
<pre>
+
* Search view (leftmost): This view lists bookmarks and results of search commands.
-
$ EHOME=/usr/local/e
+
-
</pre>
+
-
The shell commands below generates a trace of <code>seller.e</code> in the current directory. (The tracing system is very finicky about not overwriting files, so if you want to save a trace log to a specific name by using the <code>TraceLog_dir</code> and <code>TraceLog_name</code> properties, be sure to delete any previous trace of that name.)
+
* Process-order view: This view lists events in chronological order, organized by vat. For example, clicking the "buyer" tab shows all events logged by the buyer vat. The events are ordered by turn number and within each turn, an intra-turn number.
-
<pre>
+
* Message-order view: This view shows the order in which events caused other events by sending messages. Message order is reflected in the outline structure: nested events were caused by the parent event. Causeway assigns each vat a color so we can see when message flow crosses vat boundaries.  
-
$ rm -f seller.txt
+
-
$ rune -DTraceLog_dir=. -DTraceLog_name=seller.txt &quot;$EHOME/scripts/test/causeway/seller.e&quot;
+
-
waiting...
+
-
</pre>
+
-
Wait for the seller to print &quot;waiting...&quot;, as shown above. Then, in a separate shell window, generate a trace of <code>buyer.e</code> by saying
+
* Stack Explorer and Source view: These views are familiar from sequential debugging.
 +
 
 +
=== Menu Commands ===
 +
 
 +
'''File >> Set Source Root...''' point at the root directory of the sources.
 +
 
 +
For example, for the trace record pathname shown below,
<pre>
<pre>
-
$ rm -f buyer.txt
+
"trace" : {
-
$ rune -DTraceLog_dir=. -DTraceLog_name=buyer.txt &quot;$EHOME/scripts/test/causeway/buyer.e&quot;
+
  "calls" : [ {
-
***-----------------------***
+
      "name" : "AsyncAnd.run",
-
*** Trace data written to ***
+
      "source" : "org/waterken/purchase_ajax/AsyncAnd.java"
-
'/home/markm/buyer.txt'
+
    } ]
-
***-----------------------***
+
}
-
$
+
</pre>
</pre>
-
After <code>buyer.e</code> has terminated, currently, you must manually kill <code>seller.e</code>.
+
set the source root to the underlined prefix.
-
The first time a process actually writes something to its trace file, it will also print a diagnostic to its stderr with the name of that trace file, as shown above. After the trace has completed, rename the trace files to shorter, more memorable names. For example, <code>etrace.2004-11-20T01_17_58.600Z.txt</code> could be renamed to <code>seller.txt</code>.
+
<u>~/Desktop/waterken/example/src/</u>org/waterken/purchase_ajax/AsyncAnd.java
-
Causeway needs deep thread stacks to run. When using standard JDKs, you can tell rune to give its program deeper stacks by using, for example, the &quot;-J-Xss8m&quot; option, to give it 8 MBytes. Assuming your trace files are <code>seller.txt</code> and <code>buyer.txt</code> in the current directory, you can run Causeway as follows:
+
''(Limitation: Cannot set multiple source roots.)''
 +
 
 +
 
 +
'''File >> Open''' select log files to open.
 +
 
 +
''(Limitation: Cannot select a folder. Must go into the folder and select the multiple files.)''
 +
 
 +
 
 +
'''File >> Export...''' translates Causeway's message graph (DAG) to the GraphViz DOT format and writes the dot file to a local disk. The dot file is a human-readable text file. It specifies a graph using the DOT language. [http://www.graphviz.org/ GraphViz] must be downloaded and installed to see the graph visualization. The graph below was generated for the Waterken example described above.
 +
 
 +
 
 +
[[Image:messageGraph.png|alt=Causeway's DAG as GraphViz graph]]
 +
 
 +
 
 +
'''Search >> Find Lost Messages''' reports sent messages that were apparently not received.
 +
 
 +
'''Tools >> Set Filter Options...''' presents all source files seen during parsing of the trace logs. Individual files can be filtered out.
 +
 
 +
 
 +
[[Image:filter-async-and.png|alt=Filter options dialog]]
 +
 
 +
 
 +
Filtering <code>AsyncAnd.java</code> hides all stack frames whose source is in the <code>AsyncAnd</code> class, resulting in simpler, more abstract message-order view and GraphViz graph (as shown below).
 +
 
 +
 
 +
[[Image:filtered-mov.png|alt=Message-order view with AsyncAnd.java filter]]
 +
 
 +
 
 +
[[Image:filtered-mg.png|alt=Message graph with AsyncAnd.java filter]]
 +
 
 +
 
 +
''(Limitation: These settings are not persistent across launches.)''
 +
 
 +
=== Context Menu Commands ===
 +
 
 +
'''Bookmark''' bookmarks the currently selected event.
 +
 
 +
 
 +
[[Image:bookmark.png|alt=Bookmark menu item]]
 +
 
 +
 
 +
'''Find Multiples''' finds the multiple causes of a joining event, e.g., finds the multiple sends to a target showing a multiples icon.
 +
 
 +
 
 +
[[Image:find-multiples.png|alt=Find multiples menu item]]
 +
 
 +
 
 +
'''Search Stacks''' finds all events that contain a stack frame for this source line, e.g., all events passing through this source code.
 +
 
 +
 
 +
[[Image:search-stacks.png|alt=Search stacks menu item]]
 +
 
 +
 
 +
''(Limitation: Matching events must match exactly on source and line position as specified in the trace logs. Not all lines in the source view can be searched and currently, there is no visual indication identifying the lines that can be searched.)''
 +
 
 +
== See Also ==
 +
 
 +
[[Causeway Platform Developer]] to understand how to instrument a platform to generate Causeway trace logs.
 +
 
 +
[http://www.hpl.hp.com/techreports/2009/HPL-2009-78.html HP Labs Technical Report] presents our experience with the Waterken web server which we have instrumented to generate Causeway's language-neutral trace log format.
 +
 
 +
[http://www.youtube.com/watch?v=QeqcGa7HlBk Screencast] presents a brief demonstration of Causeway, using the example from the HP Tech Report.
 +
 
 +
[http://waterken.sourceforge.net/debug/ Debugging a Waterken application] explains how to configure the Waterken server to emit the [[wikipedia:JSON|JSON]] debugging records understood by Causeway.
 +
 
 +
[http://code.google.com/p/ambienttalk/wiki/Debugging Debugging AmbientTalk using Causeway] explains how to emit Causeway debugging records, in order to use Causeway to debug AmbientTalk applications.
-
<pre>
 
-
$ rune -J-Xss8m &quot;$EHOME/scripts/causeway.e-swt&quot; buyer.txt seller.txt
 
-
</pre>
 
-
Causeway has a long startup time, so be prepared to wait a while until your window appears. If you have any problems, suggestions, or other reactions, please let us know.
+
[[Category:Applications]]

Latest revision as of 22:50, 26 May 2010

Contents

Causeway: A message-oriented distributed debugger

Causeway is an open source postmortem distributed debugger for examining the behavior of distributed programs built as communicating event loops. Its message-oriented approach follows the flow of messages across process and machine boundaries.

Getting Started

The simplest way to get started is to launch Causeway from a command line shell and then open one of the examples.

$ cd e/src/esrc/scripts
$ rune causeway.e-swt

From the Welcome view select an example program from the Help menu.


Image:Welcome-help.png


Optionally, the sources and trace logs can be specified on the command line.

$ rune -Dsrc=<srcRootDir> causeway.e-swt <logs>...

Java's default memory settings are sufficient for the examples but larger programs need more stack and heap space. Use the -Xss (stack) and -Xmx (heap) options to increase Java default memory sizes. Follow the amount with m for Mb or k for Kb. Notice the format does not follow the name=value convention. The J option tells rune to pass the option to Java.

$ rune -J-Xss128m -J-Xmx128m causeway.e-swt 

Waterken Example (Ajax-style)

This Java program ran on the Waterken server instrumented to generate Causeway's trace log format. The program is a distributed implementation of a procedure for handling new purchase orders.

Before an order is placed, certain conditions must be met: the item is in stock and available, the customer’s account is in good standing, and the delivery options are up to date. An object residing in the “buyer” process (or vat) has remote references to objects residing in the “product” and “accounts” processes. The buyer queries the remote objects with asynchronous message sends. This example uses Ajax-style continuation-passing: a request carries a callback argument to which a response should be sent.

The screenshot below shows the principal views from Causeway's postmortem display for this example.

Causeway Viewer

Causeway Viewer


  • Search view (leftmost): This view lists bookmarks and results of search commands.
  • Process-order view: This view lists events in chronological order, organized by vat. For example, clicking the "buyer" tab shows all events logged by the buyer vat. The events are ordered by turn number and within each turn, an intra-turn number.
  • Message-order view: This view shows the order in which events caused other events by sending messages. Message order is reflected in the outline structure: nested events were caused by the parent event. Causeway assigns each vat a color so we can see when message flow crosses vat boundaries.
  • Stack Explorer and Source view: These views are familiar from sequential debugging.

Menu Commands

File >> Set Source Root... point at the root directory of the sources.

For example, for the trace record pathname shown below,

"trace" : {
  "calls" : [ {
      "name" : "AsyncAnd.run",
      "source" : "org/waterken/purchase_ajax/AsyncAnd.java"
    } ]
}

set the source root to the underlined prefix.

~/Desktop/waterken/example/src/org/waterken/purchase_ajax/AsyncAnd.java

(Limitation: Cannot set multiple source roots.)


File >> Open select log files to open.

(Limitation: Cannot select a folder. Must go into the folder and select the multiple files.)


File >> Export... translates Causeway's message graph (DAG) to the GraphViz DOT format and writes the dot file to a local disk. The dot file is a human-readable text file. It specifies a graph using the DOT language. GraphViz must be downloaded and installed to see the graph visualization. The graph below was generated for the Waterken example described above.


Causeway's DAG as GraphViz graph


Search >> Find Lost Messages reports sent messages that were apparently not received.

Tools >> Set Filter Options... presents all source files seen during parsing of the trace logs. Individual files can be filtered out.


Filter options dialog


Filtering AsyncAnd.java hides all stack frames whose source is in the AsyncAnd class, resulting in simpler, more abstract message-order view and GraphViz graph (as shown below).


Message-order view with AsyncAnd.java filter


Message graph with AsyncAnd.java filter


(Limitation: These settings are not persistent across launches.)

Context Menu Commands

Bookmark bookmarks the currently selected event.


Bookmark menu item


Find Multiples finds the multiple causes of a joining event, e.g., finds the multiple sends to a target showing a multiples icon.


Find multiples menu item


Search Stacks finds all events that contain a stack frame for this source line, e.g., all events passing through this source code.


Search stacks menu item


(Limitation: Matching events must match exactly on source and line position as specified in the trace logs. Not all lines in the source view can be searched and currently, there is no visual indication identifying the lines that can be searched.)

See Also

Causeway Platform Developer to understand how to instrument a platform to generate Causeway trace logs.

HP Labs Technical Report presents our experience with the Waterken web server which we have instrumented to generate Causeway's language-neutral trace log format.

Screencast presents a brief demonstration of Causeway, using the example from the HP Tech Report.

Debugging a Waterken application explains how to configure the Waterken server to emit the JSON debugging records understood by Causeway.

Debugging AmbientTalk using Causeway explains how to emit Causeway debugging records, in order to use Causeway to debug AmbientTalk applications.

Personal tools
more tools