Causeway

Causeway: A message-oriented distributed debugger
Causeway is an open source postmortem distributed debugger for examining the behavior of distributed programs built as communicating event loops. Its message-oriented approach follows the flow of messages across process and machine boundaries.

Getting Started
The simplest way to get started is to launch Causeway from a command line shell and then open one of the examples.

$ cd e/src/esrc/scripts $ rune causeway.e-swt

From the Welcome view select an example program from the Help menu.



Optionally, the sources and trace logs can be specified on the command line.

$ rune -Dsrc= causeway.e-swt ...

Java's default memory settings are sufficient for the examples but larger programs need more stack and heap space. Use the -Xss (stack) and -Xmx (heap) options to increase Java default memory sizes. Follow the amount with m for Mb or k for Kb. Notice the format does not follow the name=value convention. The J option tells rune to pass the option to Java.

$ rune -J-Xss128m -J-Xmx128m causeway.e-swt

Waterken Example (Ajax-style)
This Java program ran on the Waterken server instrumented to generate Causeway's trace log format. The program is a distributed implementation of a procedure for handling new purchase orders.

Before an order is placed, certain conditions must be met: the item is in stock and available, the customer’s account is in good standing, and the delivery options are up to date. An object residing in the “buyer” process (or vat) has remote references to objects residing in the “product” and “accounts” processes. The buyer queries the remote objects with asynchronous message sends. This example uses Ajax-style continuation-passing: a request carries a callback argument to which a response should be sent.

The screenshot below shows the principal views from Causeway's postmortem display for this example.

Causeway Viewer



 * Search view (leftmost): This view lists bookmarks and results of search commands.


 * Process-order view: This view lists events in chronological order, organized by vat. For example, clicking the "buyer" tab shows all events logged by the buyer vat. The events are ordered by turn number and within each turn, an intra-turn number.


 * Message-order view: This view shows the order in which events caused other events by sending messages. Message order is reflected in the outline structure: nested events were caused by the parent event. Causeway assigns each vat a color so we can see when message flow crosses vat boundaries.


 * Stack Explorer and Source view: These views are familiar from sequential debugging.

Menu Commands
File >> Set Source Root... point at the root directory of the sources.

For example, for the trace record pathname shown below,

"trace" : { "calls" : [ { "name" : "AsyncAnd.run", "source" : "org/waterken/purchase_ajax/AsyncAnd.java" } ] }

set the source root to the underlined prefix.

~/Desktop/waterken/example/src/ org/waterken/purchase_ajax/AsyncAnd.java

(Limitation: Cannot set multiple source roots.)

File >> Open select log files to open.

(Limitation: Cannot select a folder. Must go into the folder and select the multiple files.)

File >> Export... translates Causeway's message graph (DAG) to the GraphViz DOT format and writes the dot file to a local disk. The dot file is a human-readable text file. It specifies a graph using the DOT language. GraphViz must be downloaded and installed to see the graph visualization. The graph below was generated for the Waterken example described above.



Search >> Find Lost Messages reports sent messages that were apparently not received.

Tools >> Set Filter Options... presents all source files seen during parsing of the trace logs. Individual files can be filtered out.



Filtering  hides all stack frames whose source is in the   class, resulting in simpler, more abstract message-order view and GraphViz graph (as shown below).





(Limitation: These settings are not persistent across launches.)

Context Menu Commands
Bookmark bookmarks the currently selected event.



Find Multiples finds the multiple causes of a joining event, e.g., finds the multiple sends to a target showing a multiples icon.



Search Stacks finds all events that contain a stack frame for this source line, e.g., all events passing through this source code.



(Limitation: Matching events must match exactly on source and line position as specified in the trace logs. Not all lines in the source view can be searched and currently, there is no visual indication identifying the lines that can be searched.)