From Erights

Revision as of 18:14, 7 December 2008 by (Talk)
Jump to: navigation, search

Causeway: A message-oriented distributed debugger

Causeway, an open source distributed debugger written in E, lets you browse the causal graph of events in a distributed computation.

Causeway provides a post-mortem view, gathered from trace files written by the processes you wish to debug. (We explain below how to enable this tracing.) In the top-left pane, each tab represents the full order of events recorded by each process. This gives a "follow the process" view common to conventional distributed debuggers. In the Message tree pane, we see an alternative "follow the conversation" outline view, in which each event expands to show the events it causes.


Causeway presents several different views of the causal relations. The views are coordinated such that, selecting an item in one view causes corresponding selections in other views.

  • Process-order View (top-left pane)

This view lists events in chronological order, organized by vat. It's a tabbed view with one tab per vat.

An entry in the process-order view is a 2-level subtree. The parent item represents an event; each nested item represents an eventual send that occurred during the parent event.

In the screenshot, the selected item is the currently selected event in the message-order view. Synchronized selection between the process-order and message-order views is especially useful since, taken together, they convey the equivalent of spacetime diagrams.

  • Message-order View (top-right pane) This view shows the order in which events caused other events by sending messages. This message order is reflected in the outline structure; nested events were caused by the parent event.

When an event has multiple causes it is a joining event. A joining event appears directly under each of its causes and is marked with a right arrow icon.

Each tree item represents a message target and is identified by a vat name and turn. The descriptive label depends on the information available in the trace record for the event and is one of the following.

The "text" field string. This field is required for Comment records. It is optional for Sent, SentIf, and Resolved records. A single line of source code from the source file specified in the top stack entry. The source file name and function name specified in the top stack entry.

  • Stack Explorer (bottom-left pane) As with sequential debugging, the question is often: How did we get here -- what chain of activations led to the current event? Causeway's stack explorer answers this question by looking back in time and presenting both eventual sends and immediate calls that led to the current event.

An entry in the stack explorer is a 2-level subtree. The parent item represents an event; its nested items represent the stack trace captured for that event.

In the screenshot, the top entry is the currently selected event in the message-order view. Subsequent entries are built by following the message graph back in time to sending events. An event having multiple causes has multiple paths but only the last cause is followed. Being the last in chronological order, it is expected to be the most useful for following the interesting causality.

  • Source View (bottom-right pane) This view shows the source code for the currently selected item in the stack explorer and indicates the corresponding source span.

Generating causality traces

This section is currently obsolete. E-on-Java does not yet generate the trace records that Causeway 0.9.2 will expect.

E supports tracing and there are many options for turning on various kinds of traces, generated at different levels of detail. Tracing can be turned on using command line properties, or by explicit actions within the programs. The example programs "seller.e" and "buyer.e" turn causality tracing on and off in order to bracket just that portion of their computation that would be interesting to view in Causeway. We recommend this technique to avoid being swamped in irrelevant detail.

In the code below, the trace begins with the eventual send to x and ends eventually, sometime after answer is resolved. The trace includes the message sends (effects) caused by x <- question() (the cause) and the message sends caused by these effects, and so on.

# import tracing support
def tcr := <unsafe:org.erights.e.develop.trace.TraceController>

# ...

# turn causality tracing on
tcr.setProperty("TraceLog_causality", "debug")

def answer := x <- question()
when (answer) -> done(_) :any {
    # ...
} catch problem {
    # ...
} finally {
    # turn causality tracing off
    tcr.setProperty("TraceLog_causality", "warning")

So that the examples below will be more reusable, first set a shell variable EHOME to the location where you installed E. On MSWindows in the cygwin bash shell, this would typically look like

$ EHOME="c:/Program Files/"

In a bash shell on a Unixoid system, this would typically be

$ EHOME=/usr/local/e

The shell commands below generate a trace of seller.e in the current directory. (The tracing system is very finicky about not overwriting files, so if you want to save a trace log to a specific name by using the TraceLog_dir and TraceLog_name properties, be sure to delete any previous trace of that name.)

$ rm -f seller.txt
$ rune -DTraceLog_dir=. -DTraceLog_name=seller.txt "$EHOME/scripts/test/causeway/seller.e"

Wait for the seller to print "waiting...", as shown above. Then, in a separate shell window, generate a trace of buyer.e by saying

$ rm -f buyer.txt
$ rune -DTraceLog_dir=. -DTraceLog_name=buyer.txt "$EHOME/scripts/test/causeway/buyer.e"
*** Trace data written to ***

After buyer.e has terminated, currently, you must manually kill seller.e.

The first time a process actually writes something to its trace file, it will also print a diagnostic to its stderr with the name of that trace file, as shown above. After the trace has completed, rename the trace files to shorter, more memorable names. For example, etrace.2004-11-20T01_17_58.600Z.txt could be renamed to seller.txt.

Causeway has a long startup time, so be prepared to wait a while until your window appears. If you have any problems, suggestions, or other reactions, please let us know.

See Also

Causeway draft paper

Our current development effort is to generalize Causeway to support asynchronous message-passing programs running on event loop-based platforms in general, not just E. Our initial focus has been on the Waterken server. Using Causeway to debug Waterken applications

Personal tools
more tools