From Erights

Revision as of 16:08, 7 August 2008 by Tstanley (Talk)
Jump to: navigation, search

Causeway: A message-oriented distributed debugger

Causeway, an open source distributed debugger written in E, lets you browse the causal graph of events in a distributed computation.

Causeway provides a post-mortem view, gathered from trace files written by the processes you wish to debug. (We explain below how to enable this tracing.) In the top-left pane, each tab represents the full order of events recorded by each process. This gives a "follow the process" view common to conventional distributed debuggers. In the Message tree pane, we see an alternative "follow the conversation" outline view, in which each event expands to show the events it causes.


Causeway presents several different views of the causal relations. The views are coordinated such that, selecting an item in one view causes corresponding selections in other views.

  • Message Tree View This outline represents activation order: Which events caused which other events by sending messages.
  • Process-order View (top-left) This tabbed folder view includes a tab for each vat. Within each tab, there's a 2-level outline. The outer level is each successive turn in that vat. The inner level is all the turns caused by that turn. Overall, each tab is a complete chronological order of what happened in that vat.
  • Stack Explorer (bottom-left) A 2-level outline showing the activation history that resulted in a particular event. The outer level is the activation sequence of prior events. Within each event, the inner level is the call stack.
  • Source Code View (bottom-right) This view shows the filename and E source code, if available.

Generating causality traces

This section is currently obsolete. E-on-Java does not yet generate the trace records that Causeway 0.9.2 will expect.

E supports tracing and there are many options for turning on various kinds of traces, generated at different levels of detail. Tracing can be turned on using command line properties, or by explicit actions within the programs. The example programs "seller.e" and "buyer.e" turn causality tracing on and off in order to bracket just that portion of their computation that would be interesting to view in Causeway. We recommend this technique to avoid being swamped in irrelevant detail.

In the code below, the trace begins with the eventual send to x and ends eventually, sometime after answer is resolved. The trace includes the message sends (effects) caused by x <- question() (the cause) and the message sends caused by these effects, and so on.

# import tracing support
def tcr := <unsafe:org.erights.e.develop.trace.TraceController>

# ...

# turn causality tracing on
tcr.setProperty("TraceLog_causality", "debug")

def answer := x <- question()
when (answer) -> done(_) :any {
    # ...
} catch problem {
    # ...
} finally {
    # turn causality tracing off
    tcr.setProperty("TraceLog_causality", "warning")

So that the examples below will be more reusable, first set a shell variable EHOME to the location where you installed E. On MSWindows in the cygwin bash shell, this would typically look like

$ EHOME="c:/Program Files/"

In a bash shell on a Unixoid system, this would typically be

$ EHOME=/usr/local/e

The shell commands below generate a trace of seller.e in the current directory. (The tracing system is very finicky about not overwriting files, so if you want to save a trace log to a specific name by using the TraceLog_dir and TraceLog_name properties, be sure to delete any previous trace of that name.)

$ rm -f seller.txt
$ rune -DTraceLog_dir=. -DTraceLog_name=seller.txt "$EHOME/scripts/test/causeway/seller.e"

Wait for the seller to print "waiting...", as shown above. Then, in a separate shell window, generate a trace of buyer.e by saying

$ rm -f buyer.txt
$ rune -DTraceLog_dir=. -DTraceLog_name=buyer.txt "$EHOME/scripts/test/causeway/buyer.e"
*** Trace data written to ***

After buyer.e has terminated, currently, you must manually kill seller.e.

The first time a process actually writes something to its trace file, it will also print a diagnostic to its stderr with the name of that trace file, as shown above. After the trace has completed, rename the trace files to shorter, more memorable names. For example, etrace.2004-11-20T01_17_58.600Z.txt could be renamed to seller.txt.

Causeway has a long startup time, so be prepared to wait a while until your window appears. If you have any problems, suggestions, or other reactions, please let us know.

See Also

Causeway draft paper

Our current development effort is to generalize Causeway to support asynchronous message-passing programs running on event loop-based platforms in general, not just E. Our initial focus has been on the Waterken server. Using Causeway to debug Waterken applications

Personal tools
more tools