line number mapping for debug

[pmuellr]

There's been some email chit-chat regarding support coffeescript in the existing browser debuggers. The basic idea is to include information in the generated JavaScript files that the debuggers can make use of to display the original coffeescript source, and deal with line mapping issues between the CoffeeScript source and generated JavaScript.

Thought I'd go ahead and open up an issue for discussion.

[pmuellr]

So some thoughts:

• CoffeeScript source location - somehow we need the coffeescript source. See Issue add compile option to output a "//@ sourceURL=foo" comment #557 as one possible solution. The debugger would then be responsible for finding the source given that "URL". Java current just uses the basename() of the file as the file name in it's debug info, but of course it also has the full class name available to it as well. Not sure how we might get some "relative" name in there, so it seems like basename() or abspath() of the input file are the choices. abspath() exposes more information than it should, basename() will cause naming conflicts for same-named files in different directories
• line number information - The debugger will eventually want this in hash tables, both directions. Go from a cs line # to js line # when setting breakpoints, and js line # to cs line # when the JS vm is communicating to the debugger (eg, "stopped at line 47"). Presumably the actual data for the line number data could be in JSON format (eg, an object literal mapping cs #'s to an array of js #'s).

[mrblonde]

Might be worth looking at this also:
http://nex-3.com/posts/92-firesass-bridges-the-gap-between-sass-and-firebug

Maybe this same type of strategy could be used so that firebug could step through javascript line by line, but display the coffeescript code instead.

Tough project, but could turn into something pretty killer!

[jashkenas]

I'd be glad to help someone out with this ticket, but I'm not going to be spearheading it myself ... so closing as frozen for the time being. If someone wants to step up and do a port of FireSass for CoffeeScript, do let us know.

[jashkenas]

Alright -- this ticket now has life again, in Firefox at least:

https://bugzilla.mozilla.org/show_bug.cgi?id=618650

Reopening...

[subtleGradient]

Quick helpful workaround:

Include each line of the original coffeescript source inside a block comment before the js code that it generated. You could include the original coffeescript source lune numbers in the comments.

That would at least help simplify the learning and debugging process.

[StanAngeloff]

Mascara has a source-line mapping utility that only works in Firefox. It's quite possible to implement something similar:

try
  console.log 'Hello'
  a for i in [0..10]
catch error
  lines = { 31: 3, 32: 3, 33: 3 }
  error.message = error.message + ' on line ' + lines[error.stack.split('\n').shift().split(':').pop()]
  throw error

Run on Try CoffeeScript. Firefox only, though.

[andrewschaaf]

Related code:

Closure Compiler (JS -> JS)

...can generate source-to-source mappings:

java -jar $CLOSURE_JAR --js example.js --create_source_map ./map --js_output_file compiled.js

SourceMapLegacy.java has comments describing the format.

_PYXC-PJ (PY -> JS)_

...can generate mappings using the same format, but with more file info: [{"sha1":"...",...}] instead of []

commit: "changed emit() return type from a string to list of strings and kids"

commit: "source-to-source mappings! (same format as Closure Compiler)"

** exception-server **

An exception logging/viewing server has been written (in NodeJS) that uses these mappings to convert an exception's stack information into pre-compilation code snippets with TextMate URLs.

(your compilers/scripts send the server (code_sha1, mapping)s and (sha1, code)s at compile-time)

I'll post it [REDACTED TIMEFRAME] when it's more documented/polished.

** common-exception **

...can extract file/line/[col] information from NodeJS, Firefox, Chrome, Safari

[lucaswoj]

Crazy idea: What if when the CoffeeScript was compiled to JavaScript, it was done such that the lines corresponded 1:1 by placing multi-line JavaScript conversions on a single line.

[pmuellr]

I suspect it is crazy, but it would be useful. I actually do that with https://github.com/pmuellr/scooj , but only because the source is largely unprocessed JavaScript anyway. The nice thing is that even though you see some grungy code in the debugger, you do know the exact line number to go to in your editor.

[alexandertrefz]

I suggest a --debug coffee parameter that does something like this:

# Add Core-Utils to Underscore Namespace
_.ducktype = (obj, methods...) ->
  some("code")

to become:

var __slice = Array.prototype.slice;
// Line 2
_.ducktype = function() {
  var methods, obj;
  obj = arguments[0], methods = 2 <= arguments.length ? __slice.call(arguments, 1) : [];
  // Line 3
  return some("code");
};

this would work Browser independent and i think(im not a coffescript compiler expert, though) it could be implement without big hassles.

[andrewschaaf]

Let's define some terms:

Def a "compiler" compiles a non-empty set of "source file"s to exactly one "compiled file"

Defs

• a "(coord to coord)-mapping" is a function from (line, col) in the compiled file to (source_file_ref, line, col)
• a "(line to line)-mapping" is a function from (line) in the compiled file to (source_file_ref, line)
• a compiled file is "{coord,line}-{coord,line}-commented" if it contains comments in some format which provide a ({coord,line} to {coord,line})-mapping
• a "(...mapping...) file" is a byte-string in some format which contains a representation of a (...mapping...)

Notes:

• Don't worry, the representations of these mappings are usually much more space-efficient than the ultra-trivial {coord1:info1, ...}
• source_file_ref should include sha1(source_file) if possible, because that's really useful when used with my not-yet-released exception server

Defs

• "CCMF" is a (coord to coord)-mapping format. It's what Closure Compiler generates
• "CCMFH" is a backward-compatible extension with required hashes and optional other info

See my writeup of CCMF[H]: https://gist.github.com/769845

[andrewschaaf]

.lineno

Recall: adding .lineno to each node instance is a solved problem:

#955

With no side effects, that grammar.coffee change could be merged at any time.

[andrewschaaf]

Let's get this done! (Please? Please? Please?)

CoffeeScript should support both

1. {coord,line}-line-commenting in whatever that format turns out to be (e.g. via --line-commenting)
2. exporting CCMFH files (e.g. via --ccmfh-out=)

(1) might need to wait for some format finalization, but (2) does not.

And the nodes.coffee work required for (2) will apply to (1).

I'm trying to do a FOSS triple-launch in [REDACTED TIMEFRAME]:

• a NodeJS web framework
• a coherent Freedom Box attempt
• an exception server with awesome mapping support and TextMate links to your uncompiled source files
  • Screenshot of a mapping-following representation of an exception that occured while NodeJS was running code compiled by PYXC-PJ from Python syntax

(all three are written in CoffeeScript)

...and it would be really, really awesome to have CoffeeScript CCMFH support before then so I can integrate it.

Imagine being able to hit Command-R and have jashkenas/coffee-script-tmbundle compile, invoke node, and then represent the resulting exception in a beautiful and convenient mapping-following way with CoffeeScript source links and code snippets...

[andrewschaaf]

Some modest proposals

The Good (but more complex and possibly slower)

Something like what PYXC-PJ does.

EDIT: e.g.

Throw
  compileNode: (o) ->
    ...
    something(o, ["throw ", @expression, ";"])

Instead of having the main compilation function return a string, have it return a BAR.

BAR ::= ELEMENT-list

ELEMENT ::= string | FOO | BAR

FOO ::= something which can
    1. provide the code fragment for that subtree
    2. provide the ((line, col) -> (source line))
       mapping information for that fragment

Some function can then use the resulting BAR to generate both

1. the usual JavaScript
2. a CCMFH file

PYXC-PJ commits:

• "changed emit() return type from a string to list of strings and kids"
• "source-to-source mappings! (same format as Closure Compiler)"

I'm not a nodes.coffee expert (yet) so I'm not sure if subnode.compile(o) can be deferred. If so would let FOO be, e.g., {subnode:@expression, o:o}

Would o have been destructively modified by the time foo.subnode(foo.o) gets run?

I think The Good could be done without deferring .compile, but that might be harder.

The Bad (but simple (though verbose) to implement and possibly faster)

Something like what Closure Compiler does.

Instead of the beatifully concise current compileNode methods, do something like:

    something.startNode this
    something.add           "throw {"
    something.addSubnode    @expression, o
    
    # addSubnode: (node, o) ->
    #   ...
    #   node.compile o
    #   ...
    
    something.add           ";"
    something.endNode()

This would invoke the .compiles in the same order as the current implementation.

The Fugly

Have whatever methods invoke compileNode replace ([...].compileNode([...])) with

((() ->
    token = newUniqueToken()
    (
        "/*FUGLY:#{token}:@lineno*/" + 
        [...].compileNode([...]) +
        "/*/FUGLY:#{token}*/"
    )
)())

...and write a crazy function that uses the resulting text monstrosity to generate (comment-free JS, a CCMFH file)

This would be totally insane, but it would work and it would involve the least modification to nodes.coffee

[andrewschaaf]

I'll work on a

something(o, ["throw ", @expression, ";"])

approach

[andrewschaaf]

BAR-ification of nodes.coffee in progress:

https://github.com/andrewschaaf/coffee-script/commits/master

After each change, I run

git checkout -- lib && cake build && cake build && cake test && git status

and confirm that the tests pass and that no lib/*.js files other than nodes.coffee are changed

[andrewschaaf]

From #coffeescript yesterday:

my point about a codegen method's "//#{@child.compile o}" vs "#{@child.compile o}//" was that the codegen method currently discards the information distinguishing the two.

the relativeCharOffset of each kid matters.

one option would be for the codegen method to store that information in its kids

e.g. for Throw ("throw #{@expression...};"),

@expression.relativeCharOffset = "throw ".length

but this could make the codegen methods uglier than with the BAR-returning approach

with the BAR-returning approach, the Base method calling the codegen method can process the BAR immediately, computing the flattenedCodeString as usual and (processing the mapping implications if mapping:true)

BAR-returning codegen methods are a way to abstract out the mapping-handling code

from the outside, you're still calling the top node's .compile or .compileWithMapping

[andrewschaaf]

From #coffeescript yesterday:

my point about a codegen method's "//#{@child.compile o}" vs "#{@child.compile o}//" was that the codegen method currently discards the information distinguishing the two.

the relativeCharOffset of each kid matters.

one option would be for the codegen method to store that information in its kids

e.g. for Throw ("throw #{@expression...};"),

@expression.relativeCharOffset = "throw ".length

but this could make the codegen methods uglier than with the BAR-returning approach

with the BAR-returning approach, the Base method calling the codegen method can process the BAR immediately, computing the flattenedCodeString as usual and (processing the mapping implications if mapping:true)

BAR-returning codegen methods are a way to abstract out the mapping-handling code

from the outside, you're still calling the top node's .compile or .compileWithMapping

[ozra]

About ten years ago I wrote som crude source-to-source compilers for Perl and C++ which I called Pyrl and Cython. They simply gave you pythonish indent-syntax. They had the flag -p for pretty output and some other for line-to-line, which esentially means that it simply made sure that lines in the output ended up in the same line number as in the input, no matter if certain lines became 'ugly' because of code stacking.

Simple solution. Works in all environments (like Jake that compiles the coffee-makefile from inside it's script), any editor, etc., shouldn't be to hard to implement?

[ghost]

How hard would it be to add an option to have the js output include a call to a custom trace function on every function call? The param could be the callee function name and optionally the original line number and file. This would really help debug those cases where the code doesn't finish because I forgot to call something else at the end of a callback. These cases drive me crazy and cause me to put log calls all over the place.

The convention could be that if a specific function exists at the top level then the feature is enabled and that function is called.

debugFnCallLog = (funcName, fileName, lineNum) ->
    myCustomLogRoutine ...

[mathieuravaux]

For reference, here is a related webkit ticket: https://bugs.webkit.org/show_bug.cgi?id=30933

[ghost]

In my fantasy world V8 and other engines would be broken into a separate parser and byte-code interpreter while retaining the same JS semantics.

[aseemk]

Just want to toss out a major +1 to preserving line numbers, i.e. mapping 1:1 from source Coffee line # to generated JS line #. The excellent Streamline.js library has this option for its transformations, and it has proved invaluable.

Btw, if it helps, our platform is Node. We too compile dynamically w/out generating JS files, so things like line-numbers-in-comments aren't helpful. Our use case also isn't e.g. Firebug debugging generally but more "my program threw an error on line 512, where is that?"

Thanks in advance! Excitedly looking forward to seeing what comes of this.

[thelinuxlich]

+1 for this

[andrewschaaf]

Anyone in the NYC area interested in meeting up for a line-mapping mini-hackathon on Sunday the 15th?

[lucaswoj]

No, but I'd personally donate beer money. This needs to be done.

On May 4, 2011, at 10:27 AM, andrewschaafreply@reply.github.com wrote:

Anyone in the NYC area interested in meeting up for a line-mapping mini-hackathon on Sunday the 15th?

Reply to this email directly or view it on GitHub:
#558 (comment)

[tanepiper]

+1 I'd send some coffee + beer money too for this, it would be the biggest evolution for CoffeeScript

[ghost]

I'm surprised that no one has ever implemented one of the several quick
kludges that have been suggested.

On Wed, May 4, 2011 at 6:14 PM, tanepiper <
reply@reply.github.com>wrote:

+1 I'd send some coffee + beer money too for this, it would be the biggest
evolution for CoffeeScript

Reply to this email directly or view it on GitHub:
#558 (comment)

Mark Hahn
Website Manager
mark@boutiquing.com
949-229-1012

[michaelficarra]

I'll actually be in the NYC area on the 15th with nothing to do. I may be up for this.

[lephyrius]

+1 for some way of debugging coffeescript. It would be really helpful.

[jashkenas]

That's fantastic. I'd love to try and get one of the branches merged, and a release cut, for when it lands in Chrome proper.

[jimtla]

Source maps are in chrome proper - version 18 has support.

There's a nice explanation here which, incidentally, calls us out.

[ehartford]

Is there a coffee fork that enables source maps compatible with chrome 18's new feature?

[brokenseal]

I'm looking forward to test this in chrome 18 :)

[gimmi]

+1

I'm waiting for this feature to start taking seriously coffeescript

[studgeek]

Note IDEs that provide their own debugging environment can do the source mapping as well, they just need the sourcemap from coffeescript. For example JetBrains IDEA has a feature request for this (http://youtrack.jetbrains.com/issue/IDEA-84442) and Firefox still has this issue that folks can vote for (https://bugzilla.mozilla.org/show_bug.cgi?id=618650), but neither is going to move until they have something generating sourcemaps for them to play with.

[emilecantin]

Chrome 19 is now out, is there any progress on this? I'm really looking forward to debug directly in CoffeeScript.

[jashkenas]

'Fraid there's nothing to report yet progress-wise, but it's next on the list.

[vosechu]

You're all amazing and we love you. I've been using Coffeescript on a production project for the last two months and I'm absolutely in love with it. This would just be a little extra frosting on the cake.

[simonexmachina]

Awesome guys, can't wait for this to land.

[gojomo]

I realize I'm way late to an old and overlong thread, but wanted to throw another offbeat hybrid idea out there:

Indicate the two different "line" numbers by using two different line-end conventions in the same compiled file. For example, the compiled/JS line numbers would be indicated by traditional line-ends (CR/LF/CRLF), and the original-source line-numbers by some other harmless whitespace, such as FF (0x0C) or VB (0x0B).

That is, every JS line has a traditional line-length and line-end (no overlong lines), but only those representing another line forward in the CS source also include the new-convention secondary line-end. (And, this approach also allows for multiple source-line-ends to a single output line-end.)

Code editors and line-reporting code could offer a toggle for how they number lines.. or simply report both when dual conventions are noticed in the file. For example...

underscore_coffee.js:64;83

...where the ';83' indicates the secondary-convention line-number.

[vendethiel]

Check out https://github.com/michaelficarra/CoffeeScriptRedux

[wamatt]

Given that Redux now supports this. What is the implication for this repo?

Politics aside, and speaking strictly pragmatically for the benefit of new users such as myself, should we be switching to redux as the future compiler of choice?

[michaelficarra]

@wamatt: I would recommend sticking with this more stable compiler for serious projects for now as the kinks are worked out of my compiler over the next few months. That said, if you can go over the compiled output manually (or if your program has good test coverage and your tests still pass), and it looks good to you -- which it really should for most programs -- go ahead and take advantage of the source map support.

[wamatt]

@michaelficarra Thanks, that sounds sensible. Keep up the great work! :)

[devongovett]

What's the status on this? @michaelficarra's Redux compiler seems to support them and there is at least one pull request for support in this compiler but relatively little discussion from the maintainers. Is sourcemap support still something being planned? Is something holding it back? Just looking for updates.

[jashkenas]

@devongovett -- I think the status is basically what you said ... they work in Redux, and there are a couple of probably working branches with pull requests over here as well. One in particular I've been keeping my eye on and corresponding via email about, that looks very promising.

That said, I'd like to wrap up one new release of CoffeeScript with official "literate" support before starting to look at 'em in earnest. If you want to help move things along, feel free.

[vendethiel]

PONG, 3 years after

[michaelficarra]

Oh, wow, we can finally close this. The oldest open CoffeeScript issue. Well, here goes 😄.

edit: Actually, I forgot about the recently re-opened issue #77. That issue's even older than me.

[andrewschaaf]

Congratulations!

[sheerun]

Unfortunately 🍰 was a lie ;-) Thank you!

[wamatt]

Woot!

[terrycojones]

Awesome ticket :-)

[ajumell]

Check this project. https://npmjs.org/package/coffee-join/

[JJ]

wow