Million years ago, way before the ice age, I was preparing small C++ project for “Unix Programming” university course and at some point had to debug it via command line. That was mind blowing. And surprisingly productive. Apparently, when nothing stands in the way, especially UI, debugging can become incredibly focused.
Since .NET Framework got his cross platform twin brother .NET Core, I was looking forward to repeat the trick and debug .NET Core app on Ubuntu via command line. Few days ago it finally happened and even though it wasn’t a smooth ride, that was quite an interesting experience. So, let’s have look.
We’ll need Ubuntu, .NET Core SDK, lldbdebugger and a sample app. Late April, 2018 was the month of updates, so now we have shiny new Ubuntu 18.04and . NET Core SDK 2.1 RC1, which finally got its libsosplugin.socompiled against lldb-3.9, so v3.6we had to use for previous .NETs finally can rest in peace. As for demo project, any .NET Core hello-world wannabe with local variables and call stacks will do.
I’ll install all of that in a VM and here’s Vagrantfile with its provision.shfile to do so:
| 1 2 3 4 5 6 7 8 9 | Vagrant.configure("2")do|config| config.vm.box="ubuntu/bionic64" config.vm.provider"virtualbox"do|v| v.memory=1024 end config.vm.provision"shell",path:"provision.sh" end |
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | #!/bin/bash functioninstall_lldb39{ echo"deb http://llvm.org/apt/trusty/ llvm-toolchain-trusty-3.9 main"|tee/etc/apt/sources.list.d/llvm.list wget-O-http://llvm.org/apt/llvm-snapshot.gpg.key|apt-keyadd- apt-getupdate-y apt-getinstall-ylldb-3.9 } functioninstall_ms_package_source{ wget-qO-https://packages.microsoft.com/keys/microsoft.asc|gpg--dearmor>microsoft.asc.gpg mvmicrosoft.asc.gpg/etc/apt/trusted.gpg.d/ wget-qhttps://packages.microsoft.com/config/ubuntu/18.04/prod.list mvprod.list/etc/apt/sources.list.d/microsoft-prod.list } functioninstall_netsdk21rc1{ install_ms_package_source apt-getinstall-yapt-transport-https apt-getupdate-y apt-getinstall-ydotnet-sdk-2.1.300-rc1-008673 } functionmain{ install_netsdk21rc1 install_lldb39 } main |
It’s very simple. Let’s have a function that returns a number of ticks passed since last measurement. It’s absolutely useless except for the fact that it will have local variables and some arguments to examine later.
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | usingSystem; namespaceconsole { classProgram { staticvoidMain(string[]args) { while(true) { varlastTick=DateTime.Now.Ticks; System.Threading.Thread.Sleep(2000); varticksElapsed=GetTicksElapsed(lastTick); } } staticlongGetTicksElapsed(longlastTicks) { varcurrentTicks=DateTime.Now.Ticks; vardelta=lastTicks-currentTicks; returndelta; } } } |
OK, so let’s start the whole thing and try to connect debugger to it:
| 1 2 3 4 | dotnetbuild # ... dotnetbin/Debug/netcoreapp2.1/console.dll& # [1] 6124 |
The program is producing no output and running in a background thread, so knowing the process id ( 6124) I can start lldb, load SOS plugin and connect to dotnet process:
| 1 2 3 4 5 6 7 8 | find/usr-namelibsosplugin.so# find SOS plugin # /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.0-rc1/libsosplugin.so sudolldb-3.9 # start LLDB (lldb)pluginload/usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.0-rc1/libsosplugin.so (lldb)processattach-p6124 # Process 6124 stopped # * thread #1: tid = 6124, 0x00007fef21a92ed9 libpthread.so.0`__pthread_cond_timedwait + 649, name = 'dotnet', stop reason = signal SIGSTOP # ... |
Nothing difficult. We can make sure that SOS has kicked in by examining e.g. managed threads before moving on to setting up an actual breakpoint:
| 1 2 3 4 5 6 7 8 9 10 11 | (lldb)clrthreads # ThreadCount: 2 # UnstartedThread: 0 # BackgroundThread: 1 # PendingThread: 0 # DeadThread: 0 # Hosted Runtime: no # Lock # ID OSID ThreadOBJ State GC Mode GC Alloc Context Domain Count Apt Exception # 1 1 17ec 00000000022884A0 2020020 Preemptive 00007FEE8002DE50:00007FEE8002FB30 000000000226F620 0 Ukn # 7 2 17f2 00000000022AB0D0 21220 Preemptive 0000000000000000:0000000000000000 000000000226F620 0 Ukn (Finalizer) |
For that we can use bpmd command introduced by SOS plugin. The only parameter it needs is a method name or its descriptor address, so in order to set up a breakpoint in a method called GetTicksElapsed, which is a member of Program class inside of console.dllassembly, here’s what I’d do:
| 1 2 3 4 | (lldb)bpmdconsole.dllProgram.GetTicksElapsed MethodDesc=00007FEEA63F57E8 Settingbreakpoint:breakpointset--address0x00007FEEA7061777[console.Program.GetTicksElapsed(Int64)] Addingpendingbreakpoints... |
As a side note, sometimes it might be actually easier to add a breakpoint by method descriptor address instead. Those are all over the place – in call stacks, instruction pointers, in class method tables. For instance, this is the call stack of currently selected thread:
| 1 2 3 4 5 6 7 8 | (lldb)clrstack # OS Thread Id: 0x17ec (1) # Child SP IP Call Site # 00007FFE0D679AA8 00007fef21a92ed9 [HelperMethodFrame: 00007ffe0d679aa8] System.Threading.Thread.SleepInternal(Int32) # 00007FFE0D679BF0 00007FEEA7165ABB System.Threading.Thread.Sleep(Int32) # 00007FFE0D679C00 00007FEEA70616FC console.Program.Main(System.String[]) [/home/vagrant/console/Program.cs @ 12] # 00007FFE0D679F10 00007fef2023de1f [GCFrame: 00007ffe0d679f10] # 00007FFE0D67A310 00007fef2023de1f [GCFrame: 00007ffe0d67a310] |
And this how setting up a breakpoint in Main method might look like:
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | (lldb)ip2md00007FEEA70616FC # MethodDesc: 00007feea63f57d8 # Method Name: console.Program.Main(System.String[]) # Class: 00007feea7131088 # MethodTable: 00007feea63f5800 # mdToken: 0000000006000001 # Module: 00007feea63f43e0 # IsJitted: yes # Current CodeAddr: 00007feea7061690 # Code Version History: # CodeAddr: 00007feea7061690 (Non-Tiered) # NativeCodeVersion: 0000000000000000 # Source file: /home/vagrant/console/Program.cs @ 12 (lldb)bpmd-md00007feea63f57d8 # MethodDesc = 00007FEEA63F57D8 # Setting breakpoint: breakpoint set --address 0x00007FEEA7061690 [console.Program.Main(System.String[])] |
Easy peasy. Now, let’s resume the program with program continue command and see how it almost immediately stops at GetLastTicks:
| 1 2 3 4 5 6 | (lldb)processcontinue # Process 6124 resuming (lldb)Process6124stopped # * thread #1: tid = 6124, 0x00007feea7061777, name = 'dotnet', stop reason = breakpoint 1.1 # frame #0: 0x00007feea7061777 # ... |
Good. Now it’s time to look around.
clrstack (or sos ClrStack) does exactly that:
| 1 2 3 4 5 6 | (lldb)clrstack # OS Thread Id: 0x17ec (1) # Child SP IP Call Site # 00007FFE0D679BB0 00007FEEA7061777 console.Program.GetTicksElapsed(Int64) [/home/vagrant/console/Program.cs @ 18] # 00007FFE0D679C00 00007FEEA7061706 console.Program.Main(System.String[]) [/home/vagrant/console/Program.cs @ 13] # ... |
The output is pretty straightforward. Even line numbers and file names are there. But clrstack can do more than that. For instance, it also has -p parameter, which will include function arguments into the output, and that’s something really, really useful:
| 1 2 3 4 5 6 7 8 9 10 11 | (lldb)clrstack-p # OS Thread Id: 0x17ec (1) # Child SP IP Call Site # 00007FFE0D679BB0 00007FEEA7061777 console.Program.GetTicksElapsed(Int64) [/home/vagrant/console/Program.cs @ 18] # PARAMETERS: # lastTicks (0x00007FFE0D679BE0) = 0x08d5bacfae88e0b3 # # 00007FFE0D679C00 00007FEEA7061706 console.Program.Main(System.String[]) [/home/vagrant/console/Program.cs @ 13] # PARAMETERS: # args (0x00007FFE0D679C40) = 0x00007fee8001e5c0 # ... |
Let’s examine what we’ve got. lastTicks value is 0x8d5bacfae88e0b3, which is 636620323491995827 in decimal, which indeed corresponds to current UTC date:
As for args argument, most likely that’s program’s command line arguments and this is something easy to confirm:
| 1 2 3 4 5 6 7 8 | (lldb)dumpobj0x00007fee8001e5c0 # Name: System.String[] # MethodTable: 00007feea6e26308 # EEClass: 00007feea65c64a8 # Size: 24(0x18) bytes # Array: Rank 1, Number of elements 0, Type CLASS # Fields: # None |
Yup, that’s empty array of strings allright.
Bad news here. clrstack -i is the command to see local variables and their values. However, in lldb-3.9 and libsosplugin.so, which comes with .NET Core 2.1 RC1, this command immediately causes segmentation fault and crash of lldbprocess. Haven’t checked it in earlier versions, but here it happens 100% of the time.
Unlike with WinDBG, it doesn’t look like there are SOS commands for stepping in, out or over the next statement or function. However, there’re still native commands, which will step over assembly instructions, but it’s better than nothing. Especially when we can call clrstackand check where in managed realm we currently are:
| 1 2 3 4 5 6 7 | (lldb)next (lldb)Process6124stopped # * thread #1: tid = 6124, 0x00007feea7061778, name = 'dotnet', stop reason = instruction step into # frame #0: 0x00007feea7061778 # -> 0x7feea7061778: callq 0x7feea6af3cd0 # 0x7feea706177d: movq %rax, -0x38(%rbp) # 0x7feea7061781: movq -0x38(%rbp), %rdi |
callq, if I’m not mistaken, is the instruction to execute procedure at given address, so step will probably jump us somewhere:
| 1 2 3 4 5 6 7 8 9 10 11 | (lldb)step (lldb)Process6124stopped # * thread #1: tid = 6124, 0x00007feea6af3cd0, name = 'dotnet', stop reason = instruction step into # frame #0: 0x00007feea6af3cd0 # -> 0x7feea6af3cd0: pushq %rbp # ... (lldb)clrstack # OS Thread Id: 0x17ec (1) # Child SP IP Call Site # 00007FFE0D679BA8 00007FEEA6AF3CD0 System.DateTime.get_Now() # 00007FFE0D679BB0 00007FEEA706177D console.Program.GetTicksElapsed(Int64) [/home/vagrant/console/Program.cs @ 19] |
Yup, we’re in DateTime.Now‘s getter now. I actually could test where callq leads without stepping into the method itself. Checking callq‘s argument would tell the same story:
| 1 2 3 4 | (lldb)ip2md0x7feea6af3cd0 # MethodDesc: 00007feea66448c0 # Method Name: System.DateTime.get_Now() # ... |
Stepping out from current procedure could’ve been easy, if it didn’t jump 2 levels up instead of one every other time I used it. Maybe it has something to do with the fact that real call stack is actually a mixture of managed and unmanaged entries, whereas clrstackcommand shows only managed ones ( -f argument would show all of them). Then, I couldn’t find an alias for step-out command and had to use its full form instead: thread step-out.
So this is how debugging a .NET Core app from a command line on Linux feels like. It’s surprisingly hard to find any documentation for it, so probably I missed a command or to. Faulting clrstack -i also doesn’t make the debugging easier. But still it’s really cool to be able to add a breakpoint, see how execution goes and examine call stack parameters – all from a command line, on any machine and for any project. It’s also surprising to see how thin the layer between a managed code and assembly language is. If I was able to convert callq instruction argument to managed method description, maybe I’ll be able to convert CPU register values to managed objects as well. Who knows.