[DynInst_API:] "Bad addressing mode!" in F23 libm.so

[Josh Stone <jistone@xxxxxxxxxx>]

Working with Dyninst master, I'm getting a "Bad addressing mode!"
assertion failed in ia32_decode_operands, parsing F23 libm.so,
specifically from glibc-2.22-11.fc23.x86_64.  I do have the debuginfo
package for this installed too.

The assertion fails in the default case for switch(op.admet), because it
doesn't have an entry for 28 -> am_VR.  But I think this instruction is
bogus anyway.

Here's some basic info from gdb below.

> #4  0x00007ffff4a889bf in NS_x86::ia32_decode_operands (pref=..., gotit=..., addr=0x7fffe2446721 ")\317\305\301T\376\305\323\\\350ÅX\340\305\323X\311\305\371(\350\305\321T\356\305\371.\375\017\206\276\t", instruct=..., mac=0x0)
>     at /home/jistone/paradyn/dyninst/common/src/arch-x86.C:9193
> 9193            assert(0 && "Bad addressing mode!");
> (gdb) info locals
> op = @0x7ffff4d18ddc: {admet = 28, optype = 10}
> i = 1
> loc = 0x63ffc50
> nib = 1
> addrSzAttr = 4
> operSzAttr = <optimized out>
> __PRETTY_FUNCTION__ = "unsigned int NS_x86::ia32_decode_operands(const NS_x86::ia32_prefixes&, const NS_x86::ia32_entry&, const unsigned char*, NS_x86::ia32_instruction&, NS_x86::ia32_memacc*)"
> sema = <optimized out>
> (gdb) info args
> pref = @0x63a7124: {count = 2, prfx = "\000\000\000\000", opcode_prefix = 102 'f', vex_present = true, vex_prefix = "\305\000\000", vex_l = 1 '\001', vex_w = 255 '\377', sse_mult = 0}
> gotit = @0x7ffff4d18dc8: {id = e_extrq, otable = 99, tabidx = 0 '\000', hasModRM = true, operands = {{admet = 20, optype = 5}, {admet = 28, optype = 10}, {admet = 0, optype = 0}}, legacyType = 0, opsema = 6}
> addr = 0x7fffe2446721 ")\317\305\301T\376\305\323\\\350ÅX\340\305\323X\311\305\371(\350\305\321T\356\305\371.\375\017\206\276\t"
> instruct = @0x63a7120: {size = 3, prf = {count = 2, prfx = "\000\000\000\000", opcode_prefix = 102 'f', vex_present = true, vex_prefix = "\305\000\000", vex_l = 1 '\001', vex_w = 255 '\377', sse_mult = 0}, mac = 0x0, cond = 0x0, 
>   entry = 0x0, loc = 0x63ffc50, legacy_type = 0, rip_relative_data = false}
> mac = 0x0
> (gdb) backtrace
> #0  0x00007ffff6426a98 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55
> #1  0x00007ffff642869a in __GI_abort () at abort.c:89
> #2  0x00007ffff641f227 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x7ffff4ae0d49 "0 && \"Bad addressing mode!\"", file=file@entry=0x7ffff4ae0c10 "/home/jistone/paradyn/dyninst/common/src/arch-x86.C", 
>     line=line@entry=9193, 
>     function=function@entry=0x7ffff4ae07c0 <NS_x86::ia32_decode_operands(NS_x86::ia32_prefixes const&, NS_x86::ia32_entry const&, unsigned char const*, NS_x86::ia32_instruction&, NS_x86::ia32_memacc*)::__PRETTY_FUNCTION__> "unsigned int NS_x86::ia32_decode_operands(const NS_x86::ia32_prefixes&, const NS_x86::ia32_entry&, const unsigned char*, NS_x86::ia32_instruction&, NS_x86::ia32_memacc*)") at assert.c:92
> #3  0x00007ffff641f2d2 in __GI___assert_fail (assertion=assertion@entry=0x7ffff4ae0d49 "0 && \"Bad addressing mode!\"", file=file@entry=0x7ffff4ae0c10 "/home/jistone/paradyn/dyninst/common/src/arch-x86.C", line=line@entry=9193, 
>     function=function@entry=0x7ffff4ae07c0 <NS_x86::ia32_decode_operands(NS_x86::ia32_prefixes const&, NS_x86::ia32_entry const&, unsigned char const*, NS_x86::ia32_instruction&, NS_x86::ia32_memacc*)::__PRETTY_FUNCTION__> "unsigned int NS_x86::ia32_decode_operands(const NS_x86::ia32_prefixes&, const NS_x86::ia32_entry&, const unsigned char*, NS_x86::ia32_instruction&, NS_x86::ia32_memacc*)") at assert.c:101
> #4  0x00007ffff4a889bf in NS_x86::ia32_decode_operands (pref=..., gotit=..., addr=0x7fffe2446721 ")\317\305\301T\376\305\323\\\350ÅX\340\305\323X\311\305\371(\350\305\321T\356\305\371.\375\017\206\276\t", instruct=..., mac=0x0)
>     at /home/jistone/paradyn/dyninst/common/src/arch-x86.C:9193
> #5  0x00007ffff4a89341 in NS_x86::ia32_decode (capa=capa@entry=1, addr=<optimized out>, instruct=...) at /home/jistone/paradyn/dyninst/common/src/arch-x86.C:8283
> #6  0x00007ffff4d62ba6 in Dyninst::InstructionAPI::InstructionDecoder_x86::doIA32Decode (this=0x63a4090, b=...) at /home/jistone/paradyn/dyninst/instructionAPI/src/InstructionDecoder-x86.C:1292
> #7  0x00007ffff4d634a9 in Dyninst::InstructionAPI::InstructionDecoder_x86::decodeOpcode (this=<optimized out>, b=...) at /home/jistone/paradyn/dyninst/instructionAPI/src/InstructionDecoder-x86.C:1352
> #8  0x00007ffff4e5eec7 in Dyninst::InstructionAPI::InstructionDecoderImpl::decode (this=0x63a4090, b=...) at /home/jistone/paradyn/dyninst/instructionAPI/src/InstructionDecoderImpl.C:56
> #9  0x00007ffff4d621c9 in Dyninst::InstructionAPI::InstructionDecoder_x86::decode (this=<optimized out>, b=...) at /home/jistone/paradyn/dyninst/instructionAPI/src/InstructionDecoder-x86.C:1406
> #10 0x00007ffff4d61ff6 in Dyninst::InstructionAPI::InstructionDecoder::decode (this=this@entry=0x7fffffffbfc0) at /home/jistone/paradyn/dyninst/instructionAPI/src/InstructionDecoder.C:65
> #11 0x00007ffff5581500 in hd::ProbabilityCalculator::decodeInstruction (this=this@entry=0x7fffffffc1f0, data=..., addr=addr@entry=341790) at /home/jistone/paradyn/dyninst/parseAPI/src/ProbabilisticParser.C:493
> #12 0x00007ffff5582162 in hd::ProbabilityCalculator::calcForwardWeights (this=0x7fffffffc1f0, cur=0, addr=341790, tree=0x7fffffffc1f8, valid=@0x7fffffffc0ef: true) at /home/jistone/paradyn/dyninst/parseAPI/src/ProbabilisticParser.C:416
> #13 0x00007ffff558245b in hd::ProbabilityCalculator::calcProbByMatchingIdioms (this=this@entry=0x7fffffffc1f0, addr=addr@entry=341790) at /home/jistone/paradyn/dyninst/parseAPI/src/ProbabilisticParser.C:331
> #14 0x00007ffff5516ce0 in Dyninst::ParseAPI::Parser::probabilistic_gap_parsing (this=0x4eff830, cr=cr@entry=0x4effd40) at /home/jistone/paradyn/dyninst/parseAPI/src/Parser-speculative.C:361
> #15 0x00007ffff550c072 in Dyninst::ParseAPI::CodeObject::parseGaps (this=<optimized out>, cr=cr@entry=0x4effd40, type=type@entry=Dyninst::ParseAPI::IdiomMatching) at /home/jistone/paradyn/dyninst/parseAPI/src/CodeObject.C:188
> #16 0x00007ffff7664bdc in image::analyzeImage (this=this@entry=0x4d18b20) at /home/jistone/paradyn/dyninst/dyninstAPI/src/image.C:1201
> #17 0x00007ffff7664ca1 in image::analyzeIfNeeded (this=0x4d18b20) at /home/jistone/paradyn/dyninst/dyninstAPI/src/image.C:1165
> #18 0x00007ffff7666ca8 in image::findBlocksByAddr (this=0x4d18b20, addr=34825, blocks=std::set with 0 elements) at /home/jistone/paradyn/dyninst/dyninstAPI/src/image.C:1673
> #19 0x00007ffff7670a2e in mapped_object::findBlocksByAddr (this=0x4f77820, addr=addr@entry=140737281382409, blocks=std::set with 0 elements) at /home/jistone/paradyn/dyninst/dyninstAPI/src/mapped_object.C:624
> #20 0x00007ffff7672de5 in mapped_object::findOneBlockByAddr (this=<optimized out>, addr=addr@entry=140737281382409) at /home/jistone/paradyn/dyninst/dyninstAPI/src/mapped_object.C:2072
> #21 0x00007ffff760ae22 in BPatch_object::findPoints (this=this@entry=0xd11ff70, addr=addr@entry=140737281382409, points=std::vector of length 0, capacity 0) at /home/jistone/paradyn/dyninst/dyninstAPI/src/BPatch_object.C:154
> #22 0x0000000000412bf8 in mutatee::instrument_dynprobe_target (this=this@entry=0x64c5870, object=object@entry=0xd11ff70, target=...) at ../../stapdyn/mutatee.cxx:344
> #23 0x0000000000416476 in mutatee::instrument_object_dynprobes (this=this@entry=0x64c5870, object=0xd11ff70, targets=std::vector of length 7, capacity 8 = {...}) at ../../stapdyn/mutatee.cxx:472
> #24 0x000000000041661f in mutatee::instrument_dynprobes (this=0x64c5870, targets=std::vector of length 7, capacity 8 = {...}) at ../../stapdyn/mutatee.cxx:590
> #25 0x000000000040db1f in mutator::create_process (this=0x6765a0, command="stap -l begin") at ../../stapdyn/mutator.cxx:282
> #26 0x00000000004076f6 in main (argc=<optimized out>, argv=<optimized out>) at ../../stapdyn/stapdyn.cxx:160

Using the process mapping, I can see that addr is offset 0x53271.
Compare to the disassembly:

(from frame #4)
> (gdb) x/8b addr                                                                                                         
> 0x7fffe2446721: 0x29    0xcf    0xc5    0xc1    0x54    0xfe    0xc5    0xd3

> 0000000000051b50 <__ieee754_atan2_fma4>:
[...]
>    5371f:       c5 79 29 cf             vmovapd %xmm9,%xmm7
>    53723:       c5 c1 54 fe             vandpd %xmm6,%xmm7,%xmm7
>    53727:       c5 d3 5c e8             vsubsd %xmm0,%xmm5,%xmm5

Those bytes match, but they start in the middle of an instruction!


In my own code, frame #22, it's trying to find offset 0x8809, which I
confirmed is a simple NOP for one of libm's SDT probes.  That was fed to
BPatch_object::fileOffsetToAddr() to get address 0x7ffff3a9d809, which
still makes sense from what I can see of the mutatee mapping:
> 00007ffff3a95000   1028K r-x-- libm-2.22.so
> 00007ffff3b96000   2044K ----- libm-2.22.so

0x7ffff3a9d809 - 0x7ffff3a95000 == 0x8809, OK.  And that address seen in
frame #18 image::findBlocksByAddr is back to 34825 == 0x8809.


I also took a log with DYNINST_DEBUG_PARSING=1, but that's huge.  I've
uploaded it here:
https://fedorapeople.org/~jistone/dyninst-bad-addressing-mode-parsing.log.bz2