This question relates to some of my other posts, if you would like to know what it is for look at them. I was wondering if in memory were the code is stored(machine code), if there is a symbol or number that tells the CPU where the instruction ends and a new one begins, like an end symbol. I figure that there must be since the CPU has to know were different instructions are, because instructions are different sizes and are not consistent. Also a disassembler has to have some way to detect instruction start and ends to convert them back to assembler.
the CPU microcode can determine the size from the first 1 or 2 bytes of the opcode
So depending on the first opcode it can determine the size and were it ends?
Quote from: www.:).com on November 07, 2010, 12:33:54 AM
So depending on the first opcode it can determine the size and were it ends?
Depending on the first
few bytes of an opcode it can determine the size and were it ends
ok
The size of an instruction is determined by the encoding of it for example:
- Prefixs Byte (66, F3, etc) - (Word prefix, REP, Segment, etc)
- OpcodeBase Byte 1 (0Fh)
- OpcodeBase Byte 2 (XX)
- MOD/REG Byte (XX)
- SIB byte (XX)
- Displacement (Byte, Word, Dword)
- Inmmediate (Byte, Word, Dword)
But eye on that for example not all instructions use the MOD/REG byte, special encodings for the EAX regist uses combined MOD/REG byte inside the OpcodeBase Byte.
Also are alot of exceptions to this rules, for a detailed information check:
http://www.sandpile.org/ia32/opc_enc.htm
Finally i believe you should do this by try and error test. Since there are some opcodes wich doesnt follow the common encoding compared to others, makes you wonder in what was intel thinking... :U
i think all of them can be determined by the first 1 or 2 bytes
that does not include over-ride operators like size, segment, REP, etc
Quote from: dedndave on November 07, 2010, 01:35:14 AM
i think all of them can be determined by the first 1 or 2 bytes
that does not include over-ride operands like size, segment, REP, etc
Yes, when you mean 1-2 bytes you mean OpcodeBase byte + MOD/REG with those ones you can check if it got displacement, if it have scalar and such on...
With an opcode table, fixed opcodes can be calculated by just checking the first byte (OpcodeBase one). Ofcourse the prefixs are another deal...
I still think intel mess somethings, some opcodes doenst follow this encoding and you need to treath them has "unique" for example ENTER opcode in no place says it have 2 Inmmeds where 1 is a word and another is a byte.
I found out that this is pretty lame and when making the opcode analyzer i have to waste time for check those "special" opcodes. :snooty:
When you check the first 1-2 bytes does that require knowledge of the processor opcode instructions or will it hold true for all instructions. If so is the an easier way to detect the size of the instruction?
Quote from: www.:).com on November 07, 2010, 02:05:07 AM
When you check the first 1-2 bytes does that require knowledge of the processor opcode instructions or will it hold true for all instructions. If so is the an easier way to detect the size of the instruction?
No, when you check the first byte instruction, usually you check for bitfields that represent something for example the last bit field of the OpcodeBase byte represent if the operation is 16 bit or 32 bit.
While the 2 bitfield of the OpcodeBase represent the SOURCE/DEST order.
The MOD/REG have inside encoded 1 or 2 regs (depends of the opcode) and also have a representative bitfield for check if opcode have a memory operand and SIB byte.
So on you have to check for get the size of the instruction and no there is no easy way. And no its not easy, some instruction need to be treated has unique and encode is different for them.
So the instructions in memory contained in their first 1-2 bytes of each instruction are the formatting for the rest of the instruction?
that's about the size of it :lol
if you are interested in this stuff, i suggest you have a look at the Intel manuals...
http://website.masm32.com/reference.htm
Ok, I will take a look at the manuals.
Decoding instructions is not nearly as easy as "the first few bytes determine the size". Just to give you an idea of how complex it really is, an instruction can have up to four optional prefixes from four different groups of prefixes, even before the opcode itself starts. After that, different opcodes have different lengths, and different bits within that determines how many bytes follow. It is not an trivial task, and every hard-and-fast rule has exceptions.
-r
I realize that it will not be easy, buy i'm not looking to decode them just isolate them by using their size.
- such as i don't want to know what the instruction is, just the size of it.
Quote from: redskull on November 07, 2010, 04:19:13 PM
Decoding instructions is not nearly as easy as "the first few bytes determine the size". Just to give you an idea of how complex it really is, an instruction can have up to four optional prefixes from four different groups of prefixes, even before the opcode itself starts. After that, different opcodes have different lengths, and different bits within that determines how many bytes follow. It is not an trivial task, and every hard-and-fast rule has exceptions.
-r
Those prefixes still using 1 byte so its nothing complex... Just with a opcode table you can get fixed opcodes size and with bitfield analyzer from the OpcodeBase + MOD/REG you can get the displacement + SIB byte.
Problem comes for example to get the Inmmed size or with some instructions that doesnt follow the normal encoding.
QuoteI realize that it will not be easy, buy i'm not looking to decode them just isolate them by using their size.
- such as i don't want to know what the instruction is, just the size of it.
Well getting their size is decode them... You cant really think to do this without any information or a deep lecture of the intel encoding...
Here you have an example of getting an opcode size (its not complete ofcourse, its just an snippet, i wont do your homework):
Mov Eax, OpcodeAddr
Xor Ecx, Ecx
.If (Byte Ptr [Eax] == 0Fh) ;Check 2 byte Opcode Base
Add Ecx, 1 ;+1 to the opcode size
.EndIf
Add Ecx, 2 ;+2 (Opcode Base Byte + MOD/REG byte)
Movzx Edx, Byte Ptr [Eax+1] ;Get MOD/REG Byte
.If (Ebx < 40h) ;Check if it have mem addr operation inside
And Ebx, 7 ;Check for SIB byte present
.If (Ebx == 5)
Add Ecx, 4 ;Its literal address DWORD PTR DS:[100000]
.Elseif (Ebx == 4)
Add Ecx, 1 ;+1 to the opcode size (SIB Byte)
Movzx Ebx, Byte Ptr [Eax+2] ;Get the SIB byte
.If (Ebx >= 40H) ;Check for Scalar
And Ebx, 7
.If (Ebx == 5) ;[REG32*SCALAR+DISPLACEMENT]
Add Ecx, 4 ;+4 opcode size
.EndIf
.EndIf
.EndIf
.EndIf
And so on you have to analyze the opcode bytes with their bitfield for have extra information. While you doing this youll realise that some opcode have their own encoding, thats why youll need an opcode table where you can process fixed opcode sizes without analyze it (faster).
Here you have an excelent link that help me with this task of making the opcode table:
http://ref.x86asm.net/geek32.html
PS: I use PRE formated text and still my commentaries are not aligned :(
Well it will certainly take a lot of work to get it to work flawlessly for the 386 through the i7. What exactly is the point of this exercise?
Quote from: theunknownguy on November 07, 2010, 05:34:50 PM
PS: I use PRE formated text and still my commentaries are not aligned :(
Stop using HARD TABS, my guess is that it expands them to 8 space alignment, and you're using something else. Try using a soft tab setting on your editor, with you own indentation preference.
Quote from: clive on November 07, 2010, 05:46:59 PM
Well it will certainly take a lot of work to get it to work flawlessly for the 386 through the i7. What exactly is the point of this exercise?
I think he wants to build something with self modify opcodes (not virus - i dont know) probably releated to its thread in DOS...
I have a flawless version of this. Support XMM/FPU/SSE (not SSE4 i havent add support). But cant share its part of my work, so i just give him an snippet of getting an literal address encoding and a scalar one example. (Its better give something for start than nothing)
Quote
Stop using HARD TABS, my guess is that it expands them to 8 space alignment, and you're using something else. Try using a soft tab setting on your editor, with you own indentation preference.
Soft Tab setting?... Any suggestions?
Thank you for the example code(answer for theunknownguy). :clap:
- the practice of this code is as part of a dual thread handler(answer for clive)
This is an old proc that i founded years ago on internet:
Credits to: I dont know (wish i could remember the site) :dazzled:
Local Modrm:Byte
Local Prefix:Byte
Push Esi
Push Edi
Push Ebx
Xor Ebx, Ebx
Xor Eax, Eax
Xor Ecx, Ecx
Mov Prefix, Al ;Clean Prefix lame way
Mov Edx, API ;EDX = pMemory
PrefixLoop:
.If (Eax == 8)
.Else
Mov Esi, Offset OpcodeTable
Mov Cl, Byte Ptr [Edx]
Add Esi, Ecx
.If Byte Ptr [Esi] == 40H
.If Cl == 66H
Mov Prefix, 1H
.ElseIf Cl == 67H
Mov Prefix, 2H
.EndIf
Add Eax, 1H
Add Edx, 1H
Jmp PrefixLoop
.EndIf
Mov Esi, Offset OpcodeTable
Mov Cl, Byte Ptr [Edx]
Add Esi, Ecx
Mov Bl, Byte Ptr [Esi]
.If Cl == 0F6H || Cl == 0F7H
Mov Cl, Byte Ptr [Edx + 1H]
Mov Edi, Ecx
And Edi, 38H
.If Edi != NULL
Mov Ebx, 20H
.EndIf
.EndIf
.If Prefix == 1H
.If Ebx & 10H
Push Eax
Mov Eax, 10H
Not Eax
And Ebx, Eax
Or Ebx, 8H
Pop Eax
.EndIf
.EndIf
.If Prefix == 2H
Push Eax
Mov Eax, 10H
Not Eax
And Ebx, Eax
Mov Eax, 4H
Not Eax
And Ebx, Eax
Or Ebx, 8H
Pop Eax
.EndIf
.If Ebx == 2H
Add Eax, 1H
Add Edx, 1H
Mov Esi, Offset OpcodeTable
Add Esi, 0FFH
Mov Cl, Byte Ptr [Edx]
Add Esi, Ecx
Mov Bl, Byte Ptr [Esi]
.EndIf
.If Ebx == 1H
Add Eax, 1H
.EndIf
.If Ebx == 4H
Add Eax, 2H
.EndIf
.If Ebx == 8H
Add Eax, 3H
.EndIf
.If Ebx == 0CH
Add Eax, 4H
.EndIf
.If Ebx == 10H
Add Eax, 5H
.EndIf
.If Ebx == 18H
Add Eax, 7H
.EndIf
.If EBX & 20h
Add Eax, 2H
Add Edx, 1H
Mov Cl, Byte Ptr [Edx]
Mov Modrm, Cl
Shr Cl, 6H ;Prefix Scalar
.If Cl == NULL
Mov Dl, Modrm
And Dl, 7H
.If Dl == 5H
Add Eax, 4H
.EndIf
.EndIf
.If Cl != 3H
Mov Dl, Modrm
And Dl, 7H
.If Dl == 4H
Add Eax, 1H
.If CL == 1h
Mov Dl, Modrm
And Dl, 7H
.If Dl == 4H
Add Eax, 1H
.EndIf
.EndIf
.If Cl == 2H
Mov Dl, Modrm
And Dl, 7H
.If Dl == 4H
Add Eax, 4H
.EndIf
.EndIf
.Endif
.EndIf
.If Modrm >= 40H && Modrm <= 7FH
Mov Dl, Modrm
And Dl, 7H
.If Dl != 4H
Add Eax, 1H
.EndIf
.EndIf
.If Modrm >= 80H && Modrm <= 0BFH
Mov Dl, Modrm
And Dl, 7H
.If Dl != 4H
Add Eax, 4H
.EndIf
.EndIf
.If Ebx & 10H
Add Eax, 4H
.EndIf
.If Ebx & 8H
Add Eax, 2H
.EndIf
.If Ebx & 4H
Add Eax, 1H
.EndIf
.EndIf
.If Ebx == NULL
Mov Eax, 0FFFFFFFFH
.EndIf
.EndIf
Pop Ebx
Pop Edi
Pop Esi
Ret
I dont have the opcode table, but sure it will help you, i doubt it have XMM or SSE support
Quote from: www.:).com
- the practice of this code is as part of a dual thread handler(answer for clive)
Yeah, I'm convinced you have conceived the most inefficient way of doing that possible, perhaps you could trace/single-step the code and have the CPU work out the instruction length for you. It would be faster.
Quote from: clive on November 07, 2010, 06:00:02 PM
Quote from: www.:).com
- the practice of this code is as part of a dual thread handler(answer for clive)
Yeah, I'm convinced you have conceived the most inefficient way of doing that possible, perhaps you could trace/single-step the code and have the CPU work out the instruction length for you. It would be faster.
:lol :lol
Dont blame the guy for trying crazy ideas, at least he will learn alot :green
after he's been writing code for a while, he will know how many bytes each instruction takes :bg
but, i can understand his curiosity
interesting how we each have a different mode when we are learning :P
Quote from: theunknownguyI have a flawless version of this. Support XMM/FPU/SSE (not SSE4 i havent add support).
And I have decoders through SSE5/VMX, but I'm pretty sure they aren't flawless or handle some undocumented cases.
QuoteSoft Tab setting?... Any suggestions?
How it indents, with hard tabs (ASCII 9) and with what column settings, or if it expands the TAB key into a number space (ASCII 32) characters to acheive the desired alignment. Most editors or IDEs have options, for assembler I usually use 8, for C 2, but it depends on personal preference, which is why most provide configuration options. Where and how on the tools you are using, I don't know. Assume the forum is using 8
Also if I was decoding instructions lengths, over half the benefit from all the work would be to know what the instruction was. For validation tasks I'd rather have the processor provide the length information, by tracing or other means.
a debugger will do it for you :bg
Quote from: theunknownguy
Dont blame the guy for trying crazy ideas, at least he will learn alot
This might have been interesting years ago, but the non-linear code processing might stick a spanner in the works. Perhaps modelling how the instructions pair, assign to different execution units, pipeline, hyperthread, stall, select/retire registers, etc might give a clearer view of what's actually happening internally in a single cycle.
Or I could just use two task stacks and switch context between them.
Quote from: clive on November 07, 2010, 06:11:13 PM
Quote from: theunknownguyI have a flawless version of this. Support XMM/FPU/SSE (not SSE4 i havent add support).
And I have decoders through SSE5/VMX, but I'm pretty sure they aren't flawless or handle some undocumented cases.
QuoteSoft Tab setting?... Any suggestions?
How it indents, with hard tabs (ASCII 9) and with what column settings, or if it expands the TAB key into a number space (ASCII 32) characters to acheive the desired alignment. Most editors or IDEs have options, for assembler I usually use 8, for C 2, but it depends on personal preference, which is why most provide configuration options. Where and how on the tools you are using, I don't know. Assume the forum is using 8
Also if I was decoding instructions lengths, over half the benefit from all the work would be to know what the instruction was. For validation tasks I'd rather have the processor provide the length information, by tracing or other means.
I havent encounter any flawless till SSE3... I still have to add alot of support meaby ill find problems later :lol
Still havent found any virus maker that uses SSE5... Barely they use FPU.
Thanks for the TAB tip i will try it :clap:
Quote
This might have been interesting years ago, but the non-linear code processing might stick a spanner in the works. Perhaps modelling how the instructions pair, assign to different execution units, pipeline, hyperthread, stall, select/retire registers, etc might give a clearer view of what's actually happening internally in a single cycle.
Sounds like instead of go to the NASA training program... just jump into the rocket...
I think in order to get to that point you must learn alot of things, instead of just jump to that area...
nahhhh
we are assembly programmers!
we strap in and learn the ropes when we get to the outer atmoshpere :lol
Quote from: dedndave on November 07, 2010, 06:06:12 PM
after he's been writing code for a while, he will know how many bytes each instruction takes :bg
but, i can understand his curiosity interesting how we each have a different mode when we are learning :P
I recognize that, but half the trick with software development is to find the easiest/quickest/efficient way of doing something. And realizing quickly that a particular way of attacking a problem is a time-sink/dead-end.
Counting instruction bytes has no useful bearing on execution speed in 2010.
Quote from: dedndave on November 07, 2010, 06:22:26 PM
nahhhh
we are assembly programmers!
we strap in and learn the ropes when we get to the outer atmoshpere :lol
:lol you guys are so rude with newcomers...
Telling the poor guy instead of learning how get instruction size or their encoding go and check how the whole deal works for inside... Ye sure go learn how fetch on CPU work, uops, and why not, the execution units, pipeline, etc.
Yes... guys are scared to come and ask questions here :lol
Learning to swim by jumping into the deep end is one thing, but being hell-bent on attempting an impossible project and ignoring the advice you ask for is something else.
-r
Quote from: redskull on November 07, 2010, 06:27:53 PM
Learning to swim by jumping into the deep end is one thing, but being hell-bent on attempting an impossible project and ignoring the advice you ask for is something else.
-r
Is it impossible? Caused it really takes a day or two to add the normal instructions + FPU and XMM... (I think he will be more than happy with that)
Dont say a guy what is impossible or what is not on his level, you only damage its confident on reach its goal... Instead of that you guys could use the experience you have on giving some snippets and he can start from there.
Sounds: George Michael - Freedom (Yes the guy have the freedom to take your advises or keep his idea)
theunknownguy, trying to understand your code and I read up on "movzx" -- "move zero extension" what dose it do with the zeros put them in front(01) or in back(10)?
if he wants to learn about opcodes, there is no better place than the Intel manuals
i gave him a link to get them - i haven't been rude
MOVZX puts them "in front", or "above" in intel terms
if you have a byte value of FAh...
movzx eax,ByteValue
will place FAh into the AL portion of EAX and fill the rest of it with 0's
so, EAX = 0000 00FA h
there is another instruction - MOVSX
it sign-extends the value
if bit 7 is 0, it fills the remainder of the regsiter with 0's
if bit 7 is 1, it fills the remainder of the regsiter with 1's
movsx eax,ByteValue
EAX = FFFF FFFA h
Quote from: www.:).com on November 07, 2010, 06:37:42 PM
theunknownguy, trying to understand your code and I read up on "movzx" -- "move zero extension" what dose it do with the zeros put them in front(01) or in back(10)?
I dont like to use partial regist operation, i hold everything on 32 bit REG instead of 8 bit. Of course if you like to use 8 bit reg (wich i dont see a reason why would you like) Just use MOV DL, BYTE PTR [EAX+1]
Quoteif he wants to learn about opcodes, there is no better place than the Intel manuals
i gave him a link to get them - i haven't been rude
No, not you dedndave you actually the less rude guy here...
I ment clive for this line:
QuoteThis might have been interesting years ago, but the non-linear code processing might stick a spanner in the works. Perhaps modelling how the instructions pair, assign to different execution units, pipeline, hyperthread, stall, select/retire registers, etc might give a clearer view of what's actually happening internally in a single cycle.
PS:
Everyone give links, i give some snipets and the only thing left is you www.:).com start reading and then come with good questions
ok
Quote from: theunknownguy
I ment clive for this line:
This was some sarcasm on my part because I still fail to understand the connection between instruction encoding lengths, and two threaded execution.
My more broader point was that a super-scaler CPU is not doing one thing at a time, and using a 80386 mindset is not a good approach to modelling the CPU complexity.
Quote from: theunknownguy on November 07, 2010, 06:31:07 PM
Is it impossible?
The decoder is not impossible, but his proposed DOS-based thread handler is, for reasons discussed in the other thread. After he was told this by several members, he simply brushed it off and started a new thread; apparently no project is too hard to have us help him do it, and
that's what's really rude. Maybe for his next project he'll do a 100-to-1 compressor :toothy
-r
Quote from: redskull on November 07, 2010, 07:10:34 PM
The decoder is not impossible, but his proposed DOS-based thread handler is, for reasons discussed in the other thread. After he was told this by several members, he simply brushed it off and started a new thread; apparently no project is too hard to have us help him do it, and that's what's really rude. Maybe for his next project he'll do a 100-to-1 compressor :toothy
-r
Yes, i did start multiple new threads, but for sub topics on my first thread one for the main topic and 2 others. Another one for instruction isolating in memory and another for moving the locations of that isolated memory.
Quote from: dedndave on November 07, 2010, 01:11:04 PM
that's about the size of it :lol
if you are interested in this stuff, i suggest you have a look at the Intel manuals...
http://website.masm32.com/reference.htm
Such a very amazing link!
Thanks for the post.
Bye bye tracy.
hi Tracy - welcome to the forum :U
that link comes from this page, which is in the upper right corner of the forum page
Forum Links and Website (http://website.masm32.com/)
Hutch has put several other goodies there, as well :bg
Quote from: dedndave on November 10, 2010, 02:20:36 PM
hi Tracy - welcome to the forum :U
that link comes from this page, which is in the upper right corner of the forum page
Forum Links and Website (http://website.masm32.com/)
Hutch has put several other goodies there, as well :bg
I guess you're replying to a spam bot :bg
what makes you say that ? :red
Quote from: dedndave on November 10, 2010, 03:21:04 PM
what makes you say that ? :red
I'm not sure, but ... it talks like a spam-bot, it looks like a spam-bot and it smells like a spam-bot ... so I came to think that it probably IS a spam-bot.
time will out :bg