Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length

 
Advanced search

1352552 Posts in 62401 Topics- by 54120 Members - Latest Member: JamesLewisWhite

December 11, 2018, 02:15:08 AM

Need hosting? Check out Digital Ocean
(more details in this thread)
TIGSource ForumsDeveloperTechnical (Moderator: ThemsAllTook)<removed>
Pages: [1]
Print
Author Topic: <removed>  (Read 1156 times)
Daywalker
Level 0
**



View Profile
« on: June 15, 2018, 07:41:31 AM »

<removed>
« Last Edit: August 31, 2018, 01:43:01 PM by Daywalker » Logged

but a shadow of the past ..
Daid
Level 2
**



View Profile
« Reply #1 on: June 16, 2018, 02:26:21 AM »

First question that comes to mind is, why? For using it, it's not like there are no valid alternatives: https://en.wikipedia.org/wiki/Comparison_of_assemblers#x86_assemblers

For using it, everything is moving to 64bit these days. So x86-64 makes a lot more sense from that perspective if you want to squeeze performance.

As a practice project, IA-32 is only of the most complexly large instruction sets available, making your life complex. You would be better off with something like Z80 (used in the gameboy, and thus easy to test in an emulator)


(Getting down to the bare metal of a machine is my area of expertise, so feel free to shoot any questions here. But I won't directly help with your project, due to me spending my main time somewhere else)
Logged

Polly
Level 6
*



View Profile
« Reply #2 on: June 16, 2018, 09:10:41 AM »

If you've never written a assembler* before, i'd recommend taking a look at a open-source assembler for a simple CPU first ( for example this DCPU-16 assembler ) It should give you a general idea on how to write your own assembler.

*A "assembly compiler" is generally referred to as a "assembler" Wink
Logged
Daid
Level 2
**



View Profile
« Reply #3 on: June 17, 2018, 12:40:01 AM »

*A "assembly compiler" is generally referred to as a "assembler" Wink
I think this might be hitting more mark then you think.

Reading your 2nd (longer) post. I think you have some terminology to catch up to.
https://www.tutorialspoint.com/compiler_design/compiler_design_overview.htm


I assumed you wanted to make the assembler. But reading your 2nd post, I think you might want to make the whole thing, from high level language to machine instructions.
I've been there. And I can tell you, it's a lot of work. I've only made functional script compilers, my other compiler projects died before doing anything functional. Script compilers generally compile to an intermediate format that needed an engine to run in.


But, basic steps you will need then:
Tokenizer: Converting lines into identifiers, numbers, etc. It's quite an easy step, but it makes the next step a lot easier to build.
Parser: I recommend a "recursive descent parser". As a result, you will get an "Abstract syntax tree" (AST). This step is quite complex already.
Code generator: For each function in your AST, you'll need to walk to tree, and generate the right machine instructions.


Naturally, there is a lot more you could do. Pre-processing, optimizing. But those are all extra steps you can do. Not really required to get a working compiler from A to Z.

Why did I only do scripting? Because the code generation becomes complex very fast with different variable types and register allocation. My script parser just did a pure stack based implementation, which was a lot easier.


Now, other things you can read into:
The python AST module: https://docs.python.org/3/library/ast.html gives full access to the AST of python code. Can give more insight into ASTs themselves.
Python also has it's tokenizer available for experimentation: https://docs.python.org/3/library/tokenize.html
LUA is a stack based scripting language. Understanding how LUA works with it's stack can help in understanding lower level machine based programming, which also includes a stack: https://www.lua.org/pil/contents.html (Especially the C API)


Another thing you could investigate, is just making a new "front" for the GNU compiler, or LLVM.
Logged

Polly
Level 6
*



View Profile
« Reply #4 on: June 17, 2018, 04:45:07 AM »

I assumed you wanted to make the assembler. But reading your 2nd post, I think you might want to make the whole thing, from high level language to machine instructions.

Since he specifically mentions FASM ( which is a x86 assembler ) i actually doubt that.
Logged
Crimsontide
Level 5
*****


View Profile
« Reply #5 on: June 17, 2018, 05:25:39 AM »

Ya, but  he also mentions his own language so...  TBH I'm kinda confused at what the end game is as well.
Logged
Daid
Level 2
**



View Profile
« Reply #6 on: June 17, 2018, 09:03:39 AM »

Another thing you could investigate, is just making a new "front" for the GNU compiler, or LLVM.
Not sure what that means, though I might look into it
The "backends" behind gcc and clang (both C compilers) support multiple languages. So it might be possible to add your own language to this as well. But I have no clue how complex that would be.
Logged

Polly
Level 6
*



View Profile
« Reply #7 on: June 17, 2018, 10:19:25 AM »

Ah, if your end goal is a compiler for a new high-level language that changes things a bit Tongue

If you're going to write a compiler ( that doesn't need inline assembly support ) it's common to use a "integrated assembler" .. which basically means that the compiler generates machine code directly instead of generating the intermediate assembly instructions and invoking a assembler. So as mentioned by Daid, you might want to take a look at ( for example ) LLVM instead of writing everything from scratch yourself.

For instance, Jonathan Blow has been working on his own programming language ( Jai ) since 2014 and is using LLVM as backend as well.
Logged
Daid
Level 2
**



View Profile
« Reply #8 on: June 18, 2018, 08:24:55 AM »

Only slightly related, but you mention dislike for mainstream OSes as well. https://wiki.osdev.org/Expanded_Main_Page is also a treasure of information. Especially if you like the low level stuff, as you indicate you do.
Logged

Crimsontide
Level 5
*****


View Profile
« Reply #9 on: June 19, 2018, 11:57:41 AM »

I started work on a small assembler of sorts years ago in C++.  The idea was a little different than yours.  Rather than output to a file, it would output the assembly to a chunk of memory, and wrap it like a standard function object.  The idea being I could dynamically generate code 'on the fly'.

I got it working for a few instructions before I got bored (or busy, I'm not sure I was in university at the time), but I can post a few code snippets to give a general idea of the structure.  This is all old code while I was still learning C++ (so while it did work, and the idea was sound IMHO, its still rather nooby-ish), but maybe it'll give you some ideas.

The class looked like this:

Code:
// ----- FunctorX64 -----
template<class F> class FunctorX64 : public Core::FunctorTemplate<F,boost::function_traits<F>::arity> {

protected:
// register enumerations
enum Reg8 { al, bl, cl, dl, ah, bh, ch, dh, sil, dil, bpl, spl, r8b, r9b, r10b, r11b, r12b, r13b, r14b, r15b };
enum Reg16 { ax, bx, cx, dx, si, di, bp, sp, r8w, r9w, r10w, r11w, r12w, r13w, r14w, r15w };
enum Reg32 { eax, ebx, ecx, edx, esi, edi, ebp, esp, r8d, r9d, r10d, r11d, r12d, r13d, r14d, r15d };
enum Reg64 { rax, rbx, rcx, rdx, rsi, rdi, rbp, rsp, r8, r9, r10, r11, r12, r13, r14, r15 };
enum RegXMM { xmm0, xmm1, xmm2, xmm3, xmm4, xmm5, xmm6, xmm7, xmm8, xmm9, xmm10, xmm11, xmm12, xmm13, xmm14, xmm15 };

private:
// register type info (a little tedious this way, but easier than handling function template specializations of enums in a template class)
bool IsReg (Reg8) const;
bool IsReg (Reg16) const;
bool IsReg (Reg32) const;
bool IsReg (Reg64) const;
bool IsReg (RegXMM) const;

bool IsReg8 (Reg8) const;
bool IsReg8 (Reg16) const;
bool IsReg8 (Reg32) const;
bool IsReg8 (Reg64) const;
bool IsReg8 (RegXMM) const;

bool IsReg16 (Reg8) const;
bool IsReg16 (Reg16) const;
bool IsReg16 (Reg32) const;
bool IsReg16 (Reg64) const;
bool IsReg16 (RegXMM) const;

bool IsReg32 (Reg8) const;
bool IsReg32 (Reg16) const;
bool IsReg32 (Reg32) const;
bool IsReg32 (Reg64) const;
bool IsReg32 (RegXMM) const;

bool IsReg64 (Reg8) const;
bool IsReg64 (Reg16) const;
bool IsReg64 (Reg32) const;
bool IsReg64 (Reg64) const;
bool IsReg64 (RegXMM) const;

bool IsRegXMM (Reg8) const;
bool IsRegXMM (Reg16) const;
bool IsRegXMM (Reg32) const;
bool IsRegXMM (Reg64) const;
bool IsRegXMM (RegXMM) const;

// register info
bool IsExtendedRegister (Reg8) const; // returns true for extended registers, any register that requires the Rex.R or Rex.B bits to be set
bool IsExtendedRegister (Reg16) const;
bool IsExtendedRegister (Reg32) const;
bool IsExtendedRegister (Reg64) const;
bool IsExtendedRegister (RegXMM) const;

bool IsRexRegister (Reg8) const; // true for registers which require a rex prefix possibly without the Rex.B or Rex.R bits set (sil, dil, bpl, spl, r8b - r15b)
bool IsRexRegister (Reg16) const; // same as IsExtendedRegister
bool IsRexRegister (Reg32) const; // same as IsExtendedRegister
bool IsRexRegister (Reg64) const; // same as IsExtendedRegister
bool IsRexRegister (RegXMM) const; // same as IsExtendedRegister

bool IsHighRegister (Reg8) const; // returns true for ah, bh, ch, dh
bool IsHighRegister (Reg16) const; // returns false
bool IsHighRegister (Reg32) const; // returns false
bool IsHighRegister (Reg64) const; // returns false
bool IsHighRegister (RegXMM) const; // returns false

bool IsLowRegister (Reg8) const; // returns true for al, bl, cl, dl
bool IsOldRegister (Reg8) const; // returns true for al, bl, cl, dl, ah, bh, ch, dh
bool IsNewRegister (Reg8) const; // returns true for al, bl, cl, dl, sil, dil, bpl, spl, r8b, ..., r15b

// helper functions
byte RegisterValue (Reg8) const;
byte RegisterValue (Reg16) const;
byte RegisterValue (Reg32) const;
byte RegisterValue (Reg64) const;
byte RegisterValue (RegXMM) const;
byte GetScale (int) const;

// prefix's
void OperandSizePrefix (); // changes the default operand size from 32 to 16 bit, must be used before RexPrefix
void AddressSizePrefix (); // changes the default address size from 64 to 32 bit, must be used before RexPrefix
void RexPrefix (bool W, bool R, bool X, bool B); // emits a Rex prefix with the associated WRXB bits set

// output ModRM byte (if D bit in OP is set rm = src, reg = dest, otherwise reg = src, rm = dest), SIB, Disp, as needed
void ModRM (Reg8 reg, Reg8 rm);
void ModRM (Reg16 reg, Reg16 rm);
void ModRM (Reg32 reg, Reg32 rm);
void ModRM (Reg64 reg, Reg64 rm);

// op with reg / reg operands (if D bit in OP is set rm = src, reg = dest, otherwise reg = src, rm = dest), SIB, Disp, as needed
void OpRR (byte op, Reg8 reg, Reg8 rm);
void OpRR (byte op, Reg16 reg, Reg16 rm);
void OpRR (byte op, Reg32 reg, Reg32 rm);
void OpRR (byte op, Reg64 reg, Reg64 rm);

// op with an immediate operand (register is encoded in OP, set rexb if the encoded register needs rex.b set)
void OpI (byte op, uint8, bool rex, bool rexb); // rex needs to be set if sil, dil, bpl, spl are to be used, rex and rexb need to be set for r8b - r15b
void OpI (byte op, uint16, bool rexb);
void OpI (byte op, uint32, bool rexb);
void OpI (byte op, uint64, bool rexb);

// op with reg / memory access
template<class TR> void OpRM (byte op, TR reg, int32 disp); // absolute displacement
template<class TR> void OpRP (byte op, TR reg, int32 disp); // displacement from RIP

template<class TR, class TA> void OpRM (byte op, TR reg, TA base, int32 disp);
template<class TR, class TA> void OpRM (byte op, TR reg, int scale, TA index, int32 disp);
template<class TR, class TA> void OpRM (byte op, TR reg, TA base, int scale, TA index, int32 disp);

protected:
// basic operations
void NOP ();
void RET (); // near (same segment) return
void FARRET (); // far (different segment) return

// move reg to reg
void MOV (Reg8 dest, Reg8 src);
void MOV (Reg16 dest, Reg16 src);
void MOV (Reg32 dest, Reg32 src);
void MOV (Reg64 dest, Reg64 src);

// move immediate to reg
void MOV (Reg8 dest, uint8 i);
void MOV (Reg16 dest, uint16 i);
void MOV (Reg32 dest, uint32 i);
void MOV (Reg64 dest, uint64 i);

// move absolute address (access global / absolute data)
void LOD (Reg8 dest, const void* src);
void LOD (Reg16 dest, const void* src);
void LOD (Reg32 dest, const void* src);
void LOD (Reg64 dest, const void* src);

void SAV (Reg8 src, void* dest);
void SAV (Reg16 src, void* dest);
void SAV (Reg32 src, void* dest);
void SAV (Reg64 src, void* dest);

// load from memory
void LOD (Reg8 dest, Reg32 base, int scale, Reg32 index, int32 disp = 0); // loads from base + scale * index + disp


// save to memory
void SAV (Reg8 src, Reg32 base, int scale, Reg32 index, int32 disp = 0); // save to base + scale * index + disp

// load from indirect address (no displacement)
void LOD (Reg8 dest, Reg32 srcAddr);
void LOD (Reg16 dest, Reg32 srcAddr);
void LOD (Reg32 dest, Reg32 srcAddr);
void LOD (Reg64 dest, Reg32 srcAddr);

void LOD (Reg8 dest, Reg64 srcAddr);
void LOD (Reg16 dest, Reg64 srcAddr);
void LOD (Reg32 dest, Reg64 srcAddr);
void LOD (Reg64 dest, Reg64 srcAddr);

// function prolog/epilog code

// debug

public:
// constructor
FunctorX64 ();
};

The registers were all enums, with a bit of function overloading you can make the assembly both easy to use and easier to code.

Most of the ops follow a few simple templates, for example 'RR' ops (operations that access two registers) might look like:

Code:
// ----- op reg / reg -----
template<class F> void FunctorX64<F>::OpRR (byte op, Reg8 reg, Reg8 rm) {

// attempt to encode without REX
if (IsOldRegister(src) && IsOldRegister(dest)) {
AddOp(op);
ModRM(reg,rm);
return;
}

// encode with REX
if (IsNewRegister(src) && IsNewRegister(dest)) {
RexPrefix(false,IsExtendedRegister(src),false,IsExtendedRegister(dest));
AddOp(op);
ModRM(src,dest);
return;
}

// thrown by using the old high 8-bit registers and the new registers in the same op (bh -> r9b for example)
throw InvalidRegisterCombination("template <typename F> void FunctorX64<F>::AddOpRR (byte op, Reg8 reg, Reg8 rm) - invalid register combination");
}

another examples, operand with an immediate encoded, shows how operator overloading can make things quite clean:
Code:
// --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
// op with an immediate operand
// --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

// ----- op immediate -----
template<class F> void FunctorX64<F>::OpI (byte op, uint8 i, bool rex, bool rexb) {
if (rex | rexb) RexPrefix(false, false, false, rexb);
AddOp(op);
AddImmediate(i);
}

template<class F> void FunctorX64<F>::OpI (byte op, uint16 i, bool rexb) {
OperandSizePrefix(); // encode 16 bit operand
if (rexb) RexPrefix(false, false, false, true); // encode extended register
AddOp(op); // encode op
AddImmediate(i); // encode immediate value
}

template<class F> void FunctorX64<F>::OpI (byte op, uint32 i, bool rexb) {
if (rexb) RexPrefix(false, false, false, true);
AddOp(op);
AddImmediate(i);
}

template<class F> void FunctorX64<F>::OpI (byte op, uint64 i, bool rexb) {
RexPrefix(true, false, false, rexb);
AddOp(op);
AddImmediate(i);
}

An op code might be encoded like:

Code:
template<class F> void FunctorX64<F>::LOD(Reg8 dest, Reg32 srcAddr) { OpRM(0x8a, dest, srcAddr, 0); }

As far as to all the details of how intel assembly works at the hex/machine code level.  Well you're just going to have to download and read the official docs, which you can download straight from intel last I checked.  x86/x64 are very complex instruction sets with a ton of 'gotchas'.

Good luck.
Logged
Crimsontide
Level 5
*****


View Profile
« Reply #10 on: June 21, 2018, 04:50:17 PM »

The operand size prefix changes the operand from a 32-bit operation to a 16 bit operation.  Whether this matters depends on the instruction.  You'll find that in x86/x64 assembly there are often multiple ways of encoding the same, or similar, operations.
Logged
Crimsontide
Level 5
*****


View Profile
« Reply #11 on: June 21, 2018, 08:07:51 PM »

Well back in the day when they moved from 8-bit to 16-bit they just added new op codes, but quickly ran out.  So for 32 bit they added the prefix stuff to sort of 'tack on' 32-bit (they did the same thing with the REX prefix and 64 bit).  So while in 16 bit mode you could switch to 32-bit with the prefix, and vice versa; but for all intents and purposes 16 bit mode no longer exists.  I wish they'd completely deprecate 32-bit as well...

As far as add goes, the result would be the same (32 or 16 bit) if all you care about are the low 16 bits.  If the high bits matter then the prefix (or lack of) matters.
Logged
Crimsontide
Level 5
*****


View Profile
« Reply #12 on: June 22, 2018, 06:42:15 AM »

Though I do agree, it's the whole backward-compatibility thing that kinda holds it back as far as I know.. and the fact that programmers have to catch up

No its a business thing.  Intel and AMD own the x86/x64 IP (the details of who owns what exactly are complicated), so only they can release processors for those instruction sets.  If they make significant changes it would most likely mean they would infringe on someone else's IP (like ARM or Samsung or something), and would be forced to cross license.  That would mean an end to Intel's virtual monopoly (Intel keeps AMD kicking around, much like MS did for Apple back in the day, simply to keep the anti-trust laws from kicking in).

It'd be trivial to migrate to something better while still maintaining backwards compatibility.
Logged
Pages: [1]
Print
Jump to:  

Theme orange-lt created by panic