CSL '0401' Program Binary Disassembly Notes

[terra]

Well this screenshot doesn't look like much, but it's a big milestone.



For some time I've been quietly working on reconstructing C source code from the disassembly work, initially with the primary goal of making the codebase easier to read and understand, but more recently with the goal of being able to compile the source into a binary.

Vast amounts of effort have yielded the ability to compile (from C code) a full binary that is address-exact with 0401. What I mean by that is that the compiled binary has every RAM variable and every tune parameter at the exact memory address it is located at in 0401 (by linking files and sections in the correct order, not by forcing the location of each variable manually. This means any intentional changes to the codebase will correctly reflow). The actual program code is not a byte-exact match with 0401 because I'm using modern GCC whereas I *think* in the 90's and 00's BMW used a Microtek compiler. The compilers have their own strategies and optimisations, hence the program code is not exactly the same in structure (although my goal is of course that actual behaviour is identical).

I reached the point this weekend where the next step was to do an initial test/validation. The approach I decided on was to BDM the full binary (bootloader, program and tune) onto the DME. I modified the boot mode variable at 0x4800 to prevent the bootloader from jumping to the main OS, this forces the bootloader to stay in boot mode and service DS2 communication from the bootloader (effectively limiting the scope of what I was testing).

The screenshot above shows the success. The DME booted into the bootloader, I was able to connect over DS2 (OBD cable) and identify the DME as you can see in the screenshot. My VIN, serial number, and AIF records are shown because I copied over these sections of EEPROM when I BDM'd as I didn't want to cause any EWS de-sync issues, etc.

I was then able to read off the tune from the DME as a test that the major DS2 functions in the bootloader are working.

To have got this far is incredible, but there's obviously a lot more work to be done. I'm not going to reflect on that too much right now and just be pleased with this milestone!

Amazing work.

Hypothetically... do you think the code is portable enough that it could be compiled for other embedded platforms?

[Obioban]

Woooooot! Excellent work!

[heinzboehmer]

Compilable source!? That's incredible. Makes the barrier for actually modifying the program so much lower. Nice work man.

What I mean by that is that the compiled binary has every RAM variable and every tune parameter at the exact memory address it is located at in 0401 (by linking files and sections in the correct order, not by forcing the location of each variable manually. This means any intentional changes to the codebase will correctly reflow).

Niiice. That's not an easy thing to get right. Super cool.

Also, your screenshot seems to be broken.

[0-60motorsports]

Well this screenshot doesn't look like much, but it's a big milestone.



For some time I've been quietly working on reconstructing C source code from the disassembly work, initially with the primary goal of making the codebase easier to read and understand, but more recently with the goal of being able to compile the source into a binary.

Vast amounts of effort have yielded the ability to compile (from C code) a full binary that is address-exact with 0401. What I mean by that is that the compiled binary has every RAM variable and every tune parameter at the exact memory address it is located at in 0401 (by linking files and sections in the correct order, not by forcing the location of each variable manually. This means any intentional changes to the codebase will correctly reflow). The actual program code is not a byte-exact match with 0401 because I'm using modern GCC whereas I *think* in the 90's and 00's BMW used a Microtek compiler. The compilers have their own strategies and optimisations, hence the program code is not exactly the same in structure (although my goal is of course that actual behaviour is identical).

I reached the point this weekend where the next step was to do an initial test/validation. The approach I decided on was to BDM the full binary (bootloader, program and tune) onto the DME. I modified the boot mode variable at 0x4800 to prevent the bootloader from jumping to the main OS, this forces the bootloader to stay in boot mode and service DS2 communication from the bootloader (effectively limiting the scope of what I was testing).

The screenshot above shows the success. The DME booted into the bootloader, I was able to connect over DS2 (OBD cable) and identify the DME as you can see in the screenshot. My VIN, serial number, and AIF records are shown because I copied over these sections of EEPROM when I BDM'd as I didn't want to cause any EWS de-sync issues, etc.

I was then able to read off the tune from the DME as a test that the major DS2 functions in the bootloader are working.

To have got this far is incredible, but there's obviously a lot more work to be done. I'm not going to reflect on that too much right now and just be pleased with this milestone!

This is so awesome. I love reading all this stuff and I know I’m still not smart enough to understand it LOL. Thanks for doing this brother.

[Bry5on]

That is absolutely epic, man you must have been chugging on this in the shadows for quite some time! Great work!

[karter16]

Well this screenshot doesn't look like much, but it's a big milestone.



For some time I've been quietly working on reconstructing C source code from the disassembly work, initially with the primary goal of making the codebase easier to read and understand, but more recently with the goal of being able to compile the source into a binary.

Vast amounts of effort have yielded the ability to compile (from C code) a full binary that is address-exact with 0401. What I mean by that is that the compiled binary has every RAM variable and every tune parameter at the exact memory address it is located at in 0401 (by linking files and sections in the correct order, not by forcing the location of each variable manually. This means any intentional changes to the codebase will correctly reflow). The actual program code is not a byte-exact match with 0401 because I'm using modern GCC whereas I *think* in the 90's and 00's BMW used a Microtek compiler. The compilers have their own strategies and optimisations, hence the program code is not exactly the same in structure (although my goal is of course that actual behaviour is identical).

I reached the point this weekend where the next step was to do an initial test/validation. The approach I decided on was to BDM the full binary (bootloader, program and tune) onto the DME. I modified the boot mode variable at 0x4800 to prevent the bootloader from jumping to the main OS, this forces the bootloader to stay in boot mode and service DS2 communication from the bootloader (effectively limiting the scope of what I was testing).

The screenshot above shows the success. The DME booted into the bootloader, I was able to connect over DS2 (OBD cable) and identify the DME as you can see in the screenshot. My VIN, serial number, and AIF records are shown because I copied over these sections of EEPROM when I BDM'd as I didn't want to cause any EWS de-sync issues, etc.

I was then able to read off the tune from the DME as a test that the major DS2 functions in the bootloader are working.

To have got this far is incredible, but there's obviously a lot more work to be done. I'm not going to reflect on that too much right now and just be pleased with this milestone!

[karter16]

That would be a fun exercise. Curious to know how stressed these things are (i.e. how much more stuff we can make them do).

I've got both a stock non-HP and a HW converted HP to test with, in case you end up porting the logging!

Okay awesome I do need to do at least some disassembly of a non-HP program for something else I'm working on so it will be on the cards at some point. The DME bmwfnatic has sent arrived today as well so I now have a second DME for testing purposes too which is amazing!

[heinzboehmer]

Yep that's exactly what I'm thinking. Ideally need to do it on both an HP and a non-HP, however I'd have to port the CAN logging to a non-HP version of the code :-)

That would be a fun exercise. Curious to know how stressed these things are (i.e. how much more stuff we can make them do).

I've got both a stock non-HP and a HW converted HP to test with, in case you end up porting the logging!

[karter16]

I see.

So what I'm hearing is that we can pipe `db_tbgnd` out through CAN, then run the car for some period of time and work out average CPU utilization?

Yep that's exactly what I'm thinking. Ideally need to do it on both an HP and a non-HP, however I'd have to port the CAN logging to a non-HP version of the code :-)

[heinzboehmer]

Sorry should have made that clear. It's essentially "main()" for the OS. It consists of a while(true) loop and continuously executes when no higher-priority task is running. It's actually doing work, a large number of non-time-critical functions (e.g. table lookups with slow moving inputs like tmot, etc. etc.). So in the true sense of the word it's not really actually a "task" as such in that it's not triggered by interrupt/scheduler.

I see.

So what I'm hearing is that we can pipe `db_tbgnd` out through CAN, then run the car for some period of time and work out average CPU utilization?

[karter16]

What is the background task in this context? Is it what gets scheduled when the system is idle? Or is it actually doing work?

Sorry should have made that clear. It's essentially "main()" for the OS. It consists of a while(true) loop and continuously executes when no higher-priority task is running. It's actually doing work, a large number of non-time-critical functions (e.g. table lookups with slow moving inputs like tmot, etc. etc.). So in the true sense of the word it's not really actually a "task" as such in that it's not triggered by interrupt/scheduler.

[heinzboehmer]

Yep great point - and indeed there actually IS an easy way to test this:


db_performance_control() runs in the Master and Slave background tasks. db_tbgnd, db_tbgnd_av and db_tbgnd_max give the current, average and max interval times between runs of the background task.

What is the background task in this context? Is it what gets scheduled when the system is idle? Or is it actually doing work?

[karter16]

Presumably they did bump the clock speed for a reason. I suppose it could simply be that the memory fast enough for the higher clocks wasn’t available (cheaply enough) in 1999, but was by 2003

Yep great point - and indeed there actually IS an easy way to test this:


db_performance_control() runs in the Master and Slave background tasks. db_tbgnd, db_tbgnd_av and db_tbgnd_max give the current, average and max interval times between runs of the background task.

[terra]

I'm pretty sure the main reason BMW went to the HP is that they needed more parameter (tune) space. 0401 only *just* leaks into the second half of the program ROM that the HP added, but consumes the majority of the doubled parameter space. It's definitely possible that it was a CPU utilisation issue as well, but I don't really have a good way to evidence / prove that, without making a cut down version of 0401 and trying to run it on a standard MSS54.

Presumably they did bump the clock speed for a reason. I suppose it could simply be that the memory fast enough for the higher clocks wasn’t available (cheaply enough) in 1999, but was by 2003

[karter16]

If nothing else, the models that have likely been used to help the development have 1000% been trained on karter16's work.

I must say it was kinda funny when I reached the point with my research that sometimes the only thing google search was giving me in response to my query string was my own posts in this thread. Flattering, but not particularly helpful lol.