CSL '0401' Program Binary Disassembly Notes

[heinzboehmer]

Yes this was by far the biggest issue in my mind when I first thought about this. The fact that the 68k's RAM is byte addressable though vastly simplifies this issue.

Oh clever! I completely missed that in your previous post, but yes, doing it byte by byte would greatly simplify things.

If there's nothing in the DPR or the RAM that might make things misbehave (not sure why there would be, but I haven't actually looked at the code), then I agree than validating based on address ranges is a viable option.

Don't get me wrong, I'm all for the generalized solution. I just think it's significantly more work to get right.

[karter16]

I was thinking about this today and I have to agree with Bryson.

Adding a new CAN message with statically mapped variables is almost trivial when it comes to potential bugs.

Adding the ability to dynamically assign these variables requires a significant amount of additional effort, if you want to build a truly comprehensive solution. And, of course, more complicated means more bug prone.

Just off the top of my head, these things would need to be addressed by this new dynamic infra:

• Variable address validation
  • How will you make sure the supplied addresses are actually for pertinent variables and not just random memory addresses?
  • What's stopping someone from assigning 0x0 as an address for one of the variables?
• Size
  • What's stopping someone from providing the addresses of 8 words?
  • What's the plan for data overflows?
    • Truncate?
    • Don't send the bad value?
    • Error out and skip sending the entire CAN message?
• Error reporting
  • If there is an error in the provided variable addresses, how do you communicate this?
    • Maybe reserve the first byte in one the CAN messages for flags?
    • Raise a new DTC?
    • Unfortunately, both of those ideas put the validation responsibility on the user, so what's stopping an inexperienced user from incorrectly setting this up, then ignoring the errors and trusting malformed data?
• Testing
  • How will you test this new code? It seems complex enough that (at least) unit tests would be required, but that's not exactly the easiest thing to do with the current development setup.

I think the idea of making this modular is super cool. Unfortunately, I also think that the amount of work required to make this work well is orders of magnitude bigger than just adding a new CAN message with statically mapped variables.

Thank you so much for taking the time to think this through and detail your thoughts, I really appreciate it!

You're absolutely right, this is certainly a bigger, more complex exercise. I wouldn't view this as precluding simply adding a second fixed CAN message as well for those who prefer that. I guess I'm thinking (a) of myself in that it's something I would find quite useful with the tuning work I still have ahead of me and (b) it would also mean that those who wanted to could configure as they prefer without it being reliant on someone (me) crafting different ROMs to suit each purpose.

I've been thinking the last few days about the first couple of items you mention:

Variable address validation
You're right, it would be necessary to ensure that valid addresses were configured. It wouldn't be too bad if someone enters a memory address that references program in that the value, although not useful, would be returned. But if someone references memory that doesn't exist that would cause a page fault at least.

Validating the addresses are valid isn't actually too bad as there are only two address ranges which matter for this purpose. The DPR at 0x00ff8000 through about 0x00ff8400, and the RAM at 0x00ffe500 through 0x00ffefff (although would probably limit this to 0xffe600 through 0x00ffefff as the first 256 bytes of that range are used by the operating system itself and wouldn't be useful to expose on the CAN bus). So essentially each configured address needs to be checked to ensure it belongs to one of those two ranges.

The CPU has a long list of initialization functions it runs once on startup, and I'd inject this additional check there.


Size
Yes this was by far the biggest issue in my mind when I first thought about this. The fact that the 68k's RAM is byte addressable though vastly simplifies this issue.

(FYI heinzboehmer I realize that I don't need to explain to you the below in the great level of detail that I have. I've detailed it to this level for future reference for others :-) )



Let's take the example of AQ_REL. AQ_REL is a word (16 bit) variable which resides at 0x00ff80de



The 68k architecture is big endian, so the most significant byte comes first at 0x00ff80de and the least significant byte at 0x00ff80df.

Now if you want to access the variable the usual way in code is to do something like this:



move.w moves the word (16 bits) to the location specified. But this isn't the only way you can access the variable.

The MC68336, like all 68k processors, is byte addressable, which means that each and every byte in the RAM is individually addressable. Which means you can do things like this:

move.b (A2) =>AQ_REL, aq_rel_msb

This moves the byte at the address AQ_REL (0x00ff80de) refers to into an (imaginary) variable that I named aq_rel_msb for the purposes of this example.

If AQ_REL = 0x8765 then the value that would be in aq_rel_msb would be 0x87.

Likewise we could do (pseudo-code):

move.b (A2) =>AQ_REL+1, aq_rel_lsb

Which would move the byte at the address AQ_REL+1 (0x00ff80df) to the imaginary variable aq_rel_lsb. Using the same example then the value in aq_rel_msb would be 0x65.


For the purposes of configurable CAN messages this means that the entire thing can be simplified to byte addressing. If you wanted to expose AQ_REL on the CAN message you would configure the address parameters as follows:

CAN_Message_A_Byte_0 = 0x00ff80de
CAN_Message_A_Byte_1 = 0x00ff80df

Taking this approach means that no data conversion is needed, and therefore it's not necessary to handle overflows, etc. Simply pass on whatever the value is.


Error Reporting
In terms of error reporting my thinking would be to keep it simple and if any one memory address failed validation then the entire CAN message would be disabled. I appreciate that it's not ideal to not give any feedback to the user, but it would be clear that something was wrong, it would protect against any undesirable behavior, and for the sort of user taking advantage of this it shouldn't be too hard to troubleshoot whether the addresses have been configured correctly or not. No other error reporting would be necessary I don't think.


Testing
Agreed - testing would be manual and difficult. I certainly would be testing this extensively myself before making it available to anyone else as I agree that the risk is higher.


That's just my current thinking, I may or may not look further into this, and I certainly am grateful for you and Bry5on's thoughts on this - regardless of whether I decide to have a go at this or not it's a fun idea to consider, and in the mean time I can certainly whip up a second CAN message for you if you want to let me know exactly what you'd like on it?

Thanks!

[heinzboehmer]

Hmm, the idea sounds awesome but personally I’d stick with static settings. A new can of worms sounds a little more scary.

I was thinking about this today and I have to agree with Bryson.

Adding a new CAN message with statically mapped variables is almost trivial when it comes to potential bugs.

Adding the ability to dynamically assign these variables requires a significant amount of additional effort, if you want to build a truly comprehensive solution. And, of course, more complicated means more bug prone.

Just off the top of my head, these things would need to be addressed by this new dynamic infra:

• Variable address validation
  • How will you make sure the supplied addresses are actually for pertinent variables and not just random memory addresses?
  • What's stopping someone from assigning 0x0 as an address for one of the variables?
• Size
  • What's stopping someone from providing the addresses of 8 words?
  • What's the plan for data overflows?
    • Truncate?
    • Don't send the bad value?
    • Error out and skip sending the entire CAN message?
• Error reporting
  • If there is an error in the provided variable addresses, how do you communicate this?
    • Maybe reserve the first byte in one the CAN messages for flags?
    • Raise a new DTC?
    • Unfortunately, both of those ideas put the validation responsibility on the user, so what's stopping an inexperienced user from incorrectly setting this up, then ignoring the errors and trusting malformed data?
• Testing
  • How will you test this new code? It seems complex enough that (at least) unit tests would be required, but that's not exactly the easiest thing to do with the current development setup.

I think the idea of making this modular is super cool. Unfortunately, I also think that the amount of work required to make this work well is orders of magnitude bigger than just adding a new CAN message with statically mapped variables.

[Bry5on]

What software are you using to read the live CAN messages?

I’m using the built in logger within Gauge.S - great DIYer friendly product.

[ac427]

Beautiful. Thank you again my friend. Will test lambda integrator in the morning as it looks like I got the others sorted .. well, close on relative opening

What software are you using to read the live CAN messages?

[karter16]

A new can of worms

Haha I see what you did there 🤣

[Bry5on]

Hmm, the idea sounds awesome but personally I’d stick with static settings. A new can of worms sounds a little more scary.

[karter16]

In addition to adding another CAN frame I've also been having a think about how to make this feature a bit more generic.

We have limited frames we can use, and without a major rewrite to dynamically swap frames that is quite frankly beyond my capability and current understanding, that isn't going to change. I can already think of enough things that would be interesting to log (particularly during tuning, etc.) that they won't all fit into the available frames at once. Likewise I'm sure others will encounter similar situations.

I mentioned previously that I thought it would be possible to create the ability to provide for some degree of configuration which would mean 1 standard program binary with the ability to somewhat configure the messages via the partial binary. Having made 0x7D0 work and having done some more thinking around this I'm pretty sure that the following would be fairly easy - it would look something like this:

Master Program ROM

• Custom Program ROM with modified CAN table repurposing unused messages (propose using 0x701 as the current version does as well as 0x62F which is another safe option for the E46)
• Modify existing handler for 0x7D0 to genericise functionality
• Create new handler for "0x7D1" with generic functionality
• Insert call into CAN_Send_DME_ARBIDs_karter16 to send 0x7D1 as well and wrap both in conditional logic based on k_can_karter16_cfg

Master Tune Binary

• New parameters defined as follows:
  • k_can_karter16_cfg (bitfield to enable/disable 0x7D0 and 0x7D1)
  • An array of 8 32bit values which hold the addresses of the 8 bytes to be mapped to the 0x7D0 message
  • An array of 8 32bit values which hold the addresses of the 8 bytes to be mapped to the 0x7D1 message

And I think it's as simple as that. I initially was thinking that there would need to be logic to split up 16/32 bit values where required, but given the 68k is byte addressed it's actually very simple. If you want to send the word (2 bytes) at 0x1000 for example it's just two move byte instructions the first referencing 0x1000 and the second referencing 0x1001.

With the disassembly work that's been done it will be possible to make available a table of all valid variables in the Master + dual-ported-ram which users can then choose from themselves. Any variables from slave that aren't on the DPR would have to be inserted into the DPR (if there's any space left, I don't actually know yet) as a change to the Slave Program ROM, but again this could be done as a generic program rom change.

Edit: This approach also means that the tune binary remains backwards compatible with the standard 0401 program ROM. e.g. you could flash back to the original ROM without having to also change your tune binary. Likewise running the custom ROM with a standard 0401 tune would also be supported - 0x7D0 and 0x7D1 would simply be disabled.


What are the downsides of this

• It unfortunately doesn't solve the problem of restoring the standard functionality of 0x771 and 0x62F when the cfg parameter is disabled. I want to think more about whether this is possible.
• While the benefit is the ability to craft your own CAN messages, it means that everyone's definition of what 0x7D0 and 0x7D1 are will be different. I'm not sure whether this is a potential negative when it comes to sharing information, configs, etc.
• I'm still vaguely bothered that 0x771 might be some sort of required mechanism for some sort of BMW diagnostic process. I admit it seems unlikely, but I'm very wary of screwing up anything for anyone.

Anyway - keen for feedback on this idea/proposal

[heinzboehmer]

I selected this one to repurpose as its handler pointer was to 0x00000000. But I've been thinking about that and I can't completely rule out that it's an intentional mechanism to allow reset of the processor via CAN. It seems a bit crude and would be a crash to a hard reset without any preparation, but maybe...

Man, that would be a pretty crazy way to reset the DME!

Now I want to try and crash it over CAN

[karter16]

Datalog shows only these:

Thanks very much for this. I've been doing some more investigation. I have more work to do to understand more but have figured out the below points:

• There is a second set of 15 CAN frames which are mapped to a separate region in memory
• There is a second set of read/write handlers
• These read-write handlers are more complex and appear to be built to manage the sending/receipt of a series of messages over CAN
• I'm not sure yet but it's possible that this is support for ISO 15765-2 (ISO-TP/CAN-TP) or similar for transmission of error codes and diagnostics over CAN
• This functionality is present for both the Master (main CAN) and Slave (SMG CAN)

I've also been trying to figure out CAN ID 0x770 as it seems to be paired in some way with CAN ID 0x771 (which is what I've repurposed for 0x7D0) I selected this one to repurpose as its handler pointer was to 0x00000000. But I've been thinking about that and I can't completely rule out that it's an intentional mechanism to allow reset of the processor via CAN. It seems a bit crude and would be a crash to a hard reset without any preparation, but maybe...


Anyway. In terms of the more immediate question of a second CAN message. I'm pretty comfortable that we can use the slot for 0x62F (read) as this message is only ever received for a chassis configuration that isn't the E46.

[Bry5on]

Datalog shows only these:

[karter16]

What are the other IDs? We can log them to see if data exists, although neither of us are SMG.

That would be great thank you! - I think the most likely ones are:

0x62F
0x700
0x701
0x770
0x513

[Bry5on]

What are the other IDs? We can log them to see if data exists, although neither of us are SMG.

[karter16]

...which reminds me, karter16 any interest in adding a second CAN message with throttle pedal position? Having a reference of what you're telling the car to do with your foot is really useful. The ITBs and idle air valve on this engine (and some other stuff like DSC) make it hard to interpret the TPS logs. You never really know for certain if the throttles opened as much as they did because that's what you wanted or because the DME decided that that was how much they were gonna open.

Yes! I've actually already been planning to look at adding a second message. I think it's going to be a bit harder as, unless we can conclusively prove one of the 14 other messages is actually not used then there isn't a spot for it, which would mean additional work. I have been doing more disassembly work this weekend around the wider behavior of the CAN system to understand what options there might be.

[heinzboehmer]

Custom DME FW is working great! Just ran through a session of VE tuning on my car and it was so nice to be able to pull all that info over CAN.

Feel free to peruse: http://datazap.me/u/heinzboehmer/csl...?log=0&data=15

Everything in that log is from CAN, except for throttle pedal position and the two brake pressures.


...which reminds me, karter16 any interest in adding a second CAN message with throttle pedal position? Having a reference of what you're telling the car to do with your foot is really useful. The ITBs and idle air valve on this engine (and some other stuff like DSC) make it hard to interpret the TPS logs. You never really know for certain if the throttles opened as much as they did because that's what you wanted or because the DME decided that that was how much they were gonna open.