CSL '0401' Program Binary Disassembly Notes

[karter16]

Hmm, the idea sounds awesome but personally I’d stick with static settings. A new can of worms sounds a little more scary.

I was thinking about this today and I have to agree with Bryson.

Well against the good advice of both of you I've decided to have a go at this anyway. Mostly because it's something I'll personally find really useful, so it was enough motivation to do it.

I'll take the opportunity to review this work in a couple of days once it's out of my mind and I can look at it again with fresh eyes to see if I'm missing anything.


#1: Define new parameter space variables k_k16_can_cfg, and 64 bytes to store pointers for the 16 bytes of CAN data across the two messages.

k_k16_can_cfg: bit 0 = 7d0 enabled, bit 1 = 7d1 enabled.

The below are the configurable addresses of the RAM variables to copy into the CAN messages
k_k16_can_7d0_b0: ulong at 0x8ef04
k_k16_can_7d0_b1: ulong at 0x8ef08
k_k16_can_7d0_b2: ulong at 0x8ef0c
k_k16_can_7d0_b3: ulong at 0x8ef10
k_k16_can_7d0_b4: ulong at 0x8ef14
k_k16_can_7d0_b5: ulong at 0x8ef18
k_k16_can_7d0_b6: ulong at 0x8ef1c
k_k16_can_7d0_b7: ulong at 0x8ef20
k_k16_can_7d1_b0: ulong at 0x8ef24
k_k16_can_7d1_b1: ulong at 0x8ef28
k_k16_can_7d1_b2: ulong at 0x8ef2c
k_k16_can_7d1_b3: ulong at 0x8ef30
k_k16_can_7d1_b4: ulong at 0x8ef34
k_k16_can_7d1_b5: ulong at 0x8ef38
k_k16_can_7d1_b6: ulong at 0x8ef3c
k_k16_can_7d1_b7: ulong at 0x8ef40


#2: Define new RAM variable k16_can_st (0x00ffec1e) to store the runtime status of the two messages (e.g. they might be enabled in parameter space but disabled by sanity checks in k16_can_init())

k16_can_st: byte

This is our status byte, it is driven from k_k16_can_cfg, but is then modified based on checks that k_k16_can_cfg doesn't equal 0xff (e.g. k16 tune file is not loaded) and is also modified based on the result of the configuration validation checks.


#3: Modify the can_a_slot_cfg_table (0x3cbe8) to replace 771 and 62F with the correct config for 7D0 and 7D1.




#4: Modify the task_10ms function (0x12e9a) to call k16_can_tx_messages() a new version of can_tx_messages() which has extra logic for optionally sending the new messages.





#5: create the new k16_can_tx_messages() function

This modified function includes the 2x new CAN messages, but only if they are enabled and the configuration has passed validation.




#6: create the new k16_can_tx_7d0() function

The handler for 7D0. It looks up each address in the configuration field, then gets the value of the memory location pointed to (e.g. the RAM variable) and pushes the value to the appropriate byte of the CAN message.




#7: create the new k16_can_tx_7d1() function

The handler for 7D1 - works on the same principle as 7D0.



#8: modify the init_cpu function to call an additional k16_init() function

We need to run our k16_can_init() function once on start up to set k16_can_st, and do the validation of the config, etc. The easiest way to do this is to replace the call to an existing function with a call out to a new function that call's both the original function and the new k16_can_init() function.





#9: create the new k16_init() function at 0x3f8a0

Call the original function call we replaced, plus our new k16_can_init() function.




#10: create the new k16_can_init() function

And here's where k_k16_can_cfg is loaded and the configuration of each memory location is validated to ensure it is within range. Any situation which is not valid results in the entire CAN message being disabled.




As I said I'll come back to this to do more logic checking and bug hunting when I have a fresh mind. Looking forward to getting my gauge.s and getting setup so I can play round with this!

[karter16]

Quick update - I've now identified 42 of the CAN-related functions, there's only a few more to go.



I've got more work to do on this and some more to understand, but should be able to get to the point of documenting this pretty-much in full by the time I'm done.

[heinzboehmer]

Yes this was by far the biggest issue in my mind when I first thought about this. The fact that the 68k's RAM is byte addressable though vastly simplifies this issue.

Oh clever! I completely missed that in your previous post, but yes, doing it byte by byte would greatly simplify things.

If there's nothing in the DPR or the RAM that might make things misbehave (not sure why there would be, but I haven't actually looked at the code), then I agree than validating based on address ranges is a viable option.

Don't get me wrong, I'm all for the generalized solution. I just think it's significantly more work to get right.

[karter16]

I was thinking about this today and I have to agree with Bryson.

Adding a new CAN message with statically mapped variables is almost trivial when it comes to potential bugs.

Adding the ability to dynamically assign these variables requires a significant amount of additional effort, if you want to build a truly comprehensive solution. And, of course, more complicated means more bug prone.

Just off the top of my head, these things would need to be addressed by this new dynamic infra:

• Variable address validation
  • How will you make sure the supplied addresses are actually for pertinent variables and not just random memory addresses?
  • What's stopping someone from assigning 0x0 as an address for one of the variables?
• Size
  • What's stopping someone from providing the addresses of 8 words?
  • What's the plan for data overflows?
    • Truncate?
    • Don't send the bad value?
    • Error out and skip sending the entire CAN message?
• Error reporting
  • If there is an error in the provided variable addresses, how do you communicate this?
    • Maybe reserve the first byte in one the CAN messages for flags?
    • Raise a new DTC?
    • Unfortunately, both of those ideas put the validation responsibility on the user, so what's stopping an inexperienced user from incorrectly setting this up, then ignoring the errors and trusting malformed data?
• Testing
  • How will you test this new code? It seems complex enough that (at least) unit tests would be required, but that's not exactly the easiest thing to do with the current development setup.

I think the idea of making this modular is super cool. Unfortunately, I also think that the amount of work required to make this work well is orders of magnitude bigger than just adding a new CAN message with statically mapped variables.

Thank you so much for taking the time to think this through and detail your thoughts, I really appreciate it!

You're absolutely right, this is certainly a bigger, more complex exercise. I wouldn't view this as precluding simply adding a second fixed CAN message as well for those who prefer that. I guess I'm thinking (a) of myself in that it's something I would find quite useful with the tuning work I still have ahead of me and (b) it would also mean that those who wanted to could configure as they prefer without it being reliant on someone (me) crafting different ROMs to suit each purpose.

I've been thinking the last few days about the first couple of items you mention:

Variable address validation
You're right, it would be necessary to ensure that valid addresses were configured. It wouldn't be too bad if someone enters a memory address that references program in that the value, although not useful, would be returned. But if someone references memory that doesn't exist that would cause a page fault at least.

Validating the addresses are valid isn't actually too bad as there are only two address ranges which matter for this purpose. The DPR at 0x00ff8000 through about 0x00ff8400, and the RAM at 0x00ffe500 through 0x00ffefff (although would probably limit this to 0xffe600 through 0x00ffefff as the first 256 bytes of that range are used by the operating system itself and wouldn't be useful to expose on the CAN bus). So essentially each configured address needs to be checked to ensure it belongs to one of those two ranges.

The CPU has a long list of initialization functions it runs once on startup, and I'd inject this additional check there.


Size
Yes this was by far the biggest issue in my mind when I first thought about this. The fact that the 68k's RAM is byte addressable though vastly simplifies this issue.

(FYI heinzboehmer I realize that I don't need to explain to you the below in the great level of detail that I have. I've detailed it to this level for future reference for others :-) )



Let's take the example of AQ_REL. AQ_REL is a word (16 bit) variable which resides at 0x00ff80de



The 68k architecture is big endian, so the most significant byte comes first at 0x00ff80de and the least significant byte at 0x00ff80df.

Now if you want to access the variable the usual way in code is to do something like this:



move.w moves the word (16 bits) to the location specified. But this isn't the only way you can access the variable.

The MC68336, like all 68k processors, is byte addressable, which means that each and every byte in the RAM is individually addressable. Which means you can do things like this:

move.b (A2) =>AQ_REL, aq_rel_msb

This moves the byte at the address AQ_REL (0x00ff80de) refers to into an (imaginary) variable that I named aq_rel_msb for the purposes of this example.

If AQ_REL = 0x8765 then the value that would be in aq_rel_msb would be 0x87.

Likewise we could do (pseudo-code):

move.b (A2) =>AQ_REL+1, aq_rel_lsb

Which would move the byte at the address AQ_REL+1 (0x00ff80df) to the imaginary variable aq_rel_lsb. Using the same example then the value in aq_rel_msb would be 0x65.


For the purposes of configurable CAN messages this means that the entire thing can be simplified to byte addressing. If you wanted to expose AQ_REL on the CAN message you would configure the address parameters as follows:

CAN_Message_A_Byte_0 = 0x00ff80de
CAN_Message_A_Byte_1 = 0x00ff80df

Taking this approach means that no data conversion is needed, and therefore it's not necessary to handle overflows, etc. Simply pass on whatever the value is.


Error Reporting
In terms of error reporting my thinking would be to keep it simple and if any one memory address failed validation then the entire CAN message would be disabled. I appreciate that it's not ideal to not give any feedback to the user, but it would be clear that something was wrong, it would protect against any undesirable behavior, and for the sort of user taking advantage of this it shouldn't be too hard to troubleshoot whether the addresses have been configured correctly or not. No other error reporting would be necessary I don't think.


Testing
Agreed - testing would be manual and difficult. I certainly would be testing this extensively myself before making it available to anyone else as I agree that the risk is higher.


That's just my current thinking, I may or may not look further into this, and I certainly am grateful for you and Bry5on's thoughts on this - regardless of whether I decide to have a go at this or not it's a fun idea to consider, and in the mean time I can certainly whip up a second CAN message for you if you want to let me know exactly what you'd like on it?

Thanks!

[heinzboehmer]

Hmm, the idea sounds awesome but personally I’d stick with static settings. A new can of worms sounds a little more scary.

I was thinking about this today and I have to agree with Bryson.

Adding a new CAN message with statically mapped variables is almost trivial when it comes to potential bugs.

Adding the ability to dynamically assign these variables requires a significant amount of additional effort, if you want to build a truly comprehensive solution. And, of course, more complicated means more bug prone.

Just off the top of my head, these things would need to be addressed by this new dynamic infra:

• Variable address validation
  • How will you make sure the supplied addresses are actually for pertinent variables and not just random memory addresses?
  • What's stopping someone from assigning 0x0 as an address for one of the variables?
• Size
  • What's stopping someone from providing the addresses of 8 words?
  • What's the plan for data overflows?
    • Truncate?
    • Don't send the bad value?
    • Error out and skip sending the entire CAN message?
• Error reporting
  • If there is an error in the provided variable addresses, how do you communicate this?
    • Maybe reserve the first byte in one the CAN messages for flags?
    • Raise a new DTC?
    • Unfortunately, both of those ideas put the validation responsibility on the user, so what's stopping an inexperienced user from incorrectly setting this up, then ignoring the errors and trusting malformed data?
• Testing
  • How will you test this new code? It seems complex enough that (at least) unit tests would be required, but that's not exactly the easiest thing to do with the current development setup.

I think the idea of making this modular is super cool. Unfortunately, I also think that the amount of work required to make this work well is orders of magnitude bigger than just adding a new CAN message with statically mapped variables.

[Bry5on]

What software are you using to read the live CAN messages?

I’m using the built in logger within Gauge.S - great DIYer friendly product.

[ac427]

Beautiful. Thank you again my friend. Will test lambda integrator in the morning as it looks like I got the others sorted .. well, close on relative opening

What software are you using to read the live CAN messages?

[karter16]

A new can of worms

Haha I see what you did there 🤣

[Bry5on]

Hmm, the idea sounds awesome but personally I’d stick with static settings. A new can of worms sounds a little more scary.

[karter16]

In addition to adding another CAN frame I've also been having a think about how to make this feature a bit more generic.

We have limited frames we can use, and without a major rewrite to dynamically swap frames that is quite frankly beyond my capability and current understanding, that isn't going to change. I can already think of enough things that would be interesting to log (particularly during tuning, etc.) that they won't all fit into the available frames at once. Likewise I'm sure others will encounter similar situations.

I mentioned previously that I thought it would be possible to create the ability to provide for some degree of configuration which would mean 1 standard program binary with the ability to somewhat configure the messages via the partial binary. Having made 0x7D0 work and having done some more thinking around this I'm pretty sure that the following would be fairly easy - it would look something like this:

Master Program ROM

• Custom Program ROM with modified CAN table repurposing unused messages (propose using 0x701 as the current version does as well as 0x62F which is another safe option for the E46)
• Modify existing handler for 0x7D0 to genericise functionality
• Create new handler for "0x7D1" with generic functionality
• Insert call into CAN_Send_DME_ARBIDs_karter16 to send 0x7D1 as well and wrap both in conditional logic based on k_can_karter16_cfg

Master Tune Binary

• New parameters defined as follows:
  • k_can_karter16_cfg (bitfield to enable/disable 0x7D0 and 0x7D1)
  • An array of 8 32bit values which hold the addresses of the 8 bytes to be mapped to the 0x7D0 message
  • An array of 8 32bit values which hold the addresses of the 8 bytes to be mapped to the 0x7D1 message

And I think it's as simple as that. I initially was thinking that there would need to be logic to split up 16/32 bit values where required, but given the 68k is byte addressed it's actually very simple. If you want to send the word (2 bytes) at 0x1000 for example it's just two move byte instructions the first referencing 0x1000 and the second referencing 0x1001.

With the disassembly work that's been done it will be possible to make available a table of all valid variables in the Master + dual-ported-ram which users can then choose from themselves. Any variables from slave that aren't on the DPR would have to be inserted into the DPR (if there's any space left, I don't actually know yet) as a change to the Slave Program ROM, but again this could be done as a generic program rom change.

Edit: This approach also means that the tune binary remains backwards compatible with the standard 0401 program ROM. e.g. you could flash back to the original ROM without having to also change your tune binary. Likewise running the custom ROM with a standard 0401 tune would also be supported - 0x7D0 and 0x7D1 would simply be disabled.


What are the downsides of this

• It unfortunately doesn't solve the problem of restoring the standard functionality of 0x771 and 0x62F when the cfg parameter is disabled. I want to think more about whether this is possible.
• While the benefit is the ability to craft your own CAN messages, it means that everyone's definition of what 0x7D0 and 0x7D1 are will be different. I'm not sure whether this is a potential negative when it comes to sharing information, configs, etc.
• I'm still vaguely bothered that 0x771 might be some sort of required mechanism for some sort of BMW diagnostic process. I admit it seems unlikely, but I'm very wary of screwing up anything for anyone.

Anyway - keen for feedback on this idea/proposal

[heinzboehmer]

I selected this one to repurpose as its handler pointer was to 0x00000000. But I've been thinking about that and I can't completely rule out that it's an intentional mechanism to allow reset of the processor via CAN. It seems a bit crude and would be a crash to a hard reset without any preparation, but maybe...

Man, that would be a pretty crazy way to reset the DME!

Now I want to try and crash it over CAN

[karter16]

Datalog shows only these:

Thanks very much for this. I've been doing some more investigation. I have more work to do to understand more but have figured out the below points:

• There is a second set of 15 CAN frames which are mapped to a separate region in memory
• There is a second set of read/write handlers
• These read-write handlers are more complex and appear to be built to manage the sending/receipt of a series of messages over CAN
• I'm not sure yet but it's possible that this is support for ISO 15765-2 (ISO-TP/CAN-TP) or similar for transmission of error codes and diagnostics over CAN
• This functionality is present for both the Master (main CAN) and Slave (SMG CAN)

I've also been trying to figure out CAN ID 0x770 as it seems to be paired in some way with CAN ID 0x771 (which is what I've repurposed for 0x7D0) I selected this one to repurpose as its handler pointer was to 0x00000000. But I've been thinking about that and I can't completely rule out that it's an intentional mechanism to allow reset of the processor via CAN. It seems a bit crude and would be a crash to a hard reset without any preparation, but maybe...


Anyway. In terms of the more immediate question of a second CAN message. I'm pretty comfortable that we can use the slot for 0x62F (read) as this message is only ever received for a chassis configuration that isn't the E46.

[Bry5on]

Datalog shows only these:

[karter16]

What are the other IDs? We can log them to see if data exists, although neither of us are SMG.

That would be great thank you! - I think the most likely ones are:

0x62F
0x700
0x701
0x770
0x513

[Bry5on]

What are the other IDs? We can log them to see if data exists, although neither of us are SMG.