Karter16's Silbergrau E46 M3 Journal

[karter16]

Continuing to make progress:

- Lots of clean up and refinement of the elements on screen - still more to go, but I'm getting there.
- Quick pass over the UI to give it a nod to the classic "Windows Silver" #C0C0C0 colour that was prevalent when the MSS54 came into being.
- Came up with a nice way to visually represent the BRIF/ZIF/DIF data.
- Added a couple of programming actions "Clear Flash Counter" and "Insert AIF Record" which allows for the insertion of an additional AIF record into the AIF table.
- Added the ability to choose which segments of a full binary backup you want to restore from a full backup (e.g. data/tune, program, AIF sector, etc.)
- Filtering and search on logs is now functional.

[karter16]

Very cool stuff here!

So regarding baudrate -- I didn't experiment heavily but seems like sending this command after doing the seed/key routine sets the baudrate to 125k without doing anything particularly complicated?

Yep it does - however at this point the DME is in normal operating mode (eg the OS is running and it can't keep up with 38400 or 125000 baud rates particularly for bulk read or writes) once it's in boot mode it's not running the OS, literally just sits in a loop to process the DS2 commands and that seems to be what enables it to keep up. I would absolutely love to be proven wrong on this though or to find some cleaner way to get into boot mode. Having gone through the disassembly I'm pretty sure calling a valid wipe command is the only way (without creating a modified boot sector I mean)


Sent from my iPhone using Tapatalk

[terra]

Very cool stuff here!

So regarding baudrate -- I didn't experiment heavily but seems like sending this command after doing the seed/key routine sets the baudrate to 125k without doing anything particularly complicated?

[karter16]

Some progress updates:

- I played around with the idea of elevating to 38,400 baud for the initial identification, and backup read before entering 125,000 baud. When the DME is in normal operating mode it officially only supports 9600 baud, when it jumps to programming mode it sits in a tight loop and just processes DS2 commands and the interfaces DS2 communicates on. It's not having the overhead of the OSKAR operating system that means it can support 125,000 baud in this mode. I found even in normal OS mode 38,400 kinda worked provided the block size wasn't too big. I played around with this for a while but decided it wasn't stable enough to make an actual feature.

- The app flashing mechanism now fully supports sparse writes. The way flash works is you clear the entire segment which resets every byte to 0xFF. You then write in the new binary. If you pass DS2 a packet full of 0xFFs it'll dutifully go and write them all. The quick way to help speed things up is to skip any blocks full of 0xFFs. The better way is to determine a write plan that optimises dynamic block sizes, etc. to strike the right balance between not writing unnecessary 0xFFs but also not ending up with to much overhead from more DS2 write calls.

- I've rolled "Backups" into a more general "History" function. The app will now require a backup ahead of any destructive action where a backup doesn't already exist. Furthermore every read, write, etc. that is performed is stored as an "action" in history. This way it's easy to see exactly what you did and when, and roll back to a previous setup. I'm not sure whether anyone else really has any use for this, but it will make my own life easier and since I'm the one making this app it is now a feature :-)

- started working to simplify the UI. My approach with this was to throw all the data points on the page so I could get a good feel for what is useful where, now I'm going through re-organising and pruning anything that isn't needed.

[karter16]

I've figured it out!

So the short story is that there is no mechanism in the boot loader to put the DME into programming mode via DS2 any way other than calling a valid flash wipe.

But....

I've figured out how to get fast reads.

Essentially the trick is to read the Free Identifiers block between 0x4000 and 0x5FFF. Then wipe it, and then flash the contents back.

We can actually make this a quick process. There's actually only a couple of sub sections that hold data, but to be on the safe side the way my approach works is this.

1: When the user attempts a long read (e.g. read Tune, read full, etc.) and the connection is not already at 125000 baud, the app checks, for the current DME serial number, does it have a map of the contents of the free identifiers block. If it doesn't it takes a full backup of the block and maps the contents of it. this takes about 15 seconds.
2: Once it has the map it wipes the whole block and just writes back the non-0xFF bytes to save time. (wiping resets everything to 0xFF).
3: Then it proceeds with the read.

Even for a read of the partial/tune binary this makes the whole exercise MUCH quicker.

But we can do even better.

In future sessions when the user initiates a read (for the same DME serial number) and isn't at 125,000 baud it can lookup the previous backup to get the map of the contents. It still will actually read those regions from the DME in case anything has changed ( but only needs to read and write the specific sub-areas of the block that it knows are non-zero. (note I'm accounting for areas that actively change, flash counter, AIF, etc.)

Doing this means that in future sessions, the whole jump to 125000 baud takes approximately 3 seconds. Once it's done the connection is at 125000 baud for the rest of the session!

I've also tested and validated full program writes as well.

Current job is building out the app ui, safety checks, etc.

Few screens below - the UI is still under fairly heavy development - I know there's things that need sorting.








I'm finding this feature particularly useful. Ability to take backups at any point in time and then restore them. Coming in very handy as I build out new features and either intentionally or unintentionally corrupt the data in various blocks.




Expanded session log showing the speed of elevating to 125000 baud.

[karter16]

No point in another screenshot of the same thing, but have built out the following:

- CRC verification & correction for both Program and Tune
- Read and reset the flash counter (with complete backup and restore of the EEPROM block)
- Read the AIF sector (all history)
- Read BRIF/ZIF/ZIF Backup/DIF and verify integrity against backups.
- Flash the "full binary" program blocks (still testing this)

Still struggling to figure out the magic to get the DME into boot mode via DS2 call without erasing a sector first. I'll get there, just need to do some more experimenting. Have messed up a few times, but each time have been able to use the app to self-recover the situation - pretty cool

[karter16]

And now I can do fast writes as well :-)

Seems the DME only supports 125000 baud (BMW-FAST) when the DME is in programming mode, so now figuring out how to harmlessly trigger programming mode to get fast reads without having written something first.

[karter16]

We need to develop a MKIV launcher

Could easily fit all the important options with recreate the low res

Now that would be pretty cool!! I wonder how configurable idrive launcher is - it's been a while since I looked at it.

[0-60motorsports]

We need to develop a MKIV launcher

Could easily fit all the important options with recreate the low res

Yes! That would be awesome and a great solution because i dont see any way this BMW tech can survive much longer in our car LOL.

[Obioban]

2: I've replaced the Xtrons launcher with iDrive Launcher which is a fantastic, well put together, piece of software which further does wonders to make the unit feel like it fits in the E46 interior.

We need to develop a MKIV launcher

Could easily fit all the important options with recreate the low res

[0-60motorsports]

Today I got distracted and built my own app to connect to the DME over OBD/DS2.


Currently it can connect, retrieve system address and query for things like ZIF, AIF, etc.

It can also unlock to enable privileged commands, negotiate to change baud rate etc.

It can also read program and tune binaries.

Next step is to get it flashing 😱

That is amazing mate! Now only if someone would make an app for coding instead of people like me struggling to do simple things such as CSL Shifting Software on and SMG or just basic coding like cluster etc....

[karter16]

Today I got distracted and built my own app to connect to the DME over OBD/DS2.


Currently it can connect, retrieve system address and query for things like ZIF, AIF, etc.

It can also unlock to enable privileged commands, negotiate to change baud rate etc.

It can also read program and tune binaries.

Next step is to get it flashing 😱

[karter16]

I did a final* assembly of the gauge.s enclosure today. I say final with an asterisk as I have a a couple of final dust-coats of paint still to do on the enclosure, which, if it works like last time, will lighten up the finished surface a bit and bring the colour closer to the BMW dash colour.

Anyway - here's the *almost* finished enclosure:







I thought about finishing the entire enclosure, not just the visible face but decided that was ridiculous.


I didn't take photos of preparing the screen, but the steps were basically to take a new display, apply a red film layer, and then a layer of the same anti-glare film that I used on the xtrons. I don't like the shiny look of the gauge.s screen with the red film applied by itself, so decided to apply the anti-glare as well - which makes it look much better.

Here's a few photos of assembly - screen in:



Screen retainer:



Logic board in and sub-cover installed:



Back cover on:



Closeup of the connector:



From the front: this is the second reason why the project is quite completely finished. The red film I received has some micro impacts to it which result in unavoidable bubbles in the anti-glare layer. Not cool, I'll order some more red film and redo this. Carried on with the install cause I wanted to see the end result:






And here's how it looks in the car:









Worth noting that the softness and glow of the digits on the display is the iPhone camera struggling with the contrast, it doesn't actually look soft and dispersed like that in person.

I'm really pleased with how this has come together - couple more minor bits as mentioned to get this completely finished.


I also finally got pure-ftpd setup today (thanks to Bry5on who figured out that pure-ftpd works well with gauge.s's use of some less common ftp commands).

[karter16]

That would be a bit of a problem if the adaptations were reset lol

Maybe it is not a great idea to reset adaptations with the 75C, 70C and 55C thermostats.

I'd say it's more of a problem to not reset.

Whatever rf_psau_i is at the time the engine is switched off will be written to the adaption EEPROM. If rf_psau_i is never updated again (as a result of TMOT = 80 not being hit) then that adaption value will stick for forever. If environmental conditions have changed in the mean time then rf_psau_i (saved as an adaption) could be actively throwing your RF calculation away from where it should be.


I'd suggest for those running a lower temp thermostat to adjust what I've called k_rf_p_saug_i_tmot_min (0xE5EC in the tune file) to a lower threshold temperature.


Edit: I just had a quick look at the other parameters that use tmot as a threshold to trigger various behaviours. Seems most other modules use TMOT == 70 to 75ish. Probably means that those running lower temp thermostats on the street should consider adjusting some of these other parameters as well, depending on where they see TMOT sitting most of the time.

[Slideways]

Yup it'll not update and rf_psau_i will be stuck at whatever it was last set at as an adaption. Which, depending on whether conditions have changed, will either hinder or help.

That would be a bit of a problem if the adaptations were reset lol

Maybe it is not a great idea to reset adaptations with the 75C, 70C and 55C thermostats.