Karter16's Silbergrau E46 M3 Journal

[karter16]

Sounds like the next quick feature add is to build out the entire test suite that makes up INPA and DIS?

hahahaha stop giving me shiny things to chase. I need to finish this current shiny thing first

[heinzboehmer]

Sounds like the next quick feature add is to build out the entire test suite that makes up INPA and DIS?

[karter16]

I'm trying to quickly finish the current feature set

So I possibly lied...

Reading/clearing error memory proved surprisingly easy to build out. I started with a quick PoC cause I was interested and it was so successful I just finished it.





Also added in support for sensing the KL30 and KL15 lines cause why not.

[0-60motorsports]

Nope, the software takes a backup of everything that's in that segment of memory, wipes the segment, and then writes everything back except the bytes specific to the flash counter.


Sent from my iPhone using Tapatalk

That is great! Thanks again.

[karter16]

karter16 Would the DME be wiped if we reset the flash counter only?

Nope, the software takes a backup of everything that's in that segment of memory, wipes the segment, and then writes everything back except the bytes specific to the flash counter.


Sent from my iPhone using Tapatalk

[0-60motorsports]

karter16 Would the DME be wiped if we reset the flash counter only?

[karter16]

Id be interested in giving it a shot on an MSS54, but I don't have a way to BDM if needed. BUT I'm not worried about that as I have some spare DMEs and nothing critical in use. What do you need from me? Shoot me an email [email protected] if its easier.

Will there be an option to virginize the EWS, so we can sync a used DME with a "new" to it car?

Awesome that would be great thank you! First round of testing will be read-only anyway, as those logs should give me a fair idea of how it's all looking. After work my time I'll add you to the google drive folder that I'm using for the beta files and will flick you an email with some testing instructions - thanks very much!

Will there be an option to virginize the EWS, so we can sync a used DME with a "new" to it car?

Yep there will be for sure. Not as part of 1.0. I'm trying to quickly finish the current feature set and get it out there and then can iterate with additional features. Virginization and EWS sync are high on the list as appreciate this will be useful for quite a few people.

[George Hill]

Id be interested in giving it a shot on an MSS54, but I don't have a way to BDM if needed. BUT I'm not worried about that as I have some spare DMEs and nothing critical in use. What do you need from me? Shoot me an email [email protected] if its easier.

Will there be an option to virginize the EWS, so we can sync a used DME with a "new" to it car?

[karter16]

Continuing to make good progress towards getting this releasable.

heinzboehmer and Bry5on have kindly user-tested and given me excellent feedback to help improve the app and make it as user-friendly as possible. Laundry list of updates below:

• Fixed an edge case that cause DS2 commands to be interleaved causing a connection drop
• Added dialog to prevent app closure while DS2 connection active, if a wipe/write action is underway the user must wait until it is finished in order to leave the DME in a clean state, in any other state the user can choose to disconnect and close the app.
• Added the Flash Counter control to the Write page for better visibility, also warn the user if they attempt a write action with less than 5 flash slots remaining.
• Added graceful handling of cable/interface disconnection within the app.
• Found an issue with the Cancel button and ensured it now waits for the current DS2 command to conclude gracefully before cancelling the action.
• Disabled the cancel button while erase/write activities are in progress.
• Added full validation of Data/Program binaries. The source file is checked for version information and the app will not allow writing a program file that doesn't match the bootloader on the DME and likewise won't allow writing a data/tune file that doesn't match the program on the DME.
• Improved coverage of logging messages.
• Finished the Settings page.
• Relaxed validation (with extra user warnings) when restoring the Service segment (AIF, etc.) to allow restoration when the AIF region has been corrupted and the DME variant and VIN cannot be determined.
• Added better handling when the user connects but the car ignition is not on.
• Bryson noted that for a first time user the entry into fast read, and then the subsequent read of a tune file takes about the same amount of time as just reading the tune at 9600 baud. Have expanding settings to allow user to independently toggle fast read on and off for tune/program reads.
• Resolved a misunderstanding on my part about the persistence of the ZIF backup.

So what remains? More testing. The app has been pretty well tested on MSS54HPs, it needs testing with MSS54s. If Heinz gets a chance I think he's going to give it a go on his backup DME, if anyone else has a MSS54, the ability to BDM it if it all goes horribly wrong, and is game to be a guinea pig then let me know :-)

[0-60motorsports]

Love it! Nice work

[karter16]

Continuing to make progress:

- Lots of clean up and refinement of the elements on screen - still more to go, but I'm getting there.
- Quick pass over the UI to give it a nod to the classic "Windows Silver" #C0C0C0 colour that was prevalent when the MSS54 came into being.
- Came up with a nice way to visually represent the BRIF/ZIF/DIF data.
- Added a couple of programming actions "Clear Flash Counter" and "Insert AIF Record" which allows for the insertion of an additional AIF record into the AIF table.
- Added the ability to choose which segments of a full binary backup you want to restore from a full backup (e.g. data/tune, program, AIF sector, etc.)
- Filtering and search on logs is now functional.

[karter16]

Very cool stuff here!

So regarding baudrate -- I didn't experiment heavily but seems like sending this command after doing the seed/key routine sets the baudrate to 125k without doing anything particularly complicated?

Yep it does - however at this point the DME is in normal operating mode (eg the OS is running and it can't keep up with 38400 or 125000 baud rates particularly for bulk read or writes) once it's in boot mode it's not running the OS, literally just sits in a loop to process the DS2 commands and that seems to be what enables it to keep up. I would absolutely love to be proven wrong on this though or to find some cleaner way to get into boot mode. Having gone through the disassembly I'm pretty sure calling a valid wipe command is the only way (without creating a modified boot sector I mean)


Sent from my iPhone using Tapatalk

[terra]

Very cool stuff here!

So regarding baudrate -- I didn't experiment heavily but seems like sending this command after doing the seed/key routine sets the baudrate to 125k without doing anything particularly complicated?

[karter16]

Some progress updates:

- I played around with the idea of elevating to 38,400 baud for the initial identification, and backup read before entering 125,000 baud. When the DME is in normal operating mode it officially only supports 9600 baud, when it jumps to programming mode it sits in a tight loop and just processes DS2 commands and the interfaces DS2 communicates on. It's not having the overhead of the OSKAR operating system that means it can support 125,000 baud in this mode. I found even in normal OS mode 38,400 kinda worked provided the block size wasn't too big. I played around with this for a while but decided it wasn't stable enough to make an actual feature.

- The app flashing mechanism now fully supports sparse writes. The way flash works is you clear the entire segment which resets every byte to 0xFF. You then write in the new binary. If you pass DS2 a packet full of 0xFFs it'll dutifully go and write them all. The quick way to help speed things up is to skip any blocks full of 0xFFs. The better way is to determine a write plan that optimises dynamic block sizes, etc. to strike the right balance between not writing unnecessary 0xFFs but also not ending up with to much overhead from more DS2 write calls.

- I've rolled "Backups" into a more general "History" function. The app will now require a backup ahead of any destructive action where a backup doesn't already exist. Furthermore every read, write, etc. that is performed is stored as an "action" in history. This way it's easy to see exactly what you did and when, and roll back to a previous setup. I'm not sure whether anyone else really has any use for this, but it will make my own life easier and since I'm the one making this app it is now a feature :-)

- started working to simplify the UI. My approach with this was to throw all the data points on the page so I could get a good feel for what is useful where, now I'm going through re-organising and pruning anything that isn't needed.

[karter16]

I've figured it out!

So the short story is that there is no mechanism in the boot loader to put the DME into programming mode via DS2 any way other than calling a valid flash wipe.

But....

I've figured out how to get fast reads.

Essentially the trick is to read the Free Identifiers block between 0x4000 and 0x5FFF. Then wipe it, and then flash the contents back.

We can actually make this a quick process. There's actually only a couple of sub sections that hold data, but to be on the safe side the way my approach works is this.

1: When the user attempts a long read (e.g. read Tune, read full, etc.) and the connection is not already at 125000 baud, the app checks, for the current DME serial number, does it have a map of the contents of the free identifiers block. If it doesn't it takes a full backup of the block and maps the contents of it. this takes about 15 seconds.
2: Once it has the map it wipes the whole block and just writes back the non-0xFF bytes to save time. (wiping resets everything to 0xFF).
3: Then it proceeds with the read.

Even for a read of the partial/tune binary this makes the whole exercise MUCH quicker.

But we can do even better.

In future sessions when the user initiates a read (for the same DME serial number) and isn't at 125,000 baud it can lookup the previous backup to get the map of the contents. It still will actually read those regions from the DME in case anything has changed ( but only needs to read and write the specific sub-areas of the block that it knows are non-zero. (note I'm accounting for areas that actively change, flash counter, AIF, etc.)

Doing this means that in future sessions, the whole jump to 125000 baud takes approximately 3 seconds. Once it's done the connection is at 125000 baud for the rest of the session!

I've also tested and validated full program writes as well.

Current job is building out the app ui, safety checks, etc.

Few screens below - the UI is still under fairly heavy development - I know there's things that need sorting.








I'm finding this feature particularly useful. Ability to take backups at any point in time and then restore them. Coming in very handy as I build out new features and either intentionally or unintentionally corrupt the data in various blocks.




Expanded session log showing the speed of elevating to 125000 baud.