Previously

At the end of Part One, I found myself staring at this BMW F20 GWS (Gangwahlschalter - gear selector switch - thanks to everyone who helped dig up the German name) that I wanted to adapt into an Electric Vehicle conversion.

BMW GWS powered on bench with no LEDs lit

I had the GWS wired up and was able to see some CAN messages coming out, but despite trying I couldn't make it show any indication of the current gear. Nothing I tried sending had any measurable effect inside the GWS.

I'd put the gear selector aside on my desk and it was getting dangerously close to being retired into the Cupboard Of Forgotten Projects.

Responses To Part One

The first part got a huge response. I genuinely only expected this to resonate with a few CAN Bus loving oddballs, so I was amazed so many people responded.

Some very interesting comments, including a few new investigative techniques. I think the best idea that I hadn't thought of was making friends with a wrecker or a BMW mechanic and grabbing a CAN log from one of their cars. Maybe easier said than done, but still a good idea! It's easy to underestimate human solutions to technical problems.

Getting COVID

Anyway, back to the story. Having COVID sucks, do not recommend. Even with all the vaccines and being pretty healthy with a pretty mild infection, it really knocked me around.

At some point, sitting in a fog on the couch after watching one too many television series, I picked up my phone and resumed my obsessive hobby of "Searching for Information about BMW GWS". Miserable as I was, I noticed something...

DTC Clues

Previously, I had skimmed over the contents of BMW Device Trouble Codes (DTCs). DTCs are the short "error messages" that are read out by diagnostic scanners to get a summary of problems in the system and how frequently they happen. I'd previously stumbled across a bunch of random BMW DTCs posted to forums by people looking for help. Hidden in two different posts was:

Right there in the DTC description is 0x3FD - a value that looks exactly like a CAN message ID, sent by the transmission (EGS) to the gear selector (GWS) over the PT-CAN! How did I miss this?

There is no official list of BMW fault codes available online. Presumably BMW's own proprietary diagnostic software has this, but thankfully someone has taken this information from somewhere and put it on a website: bmwfault.codes.

By searching for more codes close to the value E09400 (and filling in a lot of captchas), I built up a list of "receiver GWS" CAN errors:

Code Description
E09400 No message (data, display, drivetrain, 0x3FD),
receiver GWS (PT-CAN), transmitter EGS (PT-CAN)
E09402 Message (data, display, drivetrain, 0x3FD) not current,
receiver GWS (PT-CAN), transmitter EGS (PT-CAN)
E09404 Message (data, display, drive train, 0x3FD) checksum error,
receiver GWS (PT-CAN), transmitter EGS (PT-CAN)
E09408 No message (terminals, 0x12F),
receiver GWS (PT-CAN), transmitter BDC (PT-CAN)
E0940A No message (dimming, 0x202),
receiver GWS (PT-CAN), transmitter BDC (PT-CAN)
E0940C No message (LCD brightness control, 0x393),
receiver GWS (PT-CAN), transmitter KOMBI (PT-CAN)
E09410 No message (data, display, transmission train, 0x3FD),
receiver GWS (PT-CAN2), transmitter EGS (PT-CAN2)
E09412 Message (data, display, transmission train, 0x3FD) not current,
receiver GWS (PT-CAN2), transmitter EGS (PT-CAN2)
E09420 No message (relative time, 0x328),
receiver GWS (PT-CAN), transmitter KOMBI (PT-CAN)
E09422 No message (vehicle condition, 0x3A0),
receiver GWS (PT-CAN), transmitter BDC-ZGM (PT-CAN)

The most promising are the first three, all related to Message (data, display, drivetrain, 0x3FD) ... transmitter EGS. There is E09400 for no message, E09402 for not current message (presumably the counter value is repeating), and E09404 for checksum error.

If it was possible to read the current DTCs from the GWS, then maybe this could be used as a feedback function to "fuzz" correct checksums and counter fields for the "data, display, drivetrain" message 0x3FD... It would certainly beat staring wide-eyed at the GWS unit, hoping it's going to blink!

The concept is little similar to a "decryption oracle" in information security, where an attacker may not have the key to decrypt a particular message but they do have a way to ask an "oracle" (say, a server) whether the encrypted message they have is valid or not, and then use that information to derive something else.

Diagnostic Access

So, how to read the diagnostic codes?

The official BMW software knows how to do this, but I don't have that software or a compatible hardware adaptor. Even if I had the software and an adaptor then I don't have a BMW to plug it into. This is particularly important in modern cars, where the excessive number of CAN buses means that the diagnostic connector may not connect to any "real" CAN bus at all. It connects to a "Diagnostic Bus" that runs back to a "CAN Gateway":

CAN Gateway diagram for BMW F01

In the blurry diagram above, the purple D-CAN lines marked "OBD" are the diagnostic connector CAN bus, the "ZGM" box is BMW's central gateway module, and the other CAN buses are all linked to it.

On some really fancy modern cars, the diagnostic part of this interface isn't even CAN - it's Ethernet or something else.

The CAN gateway reads the diagnostic messages and relays them to the correct CAN bus, possibly changing the message format as it does so. So it may not even work to connect the BMW diagnostic software to only a GWS, outside of a car and not behind a CAN gateway. It's also possible the software will flat-out refuse to talk to something that is obviously a gear selector in a trench coat, and not a real car.

Thankfully, someone named bill57p9 posted about reverse engineering the diagnostic interface on their BMW F46. Bill describes the Ethernet diagnostic messages sent to the gateway ("ZGM"), and the CAN message format sent out the other side. Woo!

The diagnostic CAN messages seem to be a modified version of the common Universal Diagnostic Services protocol (UDS, aka ISO-14229) running over the CAN ISO-TP protocol (ISO-15765-2). That's a lot of acronym soup, but the idea is that ISO-TP is a transport protocol layer on top of CAN that lets you create messages of any length with source and target module addresses, and both source and target modules will coordinate to deliver the message by sending CAN messages back and forth.

UDS is an application layer that can be implemented on top of multiple different transport protocols, including ISO-TP. It provides a way to make Remote Procedure Call style "service requests". A diagnostic tester module can make a request like "read diagnostic codes" or "enter programming mode" and receive back a "service response" with a status and data. These posts have a good explanation of the UDS protocol.

Diagram of UDS, ISO-TP and CAN as three layers

Searching further, I found a GitHub user named brandonros has published a list of "UDS functions" (service IDs) for BMW. Some of these look like standard UDS service IDs, and some look BMW-specific.

Usually, UDS uses CAN IDs as sender and receiver addresses: if a module in the car listens for diagnostic messages with CAN ID X, it will send back responses with CAN ID X+8. BMW uses a less common ISO-TP option called "Extended 11-bit Addressing". In his forum post, Bill explained that diagnostic CAN message IDs have value 0x600 plus an 8-bit number identifying the sender module of the message. Then the first byte of the CAN message data identifies the receiver module.

Wait a second, hadn't I seen some of these messages already? Recall in part one there were these intermittent unknown messages:

ID: 065e   f0 10 0a 62 17 04 e0 94

Yeah... these messages look like module 5e trying to send an ISO-TP message to module f0. Reading more of the data bytes we can see it's an ISO-TP message of length 0a bytes, starting with 62 17 04 e0 94. I don't know what this exact message means, I don't think it's a standard UDS request. We also only receive the first 5 bytes of the message because module f0 never replies to negotiate receiving the rest, as required by the ISO-TP protocol. My best guess, this is the GWS sporadically saying "hey, I've noticed something is wrong and I want to tell you about it!"

Module ID 5e also appears in another message sent by the GWS, the "Heartbeat" message. Note the ID and the last byte of the data:

ID: 055e   00 00 00 00 02 00 00 5e

Searching around confirms 5e is the GWS module ID on other BMW models. So I think this 0x500 series message on ID 0x55e is advertising "I'm here on diagnostic ID 5e!" as part of the module heartbeat.

(Afterword: In the discussion of Part One on Hacker News, halifaxbeard pointed out that the versatile "packet manipulation" tool Scapy has an impressive amount of BMW diagnostic definitions. I haven't looked into these, but they do look handy!)

Confirming UDS works

Taking the UDS standard "hard reset" request (11 01) from the list of "UDS functions" linked above, I made a quick python-can function to reset the GWS:

def hard_reset_simple(bus):
    # Message ID encodes sender '0xf1' (tester). Destination is in
    # first data byte - 5e for GWS, can also use 0xdf for broadcast
    # then send 2 byte UDS payload 0x11 0x01. response comes via ID 0x65e
    broadcast_msg = can.Message(
        arbitration_id=0x6F1, data=b"\x5e\x02\x11\x01",
        is_extended_id=False
    )
    bus.send(broadcast_msg)
    t0 = time.time()
    while time.time() < t0 + 1.0:
        r = bus.recv(0.1)
        if r and 0x600 <= r.arbitration_id < 0x700:
            print(r)

It worked! If I send the 0x202 Dimmer message that I'd discovered in Part One, the backlighting turns on. Then I send a hard reset, the GWS sends a positive response back, and finally the backlighting turns off again due to the reset:

GWS Blinking off due to hard reset

Progress, sweet blinky progress!

The very simple hand-rolled UDS request above only works because the request and the response each fit into one CAN message. By using the Python can-isotp package, it's possible to make UDS requests of any length and get back responses of any length:

def hard_reset(bus):
    return req_isotp(bus, b"\x11\x01")


def req_isotp(bus, req):
    with ThreadedBmwIsoTp(bus, 0x5E, 0xF1) as iso:
        r = iso.request(req, timeout=0.5)
        return r

(ThreadedBmwIsoTp is a class that enables the unusual "Extended 11-bits" ISO-TP addressing mode used by BMW, already supported by the isotp package.)

Reading DTCs

Going back to the list of BMW UDS service IDs, we see 19 02 0C;ReadDTC(0C). This is also a standard UDS request, to read DTCs that have certain bits set in their associated "status mask" (in this case 0x0C).

Each DTC stored in a module is associated with a DTCStatusMask field that records how and when this code was logged (for example, did it happen this "session" or in the past, is it happening intermittently or consistently, etc). The official list is locked away in the ISO-14229 UDS specification, but this PDF from Volkswagen has a good summary.

0x0C is a bit mask covering 0x04 "pending DTCs" (this is probably happening but not yet consistently) and 0x08 "confirmed DTCs" (this is definitely happening).

def get_dtcs(bus, status_mask=0x0C):
    return req_isotp(bus, [0x19, 0x02, status_mask])

The GWS sent back a response containing data like this:

59 02 ff e0 94 20 2f e0 94 22 2f e0 94 00 2f e0 94 02 2c e0 94 04 2c e0 94 08 2f e0 94 0a 2f e0 94 0c 2f e0 94 10 2f

After writing a simple decoding function this can be translated to a dict containing the DTCs and their status values:

{'e09420': 0x2f,
 'e09422': 0x2f,
 'e09400': 0x2f,
 'e09402': 0x2c,
 'e09404': 0x2c,
 'e09408': 0x2f,
 'e0940a': 0x2f,
 'e0940c': 0x2f,
 'e09410': 0x2f}

The DTCs include the ones we were expecting for the 0x3FD message! It's working...

At this point I started using the IPython terminal a lot for experimentation. It's a great command line REPL interface for exploratory programming, especially combined with the autoreload feature and configured to display all integers as hex.

Screenshot of IPython terminal showing functions used in this post

"Guess" the message structure

So, we finally have a way for the GWS to give us feedback about the valid or invalid CAN messages that it has seen. Going back to the "change one byte at a time" search method, a function sends each generated message sixteen times and then reads the diagnostic codes. If the E09404 "checksum error" DTC status has changed, the function prints the new status value and the message which caused it to change:

In [245]: bmw_gws_uds.search_valid_checksum(bus)
Base bytes 0x0
Changing offset 0
ID: 03fd   70 00 00 00 00 00 00 00
   E09404 -> 0x2e
ID: 03fd   71 00 00 00 00 00 00 00
   E09404 -> 0x2e
Changing offset 1
ID: 03fd   00 d7 00 00 00 00 00 00
   E09404 -> 0x2e
Changing offset 2
ID: 03fd   00 00 ab 00 00 00 00 00
   E09404 -> 0x2e
Changing offset 3
ID: 03fd   00 00 00 6e 00 00 00 00
   E09404 -> 0x2e
Changing offset 4
ID: 03fd   00 00 00 00 32 00 00 00
   E09404 -> 0x2e
Changing offset 5
Changing offset 6
Changing offset 7
Base bytes 0x1
Changing offset 0
ID: 03fd   33 01 01 01 01 01 01 01
   E09404 -> 0x2e
Changing offset 1
ID: 03fd   01 be 01 01 01 01 01 01
   E09404 -> 0x2e
Changing offset 2
ID: 03fd   01 01 d6 01 01 01 01 01
   E09404 -> 0x2e
[...]

Iterating through values 0x00 to 0xFF for each of the first five bytes in the message (offsets 0-4), there is always one message where the status of the E09404 "checksum error" changes from 0x2F to 0x2E. This means that the "Test Failed" status bit (0x01) has cleared showing that the error is not currently happening. (At the time I didn't bother to look the meaning of the bit up - any status change seemed good!)

So, all of the individual messages logged above should include a valid checksum. This process also reveals that the whole message is only five bytes long, because changing any later bytes in the message (offsets 5,6,7) never leads to a change in the checksum DTC status.

"Guess" the checksum

Building on the above, it's possible to write a function that sends any message data and returns True if the message has a valid checksum according to the GWS:

def verify_checksum(bus, payload):
    """Return 'True' if 'payload' appears to have a valid checksum
       according to the GWS DTC status!"""
    message = can.Message(arbitration_id=0x3FD, data=payload,
                          is_extended_id=False)
    for _ in range(16):
        bus.send(message)
        time.sleep(0.01)

    time.sleep(0.1)

    dtcs = get_dtcs(bus)
    csum_dtc = dtcs.get("e09404", "missing")
    return csum_dtc == 0x2e
In [247]: bmw_gws_uds.verify_checksum(bus, [0x70, 0x00, 0x00, 0x00, 0x00])
Out[247]: True

In [249]: bmw_gws_uds.verify_checksum(bus, [0x70, 0x00, 0x00, 0x00, 0x01])
Out[249]: False

Another incremental improvement, here's a function where you send any four byte message and it iterates through the 255 possible checksum bytes and tells you which one is valid:

def find_checksum(bus, message):
    if len(message) != 4:
        print("WARNING: Expected 4 byte message")
    for chksum in range(0x100):
        if verify_checksum(bus, [chksum] + list(message)):
            return chksum
    raise RuntimeError("No valid checksum found...")
In [256]: bmw_gws_uds.find_checksum(bus, [0x70, 0x00, 0x00, 0x03])
Out[256]: 0x27

In [257]: bmw_gws_uds.find_checksum(bus, [0x70, 0x00, 0x00, 0x13])
Out[257]: 0xea

In [258]: bmw_gws_uds.find_checksum(bus, [0x00, 0x00, 0x00, 0x00])
Out[258]: 0x32

In [259]: bmw_gws_uds.find_checksum(bus, [0x00, 0x00, 0x00, 0x01])
Out[259]: 0x2f

CRC8?

Now I had a list of valid messages including checksums, I was able to use CrcBeagle by Colin O'Flynn (of ChipWhisperer fame) to guess the actual checksum algorithm in use.

Initially this didn't work, as I'd accidentally put the last byte of each message as the checksum but the checksum was in the first byte position (same as the status message 0x197 from Part One). Once the order was correct, CrcBeagle worked immediately:

In [44]: crcbeagle.CRCBeagle().search(messages, crcs)
Input parameters:
    8-bit CRC size
    12 total messages, with:
      12 messages with 4 byte payload
NOTE: Output parameters may be specific to this message size only. Pass different length messages if possible.

Working on messages of 4 length: 
  Found single likely solution for differences of len=4, yah!
  Found single XOR-out value for len = 4: 0x70
********** example usage *************
import struct
from crccheck.crc import Crc8Base
crc = Crc8Base
def my_crc(message):
  crc._poly = 0x1D
  crc._reflect_input = False
  crc._reflect_output = False
  crc._initvalue = 0x0
  crc._xor_output = 0x70
  output_int = crc.calc(message)
  output_bytes = struct.pack("B", output_int)
  output_list = list(output_bytes)
  return (output_int, output_bytes, output_list)

m = [0, 0, 0, 0]
output = my_crc(m)
print(hex(output[0]))
**************************************
If you have multiple message lengths this solution may be valid for this only.

The CRC polynomial value is the same as the standard "CRC-8/SAE-J1850" algorithm, but this variant has custom "init" & "xor" parameters.

Time to write yet another experimental function: this one takes a 4 byte message and will calculate the checksum, send it to the GWS, and confirm that the GWS thinks the checksum is correct:

class BMW3FDCRC(crccheck.crc.Crc8Base):
    _poly = 0x1D
    _initvalue = 0x0
    _xor_output = 0x70

def bmw_3fd_crc(message):
    return BMW3FDCRC.calc(message) & 0xFF

def confirm_working_checksum(bus, message):
    """Simple function to use the DTCs to check if bmw_3fd_crc() returns correct values"""
    return verify_checksum(bus, [bmw_3fd_crc(message)] + message)
In [49]: bmw_gws_uds.confirm_working_checksum(bus, [0x23, 0x33, 0x44, 0x55])
Out[49]: True

In [50]: bmw_gws_uds.confirm_working_checksum(bus, [0x23, 0x33, 0x44, 0x77])
Out[50]: True

In [51]: bmw_gws_uds.confirm_working_checksum(bus, [0x23, 0x33, 0x44, 0x99])
Out[51]: True

Looking good...

"Guess" the counter field

The next DTC to tackle was E04902 "not current", probably relating to the counter field.

I probably could have guessed at this point that message byte 1 was the counter, after the checksum. Just in case, I made another function that ran through all four possible bytes or half bytes (masks 0xFF, 0xF0, 0x0F) and sent a sequence of messages "counting" with that byte, then looked for the E04902 not current DTC to see if it was still actively failing.

This confirmed that byte 1, or at least some of its bits, was the counter value.

Arbitrary data payloads

So, the message structure for ID 0x3FD looks like:

Byte Meaning
0 CRC8 (init=0x0, poly=0x1d, xor=0x53) of bytes 1-4
1 Counter value (must increment each message)
2 Payload byte 0
3 Payload byte 1
4 Payload byte 2

With yet another experimental function, it's possible to input arbitrary three byte payloads and have the function send a valid CAN message repeatedly every 100ms for 3 seconds, with the counter value incrementing correctly:

def send_gws_status(bus, status_bytes, tx_seconds=3):
    counter = 0
    t0 = time.time()

    while time.time() < t0 + tx_seconds:
        payload = [counter & 0xFF] + status_bytes
        payload = [bmw_3fd_crc(payload)] + payload
        message = can.Message(arbitration_id=0x3FD,
                              data=payload, is_extended_id=False)
        message.channel = 0
        bus.send(message)

        time.sleep(0.1)
        counter += 1
In [88]: bmw_gws_uds.send_gws_status(bus, [0x80, 0x00, 0x00])

In [89]: bmw_gws_uds.send_gws_status(bus, [0x40, 0x40, 0x40])

In [90]: bmw_gws_uds.send_gws_status(bus, [0x20, 0xff, 0xff])

In [91]: bmw_gws_uds.send_gws_status(bus, [0xa0, 0x00, 0x00])

Now it's possible to experiment with different values in the payload bytes, and see what happens. Sure enough, the gear selector started to respond:

BMW GWS with Drive LED lit

Decent effort to light an LED! Champagne, anyone?

Stay tuned for more about these CAN messages, and some "gearbox simulator" code to use it all together - coming soon in Part Three.

The code used in this post for experimentation, sometimes with some extra checks or features, can all be found in this car_hacking repository.

Thoughts on “BMW F Series Gear Selector, Part Two: Breakthrough