Category Archives: EPC

Samsung-Sysdump-IMS-Debug-DM-View_Cropped

VoLTE/IMS Debugging on Samsung Handsets using Sysdump & Samsung IMS Logger

Samsung handsets have a feature built in to allow debugging from the handset, called Sysdump.

Entering *#9900# from the Dialing Screen will bring up the Sysdump App, from here you can dump logs from the device, and run a variety of debugging procedures.

Samsung share information about this app publicly on their website,

Sysdump App in Samsung handsets used for debugging the device

But for private LTE operators, the two most interesting options are by far the TCPDUMP START option and IMS Logger, but both are grayed out.

Tapping on them asks for a one-time password and has a challenge key.

OTP Authentication required to unlock IMS Debugging and TCPDUMP on Samsung Sysdump tool

These options are not available in the commercial version of the OS and need to be unlocked with a one time key generated by a tool Samsung for unlocking engineering firmware on handsets.

Luckily this authentication happens client side, which means we can work out the password it’s expecting.

For those interested I’ve done a write up of how I reversed the password validation algorithm to take the key given in the OTP challenge and generate a valid response.

For those who just want to unlock these features you can click here to run the tool that generates the response.

Once you’ve entered the code and successfully unlocked the IMS Debugging tool there’s a few really cool features in the hamburger menu in the top right.

DM View

This shows the SIP / IMS Messaging and the current signal strength parameters (used to determine which RAN type to use (Ie falling back from VoLTE to UMTS / Circuit Switched when the LTE signal strength drops).

Screenshot of Samsung Sysdump tool in the IMS Debug - DM View section

Tapping on the SIP messages expands them and allows you to see the contents of the SIP messages.

Viewing SIP Messaging directly from the handset

Interesting the actual nitty-gritty parameters in the SIP headers are missing, replaced with X for anything “private” or identifiable.

Luckily all this info can be found in the Pcap.

The DM View is great for getting a quick look at what’s going on, on the mobile device itself, without needing a PC.

Logging

The real power comes in the logging functions,

There’s a lot of logging options, including screen recording, TCPdump (as in Packet Captures) and Syslog logging.

From the hamburger menu we can select the logging parameters we want to change.

Settings for Samsung IMS Logger

From the Filter Options menu we can set what info we’re going to log,

Filter options used in Dump output of Samsung IMS Logger application

If you’re leaning about VoLTE & IMS networks, or building your own, I’d suggest checking out my other posts on the topic.

PLMN Identity from Wireshark in Hex Form

PLMN Identifier Calculation (MCC & MNC to PLMN)

Note: This didn’t handle 3 digit MNCs, an updated version is available here and in the code sample below.

The PLMN Identifier is used to identify the radio networks in use, it’s made up of the MCC – Mobile Country Code and MNC – Mobile Network Code.

But sadly it’s not as simple as just concatenating MCC and MNC like in the IMSI, there’s a bit more to it.

In the example above the Tracking Area Identity includes the PLMN Identity, and Wireshark has been kind enough to split it out into MCC and MNC, but how does it get that from the value 12f410?

This one took me longer to work out than I’d like to admit, and saw me looking through the GSM spec, but here goes:

PLMN Contents: Mobile Country Code (MCC) followed by the Mobile Network Code (MNC).
Coding: according to TS GSM 04.08 [14].

If storage for fewer than the maximum possible number n is required, the excess bytes shall be set to ‘FF’. For instance, using 246 for the MCC and 81 for the MNC and if this is the first and only PLMN, the contents reads as follows: Bytes 1-3: ’42’ ‘F6′ ’18’ Bytes 4-6: ‘FF’ ‘FF’ ‘FF’ etc.

TS GSM 04.08 [14].

Making sense to you now? Me neither.

Here’s the Python code I wrote to encode MCC and MNCs to PLMN Identifiers and to decode PLMN into MCC and MNC, and then we’ll talk about what’s happening:

def Reverse(str):
    stringlength=len(str)
    slicedString=str[stringlength::-1]
    return (slicedString)    

def DecodePLMN(plmn):
    print("Decoding PLMN: " + str(plmn))
    
    if "f" in plmn:
        mcc = Reverse(plmn[0:2]) + Reverse(plmn[2:4]).replace('f', '')
        print("Decoded MCC: " + str(mcc))
        mnc = Reverse(plmn[4:6])
    else:
        mcc = Reverse(plmn[0:2]) + Reverse(plmn[2:4][1])
        print("Decoded MCC: " + str(mcc))
        mnc = Reverse(plmn[4:6]) + str(Reverse(plmn[2:4][0]))
    print("Decoded MNC: " + str(mnc))
    return mcc, mnc

def EncodePLMN(mcc, mnc):
        plmn = list('XXXXXX')
        if len(mnc) == 2:
            plmn[0] = Reverse(mcc)[1]
            plmn[1] = Reverse(mcc)[2]
            plmn[2] = "f"
            plmn[3] = Reverse(mcc)[0]
            plmn[4] = Reverse(mnc)[0]
            plmn[5] = Reverse(mnc)[1]
            plmn_list = plmn
            plmn = ''
        else:
            plmn[0] = Reverse(mcc)[1]
            plmn[1] = Reverse(mcc)[2]
            plmn[2] = Reverse(mnc)[0]
            plmn[3] = Reverse(mcc)[0]
            plmn[4] = Reverse(mnc)[1]
            plmn[5] = Reverse(mnc)[2]
            plmn_list = plmn
            plmn = ''
        for bits in plmn_list:
            plmn = plmn + bits
        print("Encoded PLMN: " + str(plmn))
        return plmn

EncodePLMN('505', '93')
EncodePLMN('310', '410')

DecodePLMN("05f539")
DecodePLMN("130014")

In the above example I take MCC 505 (Australia) and MCC 93 and generate the PLMN ID 05f539.

The first step in decoding is to take the first two bits (in our case 05 and reverse them – 50, then we take the third and fourth bits (f5) and reverse them too, and strip the letter f, now we have just 5. We join that with what we had earlier and there’s our MCC – 505.

Next we get our MNC, for this we take bytes 5 & 6 (39) and reverse them, and there’s our MNC – 93.

Together we’ve got MCC 505 and MNC 93.

The one answer I’m still looking for; why not just encode 50593? What is gained by encoding it as 05f539?

PyHSS Update – MongoDB Backend & SQN Resync

After a few quiet months I’m excited to say I’ve pushed through some improvements recently to PyHSS and it’s growing into a more usable HSS platform.

MongoDB Backend

This has a few obvious advantages – More salable, etc, but also opens up the ability to customize more of the subscriber parameters, like GBR bearers, etc, that simple flat text files just wouldn’t support, as well as the obvious issues with threading and writing to and from text files at scale.

Knock knock.

Race condition.

Who’s there?

— Threading Joke.

For now I’m using the Open5GS MongoDB schema, so the Open5Gs web UI can be used for administering the system and adding subscribers.

The CSV / text file backend is still there and still works, the MongoDB backend is only used if you enable it in the YAML file.

The documentation for setting this up is in the readme.

SQN Resync

If you’re working across multiple different HSS’ or perhaps messing with some crypto stuff on your USIM, there’s a chance you’ll get the SQN (The Sequence Number) on the USIM out of sync with what’s on the HSS.

This manifests itself as an Update Location Request being sent from the UE in response to an Authentication Information Answer and coming back with a Re-Syncronization-Info AVP in the Authentication Info AVP. I’ll talk more about how this works in another post, but in short PyHSS now looks at this value and uses it combined with the original RAND value sent in the Authentication Information Answer, to find the correct SQN value and update whichever database backend you’re using accordingly, and then send another Authentication Information Answer with authentication vectors with the correct SQN.

SQN Resync is something that’s really cryptographically difficult to implement / confusing, hence this taking so long.

What’s next? – IMS / Multimedia Auth

The next feature that’s coming soon is the Multimedia Authentication Request / Answer to allow CSCFs to query for IMS Registration and manage the Cx and Dx interfaces.

Code for this is already in place but failing some tests, not sure if that’s to do with the MAA response or something on my CSCFs,

Keep an eye on the GitLab repo!

Authentication Vectors and Key Distribution in LTE

Querying Auth Credentials from USIM/SIM cards

LTE has great concepts like NAS that abstract the actual transport layers, so the NAS packet is generated by the UE and then read by the MME.

One thing that’s a real headache about private LTE is the authentication side of things. You’ll probably bash your head against a SIM programmer for some time.

As your probably know when connecting to a network, the UE shares it’s IMSI / TIMSI with the network, and the MME requests authentication information from the HSS using the Authentication Information Request over Diameter.

The HSS then returns a random value (RAND), expected result (XRES), authentication token (AUTN) and a KASME  for generating further keys,

The RAND and AUTN values are sent to the UE, the USIM in the UE calculates the RES (result) and sends it back to the MME. If the RES value received by the MME is equal to the expected RES (XRES) then the subscriber is mutually authenticated.

The osmocom guys have created a cool little utility called osmo-sim-auth, which allows you to simulate the UE’s baseband module’s calls to the USIM to authenticate.

Using this tool I was able to plug a USIM into my USIM reader, using the Diameter client built into PyHSS I was able to ask for Authentication vectors for a UE using the Authentication Information Request to the HSS and was sent back the Authentication Information Answer containing the RAND and AUTN values, as well as the XRES value.

Wireshark Diameter Authentication Information Response message body looking at the E-UTRAN vectors
Diameter – Authentication Information Response showing E-UTRAN Vectors

Then I used the osmo-sim-auth app to query the RES and RAND values against the USIM.

Osmocom's USIM Test tool - osmo-sim-auth

The RES I got back matched the XRES, meaning the HSS and the USIM are in sync (SQNs match) and they mutually authenticated.

Handy little tool!

Information stored on USIM / SIM Card for LTE / EUTRAN / EPC - K key, OP/OPc key and SQN Sequence Number

HSS & USIM Authentication in LTE/NR (4G & 5G)

I talked a bit in my last post about using osmo-sim-auth to authenticate against a USIM / SIM card when it’s not in a phone,

I thought I’d expand a little on how the Crypto side of things works in LTE & NR (also known as 4G & 5G).

Authentication primarily happens in two places, one at each end of the network, the Home Subscriber Server and in the USIM card. Let’s take a look at each of them.

On the USIM

On the USIM we’ve got two values that are entered in when the USIM is provisioned, the K key – Our secret key, and an OPc key (operator key).

These two keys are the basis of all the cryptography that goes on, so should never be divulged.

The only other place to have these two keys in the HSS, which associates each K key and OPc key combination with an IMSI.

The USIM also stores the SQN a sequence number, this is used to prevent replay attacks and is incremented after each authentication challenge, starting at 1 for the first authentication challenge and counting up from there.

On the HSS

On the HSS we have the K key (Secret key), OPc key (Operator key) and SQN (Sequence Number) for each IMSI on our network.

Each time a IMSI authenticates itself we increment the SQN, so the value of the SQN on the HSS and on the USIM should (almost) always match.

Authentication Options

Let’s imagine we’re designing the authentication between the USIM and the Network; let’s look at some options for how we can authenticate everyone and why we use the process we use.

Failed Option 1 – Passwords in the Clear

The HSS could ask the USIM to send it’s K and OPc values, compare them to what the HSS has in place and then either accept or reject the USIM depending on if they match.

The obvious problem with this that to send this information we broadcast our supposedly secret K and OPc keys over the air, so anyone listening would get our secret values, and they’re not so secret anymore.

This is why we don’t use this method.

Failed Option 2 – Basic Crypto

So we’ve seen that sending our keys publicly, is out of the question.

The HSS could ask the USIM to mix it’s K key and OPc key in such a way that only someone with both keys could unmix them.

This is done with some cryptographic black magic, all you need to know is it’s a one way function you enter in values and you get the same result every time with the same input, but you can’t work out the input from the result.

The HSS could then get the USIM to send back the result of mixing up both keys, mix the two keys it knows and compare them.

The HSS mixes the two keys itself, and get’s it’s own result called XRES (Expected Result). If the RES (result) of mixing up the keys by the USIM is matches the result when the HSS mixes the keys in the same way (XRES (Expected Result)), the user is authenticated.

The result of mixing the keys by the USIM is called RES (Result), while the result of the HSS mixing the keys is called XRES (Expected Result).

This is a better solution but has some limitations, because our special mixing of keys gets the same RES each time we put in our OPc and K keys each time a subscriber authenticates to the network the RES (result) of mixing the keys is going to be the same.

This is vulnerable to replay attacks. An attacker don’t need to know the two secret keys (K & OPc) that went into creating the RES (resulting output) , the attacker would just need to know the result of RES, which is sent over the air for anyone to hear.
If the attacker sends the same RES they could still authenticate.

This is why we don’t use this method.

Failed Option 3 – Mix keys & add Random

To prevent these replay attacks we add an element of randomness, so the HSS generates a random string of garbage called RAND, and sends it to the USIM.

The USIM then mixes RAND (the random string) the K key and OPc key and sends back the RES (Result).

Because we introduced a RAND value, every time the RAND is different the RES is different. This prevents against the replay attacks we were vulnerable to in our last example.

If the result the USIM calculated with the K key, OPc key and random data is the same as the USIM calculated with the same K key, OPc key and same random data, the user is authenticated.

While an attacker could reply with the same RES, the random data (RAND) will change each time the user authenticates, meaning that response will be invalid.

While an attacker could reply with the same RES, the random data (RAND) will change each time the user authenticates, meaning that response will be invalid.

The problem here is now the network has authenticated the USIM, the USIM hasn’t actually verified it’s talking to the real network.

This is why we don’t use this method.

GSM authentication worked like this, but in a GSM network you could setup your HLR (The GSM version of a HSS) to allow in every subscriber regardless of what the value of RES they sent back was, meaning it didn’t look at the keys at all, this meant attackers could setup fake base stations to capture users.

Option 4 – Mutual Authentication (Real World*)

So from the previous options we’ve learned:

  • Our network needs to authenticate our subscribers, in a way that can’t be spoofed / replayed so we know who to bill & where to route traffic.
  • Our subscribers need to authenticate the network so they know they can trust it to carry their traffic.

So our USIM needs to authenticate the network, in the same way the network authenticates the USIM.

To do this we introduce a new key for network authentication, called AUTN.

The AUTN key is generated by the HSS by mixing the secret keys and RAND values together, but in a different way to how we mix the keys to get RES. (Otherwise we’d get the same key).

This AUTN key is sent to the USIM along with the RAND value. The USIM runs the same mixing on it’s private keys and RAND the HSS did to generate the AUTN , except this is the USIM generated – An Expected AUTN key (XAUTN). The USIM compares XAUTN and AUTN to make sure they match. If they do, the USIM then knows the network knows their secret keys.

The USIM then does the same mixing it did in the previous option to generate the RES key and send it back.

The network has now authenticated the subscriber (HSS has authenticated the USIM via RES key) and the subscriber has authenticated the USIM (USIM authenticates HSS via AUTN key).

*This is a slightly simplified version of how EUTRAN / LTE authentication works between the HSS and the USIM – In reality there are a few extra values, such as SQN to take into consideration and the USIM talks to to the MME not the HSS directly.

I’ll do a follow up post covering the more nitty-gritty elements, AMF and SQN fields, OP vs OPc keys, SQN Resync, how this information is transfered in the Authentication Information Answer and how KASME keys are used / distributed.

Why GTP for Mobile Networks?

Let’s take a look at GTP, the workhorse of mobile user plane packet data.

This post covers all generations of mobile data (2.5 -> 5G), so I’m using generic terms.

GSM, UMTS, LTE & NR all have one protocol in common – GTP – The GPRS Tunneling Protocol.

So why do every generation of mobile data networks from GSM/GPRS in 2000, to 5G NR Standalone in 2020, rely on this one protocol for transporting user data?

So Why GTP?

GTP – the GPRS Tunnelling Protocol, is what encapsulates and tunnels IP packets from the internet / packet data network, to and from the User.

So why encapsulate the packets? What if the Base Station had access to the internet and routed the traffic to the users?

Let’s say we did that, we’d have to have large pools of IP addresses available at each Base Station and when a user connected they’d be assigned an IP Address and traffic for these users would be routed to the Base Station which would forward it onto the user.

This would work well until a user moves from one Base Station to another, when they’d have to get a new IP Address allocated.

TCP/IP was never designed to be mobile, an IP address only exists in a single location.

Breaking out traffic directly from a base station would have other issues, such as no easy way to enforce QoS or traffic policies, meter usage, etc.

How to fix IP’s lack of mobility? GTP.

GTP addressed the mobility issue by having a single fixed point the IP Address is assigned to (In GSM/GRPS/UMTS this is the Gateway GPRS Support Node, in LTE this is the P-GW and in 5G-SA this is the UPF), which encapsulates IP traffic to/from a mobile user into GTP Packet.

You can think of GTP like GRE or any of the other common encapsulation protocols, wrapping up the IP packets into a GTP packet which we can rerouted to different Base Stations as the users move from being served by one Base Station to another.

This easy redirecting / rerouting of user traffic is why GTP is used for NR (5G), LTE (4G), UMTS (3G) & GPRS (2.5G) architectures.

GTP Packets

When looking at a GTP packet of user data you’d be forgiven for thinking nothing much goes on,

Example GTP packet containing a DNS query

Like in most tunneling / encapsulation protocols we’ve got the original network / protocol stack of IPv4 and UDP, and a payload of a GTP packet.

The packet itself is pretty bare bones, there’s flags, denoting a few basics like version number, the message type (T-PDU), the length of the GTP packet and it’s payload (used for delineating the end of the payload), a sequence number an a Tunnel Endpoint Identifier (TEID).

In the payload, we can see the network / protocol stack and application layer of the contents of the GTP packet.

From a mobility standpoint, the beauty of GTP is that it takes IP packets and puts them into a media stream of sorts, with out of band signalling, this means we can change the parameters of our GTP stream easily without touching the encapsulated IP Packet.

When a UE moves from one base station to another, all that has to happen is the destination the GTP packets are sent to is changed from the old base station to the new base station. This is signalled using GTP-C in GPRS/UMTS, GTPv2-C in LTE and HTTP in 5G-SA.

Traffic to and from the UE would look the same as the screenshot above, the only difference would be the first IPv4 address would be different, but the IPv4 address in the GTP tunnel would be the same.

Roll your own USIMs for Private LTE Networks

I wrote a while ago about USIM basics and talked about what each of the fields stored on a USIM manage, but I thought I’d talk a little about my adventures in getting custom USIMs.

I started working on a private LTE project a while ago; RAN hardware (eNodeBs) were on the way, down to a shortlist of a few EPC platforms, but I still needed USIMs before anyone was connecting to the network.

So why are custom USIMs a requirement? Can’t you just use any old USIM/SIMs?

In UMTS / LTE / NR networks there’s mutual network authentication, again I’ve written about this topic before, but unlike GSM where the network authenticates the UE, in later RAN standards, the UE also authenticates the network. (This mitigates any bad actor from setting up their own base stations and having UEs attach to it and have their traffic intercepted).

For roaming to work between carriers they’ve got to have their HSS / DRA connecting to the DRA or HSS of other carriers, to allow roaming subscribers to access the network, otherwise they too would fall foul of the mutual network authentication and the USIM wouldn’t connect to the network.

The first USIMs I purchased online through a popular online marketplace with a focus on connecting you to Chinese manufacturers.
They listed a package of USIMS, a USB reader/writer that supported all the standard USIM form factors and the software to program it, which I purchased.

The USIMs worked fairly well – They are programmable via a card reader and software that, although poorly translated/documented, worked fairly well.

USIM Programming Interface

K and OP/OPc values could be written to the card but not read, while the other values could be read and written from the software, the software also has the ability to sequentially program the USIMs to make bulk operations easier. The pricing worked out about $8 USD per USIM, which although expensive for the quantity and programmable element is pretty reasonable.

Every now and then the Crypto values for some reason or another wouldn’t get updated, which is exactly as irritating as it sounds.

Pretty quickly into the build I learned the USIMs didn’t include an ISIM service on the card, ISIM being the service that runs on the UCCID responsible for IMS / VoLTE authentication.

Again I went looking and reached out to a few manufacturers of USIMs.

The big vendors, Gemalto, Kona, etc, weren’t interested in providing USIMs in quantities less than 100,000 and their USIMs came from the factory pre-programmed, meaning the values could only be changed through remote SIM provisioning, a form of black magic.

In the end I reached out to an OEM manufacturer from China who provided programmable USIM / ISIMs for less than I was paying on the online marketplace and at any quantity I wanted with custom printing options, allocated ICCIDs, etc.

The non-programmable USIMs worked out less than $0.40 USD each in larger quantities, and programmable USIM/ISIMs for about $5 USD.

The software was almost identical except for the additional tab for ISIM operations.

USIM / ISIM programming
ISIM parameters

Smart Card Readers

In theory this software and these USIMs could be programmed by any smart card reader.

In practice, the fact that the ISO standard smart card is the same size as a credit card, means most smart card readers won’t fit the bill.

I tried a few smart card readers, from the one built into my Thinkpad, to a Bluedrive II from one of the USIM vendors, in the end the MCR3516 Smart Card Reader which reads 4FF USIMs (Standard ISO size smart card, full size SIM, Micro SIM and Nano SIM form factors, which saved on so much mucking about with form factor adapters etc.

4FF Smart Card Reader for programming SIM/USIM/ISIM

Future Projects

I’ve got some very calls “Multi Operator Neutral Host” (MoNEH) USIMs from the guys at Telet Research I’m looking forward to playing with,

eSIMs are on my to-do list too, and the supporting infrastructure, as well as Over the Air updating of USIMs.

LTE / EUTRAN – Idle Detach

In order to keep radio resources free, if a UE doesn’t send or receive data for a predefined threshold, it’ll detach from the network and call back to Idle mode.

If the UE has data to send to the network, the UE will re-attach to the network, whereas if the network has data to send to the UE, it’ll Page the UE in the tracking area it’s currently in, the UE is always listening for it’s identifier (s-TMSI) on the paging channel, and if it hears it’s identifier called, the UE will re-attach.

I’ve also attached a PCAP file of the packet flow between the eNB and the MME.

UEContextReleaseRequest [RadioNetwork-cause=user-inactivity]

The first packet is sent by the eNB to the serving MME to indicate the user wishes to detach from the network.

PCAP of UEContextReleaseRequest from eNB to MME

UEContextReleaseCommand [NAS-cause=normal-release]

The next packet is sent from the MME back to the eNB confirming UE is releasing from the network.

UEContextReleaseCommand

UEContextReleaseComplete

Finally after the UE has released it’s radio resources the eNB sends a UEContextReleaseComplete so the MME knows the UE is now in Idle state and will need to be paged.

UEContextReleaseComplete response

Subscribed-Periodic-RAU-TAU-Timer

https://github.com/open5gs/nextepc/issues/238

Recently we saw Open5Gs’s Update Location Answer response putting the Subscribed-Periodic-RAU-TAU-Timer AVP in the top level and not in the AVP Code 1400 (APN Configuration) Diameter payload from the HSS to the MME.

But what exactly does the Subscribed-Periodic-RAU-TAU-Timer AVP in the Update Location Answer response do?

Folks familiar with EUTRAN might recognise TAU as Tracking Area Update, while RAU is Routing Area Update in GERAN/UTRAN (UMTS).

Periodic tracking area updating is used to periodically notify the availability of the UE to the network. The procedure is controlled in the UE by the periodic tracking area update timer (timer T3412). The value of timer T3412 is sent by the network to the UE in the ATTACH ACCEPT message and can be sent in the TRACKING AREA UPDATE ACCEPT message. The UE shall apply this value in all tracking areas of the list of tracking areas assigned to the UE, until a new value is received.

Section 5.3.5 of 24301-9b0 (3GPP TS 24.301 V9.11.0)

So the Periodic Tracking Area Update timer simply defines how often the UE should send a Tracking Area Update when stationary (not moving between cells / tracking area lists).

The case for Header Compression in VoIP/VoLTE

On a PCM (G.711) RTP packet the payload is typically 160 bytes per packet.

But the total size of the frame on the wire is typically ~214 bytes, to carry a 160 byte payload that means 25% of the data being carried is headers.

This is fine for VoIP services operating over fixed lines, but when we’re talking about VoLTE / IMS and the traffic is being transferred over Radio Access Networks with limited bandwidth / resources, it’s important to minimize this as much as possible.

IMS uses the AMR codec, where the RTP payload for each packet is around 90 bytes, meaning up to two thirds of the packet on the wire (Or in this case the air / Uu interface) is headers.

Enter Robust Header Compression which compresses the headers.

Using ROHC the size of the headers are cut down to only 4-5 bytes, this is because the IPv4 headers, UDP headers and RTP headers are typically the same in each packet – with only the RTP Sequence number, RTP timestamp IPv4 & UDP checksum and changing between frames.

Open5Gs- Python HSS Interface

Note: NextEPC the Open Source project rebranded as Open5Gs in 2019 due to a naming issue. The remaining software called NextEPC is a branch of an old version of Open5Gs. This post was written before the rebranding.

I’ve been working for some time on Private LTE networks, the packet core I’m using is NextEPC, it’s well written, flexible and well supported.

I joined the Open5Gs group and I’ve contributed a few bits and pieces to the project, including a Python wrapper for adding / managing subscribers in the built in Home Subscriber Server (HSS).

You can get it from the support/ directory in Open5Gs.

NextEPC Python Library

Basic Python library to interface with MongoDB subscriber DB in NextEPC HSS / PCRF. Requires Python 3+, mongo, pymongo and bson. (All available through PIP)

If you are planning to run this on a different machine other than localhost (the machine hosting the MongoDB service) you will need to enable remote access to MongoDB by binding it’s IP to 0.0.0.0:

This is done by editing /etc/mongodb.conf and changing the bind IP to: bind_ip = 0.0.0.0

Restart MongoDB for changes to take effect.

$ /etc/init.d/mongodb restart

Basic Example:

import NextEPC
NextEPC_1 = NextEPC("10.0.1.118", 27017)

pdn = [{'apn': 'internet', 'pcc_rule': [], 'ambr': {'downlink': 1234, 'uplink': 1234}, 'qos': {'qci': 9, 'arp': {'priority_level': 8, 'pre_emption_vulnerability': 1, 'pre_emption_capability': 1}}, 'type': 2}]
sub_data = {'imsi': '891012222222300', \
             'pdn': pdn, \
             'ambr': {'downlink': 1024000, 'uplink': 1024001}, \
             'subscribed_rau_tau_timer': 12, \
             'network_access_mode': 2, \
             'subscriber_status': 0, \
             'access_restriction_data': 32, \
             'security': {'k': '465B5CE8 B199B49F AA5F0A2E E238A6BC', 'amf': '8000', 'op': None, 'opc': 'E8ED289D EBA952E4 283B54E8 8E6183CA'}, '__v': 0}

print(NextEPC_1.AddSubscriber(sub_data))                        #Add Subscriber using dict of sub_data

print(NextEPC_1.GetSubscriber('891012222222300'))               #Get added Subscriber's details

print(NextEPC_1.DeleteSubscriber('891012222222300'))            #Delete Subscriber

Subscriber_List = NextEPC_1.GetSubscribers()
for subscribers in Subscriber_List:
  print(subscribers['imsi'])
Open5Gs Logo

Open5GS – Splitting Network Elements

Note: NextEPC the Open Source project rebranded as Open5Gs in 2019 due to a naming issue. The remaining software called NextEPC is a branch of an old version of Open5Gs. This post was written before the rebranding.

I’ve been working for some time on Private LTE networks, and wrote my own HSS (See PyHSS – Python Home Subscriber Server).

The packet core I’m using is NextEPC, it’s well written, flexible and well supported.

I joined the Open5Gs group and I’ve contributed a few bits and pieces to the project.

One of which was how to split all the network elements in NextEPC:

NextEPC Splitting Network Elements

In a production network network elements would typically not all be on the same machine, as is the default example that ships with NextEPC.

NextEPC is designed to be standards compliant, so in theory you can connect any core network element (MME, PGW, SGW, PCRF, HSS) from NextEPC or any other vendor to form a functioning network, so long as they are 3GPP compliant.

To demonstrate this we will cover isolating each network element onto it’s on machine and connect each network element to the other. For some interfaces specifying multiple interfaces is supported to allow connection to multiple

In these examples we’ll be connecting NextEPC elements together, but it could just as easily be EPC elements from a different vendor in the place of any NextEPC network element.

ServiceIPIdentity
P-GW10.0.1.121pgw.localdomain
S-GW10.0.1.122 
PCRF10.0.1.123pcrf.localdomain
MME10.0.1.124mme.localdomain
HSS10.0.1.118hss.localdomain

External P-GW

In it’s simplest from the P-GW has 3 interfaces:

  • S5 – Connection to home network S-GW (GTP-C)
  • Gx – Connection to PCRF (Diameter)
  • Sgi – Connection to external network (Generally the Internet via standard TCP/IP)

S5 Interface Configuration

Edit /etc/nextepc/pgw.confand change the address to IP of the server running the P-GW for the listener on GTP-C and GTP-U interfaces.

pgw:
    freeDiameter: pgw.conf
    gtpc: 
      addr:
        - 10.0.1.121
     gtpu: 
      addr:
        - 10.0.1.121

Gx Interface Configuration

Edit /etc/nextepc/freeDiameter/pgwd.conf

Update ListenOn address to IP of the server running the P-GW:

ListenOn = "10.0.1.121";

Update ConnectPeer to connect to the PCRF on it’s IP.

ConnectPeer = "pcrf.localdomain" { ConnectTo = "10.0.1.123"; No_TLS; };

Restart Services

Restart NextEPC PGW Daemon:

$ sudo systemctl restart nextepc-pgwd

External S-GW

In it’s simplest form the S-GW has 2 interfaces:

  • S11 – Connection to MME (GTP-C)
  • S5 – Connection to the home network P-GW (GTP-C)

S5 Interface Configuration

Edit /etc/nextepc/sgw.confand change the address to IP of the server running the S-GW for the listener on GTP-C interface.

sgw:
    freeDiameter: pgw.conf
    gtpc: 
      addr:
        - 10.0.1.122

Restart NextEPC SGW Daemon:

$ sudo systemctl restart nextepc-sgwd

External PCRF

In it’s simplest from the PCRF has 1 network interface:

  • Gx – Connection to P-GW (Diameter)

Gx Interface Configuration

Edit /etc/nextepc/freeDiameter/hss.conf

Update ListenOn address to IP of the server running the HSS on it’s IP:

ListenOn = "10.0.1.123";

Update ConnectPeer to connect to the MME.

ConnectPeer = "pgw.localdomain" { ConnectTo = "10.0.1.121"; No_TLS; };

MongoDB Interface Configuration (NextEPC HSS only)

Edit /etc/nextepc/freeDiameter/hss.conf and change the db_uri: to point at the HSS: db_uri: mongodb://10.0.1.118/nextepc

Restart NextEPC PCRF Daemon:

$ sudo systemctl restart nextepc-pcrfd

External HSS

In it’s simplest form the HSS has 1 network interface:

  • S6a – Connection to MME (Diameter)

S6a Interface Configuration

Edit /etc/nextepc/freeDiameter/hss.conf

Update ListenOn address to IP of the server running the HSS on it’s IP:

ListenOn = "10.0.1.118";

Update ConnectPeer to connect to the MME.

ConnectPeer = "mme.localdomain" { ConnectTo = "10.0.1.124"; No_TLS; };

Restart NextEPC HSS Daemon:

$ sudo systemctl restart nextepc-hssd

MongoDB Interface Configuration (NextEPC specific)

If you are using NextEPC’s HSS you may need to enable MongoDB access from the PCRF. This is done by editing ‘‘/etc/mongodb.conf’’ and changing the bind IP to: bind_ip = 0.0.0.0

Restart MongoDB for changes to take effect.

$ /etc/init.d/mongodb restart

External MME

In it’s simplest form the MME has 3 interfaces:

  • S1AP – Connections from eNodeBs
  • S6a – Connection to HSS (Diameter)
  • S11 – Connection to S-GW (GTP-C)

S11 Interface Configuration

Edit /etc/nextepc/mme.conf, filling the IP address of the S-GW and P-GW servers.

sgw:
    gtpc:
      addr: 10.0.1.122

pgw:
    gtpc:
      addr:
        - 10.0.1.121

S6a Interface Configuration

Edit /etc/nextepc/freeDiameter/mme.conf

Update ListenOn address to IP of the server running the MME:

ListenOn = "10.0.1.124";

Update ConnectPeer to connect to the PCRF on it’s IP.

ConnectPeer = "hss.localdomain" { ConnectTo = "10.0.1.118"; No_TLS; };

Restart Services

Restart NextEPC MME Daemon:

$ sudo systemctl restart nextepc-mmed

Building Android APN / Carrier Config

As anyone who’s setup a private LTE network can generally attest, APNs can be a real headache.

SIM/USIM cards, don’t store any APN details. In this past you may remember having to plug all these settings into your new phone when you upgraded so you could get online again.

Today when you insert a USIM belonging to a commercial operator, you generally don’t need to put APN settings in, this is because Android OS has its own index of APNs. When the USIM is inserted into the baseband module, the handset’s OS looks at the MCC & MNC in the IMSI and gets the APN settings automatically from Android’s database of APN details.

There is an option for the network to send the connectivity details to the UE in a special type of SMS, but we won’t go into that.

All this info is stored on the Android OS in apns-full-conf.xml which for non-rooted (stock) devices is not editable.

Instead the devices get updates through the OS updates which pull the latest copy of this file from Google’s Android Open Source Git repo, you can view the current master file here.

This file can override the user’s APN configuration, which can lead to some really confusing times as your EPC rejects the connection due to an unrecognized APN which is not what you have configured on the UE’s operating system, but it instead uses APN details from it’s database.

The only way around this is to change the apns-full-conf.xml file, either by modifying it per handset or submitting a push request to Android Open Source with your updated settings.

(I’ve only tried the former with rooted devices)

The XML file itself is fairly self explanatory, taking the MCC and MNC and the APN details for your network:

<apn carrier="CarrierXYZ"
      mcc="123"
      mnc="123"
      apn="carrierxyz"
      type="default,supl,mms,ims,cbs"
      mmsc="http://mms.carrierxyz.com"
      mmsproxy="0.0.0.0"
      mmsport="80"
      bearer_bitmask="4|5|6|7|8|12"
/>

Once you’ve added yours to the file, inserting the USIM, rebooting the handset or restarting the carrier app is all that’s required for it to be re-read and auto provision APN settings from the XML file.

Further reading

APN and CarrierConfig | Android Open Source Project

Carrier Configuration | Android Open Source Project

UICC Carrier Privileges | Android Open Source Project

/etc/apns-full-conf.xml – Master Branch

Diameter Routing Agents (DRA)

Diameter is used extensively in 3GPP networks (Especially LTE) to provide the AAA services.

The Diameter protocol is great, and I’ve sung it’s praises before, but one issue operators start to face is that there are a lot of diameter peers, each of which needs a connection to other diameter peers.

https://en.wikipedia.org/wiki/IP_Multimedia_Subsystem#/media/File:Ims_overview.png

This diagram is an “Overview” showing one of each network element – In reality almost all network elements will exist more than once for redundancy and scalability.

What you start to end up with is a rats nest of connections, lines drawn everywhere and lots of manual work and room for human error when it comes to setting up the Diameter Peer relationships.

Let’s say you’ve got 5x MME, 5x PCRF, 2x HSS, 5x S-SCSF and 5x Packet Gateways, each needing Diameter peer relationships setup, it starts to get really messy really quickly.

Enter the Diameter Routing Agent – DRA.

Now each device only needs a connection to the DRA, which in turn has a connection to each Diameter peer. Adding a new MME doesn’t mean you need to reconfigure your HSS, just connect the MME to the DRA and away you go.

I’ll cover using Kamailio to act as a Diameter routing agent in a future post.

PyHSS – Python 3GPP LTE Home Subscriber Server

I recently started working on an issue that I’d seen was to do with the HSS response to the MME on an Update Location Answer.

I took some Wireshark traces of a connection from the MME to the HSS, and compared that to a trace from a different HSS. (Amarisoft EPC/HSS)

The Update Location Answer sent by the Amarisoft HSS to the MME over the S6a (Diameter) interface includes an AVP for “Multiple APN Configuration” which has the the dedicated bearer for IMS, while the HSS in the software I was working on didn’t.

After a bit of bashing trying to modify the S6a responses, I decided I’d just implement my own Home Subscriber Server.

The Diameter interface is pretty straight forward to understand, using a similar structure to RADIUS, and with the exception of the Crypto for the EUTRAN Authentication Vectors, it was all pretty straight forward.

If you’d like to know more you can download PyHSS from my GitLab page, and view my Diameter Primer post and my post on Diameter packet structure.

Diameter Packet Structure

We talked a little about what the Diameter protocol is, and how it’s used, now let’s look at the packets themselves.

Each Diameter packet has at a the following headers:

Version

This 1 byte field is always (as of 2019) 0x01 (1)

Length

3 bytes containing the total length of the Diameter packet and all it’s contained AVPs.

This allows the receiver to know when the packet has ended, by reading the length and it’s received bytes so far it can know when that packet ends.

Flags

Flags allow particular parameters to be set, defining some possible options for how the packet is to be handled by setting one of the 8 bits in the flags byte, for example Request Set, Proxyable, Error, Potentially Re-transmitted Message,

Command Code

Each Diameter packet has a 3 byte command code, that defines the method of the request,

The IETF have defined the basic command codes in the Diameter Base Protocol RFC, but many vendors have defined their own command codes, and users are free to create and define their own, and even register them for public use.

3GPP have defined a series of their own command codes.

Application ID

To allow vendors to define their own command codes, each command code is also accompanied by the Application ID, for example the command code 257 in the base Diameter protocol translates to Capabilities Exchange Request, used to specify the capabilities of each Diameter peer, but 257 is only a Capabilities Exchange Request if the Application ID is set to 0 (Diameter Base Protocol).

If we start developing our own applications, we would start with getting an Application ID, and then could define our own command codes. So 257 with Application ID 0 is Capabilities Exchange Request, but command code 257 with Application ID 1234 could be a totally different request.

Hop-By-Hop Identifier

The Hop By Hop identifier is a unique identifier that helps stateful Diameter proxies route messages to and fro. A Diameter proxy would record the source address and Hop-by-Hop Identifier of a received packet, replace the Hop by Hop Identifier with a new one it assigns and record that with the original Hop by Hop Identifier, original source and new Hop by Hop Identifier.

End-to-End Identifier

Unlike the Hop-by-Hop identifier the End to End Identifier does not change, and must not be modified, it’s used to detect duplicates of messages along with the Origin-Host AVP.

AVPs

The real power of Diameter comes from AVPs, the base protocol defines how to structure a Diameter packet, but can’t convey any specific data or requests, we put these inside our Attribute Value Pairs.

Let’s take a look at a simple Diameter request, it’s got all the boilerplate headers we talked about, and contains an AVP with the username.

Here we can see we’ve got an AVP with AVP Code 1, containing a username

Let’s break this down a bit more.

AVP Codes are very similar to the Diameter Command Codes/ApplicationIDs we just talked about.

Combined with an AVP Vendor ID they define the information type of the AVP, some examples would be Username, Session-ID, Destination Realm, Authentication-Info, Result Code, etc.

AVP Flags are again like the Diameter Flags, and are made up a series of bits, denoting if a parameter is set or not, at this stage only the first two bits are used, the first is Vendor Specific which defines if the AVP Code is specific to an AVP Vendor ID, and the second is Mandatory which specifies the receiver must be able to interpret this AVP or reject the entire Diameter request.

AVP Length defines the length of the AVP, like the Diameter length field this is used to delineate the end of one AVP.

AVP Vendor ID

If the AVP Vendor Specific flag is set this optional field specifies the vendor ID of the AVP Code used.

AVP Data

The payload containing the actual AVP data, this could be a username, in this example, a session ID, a domain, or any other value the vendor defines.

AVP Padding

AVPs have to fit on a multiple of a 32 bit boundary, so padding bits are added to the end of a packet if required to total the next 32 bit boundary.

Diameter Basics

3GPP selected Diameter protocol to take care of Authentication, Authorization, and Accounting (AAA).

It’s typically used to authenticate users on a network, authorize them to use services they’re allowed to use and account for how much of the services they used.

In a EPC scenario the Authentication function takes the form verifying the subscriber is valid and knows the K & OP/OPc keys for their specific IMSI.

The Authorization function checks to find out which features, APNs, QCI values and services the subscriber is allowed to use.

The Accounting function records session usage of a subscriber, for example how many sessional units of talk time, Mb of data transferred, etc.

Diameter Packets are pretty simple in structure, there’s the packet itself, containing the basic information in the headers you’d expect, and then a series of one or more Attribute Value Pairs or “AVPs”.

These AVPs are exactly as they sound, there’s an attribute name, for example username, and a value, for example, “Nick”.

This could just as easily be for ordering food; we could send a Diameter packet with an imaginary command code for Food Order Request, containing a series of AVPs containing what we want. The AVPs could belike Food: Hawian Pizza, Food: Garlic Bread, Drink: Milkshake, Address: MyHouse.
The Diameter server could then verify we’re allowed to order this food (Authorization) and charge us for the food (Accounting), and send back a Food Order Response containing a series of AVPs such as Delivery Time: 30 minutes, Price: $30.00, etc.

Diameter packets generally take the form of a request – response, for example a Capabilities Exchange Request contains a series of AVPs denoting the features supported by the requester, which is sent to a Diameter peer. The Diameter peer then sends back a Capabilities Exchange Response, containing a series of AVPs denoting the features that it supports.

Diameter is designed to be extensible, allowing vendors to define their own type of AVP and Diameter requests/responses and 3GPP have defined their own types of messages (Diameter Command Codes) and types of data to be transferred (AVP Codes).

LTE/EPC relies on Diameter and the 3GPP/ETSI defined AVP / Diameter Packet requests/responses to form the S6a Interface between an MME and a HSS, the Gx Interface between the PCEF and the PCRF, Cx Interface between the HSS and the CSCF, and many more interfaces used for Authentication in 3GPP networks.

Qos in LTE (4G) – ARP

ARP in LTE is not the Ethernet standard for address resolution, but rather the Allocation and Retention Policy.

A scenario may arise where on a congested cell another bearer is requested to be setup.

The P-GW, S-GW or eNB have to make a decision to either drop an existing bearer, or to refuse the request to setup a new bearer.

The ARP value is used to determine the priority of the bearer to be established compared to others,

For example a call to an emergency services number on a congested cell should drop any other bearers so the call can be made, thus the request for bearer for the VoLTE call would have a higher ARP value than the other bearers and the P-GW, S-GW or eNB would drop an existing bearer with a lower ARP value to accommodate the new bearer with a higher ARP value.

ARP is only used when setting up a new bearer, not to determine how much priority is given to that bearer once it’s established (that’s defined by the QCI).

QoS in LTE (4G) – MBR/AMBR/APN-MBR

MBR stands for Maximum Bit Rate, and it defines the maximum rate traffic can flow between a UE and the network.

It can be defined on several levels:

MBR per Bearer

This is the maximum bit rate per bearer, this rate can be exceeded but if it is exceeded it’s QoS (QCI) values for the traffic peaking higher than the MBR is back to best-effort.

AMBR

Aggregate Maximum Bit Rate – Maximum bit rate of all Service Data Flows / Bearers to and from the network from a single UE.

APN-MBR

The APN-MBR allows the operator to set a maximum bit rate per APN, for example an operator may choose to limit the MBR for subscriber on an APN for a MVNO to give it’s direct customers a higher speed.

(This is only applied to Non-GBR bearers)

QoS in LTE (4G) – QCI

The QCI (Quality Class Indicator) is a value of 0-9 to denote the service type and the maximum delays, packet loss and throughput the service requires.

Different data flows have different service requirements, let’s look at some examples:

A VoLTE call requires low latency and low packet loss, without low latency it’ll be impossible to hold a conversation with long delays, and with high packet loss you won’t be able to hear each other.

On the other hand a HTTP (Web) browsing session will be impervious to high latency or packet loss – the only perceived change would be slightly longer page load times as lost packets are resent and added delay on load of a few hundred ms.

So now we understand the different requirements of data flows, let’s look at the columns in the table above so we can understand what they actually signify:

GBR

Guaranteed Bit Rate bearers means our eNB will reserve resource blocks to carry this data no matter what, it’ll have those resource blocks ready to transport this data.

Even if the data’s not flowing a GBR means the resources are reserved even if nothing is going through them.

This means those resource blocks can’t be shared by other users on the network. The Uu interface in the E-UTRAN is shared between UEs in time and frequency, but with GBR bearers parts of this can be reserved exclusively for use by that traffic.

Non-GBR

With a Non-GBR bearer this means there is no guaranteed bit rate, and it’s just best effort.

Non-GBR traffic is scheduled onto resource blocks when they’re not in use by other non-GBR traffic or by GBR traffic.

Priority

The priory value is used for preemption by the PCRF.

The lower the value the more quickly it’ll be processed and scheduled onto the Uu interface.

Packet Delay Budget

Maximum allowable packet delay as measured from P-GW to UE.

Most of the budget relates to the over-the-air scheduling delays.

The eNB uses the QCI information to make its scheduling decisions and packet prioritisation to ensure that the QoS requirements are met on a per-EPS-bearer basis.

(20ms is typically subtracted from this value to account for the radio propagation delay on the Uu interface)

Packet Error Loss Rate (PELR)

This is packets lost on the Uu interface, that have been sent but not confirmed received.

The PELR is an upper boundary for how high this can go, based on the SDUs (IP Packets) that have been processed by the sender on RLC but not delivered up to the next layers (PDCP) by the receiver.

(Any traffic bursting above the GBR is not counted toward the PELR)

(The list is now larger than 0-9 with 3GPP adding extra QCI values for MCPTT, V2X, etc, the full list is available here in table 6.1.7A)