Better Core in Zen 2

Just in case you have missed it, in our microarchitecture analysis article Ian has explained in great detail why AMD claims that its new Zen2 is significantly better architecture than Zen1:

  • a different second-stage branch predictor, known as a TAGE predictor
  • doubling of the micro-op cache
  • doubling of the L3 cache
  • increase in integer resources
  • increase in load/store resources
  • support for two AVX-256 instructions per cycle (instead of having to combine two 128 bit units).

All of these on-paper improvements show that AMD is attacking its key markets in both consumer and enterprise performance. With the extra compute and promised efficiency, we can surmise that AMD has the ambition to take the high-performance market back too. Unlike the Xeon, the 2nd gen EPYC does not declare lower clocks when running AVX2 - instead it runs on a power aware scheduler that supplies as much frequency as possible within the power constraints of the platform.

Users might question, especially with Intel so embedded in high performance and machine learning, why AMD hasn't gone with an AVX-512 design? As a snap back to the incumbent market leader, AMD has stated that not all 'routines can be parallelized to that degree', as well as a very clear signal that 'it is not a good use of our silicon budget'. I do believe that we may require pistols at dawn. Nonetheless, it will be interesting how each company approaches vector parallelisation as new generations of hardware come out. But as it stands, AMD is pumping its FP performance without going full-on AVX-512.

In response to AMD's claims of an overall 15% IPC increase for Zen 2, we saw these results borne out of our analysis of Zen 2 in the consumer processor line, which was released last month. In our analysis, Andrei checked and found that it is indeed 15-17% faster. Along with the performance improvements, there have been also security hardening updates, improved virtualization support, and new but proprietary instructions for cache and memory bandwidth Quality of Service (QoS). (The QoS features seem very similar to what Intel has introduced in Broadwell/Xeon E5 version 4 and Skylake - AMD is catching up in that area).

Rome Layout: Simple Makes It a Lot Easier

When we analyzed AMD's first generation of EPYC, one of the big disadvantages was the complexity. AMD had built its 32-core Naples processors by enabling four 8-core silicon dies, and attaching each one to two memory channels, resulting in a non-uniform memory architecutre (NUMA). Due to this 'quad NUMA' layout, a number of applications saw quite a few NUMA balancing issues. This happened in almost every OS, and in some cases we saw reports that system administrators and others had to do quite a bit optimization work to get the best performance out of the EPYC 7001 series.  

The New 2nd Gen EPYC, Rome, has solved this. The CPU design implements a central I/O hub through which all communications off-chip occur. The full design uses eight core chiplets, called Core Complex Dies (CCDs), with one central die for I/O, called the I/O Die (IOD). All of the CCDs communicate with this this central I/O hub through dedicated high-speed Infinity Fabric (IF) links, and through this the cores can communicate to the DRAM and PCIe lanes contained within, or other cores.

The CCDs consist of two four-core Core CompleXes (1 CCD = 2 CCX). Each CCX consist of a four cores and 16 MB of L3 cache, which are at the heart of Rome. The top 64-core Rome processors overall have 16 CCX, and those CCX can only communicate with each other over the central I/O die. There is no inter-chiplet CCD communication.

This is what this diagram shows. On the left we have Naples, first Gen EPYC, which uses four Zepellin dies each connected to the other with an IF link. On the right is Rome, with eight CCDs in green around the outside, and a centralized IO die in the middle with the DDR and PCIe interfaces.

As Ian reported, while the CCDs are made at TSMC, using its latest 7 nm process technology. The IO die by contrast is built on GlobalFoundries' 14nm process. Since I/O circuitry, especially when compared to caching/processing and logic circuitry, is notoriously hard to scale down to smaller process nodes, AMD is being clever here and using a very mature process technology to help improve time to market, and definitely has advantages.

This topology is clearly visible when you take off the hood. 

AMD Rome chip

Main advantage is that the 2nd Gen 'EPYC 7002' family is much easier to understand and optimize for, especially from a software point of view, compared to Naples. Ultimately each processor only has one memory latency environment, as each core has the same latency to speak to all eight memory channels simultanously  - this is compared to the first generation Naples, which had two NUMA regions per CPU due to direct attached memory.

As seen in the image below, this means that in a dual socket setup, a Naples processor will act like a traditional NUMA environment that most software engineers are familiar with.

Ultimately the only other way to do this is with a large monolithic die, which for smaller process nodes is becoming less palatable when it comes to yields and pricing. In that respect, AMD has a significant advantage in being able to develop small 7nm silicon with high yields and also provide a substantial advantage when it comes to binning for frequency.

How a system sees the new NUMA environment is quite interesting. For the Naples EPYC 7001 CPUs, this was rather complicated in a dual socket setup: 

Here each number shows the 'weighting' given to the delay to access each of the other NUMA domains. Within the same domain, the weighting is light at only 10, but then a NUMA domain on the same chip was given a 16. Jumping off the chip bumped this up to 32.

This changed significantly on Rome EPYC 7002: 

Although there are situations where the EPYC 7001 CPUs communicated faster, but the fact that the topology is much simpler from the software point of view is worth a lot. It makes getting good performance out of the chip much easier for everyone that has to used it, which will save a lot of money in Enterprise, but also help accelerate adoption. 

The First Boot Experience Rome and PCIe 4.0
POST A COMMENT

184 Comments

View All Comments

  • npz - Thursday, August 8, 2019 - link

    Let me tell you: Lockheed? Linux Raytheon? Linux + Solaris, etc Aerospace Corp? Linux. You won't find Windows ANYWHERE but desktops and laptops. All confidential info and actual aerospace work resides on Linux and legacy Solaris systems.

    Big gov or Telcom? Linux and legacy Solaris. HPC? ALL -- ALL Linux. Big database? All linux. Storage? Linux. Virtualization needs? All Linux. What does Rackspace run? I've worked on and for (2nd hand as dev. engineering support) big -- I mean giant state infrastructure (think electrical grid), financial (banks, trading) and aerospace and computing corp enviornement and every single one runs Linux and/or legacy Solaris (after they moved from AIX usually, now they're moving from Solaris to Linux)

    Again you mention Active Directory -- what did I say? I stated: "companies use Windows because of familiarity for desktop support such as Active Directory for domains, but none of major critical data center centric, HPC, military, infrastructure are running Windows. Most especially not with EPYC since the Windows scheduler is broken. "

    Go outside any Active Directory needs and servers that needs to support RDP or Windows dekstops you will see Windows isn't used anywhere else
    Reply
  • npz - Thursday, August 8, 2019 - link

    examples:

    Aerospace Corp:
    https://www.glassdoor.com/job-listing/space-vehicl...
    "Space Vehicle Modeling & Simulation Engineer"
    > Familiarity with Linux and high-performance computing environments

    https://www.glassdoor.com/job-listing/data-scienti...
    "Data Scientist"
    > Working knowledge of Unix/Linux operating systems

    Citi Group:
    https://www.glassdoor.com/job-listing/infrastructu...
    "Infrastructure Solutions Engineer"
    > Must have good understanding of Linux and networking concepts

    https://www.glassdoor.com/job-listing/senior-data-...
    "Senior Data Engineer - Hadoop, VP"
    > Hands on experience with open source software platforms Linux
    Reply
  • Oliseo - Thursday, August 8, 2019 - link

    "Completely baseless claims. I have worked large scale government and military IT and Windows servers are the most common by far. "

    Working on reception will most likely be the reason for coming to that assumption, so we can go easy on you. It's an easy mistake to make.
    Reply
  • James5mith - Tuesday, August 13, 2019 - link

    FreeIPA works pretty well for us. Reply
  • FunBunny2 - Thursday, August 8, 2019 - link

    if you run an industrial strength RDBMS (Oracle, DB2, and even SS) you run on some variant of linux. Reply
  • AlyxSharkBite - Friday, August 9, 2019 - link

    NPZ, I’d say you should do some fact checking before you make a broad statement. AWS allows you to choose your OS there’s a lot of Windows Servers there. The Department of Defense uses a lot of Windows Servers see a 2016 article (https://blogs.windows.com/windowsexperience/2016/0...

    Also here’s a handy graph on server os market share a little old but numbers won’t change that much. Notice *nix only have about 20% of the market. The other 80% is Windows Server

    https://static.spiceworks.com/shared/blog_entry_in...
    Reply
  • Bonez0r - Wednesday, August 14, 2019 - link

    Wasn't the Windows scheduler fixed in a Windows update two months ago?
    https://www.reddit.com/r/Amd/comments/bz6egc/windo...
    Reply
  • Jorsher - Tuesday, September 3, 2019 - link

    NPZ - what are you talking about? I work on a military network spanning 10+ countries and tens of thousands of users and computers. The enterprise runs on Windows, with specific cases running in RHEL. I'm a fan of *nix but quit drinking the kool-aid. Windows still owns a rather large portion of the market and I don't see it changing any time soon. Reply
  • Deshi! - Thursday, August 8, 2019 - link

    I work as an application engineer for a major global finance company that develops and hosts banking and e-commerce software used by banks and major shopping outlets. 90% of all our servers are either Linux or AIX mainly running websphere or standalone Java instances. We only have a handful of Windows servers, mainly for stuff like active directory and Outlook/ SharePoint. So yeah allot of it depends on the use case, but allot of the big boys do use Linux or AIX. It's cheaper and performs better for these use cases. Reply
  • cyberguyz - Thursday, August 8, 2019 - link

    I guess we all have to ask ourselves, who are the customers that would benefit most from a 64-core, 128 gen 4 PCIe processors? SMB or huge customers that would shell out many millions of $$$ for their middleware & backend systems? @Deshi! I or one of my L3 colleagues an L3 engineer contacted by your global finance company to fix Websphere problems some years back ;) Reply

Log in

Don't have an account? Sign up now