also available for most titles (http://my.safaribooksonline.com). For more information, contact our corporate/ institutional sales department: 800-998-9938 or firstname.lastname@example.org.
Editor: Meghan Blanchette Production Editor: Nicole Shelby Copyeditor: Rachel Head Proofreader: Jasmine Kwityn March 2014:
Indexer: Judy McConville Cover Designer: Randy Comer Interior Designer: David Futato Illustrator: Rebecca Demarest
Revision History for the Second Edition: 2014-03-11:
See http://oreilly.com/catalog/errata.csp?isbn=9781449361846 for release details. Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly Media, Inc. Ethernet: The Definitive Guide, the image of an octopus, and related trade dress are trademarks of O’Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O’Reilly Media, Inc. was aware of a trademark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.
1. The Evolution of Ethernet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 History of Ethernet The Aloha Network The Invention of Ethernet Reinventing Ethernet Reinventing Ethernet for Twisted-Pair Media Reinventing Ethernet for 100 Mb/s Reinventing Ethernet for 1000 Mb/s Reinventing Ethernet for 10, 40, and 100 Gb/s Reinventing Ethernet for New Capabilities Ethernet Switches The Future of Ethernet
3 4 4 6 7 8 8 9 9 10 10
2. IEEE Ethernet Standards. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Evolution of the Ethernet Standard Ethernet Media Standards IEEE Supplements Draft Standards Differences Between DIX and IEEE Standards Organization of IEEE Standards The Seven Layers of OSI IEEE Sublayers Within the OSI Model Levels of Compliance The Effect of Standards Compliance IEEE Media System Identifiers
11 13 13 14 15 16 16 18 20 20 21 iii
10 Megabit per Second (Mb/s) Media Systems 100 Mb/s Media Systems 1000 Mb/s Media Systems 10 Gb/s Media Systems 40 Gb/s Media Systems 100 Gb/s Media Systems
21 23 24 24 25 25
3. The Ethernet System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 The Four Basic Elements of Ethernet The Ethernet Frame The Media Access Control Protocol Hardware Network Protocols and Ethernet Best-Effort Delivery Design of Network Protocols Protocol Encapsulation Internet Protocol and Ethernet Addresses Looking Ahead
27 28 30 33 36 36 37 38 39 41
4. The Ethernet Frame and Full-Duplex Mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 The Ethernet Frame Preamble Destination Address Source Address Q-Tag Envelope Prefix and Suffix Type or Length Field Data Field FCS Field End of Frame Detection Full-Duplex Media Access Control Full-Duplex Operation Effects of Full-Duplex Operation Configuring Full-Duplex Operation Full-Duplex Media Support Full-Duplex Media Segment Distances Ethernet Flow Control PAUSE Operation High-Level Protocols and the Ethernet Frame Multiplexing Data in Frames IEEE Logical Link Control
100 Gigabit Ethernet Short Copper Cable Media Systems (100GBASE-CR10) 100GBASE-CR10 Signal Encoding 100 Gigabit Ethernet Fiber Optic Media Systems Cisco CPAK Module for 100 Gigabit Ethernet 100 Gb/s Fiber Optic Media Specifications
15. Structured Cabling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 Structured Cabling Systems The ANSI/TIA/EIA Cabling Standards Solving the Problems of Proprietary Cabling Systems ISO and TIA Standards The ANSI/TIA Structured Cabling Documents Elements of the Structured Cabling Standards Star Topology Twisted-Pair Categories Minimum Cabling Recommendation Ethernet and the Category System Horizontal Cabling Horizontal Channel and Basic Link Cabling and Component Specifications Category 5 and 5e Cable Testing and Mitigation Cable Administration Identifying Cables and Components Class 1 Labeling Scheme Documenting the Cabling System Building the Cabling System Cabling System Challenges
Eight-Position (RJ45-Style) Jack Connectors Four-Pair Wiring Schemes Tip and Ring Color Codes Wiring Sequence Modular Patch Panels Work Area Outlets Twisted-Pair Patch Cables Twisted-Pair Patch Cable Quality Telephone-Grade Patch Cables Twisted-Pair Ethernet and Telephone Signals Equipment Cables 50-Pin Connectors and 25-Pair Cables 25-Pair Cable Harmonica Connectors Building a Twisted-Pair Patch Cable Installing an RJ45 Plug Ethernet Signal Crossover 10BASE-T and 100BASE-T Crossover Cables Four-Pair Crossover Cables Auto-Negotiation and MDIX Failures Identifying a Crossover Cable
Basic Switch Functions Bridges and Switches What Is a Switch? Operation of Ethernet Switches Address Learning Traffic Filtering Frame Flooding Broadcast and Multicast Traffic Combining Switches Forwarding Loops The Spanning Tree Protocol Switch Performance Issues Packet Forwarding Performance Switch Port Memory Switch CPU and RAM Switch Specifications Basic Switch Features Switch Management Packet Mirror Ports Switch Traffic Filters Virtual LANs 802.1Q Multiple Spanning Tree Protocol Quality of Service (QoS)
This is a book about Ethernet, the world’s most popular network technology, which allows you to connect a variety of computers together with a low-cost and extremely flexible network system. Ethernet is found on a wide variety of devices, and this wide‐ spread support, coupled with its low cost and high flexibility, are major reasons for its popularity. The Ethernet standard has grown to over 3,700 pages, and it covers a multitude of Ethernet technologies designed for multiple environments. Ethernet is used to build home networks, office and campus network systems, as well as wide area networks that span cities and countries. There are Ethernet systems designed for networking a neigh‐ borhood, as well as Ethernets designed for networking inside automobiles to link the multiple computers found there these days. The goal of this book is to provide a comprehensive and practical source for information on the most widely used Ethernet technologies in a single volume. This book describes the varieties of Ethernet commonly used in homes, offices, and campus networks, as well as several systems typically used in data centers and server machine rooms. These include the most widely used set of Ethernet media systems: 10 Mb/s Ethernet, 100 Mb/s Fast Ethernet, and 1000 Mb/s Gigabit Ethernet, as well as 10 Gigabit and 40 and 100 Gigabit Ethernet. We also describe full-duplex Ethernet, Ethernet Auto-Negotiation, Power over Ethernet, Energy Efficient Ethernet, structured cabling systems, network design with Ethernet switches, network management, network troubleshooting tech‐ niques, and more. To provide the most accurate information possible, we referred to the complete set of official Ethernet standards while writing this book. Our experience includes working with Ethernet technology since the early 1980s, and many hard-won lessons in network design and operation based on that experience have made their way into this edition.
Ethernet Is Everywhere Ethernet is the most widely used networking technology, and Ethernet networks are everywhere. There are a number of factors that have helped Ethernet to become so popular. Among these factors are cost, scalability, reliability, and widely available man‐ agement tools.
Cost The rapid evolution of new capabilities in Ethernet has been accompanied by an equally rapid decrease in the cost of Ethernet equipment. The widespread adoption of Ethernet technology created a large and fiercely competitive Ethernet marketplace, which serves to drive down the cost of networking components. The consumer wins out in the pro‐ cess, with the marketplace providing a wide range of competitively priced Ethernet components to choose from.
Scalability The first industry-wide Ethernet standard was published over 30 years ago, in 1980. This standard defined a 10 megabits per second (Mb/s) system, which was very fast for the time. The development of the 100 Mb/s Fast Ethernet system in 1995 provided a tenfold increase in speed. Following on that success came the development of twistedpair Gigabit Ethernet in 1999. Network interfaces that can automatically support 10, 100, and 1000 Mb/s operation of twisted-pair media systems are widely available, mak‐ ing the support of high-performance networking easy to accomplish. Applications tend to grow to fill all available bandwidth. To manage the constant in‐ crease in network usage, the 10 Gigabit Ethernet standard was developed in 2002, and most recently the 40 and 100 Gigabit systems were standardized in 2010. All of this progress in Ethernet capabilities makes it possible for a network manager to provide high-speed backbone systems and connections to high-performance servers. Desktop machines can be connected to an Ethernet link that can operate at 10 Mb/s Ethernet, 100 Mb/s Fast Ethernet, or Gigabit Ethernet speeds, as required. Network routers and switches can use 10 Gigabit and 40 or 100 Gigabit links for network back‐ bones, and data centers can connect to high-performance servers at 10, 40, or even 100 gigabits per second (Gb/s).
Reliability Ethernet is simple and robust and reliably delivers data day in and day out at sites all over the world. Ethernet based on twisted-pair media was introduced in 1987, making it possible to provide Ethernet signals over a structured cabling system.
Structured cabling provides a data delivery system for a building that is modeled on high-reliability cabling practices originally developed for the telephone system. This makes it possible to install a standards-based cabling system for Ethernet that is highly reliable and easy to manage.
Widely Available Management Tools The widespread acceptance of Ethernet brings with it the wide availability of Ethernet management and troubleshooting tools. Management tools based on standards such as the Simple Network Management Protocol (SNMP) make it possible for network ad‐ ministrators to keep track of an entire campus full of Ethernet equipment from a central location. Management capabilities embedded in Ethernet switches and computer in‐ terfaces provide powerful network monitoring and troubleshooting capabilities.
Design for Reliability A major goal of this book is to help you design and implement reliable networks, because network reliability is of paramount importance to users and organizations. Access to the Internet and information sharing between networked computers is an essential fea‐ ture of today’s world, and if the network fails, everything comes to a halt. This book shows you how to design reliable networks, how to monitor them and keep them work‐ ing reliably, and how to fix them should something fail. The wide range of Ethernet components and cabling systems available today provides enormous flexibility, making it possible to build an Ethernet to fit just about any cir‐ cumstance. However, all this flexibility does have a price. The many varieties of Ethernet each have their own components and their own configuration rules, which can make the life of a network designer complex. Designing and implementing a reliable Ethernet system requires that you understand how all the bits and pieces fit together, and that you follow the official guidelines for the configuration of the media systems. To help you with that task, this book provides the configuration guidelines for the widely used media systems.
Downtime is Expensive Avoiding network downtime is important for a number of reasons, not least of which is the cost of a network outage. Some quick “back of the envelope” calculations can show how expensive network downtime can be. Let’s assume that there are 1,000 network users at the Amalgamated Widget Company, and that their average annual salary in‐ cluding all overhead (benefits, etc.) is $100,000. That comes to $100 million a year in employee costs. Let’s further assume that everyone in the company depends on the network to get their work done, and that the network is used 40 hours a week, for about 50 weeks of the year. Preface
That’s 2,000 hours of network operation. Dividing the annual employee cost by the hours of network operation shows that the network is supporting $50,000 per hour of em‐ ployee cost during the year. Let’s further assume that when we total up all of the network outages over the period of a year in our hypothetical corporation, we find that the network was down just 1% of the time (99% uptime, or “two nines”). That sounds like really good uptime, but that small fraction of 2,000 hours represents a total of 20 hours of network outage. Twenty hours of network downtime at $50,000/hour is $1,000,000 in lost productivity due to network outage. Obviously, our example is very “quick and dirty.” We didn’t bother to calculate the impact of network outages during times when no one is around but when the network is still nevertheless supporting critically important servers. Also, we’re assuming that a net‐ work failure brings all operations to a halt, instead of trying to factor in the varying effects of localized failures that cause outages on only a portion of the network system. Nor do we try to estimate how much other work people could get done while the network is down, which would tend to lessen the impact. However, the main point is clear: even relatively small amounts of network downtime can cost quite a lot in lost productivity. That’s why it’s worth investing extra time, effort, and money to create the most reliable network system you can afford.
How to Use This Book The goal of this book is to provide the information needed for you to understand and operate any Ethernet system. For example, if you are a newcomer to Ethernet and you need to know how twisted-pair Ethernet systems work, then you can start with Part I. After reading those chapters, you can go to the twisted-pair media chapters in Part II, as well as the twisted-pair cabling information in Part III. Twisted-pair cables are con‐ nected together to form a network using switches, and these are described in Part IV. Experts in Ethernet can use the book as a reference guide and jump directly to those chapters that contain the information they need.
Organization of This Book The purpose of this book is to provide a comprehensive and practical guide to the Ethernet system and the Ethernet devices and components commonly used in office and building networks. The emphasis is on practical issues, with minimal theory and jargon. Chapters are kept as self-contained as possible, and many examples and illus‐ trations are provided. The book is organized into six parts to make it easier to find the specific information you need.
Here’s what you’ll find in each of these parts: • Part I provides an introduction to the Ethernet standard and a description of Ether‐ net theory and operation. The chapters in this part cover those portions of Ethernet operation that are common to all Ethernet media systems, including the Ethernet frame, the operation of the media access control system, full-duplex mode, and the Auto-Negotiation protocol. • Part II contains a description of each of the Ethernet media systems. It begins with the basics of Ethernet media signaling in Chapter 7, which also covers the Energy Efficient Ethernet system that saves power by modifying the media signaling during idle periods. Chapters 8 through 14 describe specific media systems, including 10, 100, and 1000 Mb/s, and 10, 40, and 100 Gb/s systems. • Part III offers a description of structured cabling systems and the components and cables used in building your Ethernet system, including a discussion of the struc‐ tured cabling standards and details on twisted-pair and fiber optic cabling. • Part IV describes the fundamentals of network design, including how to design and build Ethernet systems using Ethernet switches. • Part V covers Ethernet performance and troubleshooting. • Part VI contains the appendixes and glossary.
Disclaimer While every precaution has been taken in the preparation of this work, the authors assume no responsibility for errors or omissions, or for damages resulting from the use of information contained herein. We make no claims about the completeness or the accuracy of the information as it may apply to any field conditions.
Conventions Used in This Book Italic Used for filenames, new terms, and URLs. This icon designates a note, which is an important aside to its near‐ by text.
This icon designates a warning relating to the nearby text.
Safari® Books Online Safari Books Online is an on-demand digital library that delivers expert content in both book and video form from the world’s leading authors in technology and business. Technology professionals, software developers, web designers, and business and crea‐ tive professionals use Safari Books Online as their primary resource for research, prob‐ lem solving, learning, and certification training. Safari Books Online offers a range of product mixes and pricing programs for organi‐ zations, government agencies, and individuals. Subscribers have access to thousands of books, training videos, and prepublication manuscripts in one fully searchable database from publishers like O’Reilly Media, Prentice Hall Professional, Addison-Wesley Pro‐ fessional, Microsoft Press, Sams, Que, Peachpit Press, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, Course Technol‐ ogy, and dozens more. For more information about Safari Books Online, please visit us online.
How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://oreil.ly/ethernetTDG_2e. To comment or ask technical questions about this book, send email to bookques email@example.com. For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com.
Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia
Acknowledgments This book would not have been possible without the help of many people. First and foremost, the authors would like to thank the inventors of Ethernet, Bob Metcalfe and his fellow researchers at Xerox PARC. Their work revolutionized the way computers are used, unleashing a powerful new communications technology based on information sharing on computers linked with networks. We also thank the many engineers who have voluntarily given their time in countless IEEE standards meetings to develop new capabilities for the Ethernet system and to write the Ethernet specifications. The authors would also like to thank our acquisitions editor at O’Reilly, Meghan Blanchette, and the other editors and staff of O’Reilly who have worked on this book, for their assistance and attention to detail. We’d also like to thank Tim O’Reilly for creating a technical publishing house that supports such a wide variety of information resources, and that treats both readers and writers with respect. Finally, we’d like to thank Rich Seifert, author of The Switch Book, engineer and devel‐ oper of Ethernet technology, and a participant in the creation of Ethernet standards from the earliest days of Ethernet. Rich provided in-depth reviews of the manuscript that are very much appreciated and that helped improve the final work. Of course, the authors alone are responsible for any errors.
Introduction to Ethernet
The first part of this book provides a tour of basic Ethernet theory and operation. These chapters cover the portions of Ethernet operation that are common to all Ethernet media systems, including the Ethernet frame, the operation of the media access control system, full-duplex mode, and the Auto-Negotiation protocol.