docs.confluent.io
Open in
urlscan Pro
13.32.110.76
Public Scan
Submitted URL: https://docs.confluent.io/platform/current/schema-registry/index.html#avro-json-and-protobuf-supported-formats-and-extensi...
Effective URL: https://docs.confluent.io/platform/current/schema-registry/index.html
Submission: On February 22 via api from ZA — Scanned from DE
Effective URL: https://docs.confluent.io/platform/current/schema-registry/index.html
Submission: On February 22 via api from ZA — Scanned from DE
Form analysis 1 forms found in the DOM
<form class="search-input-form">
<input type="text" class="st-search-input st-default-search-input" id="st-search-input" placeholder="Search documentation" autocomplete="off">
<div class="key-shortcut" title="Press forward slash ('/') to focus the search box">/</div>
</form>Text Content
Get Started Free
* Get Started Free
* Courses
What are the courses?
Video courses covering Apache Kafka basics, advanced concepts, setup and use
cases, and everything in between.
View all courses
Kafka® 101
Kafka® Internal Architecture
Kafka® Connect 101
Kafka® Security
Kafka Streams 101
NewDesigning Events and Event Streams
Event Sourcing and Storage
NewSchema Registry 101
Data Mesh 101
ksqlDB 101
Inside ksqlDB
Spring Frameworks and Kafka®
Building Data Pipelines
Confluent Cloud Networking
NewConfluent Cloud Security
* Learn
Pick your learning path
A wide range of resources to get you started
Start Learning
Articles
Deep-dives into key concepts
Patterns
Architectures for event streaming
FAQs
Q & A about Kafka® and its ecosystem
100 Days of Code
A self-directed learning path
Blog
The Confluent blog
Podcast
Our podcast, Streaming Audio
Confluent Developer Live
Free live professional training
Coding in Motion
Build a real-time streaming app
* Build
Design. Build. Run.
Build a client app, explore use cases, and build on our demos and resources
Start Building
Language Guides
Build apps in your favorite language
Tutorials
Hands-on stream processing examples
Demos
More resources to get you started
* Community
Join the Community
Confluent proudly supports the global community of streaming platforms,
real-time data streams, Apache Kafka®️, and its ecosystems
Learn More
Kafka Summit and Current Conferences
Premier data streaming events
Meetups & Events
Kafka and data streaming community
Ask the Community
Community forums and Slack channels
Community Catalysts
Sharing expertise with the community
* Docs
Get started for free
Use the Cloud quick start to get up and running with Confluent Cloud using a
basic cluster
Learn more
Documentation
Guides, tutorials, and reference
Confluent Cloud
Fully managed, cloud-native service
Confluent Platform
Enterprise-grade distribution of Kafka
Confluent Connectors
Stream data between Kafka and other systems
Tools
Operational and developer tools
Clients
Use clients to produce and consume messages
Courses
What are the courses?
Video courses covering Apache Kafka basics, advanced concepts, setup and use
cases, and everything in between.
View all courses
Kafka® 101
Kafka® Internal Architecture
Kafka® Connect 101
Kafka® Security
Kafka Streams 101
NewDesigning Events and Event Streams
Event Sourcing and Storage
NewSchema Registry 101
Data Mesh 101
ksqlDB 101
Inside ksqlDB
Spring Frameworks and Kafka®
Building Data Pipelines
Confluent Cloud Networking
NewConfluent Cloud Security
Learn
Pick your learning path
A wide range of resources to get you started
Start Learning
Articles
Deep-dives into key concepts
Patterns
Architectures for event streaming
FAQs
Q & A about Kafka® and its ecosystem
100 Days of Code
A self-directed learning path
Blog
The Confluent blog
Podcast
Our podcast, Streaming Audio
Confluent Developer Live
Free live professional training
Coding in Motion
Build a real-time streaming app
Build
Design. Build. Run.
Build a client app, explore use cases, and build on our demos and resources
Start Building
Language Guides
Build apps in your favorite language
Tutorials
Hands-on stream processing examples
Demos
More resources to get you started
Community
Join the Community
Confluent proudly supports the global community of streaming platforms,
real-time data streams, Apache Kafka®️, and its ecosystems
Learn More
Kafka Summit and Current Conferences
Premier data streaming events
Meetups & Events
Kafka and data streaming community
Ask the Community
Community forums and Slack channels
Community Catalysts
Sharing expertise with the community
Docs
Get started for free
Use the Cloud quick start to get up and running with Confluent Cloud using a
basic cluster
Learn more
Documentation
Guides, tutorials, and reference
Confluent Cloud
Fully managed, cloud-native service
Confluent Platform
Enterprise-grade distribution of Kafka
Confluent Connectors
Stream data between Kafka and other systems
Tools
Operational and developer tools
Clients
Use clients to produce and consume messages
Get Started Free
Confluent Documentation
/
CONFLUENT PLATFORM
* Overview
* Get Started
* What is Confluent Platform?
* Quick Start for Confluent Platform
* Kafka Basics on Confluent Platform
* Confluent Developer: What is Apache Kafka
* Videos, Demos, and Reading Material
* Scripted Confluent Platform Demo
* Overview
* Deploy Confluent Platform Environment
* Deploy Hybrid Confluent Platform and Cloud Environment
* End Demo
* Troubleshooting
* Tutorial: Introduction to Streaming Application Development
* Clickstream Data Analysis Pipeline Using ksqlDB
* RBAC Example for Confluent Platform
* Replicator Schema Translation Example for Confluent Platform
* DevOps for Kafka with Kubernetes and GitOps
* Overview
* Kafka DevOps Case Studies
* Case Study: Graduated Environments
* Case Study: Manage Cloud Secrets
* Case Study: Kafka Connect management with GitOps
* Resources
* Course: Apache Kafka 101
* Install and Upgrade
* Overview
* System Requirements
* Install Manually
* ZIP and TAR
* Ubuntu and Debian
* RHEL and CentOS
* Docker
* Install using Docker
* Docker Configuration Parameters
* Docker Image Reference
* Docker Security
* Docker Developer Guide
* Configure Automatic Startup and Monitoring
* Deploy with Ansible Playbooks
* Deploy with Confluent for Kubernetes
* License
* Upgrade Confluent Platform
* Overview
* Upgrade Confluent Platform
* Supported Versions and Interoperability
* Installation Packages
* Migrate from Apache Kafka
* Migrate an Existing Kafka Deployment
* Migrate to Confluent Server
* Build Client Applications
* Kafka Clients
* Overview
* Kafka Consumer
* Kafka Producer
* Client Guides
* Python Client
* .NET Client
* Go Client
* Java Client
* C++ Client
* JMS Client
* Client Examples
* Overview
* C/C++ Example
* .NET Example
* Go Example
* Spring Boot Example
* Java Example
* KafkaProducer Example
* Python Example
* REST Example
* Node.js Example
* Clojure Example
* Groovy Example
* Kafka Connect Datagen Example
* kafkacat Example
* Kotlin Example
* Ruby Example
* Rust Example
* Scala Example
* Client APIs
* Kafka C++ Client API
* Kafka Python Client API
* Kafka Go Client API
* Kafka .NET Client API
* JMS Client
* Overview
* JMS Client Installation
* JMS Client Development Guide
* Application Development
* MQTT Proxy
* Introduction
* Communication Security Settings
* MQTT Proxy Configuration Options
* Command Line Tools
* CLI Tools for Confluent Platform
* Confluent CLI
* kcat (formerly kafkacat) Utility
* Confluent REST APIs
* Overview
* Quick Start
* API Reference
* Production Deployment
* Confluent Server
* Configuration
* Security
* REST Proxy
* Overview
* Configuration
* Monitoring
* Security
* Connect to Confluent Cloud
* REST Proxy Tutorial
* ksqlDB and Kafka Streams
* Overview
* ksqlDB
* ksqlDB Overview
* ksqlDB Quickstart
* Install ksqlDB
* Operate ksqlDB
* Upgrade ksqlDB
* Develop applications for ksqlDB
* Run ksqlDB in Confluent Cloud
* Connect ksqlDB to Confluent Cloud
* Migrate Confluent Cloud ksqlDB applications
* Run ksqlDB in Confluent Control Center
* Connect ksqlDB to Confluent Control Center
* Secure ksqlDB with RBAC
* Frequently Asked Questions
* Troubleshoot ksqlDB issues
* Tutorials and Examples
* Tutorials overview
* ksqlDB Quick Start
* How-to Guides
* Example Code Snippets
* Materialized View/cache
* Streaming ETL Pipeline
* Event-driven Microservice
* ksqlDB with Embedded Connect
* Clickstream Data Analysis Pipeline Using ksqlDB
* Kafka Tutorials Using ksqlDB
* Kafka Streams
* Kafka Streams Overview
* Introduction
* Build your first Streams application
* Tutorial: Introduction to Streaming Application Development
* Connect Confluent Platform Components to Confluent Cloud
* Streams Concepts
* Streams Architecture
* Streams Code Examples
* Streams Developer Guide
* Writing a Streams Application
* Testing Streams Code
* Configuring a Streams Application
* Streams DSL
* Naming Kafka Streams DSL Topologies
* Optimizing Kafka Streams Topologies
* Processor API
* Data Types and Serialization
* Interactive Queries
* Memory Management
* Running Streams Applications
* Managing Streams Application Topics
* Kafka Streams Security
* Application Reset Tool
* Pipelining with Kafka Connect and Kafka Streams
* Streams Operations
* Capacity Planning and Sizing
* Monitoring Kafka Streams Applications
* Streams Upgrade Guide
* Streams FAQ
* Streams Javadocs
* Integrate External Systems to Kafka
* Overview
* Get Started with Kafka Connect
* Kafka Connect 101
* Connect to Confluent Cloud
* Connect to External Systems
* Connect on z/OS
* Connector Developer Guide
* Quick Start: Move Data In and Out of Kafka with Kafka Connect
* Kafka Connect Concepts
* Kafka Connect Reference
* Connect Javadocs
* Connect REST Interface
* Worker Configuration Properties
* Connector Configuration Properties
* Kafka Connect Licensing
* Kafka Connect Monitoring
* Kafka Connect Logging
* Kafka Connect Transformations
* Single Message Transforms for Confluent Platform
* Cast
* Drop
* DropHeaders
* ExtractField
* ExtractTopic
* Filter (Apache Kafka)
* Filter (Confluent)
* Flatten
* GzipDecompress
* HeaderFrom
* HoistField
* InsertField
* InsertHeader
* MaskField
* MessageTimestampRouter
* RegexRouter
* ReplaceField
* SetSchemaMetadata
* TimestampConverter
* TimestampRouter
* TombstoneHandler
* TopicRegexRouter
* ValueToKey
* Custom transformations
* Kafka Connect Security
* Kafka Connect Security Basics
* Kafka Connect and RBAC
* Get Started With RBAC and Kafka Connect
* Configure RBAC for a Connect Cluster
* Configure RBAC for a Connect Worker
* RBAC for self-managed connectors
* Connect Secret Registry
* Example Connect role-binding sequence
* Kafka Connect Architecture
* FileStream Connectors
* Kafka Connect FAQ
* Schema Management
* Overview
* Installing and Configuring
* Overview
* Configuration Options
* Run in Production
* Connect to Confluent Cloud
* Migrate Schemas
* Schema Registry Tutorials
* Tutorials Overview
* On-Premises
* Confluent Cloud
* Schema Linking
* Schema Validation on Confluent Server
* Monitoring
* Single and Multi-Datacenter Setup
* Schema Evolution and Compatibility
* Schemas in Control Center
* Schemas on Confluent Cloud
* Migrate Schemas
* Deleting Schemas
* Security
* Schema Registry Security Overview
* Role-Based Access Control
* Schema Registry Security Plugin
* Overview
* Install and Configure
* Schema Registry Authorization
* Supported Operations and Resources
* Role-Based Access Control
* Schema Registry ACL Authorizer
* Topic ACL Authorizer
* Developer Guide
* Overview
* Maven Plugin
* API Reference
* API Usage Examples
* Schema Formats
* Formats, Serializers, and Deserializers
* Avro
* Protobuf
* JSON Schema
* Integrate Schemas from Connectors
* Security
* General Security
* Security Overview
* Security Tutorial
* Configuring Confluent Server Authorizer
* Cluster Registry
* Security Compliance
* Prefixes for Configuring Security
* Authentication
* Authentication Methods Overview
* Authentication with SASL
* Authentication with SASL using JAAS
* Install
* Configuring GSSAPI
* Configuring OAUTHBEARER
* Configuring PLAIN
* Configuring SCRAM
* Authentication using Delegation Tokens
* Configuring Kafka Client Authentication with LDAP
* Encrypt and Authenticate with TLS
* HTTP Basic Authentication
* Adding security to a running cluster
* Authorization
* Authorization using Role-Based Access Control
* RBAC Overview
* Quick Start
* Predefined Roles
* Enable RBAC in a Running Cluster
* Discover Identifiers for Clusters
* Configuring Token Authentication
* Confluent Metadata API Reference
* RBAC Example for Confluent Platform
* Configure RBAC using the REST API
* ACLs
* Authorization using centralized ACLs
* Authorization using ACLs
* Group-Based Authorization Using LDAP
* Configuring Confluent Server Authorizer
* Configuring LDAP
* Tutorial: Group-Based Authorization Using LDAP
* Data Protection
* Audit Logs
* Audit Log Concepts
* Auditable Events
* Configure Audit Logs using the Confluent CLI
* Configure MDS to Manage Centralized Audit Logs
* MDS API Audit Log Configuration
* Configure Audit Logs using the Properties File
* Encrypt with TLS
* Secrets
* Secrets Management
* Tutorial: Secret Protection
* Redact Confluent Logs
* Component Security
* Confluent Control Center Security
* Overview
* Configure TLS/SSL
* Configure SASL
* Configure HTTP Basic Authentication
* Authorize with Kafka ACLs
* Configure LDAP
* Configure RBAC
* Manage and View RBAC Roles
* Log in to Control Center when RBAC enabled
* Manage RBAC roles with Control Center
* View your RBAC roles in Control Center
* Kafka Streams Security
* Schema Registry Security
* Kafka Connect Security
* Kafka Connect Security Basics
* Kafka Connect and RBAC
* Get Started With RBAC and Kafka Connect
* Configure RBAC for a Connect Cluster
* Configure RBAC for a Connect Worker
* RBAC for self-managed connectors
* Connect Secret Registry
* Example Connect role-binding sequence
* ksqlDB RBAC
* REST Proxy Security
* REST Proxy Security
* REST Proxy Security Plugins
* ZooKeeper Security
* Security Management Tools
* Ansible Playbooks for Confluent Platform
* Docker Security for Confluent Platform
* Multi-DC Deployment Architectures
* Overview
* Architecture Patterns and Use Cases
* Cluster Linking
* Cluster Linking
* Tutorial
* Hybrid Cloud and Bridge-to-Cloud
* Data Migration
* Mirror Topics
* Commands
* Configuration Options
* Metrics and Monitoring
* Security
* Frequently Asked Questions (FAQs)
* Multi-Region Clusters
* Overview
* Tutorial: Multi-Region Clusters
* Transition Standard Active-Passive Data Centers to a Multi-Region
Stretched Cluster
* Replicator for Multi-Datacenter Replication
* Replicator Overview
* Tutorial: Replicating Data Across Clusters
* Active-active Demo
* Download and Install
* Configure and Run
* Tuning Replicator
* Monitoring Replicator
* Replicator and Cross-Cluster Failover
* Configuration Options
* Migrate from MirrorMaker to Replicator
* Replicator Verifier
* Replicator to Cloud Configurations
* Administer
* Confluent Control Center
* Control Center Overview
* Installing and Configuring Control Center
* Configuring
* Configuration Reference
* Check Control Center Version and Enable Auto-Update
* Properties File
* Connecting Control Center to Confluent Cloud
* Confluent Monitoring Interceptors in Control Center
* Installing Control Center on Kafka
* Managing Confluent Platform Licenses
* Troubleshooting Control Center
* Upgrading Control Center
* Clusters
* Brokers
* Topics Overview
* Topics Overview
* Create
* View Topic Metrics
* Message Browser
* Manage Schemas for Topics
* Edit Configuration Settings for Topics
* Delete Topics
* Connect
* ksqlDB
* Consumers
* Replicators
* Clusters
* Control Center Alerts
* Overview
* Access Alerts and Alert History
* Configure Alerts Properties
* View and Manage Triggers
* Manage Actions
* Configure PagerDuty Emails with Alerts
* REST API for Alerts History
* Example Triggers and Actions
* Troubleshooting Alerts
* Security
* Configuration Reference
* Overview
* Kafka Broker Configurations for Confluent Platform
* Kafka Topic Configurations for Confluent Platform
* Kafka Consumer Configurations for Confluent Platform
* Kafka Producer Configurations for Confluent Platform
* Kafka Connect Configurations for Confluent Platform
* Overview
* Sink Configurations
* Source Configurations
* Kafka AdminClient Configurations for Confluent Platform
* License Configurations for Confluent Platform
* Kafka Streams Configurations for Confluent Platform
* Docker Configuration Parameters for Confluent Platform
* Control Center Configurations
* Streams Configurations
* ZooKeeper Configurations
* ksqlDB Server Configurations
* CLI Tools for Confluent Platform
* CLI Tools for Confluent Platform
* Confluent CLI
* Dynamic Kafka Configurations
* Configure Multi-Node Environment
* Metadata Service (MDS)
* Configure Metadata Service (MDS)
* Configure Confluent Platform Components to Communicate with MDS over
TLS/SSL
* Configure mTLS Authentication and RBAC for Kafka Brokers
* Configure Kerberos Authentication for Brokers Running MDS
* Configure LDAP Authentication
* Configure LDAP Group-Based Authorization for MDS
* Configure MDS to Manage Centralized Audit Logs
* Metadata Service Configuration Options
* Confluent Metadata API Reference
* Confluent Metrics Reporter
* Confluent Health+
* Health+ Overview
* Enable Health+
* Health+ Intelligent Alerts
* Health+ Monitoring Dashboard
* Confluent Telemetry Reporter
* Telemetry Reporter Metrics
* Confluent Health+ FAQ
* Confluent REST APIs
* Overview
* Quick Start
* API Reference
* Production Deployment
* Confluent Server
* Configuration
* Security
* REST Proxy
* Overview
* Configuration
* Monitoring
* Security
* Connect to Confluent Cloud
* REST Proxy Tutorial
* Kafka Operations
* Overview
* Running Kafka in Production
* Dynamic Kafka Configurations
* Post Kafka Deployment
* Self-Balancing Clusters
* Self-Balancing Overview
* Quick Start Demo (Docker)
* Tutorial: Adding and Removing Brokers
* Configuration Options and Commands
* Performance and Resource Usage
* Auto Data Balancing
* Quick Start
* Tutorial (Docker)
* Command and Configuration Options
* Monitoring Kafka with JMX
* Confluent Metrics Reporter
* Tiered Storage
* Docker Operations
* Overview
* Kafka Monitoring and Metrics Using JMX
* Configure Docker Logging
* Mounting Docker External Volumes
* Post Kafka Deployment
* ZooKeeper Operations
* Running ZooKeeper in Production
* Kafka Raft (KRaft)
* Kafka Streams Operations
* Capacity Planning and Sizing
* Monitoring Kafka Streams Applications
* ksqlDB Operations
* DevOps for Kafka with Kubernetes and GitOps
* Overview
* Kafka DevOps Case Studies
* Case Study: Graduated Environments
* Case Study: Manage Cloud Secrets
* Case Study: Kafka Connect management with GitOps
* Confluent Health+
* Health+ Overview
* Enable Health+
* Health+ Intelligent Alerts
* Health+ Monitoring Dashboard
* Confluent Telemetry Reporter
* Telemetry Reporter Metrics
* Confluent Health+ FAQ
* Confluent CLI
* Release Notes
* Release Notes
* Component Changelogs
* API and Javadocs
* Overview
* Kafka Java APIs
* Kafka Java Client APIs
* Kafka Producer Java API
* Kafka Consumer Java API
* Kafka AdminClient Java API
* Kafka Streams Java API
* Kafka Connect Java API
* Kafka Client APIs
* Kafka C++ Client API
* Kafka Python Client API
* Kafka Go Client API
* Kafka .NET Client API
* JMS Client
* Overview
* JMS Client Installation
* JMS Client Development Guide
* Confluent APIs
* Confluent Schema Registry API
* ksqlDB API
* Confluent REST API
* Confluent Metadata API
Language English日本語
Version 7.3
(current)7.27.17.06.26.16.05.5.95.5.75.5.65.5.55.5.45.5.35.5.25.5.125.5.115.5.105.5.15.5.05.4.95.4.85.4.75.4.65.4.55.4.45.4.35.4.25.4.105.4.15.4.05.3.85.3.75.3.65.3.55.3.45.3.35.3.25.3.15.3.05.2.75.2.65.2.55.2.45.2.35.2.25.2.15.2.05.1.45.1.35.1.25.1.15.1.05.0.45.0.35.0.25.0.15.0.04.1.44.1.34.1.24.1.14.1.04.0.34.0.24.0.14.0.03.3.33.3.23.3.13.3.03.2.43.2.23.2.13.2.03.1.23.1.13.1.03.0.13.0.02.0.12.0.01.0.11.0
1. Home
2. Platform
3. Schema Management
SCHEMA REGISTRY OVERVIEW¶
Looking for Schema Management Confluent Cloud docs? You are currently viewing
Confluent Platform documentation. If you are looking for Confluent Cloud docs,
check out Schema Management on Confluent Cloud.
See also
See the Schema Registry module of the free Apache Kafka 101 course to learn why
you would need a Schema Registry, what it is, and how to get started. Also see
the free Schema Registry 101 course to learn about the schema formats and how to
build, register, manage and evolve schemas.
Confluent Schema Registry provides a serving layer for your metadata. It
provides a RESTful interface for storing and retrieving your Avro®, JSON Schema,
and Protobuf schemas. It stores a versioned history of all schemas based on a
specified subject name strategy, provides multiple compatibility settings and
allows evolution of schemas according to the configured compatibility settings
and expanded support for these schema types. It provides serializers that plug
into Apache Kafka® clients that handle schema storage and retrieval for Kafka
messages that are sent in any of the supported formats.
Schema Registry lives outside of and separately from your Kafka brokers. Your
producers and consumers still talk to Kafka to publish and read data (messages)
to topics. Concurrently, they can also talk to Schema Registry to send and
retrieve schemas that describe the data models for the messages.
Confluent Schema Registry for storing and retrieving schemas¶
Schema Registry is a distributed storage layer for schemas which uses Kafka as
its underlying storage mechanism. Some key design decisions:
* Assigns globally unique ID [1] to each registered schema. Allocated IDs are
guaranteed to be monotonically increasing and unique, but not necessarily
consecutive.
* Kafka provides the durable backend, and functions as a write-ahead changelog
for the state of Schema Registry and the schemas it contains.
* Schema Registry is designed to be distributed, with single-primary
architecture, and ZooKeeper/Kafka coordinates primary election (based on the
configuration).
[1]Generally a Schema Registry cluster can include multiple tenants and each
tenant can include multiple contexts. Schema IDs are unique within a (tenant,
context) combination. For Confluent Platform Schema Registry you will only have
one tenant, so IDs will be unique within a context. Considering the
tenant/context combination as the realm in which schemas reside, schema IDs are
unique within this “world”; that is, globally unique.
Tip
To see a working example of Schema Registry, check out Confluent Platform demo.
The demo shows you how to deploy a Kafka streaming ETL, including Schema
Registry, using ksqlDB for stream processing.
SCHEMAS, SUBJECTS, AND TOPICS¶
First, a quick review of terms and how they fit in the context of Schema
Registry: what is a Kafka topic versus a schema versus a subject.
A Kafka topic contains messages, and each message is a key-value pair. Either
the message key or the message value, or both, can be serialized as Avro, JSON,
or Protobuf. A schema defines the structure of the data format. The Kafka topic
name can be independent of the schema name. Schema Registry defines a scope in
which schemas can evolve, and that scope is the subject. The name of the subject
depends on the configured subject name strategy, which by default is set to
derive subject name from topic name.
You can modify the subject name strategy on a per-topic basis. See Change the
subject naming strategy for a topic to learn more.
Following are some relevant features and guidelines to have a look at:
* The Schema Registry Tutorials shows an example of a schema definition.
* The Confluent Control Center section on how to Manage Schemas for Topics in
Control Center using Confluent Control Center includes details on Best
Practices and Pitfalls for Key Values with some good suggestions.
* Schema Linking is now available, as described in Schema Linking on Confluent
Platform.
* You can use Confluent Replicator to migrate schemas from one Schema Registry
to another, and automatically rename subjects on the target registry.
KAFKA SERIALIZERS AND DESERIALIZERS BACKGROUND¶
When sending data over the network or storing it in a file, you need a way to
encode the data into bytes. The area of data serialization has a long history,
but has evolved quite a bit over the last few years. People started with
programming language specific serialization such as Java serialization, which
makes consuming the data in other languages inconvenient, then moved to language
agnostic formats such as pure JSON, but without a strictly defined schema
format.
Not having a strictly defined format has two significant drawbacks:
1. Data consumers may not understand data producers: The lack of structure
makes consuming data in these formats more challenging because fields can be
arbitrarily added or removed, and data can even be corrupted. This drawback
becomes more severe the more applications or teams across an organization
begin consuming a data feed: if an upstream team can make arbitrary changes
to the data format at their discretion, then it becomes very difficult to
ensure that all downstream consumers will (continue to) be able to interpret
the data. What’s missing is a “contract” (cf. schema below) for data between
the producers and the consumers, similar to the contract of an API.
2. Overhead and verbosity: They are verbose because field names and type
information have to be explicitly represented in the serialized format,
despite the fact that are identical across all messages.
A few cross-language serialization libraries have emerged that require the data
structure to be formally defined by schemas. These libraries include Avro,
Thrift, Protocol Buffers, and JSON Schema . The advantage of having a schema is
that it clearly specifies the structure, the type and the meaning (through
documentation) of the data. With a schema, data can also be encoded more
efficiently. Avro was the default supported format for Confluent Platform.
For example, an Avro schema defines the data structure in a JSON format. The
following Avro schema specifies a user record with two fields: name and
favorite_number of type string and int, respectively.
{"namespace": "example.avro",
"type": "record",
"name": "user",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": "int"}
]
}
Copy
You can then use this Avro schema, for example, to serialize a Java object
(POJO) into bytes, and deserialize these bytes back into the Java object.
Avro not only requires a schema during data serialization, but also during data
deserialization. Because the schema is provided at decoding time, metadata such
as the field names don’t have to be explicitly encoded in the data. This makes
the binary encoding of Avro data very compact.
AVRO, JSON, AND PROTOBUF SUPPORTED FORMATS AND EXTENSIBILITY¶
Avro was the original choice for the default supported schema format in
Confluent Platform, with Kafka serializers and deserializers provided for the
Avro format.
Confluent Platform supports for Protocol Buffers and JSON Schema along with
Avro, the original default format for Confluent Platform. Support for these new
serialization formats is not limited to Schema Registry, but provided throughout
Confluent Platform. Additionally, Schema Registry is extensible to support
adding custom schema formats as schema plugins.
New Kafka serializers and deserializers are available for Protobuf and JSON
Schema, along with Avro. The serializers can automatically register schemas when
serializing a Protobuf message or a JSON-serializable object. The Protobuf
serializer can recursively register all imported schemas, .
The serializers and deserializers are available in multiple languages, including
Java, .NET and Python.
Schema Registry supports multiple formats at the same time. For example, you can
have Avro schemas in one subject and Protobuf schemas in another. Furthermore,
both Protobuf and JSON Schema have their own compatibility rules, so you can
have your Protobuf schemas evolve in a backward or forward compatible manner,
just as with Avro.
Schema Registry in Confluent Platform also supports for schema references in
Protobuf by modeling the import statement.
To learn more, see Formats, Serializers, and Deserializers.
SCHEMA ID ALLOCATION¶
Schema ID allocation always happens in the primary node and Schema IDs are
always monotonically increasing.
In Kafka primary election, the Schema ID is always based off the last ID that
was written to Kafka store. During a primary re-election, batch allocation
happens only after the new primary has caught up with all the records in the
store <kafkastore.topic>.
KAFKA BACKEND¶
Kafka is used as Schema Registry storage backend. The special Kafka topic
<kafkastore.topic> (default _schemas), with a single partition, is used as a
highly available write ahead log. All schemas, subject/version and ID metadata,
and compatibility settings are appended as messages to this log. A Schema
Registry instance therefore both produces and consumes messages under the
_schemas topic. It produces messages to the log when, for example, new schemas
are registered under a subject, or when updates to compatibility settings are
registered. Schema Registry consumes from the _schemas log in a background
thread, and updates its local caches on consumption of each new _schemas message
to reflect the newly added schema or compatibility setting. Updating local state
from the Kafka log in this manner ensures durability, ordering, and easy
recoverability.
Tip
The Schema Registry topic is compacted and therefore the latest value of every
key is retained forever, regardless of the Kafka retention policy. You can
validate this with kafka-configs:
kafka-configs --bootstrap-server localhost:9092 --entity-type topics --entity-name _schemas --describe
Copy
Your output should resemble:
Configs for topic '_schemas' are cleanup.policy=compact
Copy
SINGLE PRIMARY ARCHITECTURE¶
Schema Registry is designed to work as a distributed service using single
primary architecture. In this configuration, at most one Schema Registry
instance is the primary at any given moment (ignoring pathological ‘zombie
primaries’). Only the primary is capable of publishing writes to the underlying
Kafka log, but all nodes are capable of directly serving read requests.
Secondary nodes serve registration requests indirectly by simply forwarding them
to the current primary, and returning the response supplied by the primary.
Starting with Confluent Platform 4.0, primary election is accomplished with the
Kafka group protocol. (ZooKeeper based primary election was removed in Confluent
Platform 7.0.0. )
Note
Please make sure not to mix up the election modes amongst the nodes in same
cluster. This will lead to multiple primaries and issues with your operations.
KAFKA COORDINATOR PRIMARY ELECTION¶
Kafka based Schema Registry¶
Kafka based primary election is chosen when kafkastore.connection.url is not
configured and has the Kafka bootstrap brokers <kafkastore.bootstrap.servers>
specified. The Kafka group protocol, chooses one amongst the primary eligible
nodes leader.eligibility=true as the primary. Kafka based primary election
should be used in all cases. (ZooKeeper based leader election was removed in
Confluent Platform 7.0.0. See Migration from ZooKeeper primary election to Kafka
primary election.)
Schema Registry is also designed for multi-colocated configuration. See
Multi-Datacenter Setup for more details.
ZOOKEEPER PRIMARY ELECTION¶
Important
ZooKeeper leader election was removed in Confluent Platform 7.0.0. Kafka leader
election should be used instead. See Migration from ZooKeeper primary election
to Kafka primary election for full details.
HIGH AVAILABILITY FOR SINGLE PRIMARY SETUP¶
Many services in Confluent Platform are effectively stateless (they store state
in Kafka and load it on demand at start-up) and can redirect requests
automatically. You can treat these services as you would deploying any other
stateless application and get high availability features effectively for free by
deploying multiple instances. Each instance loads all of the Schema Registry
state so any node can serve a READ, and all nodes know how to forward requests
to the primary for WRITEs.
A recommended approach is to define multiple Schema Registry servers for high
availability clients using the Schema Registry instance URLs in the Schema
Registry client schema.registry.url property, thereby using the entire cluster
of Schema Registry instances and providing a method for failover.
This also makes it easy to handle changes to the set of servers without having
to reconfigure and restart all of your applications. The same strategy applies
to REST proxy or Kafka Connect.
A simple setup with just a few nodes means Schema Registry can fail over easily
with a simple multi-node deployment and single primary election protocol.
Alternatively, you can use a single URL for schema.registry.url and still use
the entire cluster of Schema Registry instances. However, this configuration
does not support failover to different Schema Registry instances in a dynamic
DNS or virtual IP setup because only one schema.registry.url is surfaced to
schema registry clients.
MIGRATE SCHEMAS (CONFLUENT CLOUD AND SELF-MANAGED)¶
Starting with Confluent Platform 7.0.0, Schema Linking is available in preview
on Confluent Platform, as described in Schema Linking on Confluent Platform.
This is the recommended method of migrating schemas. For migrating schemas from
one Confluent Cloud cluster to another, use cloud specific Schema Linking.
For pre Confluent Platform 7.0.0 releases, use Replicator with Schema
Translation to migrate schemas from a self-managed cluster to a target cluster
which is either self-managed or in Confluent Cloud. (This was first available in
Confluent Platform 5.2.0.)
* For a concept overview and quick start tutorial on migrating schemas from
self-managed clusters to Confluent Cloud using the legacy Replicator methods,
see Migrate Schemas to Confluent Cloud.
* For a demo of migrating schemas from one self-managed cluster to another
using the legacy Replicator methods, see Migrate Schemas and Replicator
Schema Translation Example for Confluent Platform.
LICENSE¶
Schema Registry is licensed under the Confluent Community License.
A Confluent Enterprise license is required for the Schema Registry Security
Plugin and for broker-side Schema Validation on Confluent Server.
You can use the plugin and Schema Validation under a 30-day trial period without
a license key, and thereafter under an Enterprise (Subscription) License as part
of Confluent Platform.
To learn more more about the security plugin, see License for Schema Registry
Security Plugin and Install and Configure the Schema Registry Security Plugin.
Tip
For complete license information for Confluent Platform, see Confluent Platform
Licenses.
COURSES AND BLOG POSTS¶
* Online Course: Schema Registry 101: Key Concepts of Schema Registry
* Schemas, Contracts, and Compatibility
* 17 Ways to Mess Up Self-Managed Schema Registry
* Yes, Virginia, You Really Do Need a Schema Registry
* How I Learned to Stop Worrying and Love the Schema
SUGGESTED READING¶
* Schema Registry Tutorials (Confluent Cloud and on-premises)
* Quick Start for Schema Management on Confluent Cloud
* Run an automated Confluent Cloud quickstart with Avro, Protobuf, and JSON
formats
* Use Control Center to manage schemas for on-premises deployments
* Installing and Configuring Schema Registry
* Running Schema Registry in Production
* Schema Validation on Confluent Server
* Schema Registry Configuration Options
* For Developers: Formats, Serializers, and Deserializers and Schema Registry
Development Overview.
Feedback
Confluent Cloud is a fully-managed Apache Kafka service available on all three
major clouds. Try it free today.
Get Started Free
* Confluent
* About
* Careers
* Contact
* Professional Services
* Product
* Confluent Cloud
* ksqlDB
* Developer
* Free Courses
* Tutorials
* Event Streaming Patterns
* Documentation
* Blog
* Podcast
* Community
* Forum
* Meetups
* Kafka Summit
* Catalysts
*
*
*
*
*
Terms & Conditions Privacy Policy Do Not Sell My Information Modern Slavery
Policy Cookie Settings Feedback
Copyright © Confluent, Inc. 2014- 2023 . Apache, Apache Kafka, Kafka, and
associated open source project names are trademarks of the Apache Software
Foundation
Feedback
On this page:
* Schema Registry Overview
* Schemas, Subjects, and Topics
* Kafka Serializers and Deserializers Background
* Avro, JSON, and Protobuf Supported Formats and Extensibility
* Schema ID Allocation
* Kafka Backend
* Single Primary Architecture
* Kafka Coordinator Primary Election
* ZooKeeper Primary Election
* High Availability for Single Primary Setup
* Migrate Schemas (Confluent Cloud and self-managed)
* License
* Courses and Blog Posts
* Suggested Reading
By clicking “Accept All Cookies”, you agree to the storing of cookies on your
device to enhance site navigation, analyze site usage, and assist in our
marketing efforts. Cookie Notice
Cookies Settings Reject All Accept All Cookies