highscalability.com Open in urlscan Pro
2a06:98c1:3120::7 Public Scan

Back to summary

URL:
http://highscalability.com/
Submission: On May 04 via api (May 4th 2022, 5:37:47 am UTC) from GB — Scanned from GB

Form analysis
1 forms found in the DOM

GET /display/Search

<form method="get" action="/display/Search">
  <input type="hidden" name="moduleId" value="4876569">
  <table class="search-form-pt-wrapper search-form-table" border="0" cellpadding="0" cellspacing="0">
    <tbody>
      <tr class="search-form-pt search-form-row">
        <td class="queryFieldCell search-form-cell">
          <span class="queryFieldWrapper">
            <input type="text" class="text queryField" name="searchQuery" value="">
          </span>
        </td>
        <td class="queryButtonCell search-form-cell">
          <span class="queryButtonWrapper">
            <input type="submit" class="button queryButton" value="→">
          </span>
        </td>
      </tr>
    </tbody>
  </table>
</form>

Text Content

* Home
* Explain the Cloud Like I'm 10
* All Time Favorites
* Real Life Architectures
* Strategies
* Support in Patreon
* Advertising
* Start Here
* contact
* RSS
* Twitter
* Facebook
* G+
* All Posts
* Amazon Store

Top
* Home
* Explain the Cloud Like I'm 10
* All Time Favorites
* Real Life Architectures
* Strategies
* Support in Patreon
* Advertising
* Start Here
* contact
* RSS
* Twitter
* Facebook
* G+
* All Posts
* Amazon Store

Recent Posts
* Sponsored Post: Wynter, Pinecone, Kinsta, Bridgecrew, IP2Location, StackHawk,
InterviewCamp.io, Educative, Stream, Fauna, Triplebyte
* Stuff The Internet Says On Scalability For January 28th, 2022
* Designing Uber
* Designing Tinder
* Designing Instagram
* Designing WhatsApp
* Designing Netflix
* Sponsored Post: Wynter, Pinecone, Kinsta, Bridgecrew, IP2Location, StackHawk,
InterviewCamp.io, Educative, Stream, Fauna, Triplebyte
* Architecture of Max reHIT Workout
* Sponsored Post: Wynter, Pinecone, Kinsta, Bridgecrew, IP2Location, StackHawk,
InterviewCamp.io, Educative, Stream, Fauna, Triplebyte

advertise
* Login
* Register

* High Scalability RSS
* High Scalability Comments RSS

Tuesday
Feb012022

SPONSORED POST: WYNTER, PINECONE, KINSTA, BRIDGECREW, IP2LOCATION, STACKHAWK,
INTERVIEWCAMP.IO, EDUCATIVE, STREAM, FAUNA, TRIPLEBYTE

Tuesday, February 1, 2022 at 8:11AM

WHO'S HIRING?

* Wynter is looking for system administrators, engineers, and developers to
join its research panel. Participate in research surveys, get paid
($90-$180/hr) for your feedback and comments. Super low key commitment, 10-15
mins per survey. Learn more and sign up.

* DevOps Engineer: At Kinsta, we set out to create the best managed hosting
platform in the world. If you are an experienced DevOps Engineer who is
constantly looking for ways to innovate and improve, we might just be the
place for you! As Kinsta’s DevOps Engineer, you will be instrumental in
making sure that our infrastructure is always on the bleeding edge of
technology, remaining stable and high-performing at all times. If you love
working with Linux, have a background in PHP, and have worked with
cutting-edge technologies, such as Ansible and LXC, check us out and apply
here.

* SysOps Engineer: As Kinsta’s SysOps Engineer, you’ll play a key role in
caring for the health of our servers, preventing issues, and responding
immediately to mitigate any problems in our infrastructure. If you have
experience in hosting and with the WordPress stack, have worked with Ubuntu
or Debian-based systems, and cutting-edge technologies, such Ansible and LXC,
you should definitely check us out and apply here!

* JavaScript Developer: Kinsta’s Development Team aims to create truly
effective solutions for our customers and our internal teams. We believe the
only way to maintain and scale our standards is to focus on quality code. If
you are a talented JavaScript developer and you would like to work with an
ownership-embracing, highly skilled team in a relaxed environment where
culture is as important as technical excellence, apply here.

* InterviewCamp.io has hours of system design content. They also do live system
design discussions every week. They break down interview prep into
fundamental building blocks. Try out their platform.

* Triplebyte lets exceptional software engineers skip screening steps at
hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart.
Make your job search O(1), not O(n). Apply here.

* Need excellent people? Advertise your job here!

COOL PRODUCTS AND SERVICES

* Search like you mean it. Pinecone is a vector database that makes it easy to
add semantic search to your applications. It combines state-of-the-art vector
search libraries like Faiss, advanced features such as filtering, and fully
managed distributed infrastructure for high performance and reliability at
any scale. Give your users better search results and recommendations and make
them happy. Get started for free.

* Bridgecrew is the cloud security platform for developers. By leveraging
automation and delivering security-as-code, Bridgecrew empowers teams to
find, fix, and prevent misconfigurations in deployed cloud resources and in
infrastructure as code. Get started for free!

* IP2Location is IP address geolocation service provider since 2002. The
geolocation database or API detects location, proxy and other >20 parameters.
The technology has been cited in more than 700 research papers and trusted by
many Fortune 500 companies. Try it today!

* Developers care about shipping secure applications. Application security
products and processes, however, have not kept up with advances in software
development. There are a new breed of tools hitting the market that enable
developers to take the lead on AppSec. Learn how engineering teams are using
products like StackHawk and Snyk to add security bug testing to their CI
pipelines.

* Stateful JavaScript Apps. Effortlessly add state to your Javascript apps with
FaunaDB. Generous free tier. Try now!

* Learn to balance architecture trade-offs and design scalable enterprise-level
software. Check out Educative.io's 5-part learning path: Scalability and
System Design for Developers. In addition to other content written by
seasoned industry experts, this path includes core material from our
bestselling course Grokking the System Design Interview. Join more than
500,000 other learners from top tech companies around the world.

* Build, scale and personalize your news feeds and activity streams with
getstream.io. Try the API now in this 5 minute interactive
tutorial. Stream is free up to 3 million feed updates so it's easy to get
started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java
and .NET. Stream is currently also hiring Devops and Python/Go developers in
Amsterdam. More than 400 companies rely on Stream for their production feed
infrastructure, this includes apps with 30 million users. With your help we'd
like to ad a few zeros to that number.

FUN AND INTERESTING EVENTS

* Advertise here!

If you are interested in a sponsored post for an event, job, or product, please
contact us for more information.

--------------------------------------------------------------------------------

MAKE YOUR JOB SEARCH O(1) — NOT O(N)

Triplebyte is unique because they're a team of engineers running their own
centralized technical assessment. Companies like Apple, Dropbox, Mixpanel, and
Instacart now let Triplebyte-recommended engineers skip their own screening
steps.

We found that High Scalability readers are about 80% more likely to be in the
top bracket of engineering skill.

Take Triplebyte's multiple-choice quiz (system design and coding questions) to
see if they can help you scale your career faster.

--------------------------------------------------------------------------------

If you are interested in a sponsored post for an event, job, or product, please
contact us for more information.

Todd Hoff | Post a Comment | Permalink | Share Article Print Article Email
Article
in sponsored post

Friday
Jan282022

STUFF THE INTERNET SAYS ON SCALABILITY FOR JANUARY 28TH, 2022

Friday, January 28, 2022 at 9:12AM

Never fear, HighScalability is here!

Think your software diagram is complex? This is a single cell modeled using
X-ray, nuclear magnetic resonance (NMR), and cryo-electron microscopy datasets.
Gael McGill

MY STUFF:

* Love this Stuff? I need your support on Patreon to keep this stuff going.
* Know anyone who needs to fix their cloud-obliviousness? My book teaches them
all they need to know about the cloud. Explain the Cloud Like I'm 10. It has
364 mostly 5 star reviews on Amazon. Here's a 100% antibody free review:
* Do you like Zone 2, Zone 5, interval, reHIT, or HIIT workouts? I made an app
for that. Max reHIT Workout. I’m not just the programmer, I’m a client. I use
it 4 times a week and if you want to retrain your mitochondria, get fit, get
healthy, and live longer, I think you’ll like it too.

NUMBER STUFF:

Click to read more ...

Todd Hoff | 3 Comments | Permalink | Share Article Print Article Email Article
in hot links

Tuesday
Jan252022

DESIGNING UBER

Tuesday, January 25, 2022 at 9:11AM

This is a guest post by Ankit Sirmorya. Ankit is working as a Machine Learning
Lead/Sr. Machine Learning Engineer at Amazon and has led several
machine-learning initiatives across the Amazon ecosystem. Ankit has been working
on applying machine learning to solve ambiguous business problems and improve
customer experience. For instance, he created a platform for experimenting with
different hypotheses on Amazon product pages using reinforcement learning
techniques. Currently, he is in the Alexa Shopping organization where he is
developing machine-learning-based solutions to send personalized reorder hints
to customers for improving their experience.

REQUIREMENTS

IN SCOPE

Click to read more ...

Todd Hoff | 6 Comments | Permalink | Share Article Print Article Email Article
in Example

Monday
Jan172022

DESIGNING TINDER

Monday, January 17, 2022 at 9:11AM

PROBLEM STATEMENT

Design a location-based social search application similar to Tinder which if
often used as a dating service. It allows users to use a swiping motion to like
(swipe right) or dislike (swipe left) other users, and allows users to chat if
both parties like each other(a “match”).

GATHERING REQUIREMENTS

Click to read more ...

Todd Hoff | 1 Comment | Permalink | Share Article Print Article Email Article
in Example

Tuesday
Jan112022

DESIGNING INSTAGRAM

Tuesday, January 11, 2022 at 9:11AM

PROBLEM STATEMENT

Design a photo-sharing platform similar to Instagram where users can upload
their photos and share it with their followers. Subsequently, the users will be
able to view personalized feeds containing posts from all the other users that
they follow.

GATHERING REQUIREMENTS

IN SCOPE

The application should be able to support the following requirements.

* Users should be able to upload photos and view the photos they have uploaded.
* Users should be able to follow other users.
* Users can view feeds containing posts from the users they follow.
* Users should be able to like and comment the posts.

OUT OF SCOPE

* Sending and receiving messages from other users.
* Generating machine learning based personalized recommendations to discover
new people, photos, videos, and stories relevant one’s interest.

HIGH LEVEL DESIGN

ARCHITECTURE

When the server receives a request for an action (post, like etc.) from a client
it performs two parallel operations: i) persisting the action in the data store
ii) publish the action in a streaming data store for a pub-sub model. After
that, the various services (e.g. User Feed Service, Media Counter Service) read
the actions from the streaming data store and performs their specific tasks. The
streaming data store makes the system extensible to support other use-cases
(e.g. media search index, locations search index, and so forth) in future.

FUN FACT: In this talk, Rodrigo Schmidt, director of engineering at Instagram
talks about the different challenges they have faced in scaling the data
infrastructure at Instagram.

SYSTEM COMPONENTS

The system will comprise of several micro-services each performing a separate
task. We will use a graph database such as Neo4j to store the information. The
reason we have chosen a graph data-model is that our data will contain complex
relationships between data entities such as users, posts, and comments as nodes
of the graph. After that, we will use edges of the graph to store relationships
such as follows, likes, comments, and so forth. Additionally, we can use
columnar databases like Cassandra to store information like user feeds,
activities, and counters.

COMPONENT DESIGN

POSTING ON INSTAGRAM

Fig 2: Synchronous and Asynchronous process for posting on Instagram

There are two major processes which gets executed when a user posts a photo on
Instagram. Firstly, the synchronous process which is responsible for uploading
image content on file storage, persisting the media metadata in graph
data-storage, returning the confirmation message to the user and triggering the
process to update the user activity. The second process occurs asynchronously by
persisting user activity in a columnar data-storage(Cassandra) and triggering
the process to pre-compute the feed of followers of non-celebrity users (having
few thousand followers). We don’t pre-compute feeds for celebrity users (have
1M+ followers) as the process to fan-out the feeds to all the followers will be
extremely compute and I/O intensive.

API DESIGN

We have provided the API design of posting an image on Instagram below. We will
send the file and data over in one request using the multipart/form-data content
type. The MultiPart/Form-Data contains a series of parts. Each part is expected
to contain a content-disposition header [RFC 2183] where the disposition type is
"form-data”.

PRECOMPUTE FEEDS

This process gets executed when non-celebrity users makes a post on Instagram.
It’s triggered when a message is added in the User Feed Service Queue. Once the
message is added in the queue, the User Feed Service makes a call to the
Followers Service to fetch the list of followers of the user. After that, the
post gets added to the feed of all the followers in the columnar data storage.

FETCHING USER FEED

When a user requests for feed then there will be two parallel threads involved
in fetching the user feeds to optimize for latency. The first thread will fetch
the feeds from non-celebrity users which the user follow. These feeds are
populated by the fan-out mechanism described in the PreCompute Feeds section
above. The second thread is responsible for fetching the feeds of celebrity
users whom the user follow. After that, the User Feed Service will merge the
feeds from celebrity and non-celebrity users and return the merged feeds to the
user who requested the feed.

API DESIGN

DATA MODELS

GRAPH DATA MODELS

We can use a graph database such as Neo4j which stores data-entities such as
user information, posts, comments, and so forth as nodes in the graph. The edges
between the nodes are used to store the relationship between data entities such
as followers, posts, comments, likes, and replies. All the nodes are added to an
index called nodeIndex for faster lookups. We have chosen this NoSQL based
solution over relational databases as it provides the scalability to have
hierarchies which go beyond two levels and extensibility due to the schema-less
behavior of NoSQL data storage.

SAMPLE QUERIES SUPPORTED BY GRAPH DATABASE

FETCH ALL THE FOLLOWERS OF JEFF BEZOS

Node jeffBezos = nodeIndex.get(“userId”, “user004”);
List jeffBezosFollowers = new ArrayList();

for (Relationship relationship: jeffBezos.getRelationships(INGOING, FOLLOWS)) {
jeffBezosFollowers.add(relationship.getStartNode());
}

FETCH ALL THE POSTS OF BILL GATES

Node billGates = nodeIndex.get(“userId”, “user001”);
List billGatesPosts = new ArrayList();

for (Relationship relationship: billGates.getRelationships(OUTGOING, POSTS)) {
billGatesPosts.add(relationship.getEndNode());
}

FETCH ALL THE POSTS OF BILL GATES ON WHICH JEFF BEZOS HAS COMMENTED

List commentsOnBillGatesPosts = new ArrayList<>();

for(Node billGatesPost : billGatesPosts) {
for (Relationship relationship: billGates.getRelationships(INGOING, COMMENTED_ON)) {
commentsOnBillGatesPosts.add(relationship.getStartNode());
}
}

List jeffBezosComments = new ArrayList();

for (Relationship relationship: jeffBezos.getRelationships(OUTGOING, COMMENTS)) {
jeffBezosComments.add(relationship.getEndNode());
}

List jeffBezosCommentsOnBillGatesPosts = commentsOnBillGatesPosts.intersect(jeffBezosComments);

COLUMNAR DATA MODELS

We will use columnar data storage such as Cassandra to store data entities like
user feed and activities. Each row will contain feed/activity information of the
user. We can also have a TTL based functionality to evict older posts. The data
model will look something similar to:

User_id -> List

FUN FACT: In this talk, Dikang Gu, a software engineer at Instagram core infra
team has mentioned about how they use Cassandra to serve critical usecases, high
scalability requirements, and some pain points.

STREAMING DATA MODEL

We can use cloud technologies such as Amazon Kinesis or Azure Stream Analytics
for collecting, processing, and analyzing real-time, streaming data to get
timely insights and react quickly to new information(e.g. a new like, comment,
etc.). We have listed below the de-normalized form of some major streaming data
entities and action.

The data entities A and B above show the containers which contain denormalized
information about the Users and their Posts. Subsequently, the data entities C
and D denote the different actions which users may take. The entity C denotes
the event where a user likes a post and entity D denotes the action when a user
follows another user. These actions are read by the related micro-services from
the stream and processed accordingly. For instance, the LikeEvent can be read by
the Media Counter Service and is used to update the media count in the data
storage.

OPTIMIZATION

We will use a cache having an LRU based eviction policy for caching user feeds
of active users. This will not only reduce the overall latency in displaying the
user-feeds to users but will also prevent re-computation of user-feeds.

Another scope of optimization lies in providing the best content in the user
feeds. We can do this by ranking the new feeds (the ones generated after users
last login) from those who the user follows. We can apply machine learning
techniques to rank the user feeds by assigning scores to the individual feeds
which would indicate the probability of click, like, comment and so forth. We
can do this by representing each feed by a feature vector which contains
information about the user, the feed and the interactions which the user has had
with the people in the feed (e.g. whether the user had clicked/liked/commented
on the previous feeds by the people in the story). It’s apparent that the most
important features for feed ranking will be related to social network. Some of
the keys of understanding the user network are listed below.

* Who is the user a close follower of? For example, one user is a close
follower of Elon Musk while another user can be a close follower of Gordon
Ramsay.
* Whose photos the user always like?
* Whose links are most interesting to the user?

We can use deep neural networks which would take the several features (> 100K
dense features) which we require for training the model. Those features will be
passed through the n-fold layers, and will be used for predicting the
probability of the different events (likes, comments, shares, etc.).

FUN FACT: In this talk, Lars Backstrom, VP of Engineering @ Facebook talks about
the machine learning done to create personalized news feeds for users. He talks
about the classical machine learning approach they used in the initial phases
for personalizing News Feeds by using decision trees and logistic regression. He
then goes to talk about the improvements they have observed in using neural
networks.

REFERENCES

* https://www.youtube.com/watch?v=_BfMH4GQWnk
* https://www.youtube.com/watch?v=hnpzNAPiC0E
* https://instagram-engineering.com/what-powers-instagram-hundreds-of-instances-dozens-of-technologies-adf2e22da2ad
* https://instagram-engineering.com/types-for-python-http-apis-an-instagram-story-d3c3a207fdb7
* http://highscalability.com/blog/2012/4/9/the-instagram-architecture-facebook-bought-for-a-cool-billio.html
* https://docs.oracle.com/cloud/latest/marketingcs_gs/OMCAC/op-api-rest-1.0-assets-image-content-post.html
* https://instagram-engineering.com/under-the-hood-instagram-in-2015-8e8aff5ab7c2

Todd Hoff | 9 Comments | Permalink | Share Article Print Article Email Article
in Example

Monday
Jan032022

DESIGNING WHATSAPP

Monday, January 3, 2022 at 9:11AM

PROBLEM STATEMENT

Design an instant messenger platform such as WhatsApp or Signal which users can
utilize tosend messages to each other. An essential aspect of the application is
that the chat messageswon’t be permanently stored in the application.

FUN FACT: Some of the chat messengers such as FB Messenger stores the chat
messages unless the users explicitly delete it. However, instant messengers such
as WhatsApp don’t save the messages permanently on their server.

GATHERING REQUIREMENTS

Click to read more ...

Todd Hoff | 8 Comments | Permalink | Share Article Print Article Email Article
in Example

Monday
Dec132021

DESIGNING NETFLIX

Monday, December 13, 2021 at 9:11AM

PROBLEM STATEMENT

Design a video streaming platform similar to Netflix where content creators can
upload their video content and viewers are able to play video on different
devices. We should also be able to store user statistics of the videos such as
number of views, video watched duration, and so forth.

GATHERING REQUIREMENTS

Click to read more ...

Todd Hoff | Post a Comment | Permalink | Share Article Print Article Email
Article
in Example

Thursday
Dec092021

SPONSORED POST: WYNTER, PINECONE, KINSTA, BRIDGECREW, IP2LOCATION, STACKHAWK,
INTERVIEWCAMP.IO, EDUCATIVE, STREAM, FAUNA, TRIPLEBYTE

Thursday, December 9, 2021 at 3:43PM

WHO'S HIRING?

* InterviewCamp.io has hours of system design content. They also do live system
design discussions every week. They break down interview prep into
fundamental building blocks. Try out their platform.

* Triplebyte lets exceptional software engineers skip screening steps at
hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart.
Make your job search O(1), not O(n). Apply here.

* Need excellent people? Advertise your job here!

COOL PRODUCTS AND SERVICES

* Stateful JavaScript Apps. Effortlessly add state to your Javascript apps with
FaunaDB. Generous free tier. Try now!

FUN AND INTERESTING EVENTS

* Advertise here!

If you are interested in a sponsored post for an event, job, or product, please
contact us for more information.

--------------------------------------------------------------------------------

MAKE YOUR JOB SEARCH O(1) — NOT O(N)

We found that High Scalability readers are about 80% more likely to be in the
top bracket of engineering skill.

Take Triplebyte's multiple-choice quiz (system design and coding questions) to
see if they can help you scale your career faster.

--------------------------------------------------------------------------------

If you are interested in a sponsored post for an event, job, or product, please
contact us for more information.

Todd Hoff | Post a Comment | Permalink | Share Article Print Article Email
Article
in sponsored post

Tuesday
Nov302021

ARCHITECTURE OF MAX REHIT WORKOUT

Tuesday, November 30, 2021 at 9:11AM

Today I have the pleasure of announcing my new app—Max reHIT Workout—on Product
Hunt. Max reHIT Workout is an exercise app that guides you through interval
workouts.

I won’t pitch the app here. I'll just say I’m proud of how it turned out and if
you want an optimal algorithm for exercising, you might like it.

I know I haven’t been writing much lately. That's because there’s been very
little evolution in software system architecture. It’s pretty much same thing,
different day. In many ways that’s good, but it’s not interesting to write
about.

This article, while definitely self serving, targets the choice of using a
native iOS environment versus a cloud environment for an app. It’s a choice
every developer must make. How do you make that choice? What are the
implications? What choice would I make next time?

OUT WITH THE OLD

Click to read more ...

Todd Hoff | 3 Comments | Permalink | Share Article Print Article Email Article

Wednesday
Oct272021

SPONSORED POST: WYNTER, PINECONE, KINSTA, BRIDGECREW, IP2LOCATION, STACKHAWK,
INTERVIEWCAMP.IO, EDUCATIVE, STREAM, FAUNA, TRIPLEBYTE

Wednesday, October 27, 2021 at 1:45PM

WHO'S HIRING?

* InterviewCamp.io has hours of system design content. They also do live system
design discussions every week. They break down interview prep into
fundamental building blocks. Try out their platform.

* Triplebyte lets exceptional software engineers skip screening steps at
hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart.
Make your job search O(1), not O(n). Apply here.

* Need excellent people? Advertise your job here!

COOL PRODUCTS AND SERVICES

* Stateful JavaScript Apps. Effortlessly add state to your Javascript apps with
FaunaDB. Generous free tier. Try now!

FUN AND INTERESTING EVENTS

* Advertise here!

If you are interested in a sponsored post for an event, job, or product, please
contact us for more information.

--------------------------------------------------------------------------------

MAKE YOUR JOB SEARCH O(1) — NOT O(N)

We found that High Scalability readers are about 80% more likely to be in the
top bracket of engineering skill.

Take Triplebyte's multiple-choice quiz (system design and coding questions) to
see if they can help you scale your career faster.

--------------------------------------------------------------------------------

If you are interested in a sponsored post for an event, job, or product, please
contact us for more information.

Todd Hoff | Post a Comment | Permalink | Share Article Print Article Email
Article
in sponsored post

highscalability.com Open in urlscan Pro 2a06:98c1:3120::7 Public Scan

Form analysis 1 forms found in the DOM

GET /display/Search

Text Content

highscalability.com Open in urlscan Pro
2a06:98c1:3120::7 Public Scan

Form analysis
1 forms found in the DOM