- 34
- 1 707 338
Interview Pen
United States
Приєднався 3 жов 2022
Our mission is to provide high-quality educational content, community, & tools to empower technologists (software engineers, etc) looking to succeed in upscaling their careers.
Design a Scalable BLOB Store | System Design
Visit Our Website: interviewpen.com/?
Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6
Like & Subscribe: youtube.com/@interviewpen
This video goes over how to design a BLOB or object storage system that can handle petabytes of data storage while handling hardware and network failures.
If you liked this video and want to learn more about fundamental systems and how they can be used to solve problems, check out our full course on interviewpen.com/? !
Table of Contents:
0:00 - Introduction
0:50 - Single Disk
1:17 - Striped Disks
2:45 - Mirrored Disks
3:40 - Parity Disks
4:47 - Sharding
6:10 - Replicated Nodes
6:56 - Parity Nodes
8:33 - Visit interviewpen.com
Socials:
Twitter: InterviewPen
LinkedIn: www.linkedin.com/company/interviewpen
Website: interviewpen.com/?
Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6
Like & Subscribe: youtube.com/@interviewpen
This video goes over how to design a BLOB or object storage system that can handle petabytes of data storage while handling hardware and network failures.
If you liked this video and want to learn more about fundamental systems and how they can be used to solve problems, check out our full course on interviewpen.com/? !
Table of Contents:
0:00 - Introduction
0:50 - Single Disk
1:17 - Striped Disks
2:45 - Mirrored Disks
3:40 - Parity Disks
4:47 - Sharding
6:10 - Replicated Nodes
6:56 - Parity Nodes
8:33 - Visit interviewpen.com
Socials:
Twitter: InterviewPen
LinkedIn: www.linkedin.com/company/interviewpen
Website: interviewpen.com/?
Переглядів: 4 975
Відео
How do Databases Work? | System Design
Переглядів 20 тис.14 днів тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Like & Subscribe: youtube.com/@interviewpen This video goes over how databases work internally specifically how they parse and execute SQL queries in the most efficient way possible. If you liked this video and want to learn more about fundamental systems and how they can be used to solve problems, check o...
Design an Email Automation Platform (Klaviyo, Mailchimp) | System Design
Переглядів 7 тис.Місяць тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Join Our Newsletter - The Blueprint: theblueprint.dev/subscribe Like & Subscribe: youtube.com/@interviewpen This is an example of a full video available on interviewpen.com. Check out our website to find more premium content like this! Problem Statement: Design an email automation platform, like Klaviyo or...
Design a Code Execution System | System Design
Переглядів 10 тис.2 місяці тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Like & Subscribe: youtube.com/@interviewpen This video shows you how to design a code execution service that can run code submitted by users and return a result. This kind of system is used on interviewpen.com for our practice problems! If you liked this video and want to learn more about fundamental syste...
Design a Data Warehouse | System Design
Переглядів 18 тис.2 місяці тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Like & Subscribe: youtube.com/@interviewpen Many large companies wind up with several sources of data and want to use all of them together to make business decisions. This video dives into the low-level details of designing a data warehousing system to bring in these data sources, transform them, and analy...
Design an ML Recommendation Engine | System Design
Переглядів 8 тис.2 місяці тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Like & Subscribe: youtube.com/@interviewpen Tons of modern software services, such as social media and ecommerce, include systems for recommending content to users. This video dives into the details of how to deploy a machine learning algorithm at scale for recommendations, and discusses how these systems ...
Shorten Unix Path | Stacks and Strings
Переглядів 1 тис.3 місяці тому
Visit Our Website: interviewpen.com/? Join Our Discord: discord.gg/Qy85PT9wj6 Join Our Newsletter - The Blueprint: theblueprint.dev/subscribe Like & Subscribe: youtube.com/@interviewpen This is an example of a full video available on interviewpen.com. Check out our website to find more premium content like this! Problem Statement: Write a function that takes in a non-empty string representing a...
What makes Kafka special? | System Design
Переглядів 15 тис.3 місяці тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Like & Subscribe: youtube.com/@interviewpen You've probably heard about Kafka, but you really know why everyone is so excited about it? This video dives into the low-level details of what makes Kafka fundamentally different from other message queue systems, and when you should use it over other solutions. ...
Design a Fault Tolerant E-commerce System | System Design
Переглядів 27 тис.4 місяці тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Like & Subscribe: youtube.com/@interviewpen This video dives into the fundamentals of designing fault tolerant systems a critical skill to succeed in system design interviews. We'll dive into a basic e-commerce system and discuss all its failure modes and how to overcome them. If you liked this video and w...
Design a Video Streaming Protocol (HLS, DASH) | System Design
Переглядів 4,9 тис.4 місяці тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Like & Subscribe: youtube.com/@interviewpen Systems like UA-cam are able to deliver videos to a wide range of clients without latency or interruption. How are these protocols designed and how do they adapt to different networks? If you liked this video and want to learn more about fundamental systems and h...
Design a Faster App Store | System Design
Переглядів 4,7 тис.4 місяці тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Join Our Newsletter - The Blueprint: theblueprint.dev/subscribe Like & Subscribe: youtube.com/@interviewpen We've all experienced times when the App Store search is slow. This video dives into a high level overview of a (consistently) performant and scalable system to enable users to search, browse, and do...
Design a Distributed Geospatial Data Platform | System Design
Переглядів 4,7 тис.5 місяців тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Join Our Newsletter - The Blueprint: theblueprint.dev/subscribe Like & Subscribe: youtube.com/@interviewpen In this video, we discuss a high-level design of a geospatial data aggregation platform. This system would be responsible for ingesting multiple formats of data from a variety of sources, aggregating...
How does Netflix's CDN scale to over 100TB/s? | System Design
Переглядів 23 тис.5 місяців тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Join Our Newsletter - The Blueprint: theblueprint.dev/subscribe Like & Subscribe: youtube.com/@interviewpen To meet the extreme demands of streaming high-quality video to ~250M users, Netflix has developed a unique content delivery solution. Netflix's CDN, called Open Connect, enables ISPs responsible for ...
Design a High-Throughput Logging System | System Design
Переглядів 42 тис.5 місяців тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Join Our Newsletter - The Blueprint: theblueprint.dev/subscribe Like & Subscribe: youtube.com/@interviewpen Logging systems are commonly found in large systems with multiple moving parts. For these high-throughput real-time systems, there are a number of challenges and considerations at scale. This video g...
How a CDN Works | System Design
Переглядів 15 тис.6 місяців тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Join Our Newsletter - The Blueprint: theblueprint.dev/subscribe Like & Subscribe: youtube.com/@interviewpen A content delivery network (CDN) is a fundamental component for any system that needs to serve content to users across the world. A CDN caches websites, images, videos, and more on servers that are p...
Docker in 5 Minutes | System Design
Переглядів 6 тис.8 місяців тому
Docker in 5 Minutes | System Design
AWS Engineer vs. Greedy Algorithm! | Software Engineering Mock Interviews (#005)
Переглядів 3,5 тис.11 місяців тому
AWS Engineer vs. Greedy Algorithm! | Software Engineering Mock Interviews (#005)
AWS Engineer Gets STUMPED In Interview! | Software Engineering Mock Interviews (#004)
Переглядів 4,3 тис.Рік тому
AWS Engineer Gets STUMPED In Interview! | Software Engineering Mock Interviews (#004)
Ex-Twitter Software Engineer vs. Breadth-First Search! | Software Engineering Mock Interviews (#003)
Переглядів 3,4 тис.Рік тому
Ex-Twitter Software Engineer vs. Breadth-First Search! | Software Engineering Mock Interviews (#003)
Amazon Software Engineer vs. Binary Search! | Software Engineering Mock Interviews (#002)
Переглядів 3,3 тис.Рік тому
Amazon Software Engineer vs. Binary Search! | Software Engineering Mock Interviews (#002)
Disjoint-Set Data Structure (Union-Find) | Fast Subset Checking
Переглядів 10 тис.Рік тому
Disjoint-Set Data Structure (Union-Find) | Fast Subset Checking
AWS Software Engineer vs String Manipulation! | Software Engineering Mock Interviews (#001)
Переглядів 4,5 тис.Рік тому
AWS Software Engineer vs String Manipulation! | Software Engineering Mock Interviews (#001)
K Closest Points to the Origin | Brute Force, Sorting, Heaps, & Reduction
Переглядів 3 тис.Рік тому
K Closest Points to the Origin | Brute Force, Sorting, Heaps, & Reduction
Why is Comparison Sorting Ω(n*log(n))? | Asymptotic Bounding & Time Complexity
Переглядів 3 тис.Рік тому
Why is Comparison Sorting Ω(n*log(n))? | Asymptotic Bounding & Time Complexity
Design Google Drive or Dropbox (Cloud File Sharing Service) | System Design Interview Prep
Переглядів 80 тис.Рік тому
Design Google Drive or Dropbox (Cloud File Sharing Service) | System Design Interview Prep
Design a Basic Search Engine (Google or Bing) | System Design Interview Prep
Переглядів 391 тис.Рік тому
Design a Basic Search Engine (Google or Bing) | System Design Interview Prep
System Design for Twitter (Timeline, Live Updates, Tweeting) | System Design Interview Prep
Переглядів 60 тис.Рік тому
System Design for Twitter (Timeline, Live Updates, Tweeting) | System Design Interview Prep
Design a Simple Authentication System | System Design Interview Prep
Переглядів 152 тис.Рік тому
Design a Simple Authentication System | System Design Interview Prep
Basic System Design for Uber or Lyft | System Design Interview Prep
Переглядів 752 тис.Рік тому
Basic System Design for Uber or Lyft | System Design Interview Prep
Looking for a System Design Lead (or Co-Founder)
Переглядів 3,1 тис.Рік тому
Looking for a System Design Lead (or Co-Founder)
What is the name of the pen softwere used for explaining on the screen?
I feel it doesn't make sense to put chunks in queue, direct client having access to a messaging-queue-system is not practically good idea from security perspective. Also it doesn't reduce load anyhow as messaging queues also need to be scaled if not injestion servers, so it is just adding extra layer just for the sake of adding. Correct me if I am wrong.
Only messaging queue for notification service makes sense.
The idea behind this was that if there are bursts of load, it wouldn't slow down users uploading their data. But I fully agree with you that it doesn't make sense for a client to have direct access, so it's not a very useful solution in this case. A better solution might be to use a tiered storage system behind our BLOB store which can provide very fast reads and writes for frequently accessed data while moving older data to cheaper storage mediums. Thanks for watching!
@@interviewpen Thanks for the prompt response and answer! Great content btw finished watching blob storage system design after this.
Can you do a design for versions control system?
I'll add it to the backlog :)
Good explanation
Thanks!
Hello! Which app do you use in your iPad to draw diagrams and explain the concepts? Great video! 👏
Thanks! We use GoodNotes.
Why not enable RAID and set the correct level?
my thought exactly. but people these days need to complicate everything
Assume you have to store across multiple computers, RAID doesn’t save you from the host attached to the RAID card dying
When we're dealing with a single node, RAID is of course a great way to implement the concepts we discussed. When dealing with more than one node, we have to do replication over a network, which requires software solutions. Thanks for watching!
You always need to think about replicas
nice
Thanks!
Its nice you showed both options using JWTs and simple session IDs. I see lots of people jump straight for JWTs because its newer, without thinking about the actual pros and cons. Generally the first question should be *can* your system actually do stateless authentication? For many financial services for example, a business requirement will be instant reflection of updates to permissions or session revocations - to implement this in JWTs you need to constantly check a blacklisted token table - immediately nullifying the benefit of stateless auth. There's also the question for many apps - realistically how many types of action can the user make anyway that won't be hitting the database / reddis at some point already? Considering the complexity that can come with JWTs, for simple apps I often find myself reaching for session IDs first, with certain components using stateless. I've yet to work in a company where even a few minute lag between permission updates is acceptable on the app itself. There's also some hybrid patterns, the satellite app authenticates with an internal auth server, recieving a JWT. That JWT is then hashed into reddis as a session token, and sent to the user for user <-> api auth. It's important people actually ask these questions to the interviewer before picking their solution!
Sir, this is golden. Thank you!
Thanks for watching!
Here are some of the issues in this design. 1. The cache and timeline construction are the most difficult to solve, yet we know too little on how it is arranged 1.1. Sure, if you have only 2 users, it will scale well. But, what if you have 180M followers to Musk? Will you fan-out 180M cache and database updates for a single tweet? 1.2. It completely ignores geography. What if the DB shard for Elon is in Texas and his followers are spread across the world? 1.2.1 Where do you store data for a user from a specific country? What if they travel? 2. Social graph. 2.1. It sounds like on every tweet write, you query the social graph to figure out where to fan out. How does this work to scale? How many shards do you need to read to know even where to fan out too? 2.2. What if a user unfollows? 2.3. Where is the graph stored. The design presented will not scale well to the millions and billions of users.
Thanks for the thoughts. You're right that this design fans out to all followers, so we're sacrificing write speed for users with many followers in order to maintain read performance for the majority of users. As you note, there's many other details we could dive into, such as using a social graph and implementing geographic sharding. Thanks for watching!
Client supplying timestamp "is a bad idea. Let's do it anyway"
Awesome, which software do you use?
We use GoodNotes on an iPad. Thanks for watching!
Thanks for the video! which platform are you using to note?
We use GoodNotes on an iPad.
Thanks, I really like the videos! But (in my personal opinion) it would be much easier to perceive information if you took some pauses when you speak. Otherwise, videos are really helpful! And one of the topics I would love to know more about is how systems like Apache Spark or Hadoop work internally, what are the specific use cases and etc. Thanks again!
Ok, thanks for the feedback. We do have multiple videos on Spark/Hadoop in our full course on interviewpen.com :)
I'm dumb so any help would be appreciated. Why are hash join on 100 rows tables faster than hash join on 10 rows tables ?
It isn’t-we prefer the plan that filters the tables first, since fewer rows have to be joined that way. Thanks!
Thank you for sharing this! This seems to be a common system design problem. I've gotten a variant of this at least twice now from companies that I've been interviewing with
would you be available for consulting
Sorry, not at the moment. Thanks for watching!
I feel like my Pokemon is about to evolve!
🎯 Key points for quick navigation: 00:00 *📚 What is a database and how queries work?* - Database takes in SQL queries to retrieve data. - SQL is declarative, focusing on what data is needed not how to get it. - Internal complexities of a database translate declarative queries into execution steps. 02:09 *🧠 Process of executing a query in a database* - Database query process involves parsing, query planning, and execution. - Parser translates SQL query into a data structure for the database to use. - Query planner optimizes query execution and evaluates different plans based on database characteristics. 03:32 *🔍 Role of parser in database query execution* - Parser converts raw SQL text into a parse tree for easier database evaluation. - Parse tree organizes query components like select, from, and where clauses for database use. - Use of parse tree allows the database to recursively evaluate query components. 08:12 *⚙️ Query planner in optimizing database query execution* - Query planner considers table statistics, cost constants, and available indexes for query optimization. - It generates multiple plans, evaluates costs for each, and outputs the most efficient plan. - Optimizing query planning leads to faster and more efficient database query execution. Made with HARPA AI
This interview misses critical discussions about the way we send new tweets to the user. I think this system doesn't work as expected. specially having a limit list in Memcached. For example, what if someone has 30 million followers?
Yes, having a large number of followers would increase latency when making a new tweet. This is a tradeoff-we’re optimizing for fast timeline loads over fast posts. Thanks for watching!
Your videos are very helpful and to the point.
Glad you like it :)
Solve the payment processing by switching to bitcoin only payments.
Awesome thanks a lot
Thanks for watching!
how algorithms works video plz
Thank you a lot for your video. Maybe next time tell us about the pros and cons of SQL? For example, that is invented in the 80s and that it is ineffective when working with a large amount of data?
Absolutely--we have a full video about different data models and storage formats (like NoSQL) on interviewpen.com :)
CS degree help me understand it pretty clearly. 😅
Nice🎉
Thanks!
Nice🎉
Thanks for watching!
Seems like I came early
🫡
First viewer. Hurrah
Nice 😛
What I like most about your videos is that they are short, crisp, to the point, great visuals and no BS
Thank you, glad you like them!
helpful but shallow. there's gotta be more details. what if the machines running the queues crash? What if publisher publishes faster than consumer consumes? what protocol is used to communicate? if multiple producers are producing, what orders are enforced?
If you want an in-depth video on how queues are used in fault-tolerant systems, check out interviewpen.com :)
Do we need those when using AWS or GCP?
Depending on what services you use for your cloud infrastructure, some of this will be managed for you. However, it’s always important to understand fault tolerance and ensure the service you’re using meets your needs.
very bad explanation. and the architecture is also not crisp. too naive
Really good explanation and walkthrough, thank you!
Thanks!
Very well done. Cool.
Thank you!
what type of education do you follow in order to know all of this stuff. its really impressive
Thanks! If you want more info on this topic, check here: openconnect.netflix.com/en/
W mans
awesome explained brother
Thanks!
awesome explained
Thank you!
Hi, I really like your channel and have learned a lot from it. However, in this particular case, I think your solution is not good enough because it mixes application with infrastructure. In this case, a Docker in Docker solution would work much better. With such a solution, you would separate the process of load balancing and providing access to your service from the application itself, for which it would be necessary to create an environment for the given language on an ongoing basis. Besides that, keep up the good work. I always look forward to your next videos!
Thanks for the suggestion, I chose Kubernetes here since it allows the infrastructure executing the code to scale separately from the API servers. Of course, the API could be deployed on any infrastructure system we want. But docker in docker would work as well and has the benefit of being easier to set up for a smaller scale system. Thanks for watching!
Too much emphasis on calculations. In a real system design interview , candidates will spend 2 mins max on calculations. Anything more you are screwed
I don’t agree-one of the most important parts of the system design interview is showing that you can translate product requirements into a solution that fits the use case. This means understanding the load that will be placed on each part of the system. Thanks for watching!
Your Channel is very good
Thank you!
Why not use Kafka for high through put?
Kafka is an event streaming platform, so it wouldn't solve any of the log storage problems we're addressing here. But if you have any thoughts on how to incorporate it, feel free to share!
@@interviewpen Use kafka stream + cassandra . process the event through consumers and save inside a Hbase db for analytics .
I had insomnia, but now, thanks to the vlog, I fall asleep within minutes watching the videos!
What software are you using on your iPad to draw and record?
We use GoodNotes on an iPad.
thank you.
Thanks for watching!
what is your work experience?
love this type of content! looking forward to more
Thanks!