Interview Pen
Interview Pen
  • 34
  • 1 707 338
Design a Scalable BLOB Store | System Design
Visit Our Website: interviewpen.com/?
Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6
Like & Subscribe: youtube.com/@interviewpen
This video goes over how to design a BLOB or object storage system that can handle petabytes of data storage while handling hardware and network failures.
If you liked this video and want to learn more about fundamental systems and how they can be used to solve problems, check out our full course on interviewpen.com/? !
Table of Contents:
0:00 - Introduction
0:50 - Single Disk
1:17 - Striped Disks
2:45 - Mirrored Disks
3:40 - Parity Disks
4:47 - Sharding
6:10 - Replicated Nodes
6:56 - Parity Nodes
8:33 - Visit interviewpen.com
Socials:
Twitter: InterviewPen
LinkedIn: www.linkedin.com/company/interviewpen
Website: interviewpen.com/?
Переглядів: 4 975

Відео

How do Databases Work? | System Design
Переглядів 20 тис.14 днів тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Like & Subscribe: youtube.com/@interviewpen This video goes over how databases work internally specifically how they parse and execute SQL queries in the most efficient way possible. If you liked this video and want to learn more about fundamental systems and how they can be used to solve problems, check o...
Design an Email Automation Platform (Klaviyo, Mailchimp) | System Design
Переглядів 7 тис.Місяць тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Join Our Newsletter - The Blueprint: theblueprint.dev/subscribe Like & Subscribe: youtube.com/@interviewpen This is an example of a full video available on interviewpen.com. Check out our website to find more premium content like this! Problem Statement: Design an email automation platform, like Klaviyo or...
Design a Code Execution System | System Design
Переглядів 10 тис.2 місяці тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Like & Subscribe: youtube.com/@interviewpen This video shows you how to design a code execution service that can run code submitted by users and return a result. This kind of system is used on interviewpen.com for our practice problems! If you liked this video and want to learn more about fundamental syste...
Design a Data Warehouse | System Design
Переглядів 18 тис.2 місяці тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Like & Subscribe: youtube.com/@interviewpen Many large companies wind up with several sources of data and want to use all of them together to make business decisions. This video dives into the low-level details of designing a data warehousing system to bring in these data sources, transform them, and analy...
Design an ML Recommendation Engine | System Design
Переглядів 8 тис.2 місяці тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Like & Subscribe: youtube.com/@interviewpen Tons of modern software services, such as social media and ecommerce, include systems for recommending content to users. This video dives into the details of how to deploy a machine learning algorithm at scale for recommendations, and discusses how these systems ...
Shorten Unix Path | Stacks and Strings
Переглядів 1 тис.3 місяці тому
Visit Our Website: interviewpen.com/? Join Our Discord: discord.gg/Qy85PT9wj6 Join Our Newsletter - The Blueprint: theblueprint.dev/subscribe Like & Subscribe: youtube.com/@interviewpen This is an example of a full video available on interviewpen.com. Check out our website to find more premium content like this! Problem Statement: Write a function that takes in a non-empty string representing a...
What makes Kafka special? | System Design
Переглядів 15 тис.3 місяці тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Like & Subscribe: youtube.com/@interviewpen You've probably heard about Kafka, but you really know why everyone is so excited about it? This video dives into the low-level details of what makes Kafka fundamentally different from other message queue systems, and when you should use it over other solutions. ...
Design a Fault Tolerant E-commerce System | System Design
Переглядів 27 тис.4 місяці тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Like & Subscribe: youtube.com/@interviewpen This video dives into the fundamentals of designing fault tolerant systems a critical skill to succeed in system design interviews. We'll dive into a basic e-commerce system and discuss all its failure modes and how to overcome them. If you liked this video and w...
Design a Video Streaming Protocol (HLS, DASH) | System Design
Переглядів 4,9 тис.4 місяці тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Like & Subscribe: youtube.com/@interviewpen Systems like UA-cam are able to deliver videos to a wide range of clients without latency or interruption. How are these protocols designed and how do they adapt to different networks? If you liked this video and want to learn more about fundamental systems and h...
Design a Faster App Store | System Design
Переглядів 4,7 тис.4 місяці тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Join Our Newsletter - The Blueprint: theblueprint.dev/subscribe Like & Subscribe: youtube.com/@interviewpen We've all experienced times when the App Store search is slow. This video dives into a high level overview of a (consistently) performant and scalable system to enable users to search, browse, and do...
Design a Distributed Geospatial Data Platform | System Design
Переглядів 4,7 тис.5 місяців тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Join Our Newsletter - The Blueprint: theblueprint.dev/subscribe Like & Subscribe: youtube.com/@interviewpen In this video, we discuss a high-level design of a geospatial data aggregation platform. This system would be responsible for ingesting multiple formats of data from a variety of sources, aggregating...
How does Netflix's CDN scale to over 100TB/s? | System Design
Переглядів 23 тис.5 місяців тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Join Our Newsletter - The Blueprint: theblueprint.dev/subscribe Like & Subscribe: youtube.com/@interviewpen To meet the extreme demands of streaming high-quality video to ~250M users, Netflix has developed a unique content delivery solution. Netflix's CDN, called Open Connect, enables ISPs responsible for ...
Design a High-Throughput Logging System | System Design
Переглядів 42 тис.5 місяців тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Join Our Newsletter - The Blueprint: theblueprint.dev/subscribe Like & Subscribe: youtube.com/@interviewpen Logging systems are commonly found in large systems with multiple moving parts. For these high-throughput real-time systems, there are a number of challenges and considerations at scale. This video g...
How a CDN Works | System Design
Переглядів 15 тис.6 місяців тому
Visit Our Website: interviewpen.com/? Join Our Discord (24/7 help): discord.gg/Qy85PT9wj6 Join Our Newsletter - The Blueprint: theblueprint.dev/subscribe Like & Subscribe: youtube.com/@interviewpen A content delivery network (CDN) is a fundamental component for any system that needs to serve content to users across the world. A CDN caches websites, images, videos, and more on servers that are p...
Docker in 5 Minutes | System Design
Переглядів 6 тис.8 місяців тому
Docker in 5 Minutes | System Design
AWS Engineer vs. Greedy Algorithm! | Software Engineering Mock Interviews (#005)
Переглядів 3,5 тис.11 місяців тому
AWS Engineer vs. Greedy Algorithm! | Software Engineering Mock Interviews (#005)
AWS Engineer Gets STUMPED In Interview! | Software Engineering Mock Interviews (#004)
Переглядів 4,3 тис.Рік тому
AWS Engineer Gets STUMPED In Interview! | Software Engineering Mock Interviews (#004)
Ex-Twitter Software Engineer vs. Breadth-First Search! | Software Engineering Mock Interviews (#003)
Переглядів 3,4 тис.Рік тому
Ex-Twitter Software Engineer vs. Breadth-First Search! | Software Engineering Mock Interviews (#003)
Amazon Software Engineer vs. Binary Search! | Software Engineering Mock Interviews (#002)
Переглядів 3,3 тис.Рік тому
Amazon Software Engineer vs. Binary Search! | Software Engineering Mock Interviews (#002)
Disjoint-Set Data Structure (Union-Find) | Fast Subset Checking
Переглядів 10 тис.Рік тому
Disjoint-Set Data Structure (Union-Find) | Fast Subset Checking
AWS Software Engineer vs String Manipulation! | Software Engineering Mock Interviews (#001)
Переглядів 4,5 тис.Рік тому
AWS Software Engineer vs String Manipulation! | Software Engineering Mock Interviews (#001)
K Closest Points to the Origin | Brute Force, Sorting, Heaps, & Reduction
Переглядів 3 тис.Рік тому
K Closest Points to the Origin | Brute Force, Sorting, Heaps, & Reduction
Why is Comparison Sorting Ω(n*log(n))? | Asymptotic Bounding & Time Complexity
Переглядів 3 тис.Рік тому
Why is Comparison Sorting Ω(n*log(n))? | Asymptotic Bounding & Time Complexity
Design Google Drive or Dropbox (Cloud File Sharing Service) | System Design Interview Prep
Переглядів 80 тис.Рік тому
Design Google Drive or Dropbox (Cloud File Sharing Service) | System Design Interview Prep
Design a Basic Search Engine (Google or Bing) | System Design Interview Prep
Переглядів 391 тис.Рік тому
Design a Basic Search Engine (Google or Bing) | System Design Interview Prep
System Design for Twitter (Timeline, Live Updates, Tweeting) | System Design Interview Prep
Переглядів 60 тис.Рік тому
System Design for Twitter (Timeline, Live Updates, Tweeting) | System Design Interview Prep
Design a Simple Authentication System | System Design Interview Prep
Переглядів 152 тис.Рік тому
Design a Simple Authentication System | System Design Interview Prep
Basic System Design for Uber or Lyft | System Design Interview Prep
Переглядів 752 тис.Рік тому
Basic System Design for Uber or Lyft | System Design Interview Prep
Looking for a System Design Lead (or Co-Founder)
Переглядів 3,1 тис.Рік тому
Looking for a System Design Lead (or Co-Founder)

КОМЕНТАРІ

  • @abdullahsharif4291
    @abdullahsharif4291 5 годин тому

    What is the name of the pen softwere used for explaining on the screen?

  • @vinayak6564
    @vinayak6564 7 днів тому

    I feel it doesn't make sense to put chunks in queue, direct client having access to a messaging-queue-system is not practically good idea from security perspective. Also it doesn't reduce load anyhow as messaging queues also need to be scaled if not injestion servers, so it is just adding extra layer just for the sake of adding. Correct me if I am wrong.

    • @vinayak6564
      @vinayak6564 7 днів тому

      Only messaging queue for notification service makes sense.

    • @interviewpen
      @interviewpen 7 днів тому

      The idea behind this was that if there are bursts of load, it wouldn't slow down users uploading their data. But I fully agree with you that it doesn't make sense for a client to have direct access, so it's not a very useful solution in this case. A better solution might be to use a tiered storage system behind our BLOB store which can provide very fast reads and writes for frequently accessed data while moving older data to cheaper storage mediums. Thanks for watching!

    • @vinayak6564
      @vinayak6564 7 днів тому

      @@interviewpen Thanks for the prompt response and answer! Great content btw finished watching blob storage system design after this.

  • @vietnguyenquoc4948
    @vietnguyenquoc4948 11 днів тому

    Can you do a design for versions control system?

  • @cool_guy_Vaibhav
    @cool_guy_Vaibhav 11 днів тому

    Good explanation

  • @CristianAbrante
    @CristianAbrante 12 днів тому

    Hello! Which app do you use in your iPad to draw diagrams and explain the concepts? Great video! 👏

  • @abhibhagat7767
    @abhibhagat7767 12 днів тому

    Why not enable RAID and set the correct level?

    • @shabarinathk8954
      @shabarinathk8954 12 днів тому

      my thought exactly. but people these days need to complicate everything

    • @ralphard
      @ralphard 12 днів тому

      Assume you have to store across multiple computers, RAID doesn’t save you from the host attached to the RAID card dying

    • @interviewpen
      @interviewpen 10 днів тому

      When we're dealing with a single node, RAID is of course a great way to implement the concepts we discussed. When dealing with more than one node, we have to do replication over a network, which requires software solutions. Thanks for watching!

  • @Zmey5656
    @Zmey5656 12 днів тому

    You always need to think about replicas

  • @ehza
    @ehza 12 днів тому

    nice

  • @tidenly
    @tidenly 13 днів тому

    Its nice you showed both options using JWTs and simple session IDs. I see lots of people jump straight for JWTs because its newer, without thinking about the actual pros and cons. Generally the first question should be *can* your system actually do stateless authentication? For many financial services for example, a business requirement will be instant reflection of updates to permissions or session revocations - to implement this in JWTs you need to constantly check a blacklisted token table - immediately nullifying the benefit of stateless auth. There's also the question for many apps - realistically how many types of action can the user make anyway that won't be hitting the database / reddis at some point already? Considering the complexity that can come with JWTs, for simple apps I often find myself reaching for session IDs first, with certain components using stateless. I've yet to work in a company where even a few minute lag between permission updates is acceptable on the app itself. There's also some hybrid patterns, the satellite app authenticates with an internal auth server, recieving a JWT. That JWT is then hashed into reddis as a session token, and sent to the user for user <-> api auth. It's important people actually ask these questions to the interviewer before picking their solution!

  • @androiduser457
    @androiduser457 13 днів тому

    Sir, this is golden. Thank you!

  • @DavidDLee
    @DavidDLee 15 днів тому

    Here are some of the issues in this design. 1. The cache and timeline construction are the most difficult to solve, yet we know too little on how it is arranged 1.1. Sure, if you have only 2 users, it will scale well. But, what if you have 180M followers to Musk? Will you fan-out 180M cache and database updates for a single tweet? 1.2. It completely ignores geography. What if the DB shard for Elon is in Texas and his followers are spread across the world? 1.2.1 Where do you store data for a user from a specific country? What if they travel? 2. Social graph. 2.1. It sounds like on every tweet write, you query the social graph to figure out where to fan out. How does this work to scale? How many shards do you need to read to know even where to fan out too? 2.2. What if a user unfollows? 2.3. Where is the graph stored. The design presented will not scale well to the millions and billions of users.

    • @interviewpen
      @interviewpen 12 днів тому

      Thanks for the thoughts. You're right that this design fans out to all followers, so we're sacrificing write speed for users with many followers in order to maintain read performance for the majority of users. As you note, there's many other details we could dive into, such as using a social graph and implementing geographic sharding. Thanks for watching!

  • @DavidDLee
    @DavidDLee 15 днів тому

    Client supplying timestamp "is a bad idea. Let's do it anyway"

  • @webcorejuan
    @webcorejuan 15 днів тому

    Awesome, which software do you use?

    • @interviewpen
      @interviewpen 15 днів тому

      We use GoodNotes on an iPad. Thanks for watching!

  • @webcorejuan
    @webcorejuan 15 днів тому

    Thanks for the video! which platform are you using to note?

  • @Travala
    @Travala 16 днів тому

    Thanks, I really like the videos! But (in my personal opinion) it would be much easier to perceive information if you took some pauses when you speak. Otherwise, videos are really helpful! And one of the topics I would love to know more about is how systems like Apache Spark or Hadoop work internally, what are the specific use cases and etc. Thanks again!

    • @interviewpen
      @interviewpen 15 днів тому

      Ok, thanks for the feedback. We do have multiple videos on Spark/Hadoop in our full course on interviewpen.com :)

  • @congminhluu5068
    @congminhluu5068 17 днів тому

    I'm dumb so any help would be appreciated. Why are hash join on 100 rows tables faster than hash join on 10 rows tables ?

    • @interviewpen
      @interviewpen 15 днів тому

      It isn’t-we prefer the plan that filters the tables first, since fewer rows have to be joined that way. Thanks!

  • @xxbighotshotxx
    @xxbighotshotxx 17 днів тому

    Thank you for sharing this! This seems to be a common system design problem. I've gotten a variant of this at least twice now from companies that I've been interviewing with

  • @djseber3357
    @djseber3357 17 днів тому

    would you be available for consulting

    • @interviewpen
      @interviewpen 15 днів тому

      Sorry, not at the moment. Thanks for watching!

  • @Kevin-cg7qk
    @Kevin-cg7qk 18 днів тому

    I feel like my Pokemon is about to evolve!

  • @tonynguyen-mq3tb
    @tonynguyen-mq3tb 18 днів тому

    🎯 Key points for quick navigation: 00:00 *📚 What is a database and how queries work?* - Database takes in SQL queries to retrieve data. - SQL is declarative, focusing on what data is needed not how to get it. - Internal complexities of a database translate declarative queries into execution steps. 02:09 *🧠 Process of executing a query in a database* - Database query process involves parsing, query planning, and execution. - Parser translates SQL query into a data structure for the database to use. - Query planner optimizes query execution and evaluates different plans based on database characteristics. 03:32 *🔍 Role of parser in database query execution* - Parser converts raw SQL text into a parse tree for easier database evaluation. - Parse tree organizes query components like select, from, and where clauses for database use. - Use of parse tree allows the database to recursively evaluate query components. 08:12 *⚙️ Query planner in optimizing database query execution* - Query planner considers table statistics, cost constants, and available indexes for query optimization. - It generates multiple plans, evaluates costs for each, and outputs the most efficient plan. - Optimizing query planning leads to faster and more efficient database query execution. Made with HARPA AI

  • @amirphl7834
    @amirphl7834 19 днів тому

    This interview misses critical discussions about the way we send new tweets to the user. I think this system doesn't work as expected. specially having a limit list in Memcached. For example, what if someone has 30 million followers?

    • @interviewpen
      @interviewpen 15 днів тому

      Yes, having a large number of followers would increase latency when making a new tweet. This is a tradeoff-we’re optimizing for fast timeline loads over fast posts. Thanks for watching!

  • @mohitnagpal
    @mohitnagpal 19 днів тому

    Your videos are very helpful and to the point.

  • @thelostandunfounds
    @thelostandunfounds 19 днів тому

    Solve the payment processing by switching to bitcoin only payments.

  • @kspv2806
    @kspv2806 19 днів тому

    Awesome thanks a lot

  • @ak-gi3eu
    @ak-gi3eu 20 днів тому

    how algorithms works video plz

  • @Zmey5656
    @Zmey5656 20 днів тому

    Thank you a lot for your video. Maybe next time tell us about the pros and cons of SQL? For example, that is invented in the 80s and that it is ineffective when working with a large amount of data?

    • @interviewpen
      @interviewpen 19 днів тому

      Absolutely--we have a full video about different data models and storage formats (like NoSQL) on interviewpen.com :)

  • @saramshshrestha7641
    @saramshshrestha7641 20 днів тому

    CS degree help me understand it pretty clearly. 😅

  • @NimaqAlizadeh
    @NimaqAlizadeh 20 днів тому

    Nice🎉

  • @NimaqAlizadeh
    @NimaqAlizadeh 20 днів тому

    Nice🎉

  • @dami970
    @dami970 20 днів тому

    Seems like I came early

  • @infinitibucket
    @infinitibucket 20 днів тому

    First viewer. Hurrah

  • @prashantkumardhotre5695
    @prashantkumardhotre5695 21 день тому

    What I like most about your videos is that they are short, crisp, to the point, great visuals and no BS

    • @interviewpen
      @interviewpen 20 днів тому

      Thank you, glad you like them!

  • @yuzhao5268
    @yuzhao5268 25 днів тому

    helpful but shallow. there's gotta be more details. what if the machines running the queues crash? What if publisher publishes faster than consumer consumes? what protocol is used to communicate? if multiple producers are producing, what orders are enforced?

    • @interviewpen
      @interviewpen 24 дні тому

      If you want an in-depth video on how queues are used in fault-tolerant systems, check out interviewpen.com :)

  • @kimchi_taco
    @kimchi_taco 26 днів тому

    Do we need those when using AWS or GCP?

    • @interviewpen
      @interviewpen 24 дні тому

      Depending on what services you use for your cloud infrastructure, some of this will be managed for you. However, it’s always important to understand fault tolerance and ensure the service you’re using meets your needs.

  • @aadill77
    @aadill77 26 днів тому

    very bad explanation. and the architecture is also not crisp. too naive

  • @ericmargolis1852
    @ericmargolis1852 28 днів тому

    Really good explanation and walkthrough, thank you!

  • @sijifalore8252
    @sijifalore8252 29 днів тому

    Very well done. Cool.

  • @shaileshchotoe
    @shaileshchotoe Місяць тому

    what type of education do you follow in order to know all of this stuff. its really impressive

    • @interviewpen
      @interviewpen 24 дні тому

      Thanks! If you want more info on this topic, check here: openconnect.netflix.com/en/

  • @coffeebytes3257
    @coffeebytes3257 Місяць тому

    W mans

  • @codewithawaisahmad
    @codewithawaisahmad Місяць тому

    awesome explained brother

  • @codewithawaisahmad
    @codewithawaisahmad Місяць тому

    awesome explained

  • @mateuszsiwiecki6267
    @mateuszsiwiecki6267 Місяць тому

    Hi, I really like your channel and have learned a lot from it. However, in this particular case, I think your solution is not good enough because it mixes application with infrastructure. In this case, a Docker in Docker solution would work much better. With such a solution, you would separate the process of load balancing and providing access to your service from the application itself, for which it would be necessary to create an environment for the given language on an ongoing basis. Besides that, keep up the good work. I always look forward to your next videos!

    • @interviewpen
      @interviewpen 24 дні тому

      Thanks for the suggestion, I chose Kubernetes here since it allows the infrastructure executing the code to scale separately from the API servers. Of course, the API could be deployed on any infrastructure system we want. But docker in docker would work as well and has the benefit of being easier to set up for a smaller scale system. Thanks for watching!

  • @pradgarimella
    @pradgarimella Місяць тому

    Too much emphasis on calculations. In a real system design interview , candidates will spend 2 mins max on calculations. Anything more you are screwed

    • @interviewpen
      @interviewpen 24 дні тому

      I don’t agree-one of the most important parts of the system design interview is showing that you can translate product requirements into a solution that fits the use case. This means understanding the load that will be placed on each part of the system. Thanks for watching!

  • @SaveCount-bh8tp
    @SaveCount-bh8tp Місяць тому

    Your Channel is very good

  • @ankushraj3599
    @ankushraj3599 Місяць тому

    Why not use Kafka for high through put?

    • @interviewpen
      @interviewpen Місяць тому

      Kafka is an event streaming platform, so it wouldn't solve any of the log storage problems we're addressing here. But if you have any thoughts on how to incorporate it, feel free to share!

    • @RaushanKumar-co3wj
      @RaushanKumar-co3wj Місяць тому

      @@interviewpen Use kafka stream + cassandra . process the event through consumers and save inside a Hbase db for analytics .

  • @Gorlung
    @Gorlung Місяць тому

    I had insomnia, but now, thanks to the vlog, I fall asleep within minutes watching the videos!

  • @TatendaZhou
    @TatendaZhou Місяць тому

    What software are you using on your iPad to draw and record?

  • @praweewongsa
    @praweewongsa Місяць тому

    thank you.

  • @pahehepaa4182
    @pahehepaa4182 Місяць тому

    what is your work experience?

  • @sambhavmishra5423
    @sambhavmishra5423 Місяць тому

    love this type of content! looking forward to more