So, Amazon actually kind of agrees! They talked about this very issue of hot keys and overprovisioning at their most recent conference. The thing is it was buried in a session on Aurora and they didn't mention DynamoDB by name -- they just called it nosql -- but they noted that a customer cut their costs 40% by moving to Aurora. Because it does automatic heat management and just bills you for what IO you use.
This is somewhat at odds with their top-level messaging which still pushes DynamoDB as the most scalable solution. And perhaps it is... there are some scalability limits to Aurora. Writes are bottlenecked by one instance. 64TB max. I think performance drops when you exceed the in memory cache. But those limits are still quite large.
Basically I sense some tension between the DynamoDB and Aurora teams and I wonder where this is all going to shake out in the long run.
Here's the full quote (I transcribed it so may contain errors):
"The one thing that surprised me is that there are some customers who are moving their nosql workload to aurora. There are two reasons for that. One, it’s a lot easier to use Aurora because it’s mysql compatible compared to nosql because the interfaces and transaction characteristics are so different. What is also interesting is people also saved money because the IO cost is much lower. In no SQL if you have a large table it gets partitioned then the IO gets partitioned across all the table partitions that you have. And if you have one partition that is hot then you have to provision based on the IO requirement of the hot partition. In the case of Aurora we do automatic heat management so we don’t have this hot parition issue. Second we don’t charge based on provisioned IO. It’s only the IO that you use. And that actually saves a lot of money. In this particular case this is a big social company, interaction company, I cannot tell the name, and they reduced their operational costs by 40% by moving from nosql to Aurora" [1]
This is somewhat at odds with their top-level messaging which still pushes DynamoDB as the most scalable solution. And perhaps it is... there are some scalability limits to Aurora. Writes are bottlenecked by one instance. 64TB max. I think performance drops when you exceed the in memory cache. But those limits are still quite large.
Basically I sense some tension between the DynamoDB and Aurora teams and I wonder where this is all going to shake out in the long run.
Here's the full quote (I transcribed it so may contain errors):
"The one thing that surprised me is that there are some customers who are moving their nosql workload to aurora. There are two reasons for that. One, it’s a lot easier to use Aurora because it’s mysql compatible compared to nosql because the interfaces and transaction characteristics are so different. What is also interesting is people also saved money because the IO cost is much lower. In no SQL if you have a large table it gets partitioned then the IO gets partitioned across all the table partitions that you have. And if you have one partition that is hot then you have to provision based on the IO requirement of the hot partition. In the case of Aurora we do automatic heat management so we don’t have this hot parition issue. Second we don’t charge based on provisioned IO. It’s only the IO that you use. And that actually saves a lot of money. In this particular case this is a big social company, interaction company, I cannot tell the name, and they reduced their operational costs by 40% by moving from nosql to Aurora" [1]
[1] https://youtu.be/60QumD2QsF0?t=17m01s