During a meeting in late December, we discussed how 2020 was the year we had built NBA Top Shot's foundation, and that 2021 would be the year we'd bring NBA Top Shot to the next level.
That prediction came true much faster than we had anticipated. January launched NBA Top Shot to new heights with exponential growth. We now have 30K+ Moment-owning collectors on the platform and have reached nearly $50 million USD in Marketplace transactions alone.
We know that Top Shot Moments are the perfect intersection between the game’s greatest highlights and revolutionary technology on the Flow blockchain. Moments are scarce and permanent. They’re also highly valuable. Still, hundreds of millions of NBA fans around the world haven’t even heard about Top Shot yet, which makes all of this even more exciting long-term.
While we look forward to what the future holds, we also recognize that the recent growth brought us some pressing challenges. I'd like to address them here, give some perspective on what’s been happening under the hood, and share with you what we're going to do to resolve these short-term hiccups.
With exponential growth, we've hit scalability issues
This chart below shows the amount of transactions per hour in the last 8 weeks on the Marketplace alone. The significant peaks in December seem minuscule now.
We envision a lot more growth from here. We have a robust infrastructure that will scale well for the future, but in recent weeks we have not been able to handle spikes in traffic as well as expected. Most of the time, our systems perform exceptionally well. When traffic is concentrated in surge -- during drops -- for instance, we’ve been unable to meet our standards.
The Stress Test program has been successful so far. It's extremely difficult to replicate the levels of load and organic traffic we're experiencing in a controlled test environment. And again, as we’ve grown each week, it’s been helpful for us to track increased loads with each test.
We've been improving our systems significantly. Our backend infrastructure is scaled and our databases have been optimized. Caching, load balancing, and several other initiatives that are going on in the background are paving the way for the future.
Many of these various changes require database migrations and require synchronization between our cached events and databases, which are hard -- if not impossible -- to do while the Marketplace is operational. This is why we've had to go into Maintenance Mode so many times in recent weeks. We repeat: the product is still very much in beta and there's no better time to do large optimizations than now, so we appreciate everyone's support.
It’s fair to hope issues will completely resolve after a Maintenance window, or new packs will be dropped as consistently as they have been historically. We also understand that it’s frustrating when new issues emerge after a long Marketplace halt. But the reality is that progress is constant and phased in multiple steps, and isn’t always a linear path from the outside looking in. At Dapper Labs and NBA Top Shot, we're thinking and prioritizing long term - your collection is here to stay for generations to come, so we are prioritizing crossing important scalability milestones over simply dropping new packs into the ecosystem prematurely.
Even with many optimizations, why is the Marketplace sometimes out of sync or slow?
We know about these common utility issues with the Marketplace:
Inability to purchase some Moments, usually the lowest ask Moments.
This issue has two factors: (1) the lowest ask Moments are usually the ones many people attempt to buy at once, and it's expected now that we'll have instances of multiple people trying to buy the same Moment at the same time (2) we've had many instances of events getting stuck in our databases at the wrong state and not processing accurately. We believe that we solved most of the bugs that cause this to happen. But it continues to be helpful to report to our team if you see any instances of listings that you're unable to purchase, no matter how much time you wait.
In any instance where there is a lag, rest assured: everything is recorded on chain and safe, but out of sync issues cause real usability issues and we've been improving it significantly over the last couple of weeks.
Inability to list or delist Moments.
There is currently a known bug that prevents collectors from listing the same exact Moment twice from our Marketplace (example: if you sell the Moment, buy it again, and try to sell one more time). The other issue is the inability to delist Moments. In the past, we've seen out of sync issues afflict collectors when they tried to delist Moments: some collectors were essentially trying to delist Moments that had, in reality, already been sold, but the site didn't reflect that appropriately. We believe most of those issues have been fixed now - but do let us know if you face it again. Finally, if you list a Moment for the wrong price, it's possible that someone snaps it quickly. We know that it's frustrating if it gets sold before you intended to sell it, but we recommend treating listings as final. We are planning to add an additional guard rail down the road, warning you when your listing price looks noticeably out of sync with other Marketplace values for the same Moment.
Overall site performance.
We've recently upgraded some of our most underperforming backend queries and, as a result, have been able to improve Marketplace performance significantly. One clear example is Marketplace buffering speed. In just the past two days it’s improved from a slow query to an impressively quick search.
Some people seemed to have received old packs or Moments recently.
Last week, we resync'd our events, which caused several people to have Moments they had previously acquired show up in their collection (which is great), but caused our Community Feed to break and show those events as if they were new. Creative theories emerged on Discord about what had happened, but the truth is: we fixed one thing and broke another. We do not intend for that to happen again. But if it does, we will certainly let you know.
I can never get a pack. WTF?
Drops continue to hit record attendance and they sell out fast. Really fast. Even Base Set packs, which are intended to always be available (and a great entry point to understanding what Top Shot is all about) have been selling out in a matter of minutes.
If you still haven't been able to purchase a pack, we understand the frustration. To be clear, it should be expected that many collectors will leave pack drops without a pack in hand. That’s part of what makes pack drops so thrilling. We promise, when you do end up obtaining a pack, it’ll make the great Moments you pull even more special.
With that being said, pack drops need to be fair. Recently, we've been introducing limits per order and maximum concurrent orders per collector. This also caused its own set of issues (raise your hand if you've seen the "please complete your pending order first" message). We are still improving our drop system relentlessly.
We're also experimenting with various ways of operationalizing pack drops. Recently we tried to distribute Cool Cats packs in new ways. After failed drop attempts, we distributed guaranteed pack reservations to many of the collectors who tried but failed to obtain a Cool Cat pack the previous week. Additionally, we ran a raffle to give more users the opportunity to collect Cool Cats packs.
The experience was interesting, but it also brought us several issues:
- This initiative put an immense load of issues on our support and community teams, which delayed other areas that need active support (more on this in a bit).
- Some collectors tried to game the raffle, despite our multiple warnings. We have various detection tools and were able to flag most attempts, but that caused additional work for our anti-fraud team and delayed other efforts.
- Some collectors successfully gamed the raffle and got multiple packs. We estimate it's been around a couple dozen collectors. We will take swift action against those accounts, and I'll share a bit more on that soon.
I'd also like to say that if you successfully got a pack for $14, it isn't cool to instantly flip it or try to exploit the system. Trading on the Marketplace is awesome -- we love when community members are able to build valuable collections, but we will continue to make sure that we're offering packs to collectors who actually want them. As a reminder, we will enable collectors to list and sell their unopened packs on the Marketplace sometime in 2021.
How about bots? Are they causing all the issues?
As with any economy, principles of supply and demand ring true on NBA Top Shot. When demand is high and supply is intentionally low, it’s impossible for everyone to get a pack. For months, we had heard complaints from many who weren’t able to get a pack saying that bots were negatively affecting the fairness of our drops. We investigated actively, and each time there was little evidence to suggest any of that was true.
In the last two weeks, real evidence has emerged that bots have come to Top Shot. It’s become clear that bots have created an uneven playing field for our collectors, especially with regards to pack drops.
These are a few dozen people who commit offences that harm the entire community. We ask you all to support our common goal with a long term mindset, and do not take a path of exploitation of fellow community members for short term gains. This is unacceptable to us, and fundamentally goes against our Code of Conduct. Going forward, we are strictly disciplining any accounts that participate in nefarious activities with bot accounts (including but not limited to using scripts directly or purchasing packs/accounts from people who used scripts to acquire packs).
Part of the beauty of blockchain and the future of a decentralized ecosystem is that automated tools and smart contracts can actually prove to be helpful. Blockchain is literally designed around the idea of programmable exchange of value, so we don’t want to go against that long term vision when it comes to our Marketplace. In the interim, however, we plan to protect our website against botting to help enforce a strong collector experience -- with attention focused toward pack drops in particular.
Consider this a warning: encouraging bot usage around pack drops is strictly prohibited and may result in severe disciplinary action.
We have created a form for collectors in the community to help us identify and stop accounts who are using bots to game the system. If you help us identify and stop a bot (or nefarious collector) through reporting in this form, we will reward you with a lucrative bounty for your help. Similarly, if you have succeeded with gaming the system and see the writing on the wall for future exploits, we encourage you to share any vulnerabilities you’ve found in our architecture. We will reward you for your help.
On a tangential note: we also urge you to consolidate your collections into one account if you haven’t done so already, as advised in our Code of Conduct. We are instituting a one month moratorium for all collectors to consolidate their collections into one account. One account per collector is our new policy, and we will be enforcing this strictly in the coming weeks.
I've written to support many times and my problem hasn’t been resolved / I haven’t received a response yet.
Until very recently, our team had just two full-time employees dedicated to Customer Support. We are scaling up this team very quickly and have already significantly bolstered the size, but hiring new members to our team is a time consuming process and can’t nor should happen overnight. We apologize for any lapses in communication or inconveniences you’ve experienced with our platform in recent weeks. We are seeing thousands of tickets per day, and we’re working through them carefully and patiently.
Not only that, but you've likely seen many other team members supporting on Discord recently and through our email. From designers, to marketing, to engineers, to our CEO, we've all been assisting support whenever we can, spending nights and weekends to go through various support issues. We are still getting better and will improve here. You have our word.
I'm not saying this as an excuse for the delay with any support you’ve needed. You are the reason we built this product and you deserve our attention to resolve issues that come up while interacting with Top Shot. I'm saying this as a way to ask for your trust and patience, and to ask you to channel all communications through our official form. It seems counter-intuitive that submitting an issue to a form will be better than pinging us on Discord, but the form gives us the information we need to diagnose the problem and is far more effective in the interim. Without your username, past communications, or Moment information, we can't understand the issues and fix bugs.
Please also review here for the FAQs that we’re seeing. Perhaps this’ll help alleviate some frustrations or answer some of your questions preemptively. Rest assured, reinforcements are on the way for our team, and process improvements are on the way with regards to important issues like withdrawals. If you’ve had an issue sitting in our queue for over a week, please fill out this form to help us prioritize the issue in our sea of tickets.
Commitments and Priorities
While we certainly look forward to implementing many of the great features that have been suggested by the community, we are prioritizing improvements to our site’s performance, enhancing pack drops, and bolstering our Customer Support team to ensure we can resolve your individual issues at a faster rate. Those are our top priorities.
On the scaling and reliability front, here’s what we’re doing:
- We've been implementing a new pack drop system to handle spikes more effectively. We could see a pack drop in the next day or two if things go well (reminder: Base Set pack releases are not pre-announced, but any Limited Edition pack drop will be announced at least 8 hours in advance.)
- We've been fixing Marketplace performance and sync issues.
- We've been scaling all of our services across the board - from load balancing tools to database replicas, and everything in between.
- We've been improving our bot recognition systems and creating more secure ways of handling exploiters.
And even if (worst case scenario) none of these work this week, we're committed to continuing to think creatively and improve NBA Top Shot as quickly as possible, we'll try new drop systems and think long term to grow the ecosystem continuously.
With regard to identity checks, withdrawals, support, and community building:
- Identity checks: Yes, they’re tedious. But they are a necessary pain point in our system. They help ensure collections are protected and collectors are able to withdraw securely. Please never share any sensitive information via email. We will never ask for you to share your government ID’s via email.. We’ll also never initiate a conversation on Discord with you soliciting private information. Always check the email address or Discord account you’re interacting with. For all your identity check needs communication will come from compliance@meetdapper.com. For this project, you’ll only ever hear from us there or at support@nbatopshot.com / support@meetdapper.com. We have heard several instances of phishing in recent weeks so please stay vigilant.
- Withdrawals: Effective as of this week, we now provide collectors with more visibility into their withdrawal status. For more information, please read here. We are working to speed up this process significantly in the future, and once you’ve successfully withdrawn once, it’ll make it a faster experience for you for each withdrawal in the future.
- Support: We are significantly bolstering our Customer Support team at Dapper Labs. This will enable us to move through tickets more efficiently than ever before. To put a more tangible number on it, we plan to quintuple the size of the team this month. We know we have ground to make up here, but rest assured that we will make it up in due time. We appreciate your continued patience.
We're making information on the site more accessible, up to date, and visible in the various user flows:
- We now have an FAQ:
- Our Code of Conduct is accessible from the Marketplace:
- Refs, support captains, onboarding captains
We could not be more proud of our All-Star roster of community Refs, support captains, onboarding captains in our Discord. They put a lot of time and effort into making this community a better place, and their hard work shows daily. We've created more channels for communication on Discord, and this group of community leaders will help us spread information and build a nice place online in the future. Please treat them with respect. They’re here to help.
Rookie Badges and other features
We decided to announce and share Rookie Badges as early as possible a while back because we knew that it would have deep implications for the NBA Top Shot Marketplace. We concluded that it was best for the entire community to learn about the badges ahead of time, even before our site could reflect their existence.
This creates an even field for people to trade with the right information in mind. That being said, we understand that collectors expected the badges soon after the announcement. I want to reiterate our commitment here again and promise that we'll ship rookie badges on NBA Top Shot as soon as sensible. Our estimates on timing for this shifted a bit given our focus and rightful priorities with scalability, but you should expect them within the next 2 months.
Share more ideas and input here: https://nbatopshot.featureupvote.com/. While we can't promise that all of the ideas will be implemented or share more about which specific features we’re already working on, I can share that this board aligns very well with our roadmap and how we're planning to evolve NBA Top Shot.
Transparency
Recently I read a comment from a collector that inspired a lot of this blog post. The comment was pointing out that NBA Top Shot's team was usually available to talk about good news and promote initiatives, but usually dodged important questions like the ones addressed above.
That isn't the intention and not how communication between our team and the community will be long-term. I recognize that we can do better in providing clarity and transparency. We aren't dodging any questions, and are 100% invested in making this project as successful as it can be. We are committed to all of the aforementioned initiatives and will go above and beyond to become more transparent and understanding. We’ll share our progress and path as soon as possible.
As a way of creating an even more collaborative dialogue with the community, we'll be running Weekly Office Hours every Friday at 10 AM PST on Discord #general-voice chat channel with Jacob and Usman from the team. Feel free to join, ask questions in our #office-hours channel, and share ideas.
Final thoughts
Wow, you made it. Wasn’t that a lot of updates? Thank you for taking the time to read this special State of NBA Top Shot edition of the Dev Diary. As early community members, you're all vital to the success of NBA Top Shot. And we could not have reached our early milestones without the love, support, and backing of our early adopters in the community. You are all vested stakeholders in the success of our platform. As we improve our beta and build the future of NBA Top Shot, we'll need your help sharing the future of collectibles with your friends and network, promoting a healthy community of Top Shot fans on social media, and holding us accountable to build the best product we can.
- Arthur Camara
