Posted on 12 Comments

How we engineer cutting-edge AI applications

Here at Topaz Labs, we train modern AI models to “auto-magically” enhance images and video, and deliver them to our customers in desktop applications that run on their own hardware.   This provides a lot of value to our customers, but raises new challenges to our engineers in ensuring reliable software delivery.

We recently released new versions of our Sharpen AI and DeNoise AI applications using a next-generation version of our AI Engine.  The AI Engine is the part of our software which takes in the AI model, and processes an image through it using a variety of hardware-optimized libraries.  This new version of the AI Engine has resulted in some incredible improvements in speed, but unfortunately the launch of the 3.0 versions of Sharpen AI and DeNoise AI was rougher than expected.

Problems

We ran into problems where after the initial release, certain combinations of operating systems, CPUs, GPUs, and drivers resulted in crashes or sub-optimal performance. Our customers posted several issues on our forums, and our support ticket volumes sky-rocketed.

Fortunately, our team rallied, and isolated a few of the major reproducible issues. And our engineers quickly diagnosed and patched numerous issues in rapid order. However, for many of our customers, the damage was already done: they had gone through an upgrade of our software, and gotten a new version that was worse than what they had had before.

The issues in our release products affected our customers across three dimensions:

The severity of the issue for a given user
The number of users affected
The length of time that a given user is affected

Each one of those dimensions has a multiplying effect on the negative impact of a given released bug, and unfortunately these releases suffered from larger than hoped-for effects across all three dimensions.

Solutions

Now that the dust is settling on those recent releases – the team met and we came up with some ways to improve the quality of our product releases in a way that also still enables us to innovate quickly on new enhancements to our products:

Better Testing

We recently had engaged a new Quality Assurance partner. But unfortunately we didn’t ask them to test across a diverse enough set of operating systems, CPUs, and GPUs. We also made large changes to our software, and the way that it is installed for our users, after the testing runs had already completed.

To address these issues, we’ve updated the testing plans to incorporate 5-times the number of tested types of machines, and instituted a stricter feature-freeze period between testing and release. This will ensure that we catch more issues earlier, and that we minimize the chances that new issues are introduced after testing.

Smaller Impact

We’ve also adjusted our release schedule, so that products with similar new functionality will go out at different times. This means that if issues with a new common piece of functionality, like this new AI Engine, are encountered, that they one affect one product at a time, and can be fixed for that one product.

Faster Fixes

One of the new features that we rolled out with the new product installers, is the ability to optimize software updates. This means that when you update a piece of Topaz software, that the installer only downloads the parts of the application which have changed since the last version.

We’ve also changed our release processes to enable more “hotfixes”, which represent a fix that the engineers feel is highly impactful for our customers, and has a low risk of introducing new problems. Close followers of past Topaz releases may have seen us do these before, we’re just further enabling the team to be able to get these fixes out faster to our customers.

Continuous Improvement through Transparency

At Topaz Labs, we feel that some of the best lessons are learned when things do not go well.  So we endeavour to foster open and honest communication internally towards learning those lessons as a team.  We also aim to keep our promise of high-quality software delivery to our customers, and be honest when we feel we have fallen short.

Hopefully others may find some of the lessons we’ve learned useful for their own projects.  We’ve certainly been encouraged that we can take steps towards even better releases in the future.

Posted on

Announcement: Ongoing Product Value and Paid Upgrades

Hello everyone! If we haven’t already been acquainted, I’m Eric, CEO of Topaz Labs. I hope you’re off to a great start in 2020. I have an announcement that may seem like bad news at first, but will end up being good news in the long run:

Starting in August 2020, we will charge for product upgrades to DeNoise AI, Sharpen AI, Gigapixel AI, and Mask AI.

Every company says that they value their customers. When we started Topaz Labs many years ago, we tried hard to figure out ways to show this through actions instead of just words. One of the ways we did this was by offering free major upgrades for all of our products. Conceptually, this felt really good to us; we loved the idea of acting differently than faceless corporations by giving ongoing value for free instead of charging for every little thing.

Within the last few years, though, we started to realize something. Product development and research is expensive! It turns out that if we can only earn money by releasing new products, there’s a very powerful internal incentive to only do that. Despite our best intentions, we found that we weren’t focusing as much as we wanted on improving our existing products. We ended up with several different products that did similar things, which caused confusion about which tool to use. Although it started out with good intentions, our free upgrade policy ultimately prevented us from delivering the best possible products to you.

Make no mistake about it: this situation was 100% our fault. We were not thoughtful enough about aligning our pricing structure with continuous improvement of our products. Since realizing this, we took some major steps to improve in 2019:

  1. Consolidated all of our offerings into a single product per category (noise reduction, sharpening, effect creation, etc.)
  2. Massively shifted our internal goals to prioritize improving existing products over releasing new ones
  3. Stopped advertising free upgrades as a policy

Despite not advertising it anymore, in 2019 alone we were proud to give away over $8M worth of claimed free upgrades to major new products like DeNoise 6 => DeNoise AI, InFocus => Sharpen AI, Adjust 6 => Adjust AI, and Studio 2. We’ve already seen a major improvement in the usefulness and experience of our products, and we’re excited to continue down this path in 2020 and beyond.

The next step is to actually move to paid upgrades to finally align our pricing policy with how we want to build products as a company. Basically: it’s our fault, yet I’m asking you to lock arms with us and walk through this phase of our development by accepting this change. I know it’s a lot to ask for, but I hope you can agree that this path really is best for continued improvements to the products that we hope you’ve grown to rely upon.

Here’s how it will work:

Upgrade license renewals

  • After buying a product for the first time, you’ll receive 1 year of free major + minor upgrades from your date of purchase.
  • After your first year of free upgrades is complete, you can buy another year at any time in the future – for any single product for $49.99. If you own multiple products, you can activate a year of account-wide upgrades for $99.99.
  • Only DeNoise, Sharpen, Gigapixel, and Mask will require an upgrade license in 2020. Other products remain under the old policy.

We want you to own your software instead of having to pay to keep accessing it. Once you buy one of our products, you can keep using your purchased version as long as you want regardless of upgrade status. We’re really proud that some of you have continued to trust our products for over a decade after buying it once, and we’d love for this to continue.

Similarly, you should also be able to choose when to upgrade and not be penalized for it. If you don’t think a certain upgrade is worth it, don’t renew your upgrade license just yet. Wait until it actually includes something you want to pay for. This also incentivizes us to make sure that the improvements we do implement are useful enough for you to upgrade.

Example

As an example, let’s say you buy Gigapixel AI for the first time today (2/13/2020), so you receive free upgrades to Gigapixel AI until 2/13/2021.

We then release Gigapixel v4.5 in March, v5 in May, v5.1 in August, and v5.2 in December 2020. You receive all these upgrades for free as part of your initial 1-year upgrade license.

Now imagine we release Gigapixel v5.3 in May 2021 and you decide it’s not worth it. Even though your upgrade license has expired, you can continue using Gigapixel v5.2 (your last owned version) for as long as you like.

Gigapixel v5.3 in May 2021 has a feature you really want, so you purchase an upgrade then. You receive Gigapixel v5.3 and another year of free upgrades starting from the time you purchased the upgrade until May 2022.

Existing customers

For the listed products above, existing customers have an upgrade license starting from a year since they purchased. For example, if you bought DeNoise AI in November 2019, your upgrade license will last until November 2020.

That said, we thought it’d be unfair to immediately expire licenses for people that have purchased over a year ago. So for anyone in this situation, we’re extending a minimum 6 months of upgrade licenses. Even if you first bought our products years ago, the earliest you’d possibly need to pay for an upgrade is in August 2020.

Starting soon, you’ll be able to see the remaining time on your upgrade licenses in your Account.

Why?

I know this is a major shift. We’ve offered free upgrades for so long that it’s become part of what we’re known for. I guess this change all comes down to a single core idea: to be truly useful to you, we believe we should be offering fewer products that go deeper rather than a buffet of shallow products.

In other words, we don’t want to offer you novelties that you’ll stop using after a month. Instead, if you buy DeNoise AI, we want to help you handle noise reduction forever through our software. This means constantly evaluating and adding the most promising cutting-edge noise reduction techniques, relentlessly improving integration with your workflow, and continually adding major new functionality like batch processing (just released!) and selections. You shouldn’t have to worry about noise reduction ever again… or at the very least, you should be able to rely on us to worry about it for you. Paid upgrades are necessary for us to support this kind of deep focus.

Again, this was not a quick or easy decision to make, but we think it’s a necessary one in order for us to deliver massive value for you in the future. All I ask of you is that you give us a chance to prove this through our product improvements in 2020 and beyond.

Thanks so much,

Eric Yang
CEO, Topaz Labs

Posted on

Enter the Studio 2 Instagram Challenge to Win Free Topaz Software!

Let's See What You Can Do in Studio 2

Interested in winning up to $99 in Topaz credit towards the purchase of any Topaz Labs software? We’ve launched our Topaz Instagram Challenge! Share your best before and after images edited in the new Topaz Studio 2 for your chance to win.

How To Enter the Challenge

Enter the challenge in just three steps:

  1. Edit one of your best photos in Topaz Studio 2
  2. Upload the before and after images to your own Instagram account
  3. Follow @TopazLabs and include the #TopazLabs hashtag in your caption

We’ll be watching for your submissions on Instagram and giving away free Topaz credits throughout the month of September, ranging from $25-$99.

Download your free trial of Topaz Studio 2 to get started. 

Let’s see what you can do in Studio 2! 

Please see the complete Instagram Challenge terms and conditions