How we engineer cutting-edge AI applications

Here at Topaz Labs, we train modern AI models to “auto-magically” enhance images and video, and deliver them to our customers in desktop applications that run on their own hardware.   This provides a lot of value to our customers, but raises new challenges to our engineers in ensuring reliable software delivery.

We recently released new versions of our Sharpen AI and DeNoise AI applications using a next-generation version of our AI Engine.  The AI Engine is the part of our software which takes in the AI model, and processes an image through it using a variety of hardware-optimized libraries.  This new version of the AI Engine has resulted in some incredible improvements in speed, but unfortunately the launch of the 3.0 versions of Sharpen AI and DeNoise AI was rougher than expected.

Problems

We ran into problems where after the initial release, certain combinations of operating systems, CPUs, GPUs, and drivers resulted in crashes or sub-optimal performance. Our customers posted several issues on our forums, and our support ticket volumes sky-rocketed.

Fortunately, our team rallied, and isolated a few of the major reproducible issues. And our engineers quickly diagnosed and patched numerous issues in rapid order. However, for many of our customers, the damage was already done: they had gone through an upgrade of our software, and gotten a new version that was worse than what they had had before.

The issues in our release products affected our customers across three dimensions:

The severity of the issue for a given user
The number of users affected
The length of time that a given user is affected

Each one of those dimensions has a multiplying effect on the negative impact of a given released bug, and unfortunately these releases suffered from larger than hoped-for effects across all three dimensions.

Solutions

Now that the dust is settling on those recent releases – the team met and we came up with some ways to improve the quality of our product releases in a way that also still enables us to innovate quickly on new enhancements to our products:

Better Testing

We recently had engaged a new Quality Assurance partner. But unfortunately we didn’t ask them to test across a diverse enough set of operating systems, CPUs, and GPUs. We also made large changes to our software, and the way that it is installed for our users, after the testing runs had already completed.

To address these issues, we’ve updated the testing plans to incorporate 5-times the number of tested types of machines, and instituted a stricter feature-freeze period between testing and release. This will ensure that we catch more issues earlier, and that we minimize the chances that new issues are introduced after testing.

Smaller Impact

We’ve also adjusted our release schedule, so that products with similar new functionality will go out at different times. This means that if issues with a new common piece of functionality, like this new AI Engine, are encountered, that they one affect one product at a time, and can be fixed for that one product.

Faster Fixes

One of the new features that we rolled out with the new product installers, is the ability to optimize software updates. This means that when you update a piece of Topaz software, that the installer only downloads the parts of the application which have changed since the last version.

We’ve also changed our release processes to enable more “hotfixes”, which represent a fix that the engineers feel is highly impactful for our customers, and has a low risk of introducing new problems. Close followers of past Topaz releases may have seen us do these before, we’re just further enabling the team to be able to get these fixes out faster to our customers.

Continuous Improvement through Transparency

At Topaz Labs, we feel that some of the best lessons are learned when things do not go well.  So we endeavour to foster open and honest communication internally towards learning those lessons as a team.  We also aim to keep our promise of high-quality software delivery to our customers, and be honest when we feel we have fallen short.

Hopefully others may find some of the lessons we’ve learned useful for their own projects.  We’ve certainly been encouraged that we can take steps towards even better releases in the future.

Share on facebook
Share on twitter
Share on linkedin
Share on email

Related Posts

12 thoughts on “How we engineer cutting-edge AI applications

  1. My approach and response may be somewhat different from others. I work in a hospital environment where everything is interconnected from billing to medical records to imaging studies and so forth. Between the threat of malware and ransomware to upgrades of one of perhaps 10 different software from 10 different vendors, where one vendor’s change may cause another’s to crash, I get it that there will be problems. So I admire your vision to improve on your past mistakes, but in my opinion, some people should just chill out a bit. I don’t think any of your clients’ businesses or hobbies will suffer lethal consequences as we might encounter in an emergency situation in the hospital, and yet when we encounter software crashes or glitches, we deal with it through phone calls to our and their IT departments.

    I have been patiently waiting for the upgrade of JPEG to RAW since I upgraded my OS from Catalina to Big Sur, but that was my fault because I failed to note your warning that there was a problem with Big Sur. I feel comfortable and am very pleased with the suite of software I purchased from Topaz and look forward new releases and perhaps even new products. I am a great fan of AI and deep convolutional constructs and am looking forward to future advances.

  2. Love you products. Keep up the great work

  3. Thank you for such a detailed explanation and your transparency. It’s refreshing, compared to your competition, and further increases my faith in your products.

  4. Dear Topaz team,
    though I was not affected by 3.0 changes, I am impressed by honesty You address the adverse situation You landed in. Not many companies or individuals has the strength and integrity to communicate in that open way. I wish You fast recovery and a lot of success.

    Best regards
    Juraj

  5. Topaz has a year’s head start on the competition but they are catching up.

    I need more warning when there is an upgrade. It’s a pain getting the notice after I[ve launched the app in photoshop or lightroom.

  6. Been using Topaz products since its inception. Love what you do. Best PS plug ins on the market.

  7. I’m glad to see TopazLabs getting so heavily into using AI to process images. I did try out the Gigapixel, Noise and Sharpen trials (only the Gigapixel product seemed to give me good results though, the other 2 didn’t seem to do much for standard-definition video capture images that I was hoping to drastically improve in quality).

    I have noticed many interesting developments when it comes to using AI to generate new synthetic images. I would be interested in a product that can do “style transfer” which lets you teach the AI how to mimic a certain style of art so that you can give it an input image and the AI will generate something in that style. The free G’mic filter for GIMP already has something like this but the results are usually not great. I’d like to be able to train an AI on the style of popular comic book artists e.g. Jack Kirby, John Byrne, Jim Lee, Marc Silvestri. Then give the AI one of my own hand-drawn illustrations and have the AI RE-draw my picture so that it more closely resembles the famous artist. I assume that would require it to recognize a hand-drawn nose, mouth, eyes, etc.

  8. Hi
    I did notice the problems with the Sharpen AI but I bided my time knowing that you would correct them quickly, and you did.
    I would like to comment on Gigapixel AI (v5.5). Over the last year our club has run all competitions on Zoom so the image size required has been 1920×1080 pix this has meant for me reducing from 6000×4000 pix. Not a difficult job but all of the image adjustments plus sharpening need to be done carefully to suit the reduced size. I thought I would see how the latest Gigapixel would perform doing most of the work.
    I did a RAW conversion with no adjustments to TIF of a landscape image that had slight noise and would require overall sharpening plus foreground focus adjustment, exposure was OK. I dropped the file into Gigapixel AI, set the output to 1080 pix high and left the rest on auto. After several minutes the result was stunning. All of the adjustments were perfect with no noise and front to back sharpness that would have taken some time to do manually.
    I have used this method several times and it always works. I feel you should point out that Gigapixel is good for reducing as well as enlarging. Was this ability intentional ?
    Regards
    Clive

  9. A good model, it seems.

    Thank you for the transparency and the information.

  10. First, thank you for outstanding software. It is unfortunate through that some of the best is no longer supported–the pre Big Sur applications.

    I wish there was a way you could have some sort of auto update feature for the software or at least a notification system that appeared before opening an image. it seems as if almost daily when working on an image the workflow is interrupted with an update notice. Thanks for listening.

  11. Sorry things went sour for you guys in these latest “improvements” From my side I have found that at the present time that AI sharpen and Denoise won’t work. But thanks for your honest communication of acknowledging that a fix is in the wings. But do keep in mind you guys make a great product. You will be nothing but better moving forward.
    Jim.

  12. I have not yet committed to purchase of your software but I like the honesty and frankness of this report of difficulties with recent releases. Some good for the future for users and your company .

Leave a Reply to Art David Cancel reply

Your email address will not be published. Required fields are marked *

You'll receive your trial information through email.
Topaz labs software have change the way I edit my photos. Support is also very good and the staff appears to have the knowledge to help.”
Johnnie Lamptey
Photographer
G-icon-png

Topaz Bundle

Manage multiple seats under a single user account.

Each license is valid for a single seat. Receive a volume discount for 10+ seats.

Save: $149.99

Total: $1349.91