{"id":49,"date":"2025-10-15T10:32:10","date_gmt":"2025-10-15T10:32:10","guid":{"rendered":"https:\/\/latenitesoft.com\/blog\/?p=49"},"modified":"2025-10-15T10:32:12","modified_gmt":"2025-10-15T10:32:12","slug":"evaluating-frontier-ai-image-generation-models","status":"publish","type":"post","link":"https:\/\/latenitesoft.com\/blog\/evaluating-frontier-ai-image-generation-models\/","title":{"rendered":"We ran over 600 image generations to compare AI models"},"content":{"rendered":"\n

tl:dr; <\/b>We’ve been making photo apps for iOS for long enough that we have gray hairs now, and using our experience we ran over 600 image generations to compare which AI models work best for which image edits. If you want, you can jump right to the image comparisons<\/a>, or the conclusion<\/a>, but promise us you won’t presumptuous comments on Hacker News until you’ve also read the background!<\/p>\n\n\n\n

Background<\/h2>\n\n\n\n

Hi! We’re LateNiteSoft, and we’ve been working on photography-related iOS apps for 15 years now. Working on market-leading apps such as Camera+, Photon<\/a> and REC<\/a>, we’ve always had our finger on the pulse on what users want out of their mobile photography.<\/p>\n\n\n\n

With the ground-breaking release of OpenAI’s gpt-image-1<\/b> image generation model earlier this year, we started investigating all the interesting use cases we could think of for AI image editing.<\/p>\n\n\n\n

But as a company that has never taken any venture capital investment, all our products have to pay for themselves. We’re in it to delight our users, not just capture market share and sell them out. When considering AI projects, one thing has been clear – we can’t take the AI startup road where you have a generous free tier, charge an unreasonably small monthly fee for “unlimited”, and hope you’re going to make it up on scale (code for “someone please acquire us”).<\/p>\n\n\n\n

All the AI-focused billing systems we could find out there were based on this. Assuming you want to claim unlimited access, and then sandbag users with “fair use” clauses and prevent them from any actual unlimited usage (which is, obviously, untenable, since you’ll end up with one $20\/mo user reselling to everyone else).<\/p>\n\n\n\n

Since we want to fairly charge our customers for what they actually use, we\u2019ve built a credit-based \u201cpay per generation\u201d-style billing system (that internally we’ve been calling CreditProxy). We\u2019ve also been planning on providing this as a service, since nobody else seems to be doing it, so if you\u2019re interested in being a trial user, get in touch!<\/a><\/p>\n\n\n\n

We released our app MorphAI<\/a> as a public proof of concept to give CreditProxy a proper real world-test, and have marketed it to the users of Camera+, which includes traditional photo-editing functionality, including a whole host of popular photo filters, giving us a built-in audience of customers ready for the next step in image editing.<\/p>\n\n\n\n

With the release of newer models like nanoBanana and Seedream, we’ve had to consider which models make sense to support. We need to explore the trade-offs between quality, prompt complexity, and pricing.<\/p>\n\n\n\n

A couple of hastily-hacked together scripts, and many, many AI generation credits later, we have some results! So that everyone else also doesn\u2019t have to waste their money, we figured we\u2019d share what we found:<\/p>\n\n\n\n

The Tests<\/h2>\n\n\n\n

Based on our experience with Camera+ and the kind of edits our users have been making with MorphAI, we picked a host of somewhat naive prompts. Veteran Midjourney users may scoff at these, but in our experience these are the kinds of prompts that our average user is likely to use.<\/p>\n\n\n\n

As for test photos, we chose some some representative things people like to take photos of: their pets, their kids, landscapes, their cars, and product photography.<\/p>\n\n\n\n

\n \n \n \n \n\n<\/p>\n\n\n\n

Image generation times are also relevant. During our test period, the generation time for all models was fairly consistent, and didn’t vary by image or prompt complexity.<\/p>\n\n\n\n

OpenAI (High)<\/strong><\/td>Gemini<\/strong><\/td>Seedream<\/strong><\/td><\/tr>
80 seconds<\/td>11 seconds<\/td>9 seconds<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

OpenAI also has a quality setting, the images included here were all generated on High quality, but we also tested Medium, and those generations averaged 36 seconds. We can include the Medium quality images as well if there is any interest!<\/p>\n\n\n\n

There are a ton of photos to compare here, so to make things easier to flip through, here are some keyboard shortcuts<\/strong> to help you out: Click on a photo to see it larger. Now you can use the arrow keys to switch between models. Press the tab key to switch between test images. Hit ESC to leave the view.<\/p>\n\n\n\n

Classic filters<\/h2>\n\n\n\n

These are the types of filters that we used to implement manually, by painstakingly hand-crafting textures and Photoshop layers and then converting those to Objective-C code. Now all you need is a few words into a language model (and to burn down half of a rain forest or so; just the cost of progress).<\/p>\n\n\n\n

Our conclusion for this category is that for photo realistic filters like this, Gemini really shines by preserving details from the original and minimizing hallucinations, but often at the expense of the strength and creativity of the effect. Especially with photos of people, Gemini seems to refuse to apply any edits at all, with a strong bias towards photo realism.<\/p>\n\n\n\n

OpenAI really likes to mess with the details of the photo, giving a characteristic “AI slop” feel, which can be a deal breaker on things like human faces.<\/p>\n\n\n\n

Grungy vintage photo<\/h3>\n\n\n
<\/div>\n